Sponsors
Sponsor Products
devops
posted by Greg Whynott  on May 7, 2018, 11 a.m. (4 months, 18 days ago)
5 Responses     0 Plus One's     0 Comments  
Is anyone doing devops at work and if so to which degree and how is it going for all involved?

If you're not sure what devops is you very likely are not, but if you care to familiarize yourself: https://theagileadmin.com/what-is-devops/


-g


Thread Tags:
  discuss-at-studiosysadmins 

Response from Will Rosecrans @ May 8, 2018, 2:55 p.m.
I think experience running renderfarms as "livestock, not pets" looks a lot like some of the modern cloudy devopsy stuff.Mounting a /software, /home, /projects NFS mount is a way to make workstations and render servers "stateless" so you can freely add and remove machines to a facility -- and it looks a lot like some of the best practices when building stateless cloud images that can go in an autoscaling group.
DevOps in a lot of ways is just a solidifying/branding/cloudifying of some practices and ideas that have been common in VFX for 20+ years.
The biggest difference is probably the adoption of Software Engineering practices, moreso than the modernization of the Ops practices.. I.e. Put a Terraform config in a git repo you can roll back and forth and merge reviewed pull-requests, rather than putting a renderfarm bootable image directly on an NFS server and modify it in place until it works. Plan around two week sprints, with story points, etc. Some of that works great for VFX. Some of it won't. We can't do *everything* in the cloud and on VM's. We can't put 100 TB of production assets into a git repo. We can't Dockerize Maya. (I found out recently xvfb-run does actually allow you to run basic OpenGL 1.0 from inside a Docker container on a headless system, which is kind of amusing, but not particularly useful.)
If you are working in commercials, a whole job might be two weeks, so the "agility" of two week planning sprints doesn't have the granularity to even notice the whole span of a project! In a classic software dev startup, using sprints and story points is a great way to avoid getting bogged down in "Hey I just need this one quick thing right now" kinds of projects, which improves overall efficiency because you aren't context switching constantly. You can just say, "Hey that sounds like a great idea. Tell my project manager to get it in the team's backlog, and will see if we can fit it into the next sprint" and go back to what you were doing. Working on a commercial, an artist says, "Hey, I need this obscure Romanian plugin that only runs on Maya 2011 to make procedurally generated paintings of sad clowns of velvet" you pretty much have to deal with it in real-time. If you offer to look at it next sprint, the job will have delivered and you'll never need paintings of sad clowns ever again.
For pipeline dev work of larger and long-term, that kind of sprint planning makes sense. "We are going to add feature X to internal webApp Y that runs on our internal VM host" totally fits into a DevOps / Agile / Sprint Planning kind of strategy. Artists know that feature X won't be available for the sad clowns job, but they can pretty confidently plan on being able to use it for the next job after, etc.


On Mon, May 7, 2018 at 7:58 AM, greg whynott <greg.whynott@gmail.com> wrote:
Is anyone doing devops at work and if so to which degree and how is it going for all involved?

If you're not sure what devops is you very likely are not, but if you care to familiarize yourself: https://theagileadmin.com/what-is-devops/


-g


To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe


0 Plus One's     0 Comments  
   

Response from Julian Firminger @ May 8, 2018, 3:45 a.m.
Firstly, Tim Loomis thanks for that amazing insight. And I agree completely with face to face story telling.
Greg,
The answer to this question is as complicated as that article "explaining" what DevOps is...
So, "yes" is my answer. Our Development team are an Agile team. Our ITOps team are not. We dont have a DevOps sysadmin, or rather, I am that sysadmin but only by virtue or being the senior admin and being the only one that understands DTAP and CI... So our Agile team of Developers use a CICD workflow driven by a suit of DevOps tools built around a DTAP deployment methodology. I maintain that pipeline from an infrastructure perspective and we cross over in the middle where the VM automated deployment (PDQDeploy) meats the CICD agents (Jenkins). <take a breath> However, the ITOps team is now adopting a DTAP approach to deploying new services. In theory we could automate that down the track a bit.
Weirdly, that article doesnt even mention DTAP and CICD. Continuous Integration / Continuous Delivery is entirely inside the Dev team. DTAP on the other hand we're finding can have a great impact on infrastructure deployment and maintenance. E.g. These days we dont deploy a new server called APPSERVER-05.ubf.nl thats going to replace 04. We deploy TSTAPPSVR-01.ubf.nl,ACCAPPSVR-01.ubf.nl andPRDAPPSVR-01.ubf.nl. We deploy to Test, break it a lot, then turn it over to Acceptance, let the users play with it and then qualify it and put it into Production. We do it with storage too. Now we always create three paths, test, acceptance, production. And we roll through them as we deploy. Then when v2.0 comes out, we upgrade TST first, and you can guess the rest. What it has meant is almost completely eliminating first time deploy or upgrade failures in production. Like, from every 3rd or 4th thing we were deploying to basically none. (Disclaimer, not our entire ITOps is running this way yet, and I still dont really know how to suck network engineering into this idea, but we're getting there)
Having said all that, I dont work at Tim's scale. And I completely agree with him that in order to "go devops" while you're in production you either need to hire a separate "going devops" team or have time, budget and great leadership in several roles to pull it off. Our Dev team is 8 people, our ITOps is 9. And it took me 3 years to get where we are now.

Julian Firminger

Senior Systems AdministratorUnited Broadcast FacilitiesAmsterdam, The Netherlands

On Tue, May 8, 2018 at 6:21 AM, Shane McEwan <shane@mcewan.id.au> wrote:
On 08/05/18 00:58, greg whynott wrote:
Is anyone doing devops at work and if so to which degree and how is it going for all involved?

Speaking of DevOps: https://www.humblebundle.com/books/devops-books

I work for a very agile company (not VFX, sadly) and I'm the embedded sysadmin in a web development team. I ensure the build and deploy pipeline is functional and that the developers have access to the tools they need. I don't do much dev stuff other than tweaking bash scripts but I feel that the "dev" in "devops" is more about empowering the developers to safely and securely build and deploy their software without necessarily knowing about AWS CloudFormation templates or Akamai caching rules.

I'm not sure how that would translate to a VFX studio, though. Artists can set their own render priorities? Artists can download and install whatever software they want to do their work? Artists can choose their own file naming schemes? Cats and dogs living together?

Shane.

To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe


0 Plus One's     0 Comments  
   

Response from Shane McEwan @ May 8, 2018, 12:25 a.m.
On 08/05/18 00:58, greg whynott wrote: > Is anyone doing devops at work and if so to which degree and how is it > going for all involved? Speaking of DevOps: https://www.humblebundle.com/books/devops-books I work for a very agile company (not VFX, sadly) and I'm the embedded sysadmin in a web development team. I ensure the build and deploy pipeline is functional and that the developers have access to the tools they need. I don't do much dev stuff other than tweaking bash scripts but I feel that the "dev" in "devops" is more about empowering the developers to safely and securely build and deploy their software without necessarily knowing about AWS CloudFormation templates or Akamai caching rules. I'm not sure how that would translate to a VFX studio, though. Artists can set their own render priorities? Artists can download and install whatever software they want to do their work? Artists can choose their own file naming schemes? Cats and dogs living together? Shane. To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe

0 Plus One's     0 Comments  
   

Response from Tim Loomis @ May 7, 2018, 4:20 p.m.
I just read through the DevOps definition at agileadmin.com and think it would apply to us and our broad, eclectic toolset. I design and operate computing systems in conjunction with UNIX admins, network overlords, security people and consultants for ArcGIS, IDL, Adobe, Drupal, DNS, storage and processing online and off, along with ancillary programs, scripts, code implementing bespoke renderings of data to feed into web sites, web maps, images, sequences, GIS services, .mp4, etc., while maintaining robust-enough feedback to catch hiccups in the data flow at the code, network and application levels. All of this work is definitely by committee as it involves satellite systems data extraction across multiple networks, applying algorithms developed by the science community and negotiating through various stakeholders the most politically fraught aspect of visualizing data, the application of color to values, as you are all well aware. Collaborating with stakeholders, scientists, educators, programmers and our communications team to write stories, make visuals and distribute through the web and social media requires constant iterations back into the systems, if not physical design past initial systems buildouts then certainly consistent coding tweaks, different philosophies of GIS layering, balancing resources to processes usually dictated by traffic volume to the sites' products, and the infinite variability a grumpy network proxied to the hilt can throw up to thwart my uptime.
Pollinating our products with standardized, consistent metadata is the last step in our development cycle and the hardest for its tediousness. There really isn't a universal, interchangeable representation standard, so I cut and paste a lot, and share my constructs within applications. Which I'm mentioning because our justification to implement the above to the commercial clouds is predicated on having the search engines parse from metadata rather than traffic volume or keyword or whatever other algorithmic chatter that equates to their proprietary relevancy score.
This is but a scattering of tinker toys sans a DevOps structure, though aspects such as network and physical infrastructure are by necessity laid out in Visio. None of us work in a vacuum, so a DevOps structure could conceivably be implemented across our organizations to compliment our hard-won development knowledge, but at least for us, defining and codifying the structure is inhibited by a lack of time, a lack of guiding organizational best practice and frankly a hesitance to commit anything to an auditable format that could potentially stifle the flexibility that we need to continue barreling forward into the implementation of emerging technologies. I haven't seen a good way to work backward, from functioning system to coherent DevOps breakout, and any way wonder if maybe the the most effective DevOps is a core group of humans living the edifice in real time. I see so much documentation become vapor the moment it's written and haven't experienced better conceptual transmission than face to face storytelling. If I had the time I would train an AI to know all of the participants, locate, monitor and assimilate the interconnections and communicate system state to all of us, but I wouldn't know where to begin jamming TensorFlow into my day. Like all of you, eyeballs on screen, fingers on keyboard and deep, useful systems access is at an absolute premium.
On Mon, May 7, 2018 at 11:05 AM, Greg Dickie <greg@justaguy.ca> wrote:
I do devops for one client. maybe 1 and a half. I think it's a good thing. IT is frequently not in a good position to handle things. When I started at discreet logic in '96 my group was basically IT for all the depts that IT did not have the expertise to handle. Essentially devops but without the cool name ;-)
On Mon, May 7, 2018 at 10:58 AM, greg whynott <greg.whynott@gmail.com> wrote:
Is anyone doing devops at work and if so to which degree and how is it going for all involved?

If you're not sure what devops is you very likely are not, but if you care to familiarize yourself: https://theagileadmin.com/what-is-devops/


-g


To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe



--


Greg Dickie
just a guy514-983-5400

To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe


0 Plus One's     0 Comments  
   

Response from Greg Dickie @ May 7, 2018, 11:10 a.m.
I do devops for one client. maybe 1 and a half. I think it's a good thing. IT is frequently not in a good position to handle things. When I started at discreet logic in '96 my group was basically IT for all the depts that IT did not have the expertise to handle. Essentially devops but without the cool name ;-)
On Mon, May 7, 2018 at 10:58 AM, greg whynott <greg.whynott@gmail.com> wrote:
Is anyone doing devops at work and if so to which degree and how is it going for all involved?

If you're not sure what devops is you very likely are not, but if you care to familiarize yourself: https://theagileadmin.com/what-is-devops/


-g


To unsubscribe from the list send a blank e-mail to mailto:studiosysadmins-discuss-request@studiosysadmins.com?subject=unsubscribe



--


Greg Dickie
just a guy514-983-5400

0 Plus One's     0 Comments