I realise this is more of an IT question than that of pipeline in general, but I’ve come across multiple mentions of keeping software, such as Maya, on version-control, such as Subversion.
For those with experience in implementing this, I’ve got a few questions for you.
From the perspective of an artist, with little or no IT experience;
- What is your motivation for storing software in version-control, as opposed to installing directly?
- What are some of your requirements of such a system?
- Has there been any challenges setting this up?
- What would a possible directory tree look like?
- Which version-control software do you find best suited for this purpose?
For those on the other end, using such a system;
- What benefits do you experience?
- What disadvantages do you suffer from?
Thanks.
Best,
Marcus
Most IT folks I know dislike storing binaries - especially large binaries - in source control. Source control is designed around managing versions of diff-able text files, it’s just an expensive storage mechanism.
To top it all off, source control not very good at actually installing software, which typically involves a complex web of dependencies in lots of different places. Most source control systems expect a single-rooted hierarchy and have a hard time moving things to a variety of locations. On windows it’ also doesn’t know how to manage the registry either.
Some people like to use disk images - that way you can go from a stored image to a complete workstation fairly fast. The main drawback to that is keeping these up to date with the constant system updates requires making new images often.
Other folks like to use configuration managers like puppet or chef, which were originally designed for making sure tons of cloud computer have identical setups. These pretty powerful but there’s a lot edge cases they have trouble with; they also work better on *Nix than windows. http://www.infoworld.com/d/data-center/puppet-or-chef-the-configuration-management-dilemma-215279
Long story short, it kind of stinks - there’s no great obvious solution.
I really hate that there is so much involvement with installing software. Nuke is a positive example:
- You can just copy the install folder and run it from anywhere (including server locations)
- It simply includes its dependencies (which on windows is only vcredists) in its install folder so that is easy to install if needed
- Beyond installation it also integrates nicely by having its startup files controlled by environment variables
- You can even do “import nuke” nowadays and use all of nuke’s python API directly from python without piping into a terminal session or handing over a script to the nuke executable.
- And in general just rocks in many way
- Yes, there are also things that suck. Just for completeness
Most software just sucks in that regard on the other hand:
- There are lengthy install operations
- Installations also include a lot of registry setup, adding files to a lot of different folders
- Installing dependencies without really showing which, nor providing standalone installers
- Many installations are hard to get to be unattendet and silent (though this is greatly reducing since quite a while luckily)
I would never store software in version control though. I imagine people use that as a replacement for proper rollout tools or images i imagine. But i am eager to hear reasons as i just would not know. From the hip:
- As mentioned, big binaries in version control are often not a good idea
- You can not use any of the standard vcs commands like making diffs or alike. This binary.
- Do you store the installer or the actual installed folder? In the first case i still really wonder why, the latter seems to make a bit of sense, but as there is so much you might have to do in addition to just cloning/checking out a folder it seems this would only work in few case,.
- You still need a lot of setup work and would prolly resort to hooks, or other automation systems or batch files or whatever, so why go VCS in the first place?
Warning: The rest is kind of offtopic in regards to storing software in VCS
We used to do images, and these days we have switched to using wpkg (we are a windows shop) for software rollouts. We have a central software repository that has the installers, scripts to do automated installs/uninstalls/updates etc. for all the deployable versions. Then you have a config xml for all of the hosts you have, and a config file for so called profiles. In the profiles you can define sets of software to be installed (e.g. have a profile “VFX” that installs Max, Maya, Houdini, shitloads of plugins and Nuke. Then maybe have an “Office” profile that installs Producer Tools, Office, Invoicing etc. etc.). In the hosts file you assign profiles to hosts (and add single packages if needed/desired).
wpkg runs an agent on every machine. If it is restarted (either remotely, manually or by booting/rebooting) it will look up what profile it is in, resolve all packages it needs, compares to installed packages and does all updates/installs/uninstalls as defined.
It’s neither perfect, nor crossplatform, but it is pretty damn neat already and a big step forward from what we used to do.
Cheers,
Thorsten
We wrote a setup app that installs and verifies software installs for all team members, and it stores profiles for each discipline. We don’t store the binaries in version control, they come from a regular network share. We can push out new versions of software using the tool, so it is pretty easy to get new people set up and keep them set up correctly.
Thanks all, this really opened my eyes to how complicated installs really are, and I can certainly imagine Windows being a bit more picky about dependencies here; quite ironic, as Unix usually seems to be the black sheep of “dependency hell”.
The reason I ask is in regards to the following problem; I’m sending a scene over to a renderfarm that has specific needs in terms of available applications, plugins and their versions. It’s important that the software is available (of course) but also that their versions match, based on a given range (say, maya > 2011) and that their plugin versions match (soup == 2014-04-27).
I can imagine the issue is similar to ensuring that a set of software is installed into a workstation - Puppet and Chef, having briefly scanned the article (thanks for that one) seem to deal with this issue - however this is secondary to my concerns.
Can I ask, are you guys all on Windows?
We are.
To make things more complex (a LOT more complex actually) dependencies are often not as easy as maya 2014 + matching soup. As soon as you get more then 1:1 dependencies things get really messy. An example would be Max + VRay + PhoenixFD. As Phoenix comes with some special VRay additions it has different Versions for different VRay versions. So if you want to support both different versions of max, vray AND phoenix you get shitloads of possible combinations instantly. Add to the mix, that there are nightlies available for both vray and phoenix and you have to use something like rez to resolve dependencies.
Cheers,
Thorsten
Darn. That does sound complicated. I’m starting to see where Rez is useful.
The spontaneous me would think that their dependencies would simply be cascading; Max requires this and that, each of this and that requires this other thing and that other thing etc. One package would ultimately have to be the one determining which dependency to go for, as long as all dependencies above are resolved, the complexity should stay rather linear, no?
Again the spontaneous me would think to map each package into a graph, and then resolve the shortest-path via standard graph-theory math; possibly working up-stream rather than down-stream, so as to eliminate options, rather than introduce new ones during the solve.
Too abstract?
Well it may be that linear. Or not heh. I guess this would only work if you have ONE version of Phoenix or VRay for each Hostversion. This is not the case though. For example we have three different vray versions (2.3 vs. 2.4 vs. 3.0) for productive. Not even thinking of betas or nightlies. Now imagine 2.3 branch being available only up to a certain point in the max release line. And it is not a pure “top down” problem as all of the downstream requirements may have introduce new dependencies, requirements, conflicts and other reasons for banging your head on the table. Also as soon as you add in inhouse stuff things explode quickly.
For example take python scripts for host applications. As soon as you are using any native python extensions they are python minor version specific as well as platform specific. Say you deploy a tool using MySQLdb. So you need to get the correct MySQLdb version for the needed python. If you use different versions of nuke, you need different versions of MySQLdb as Nuke uses different Python versions. If you do not have a specific version of MySQLdb deployed, then you get an unresolvable conflict. Now add fuzzy requirements (i want max 2014, with any vray newer than 2.4, with the blur python bridge (any version), shotgun support (which in turn requires an exact specific version of the python bridge) and an inhouse tool using python and phoenix). This gets really messy and really crazy pretty quickly. Something like “Max + Python + Inhouse tool + Shotgun” is not exactly uncommon.
I have not really gotten into the resolving itself. I hope for rez to handle that, and concentrate on providing the environment, the context and the packages 
Cheers,
Thorsten
Aw man, what have I gotten myself into? 
Fingers crossed for Rez. Save us!
Hehe, back on topic though, even in case of Rez, if you aren’t storing software on version-control, how can you possibly maintain each separate version of each app? Do you simply dump them all in a directory tree and point tools to their corresponding subdirectory?
I think a lot of people use virtual machines for their *nix render farms. These are slower but they get created fresh (usually from a stored image) so it’s much simpler to ensure they all match.
I was just going to say what Theodox did about the VMs. What you lose in terms of performance, you gain back manyfold in terms of ease of deployment and reliability. Of course, this does nothing to help you with end user setup and installation.
Well you still have to create the VM in the first place. So it does not solve the problem of a lot of possible different configurations. e.g. You can not really install different versions of vray with a single version of max. So if you want to offer vray 2.4 and 3.0 with a single max version you would have to create two different VMs. If you need to update Max, you have to update all your VMs etc.
To be honest, in the end all the different possibilities seem cumbersome. They work better or worse depending on your requirements, but there is no real solution. At least i don’t know any 
@Marcus: In regards to maintaining installer you would probably keep them in a folder structure yes. For rez you define the packages using yaml files and can have setup scripts etc. in addition to the raw files.
Cheers,
Thorsten
VMs sound interesting, and I can see how it can be both useful and not too much of a problem in the performance area. I would think stability and predictability trumps performance any day.
Off the top of my head, regarding the multitude of options required by the VM, are there other options than VirtualBox and VMWare Workstation that deal with this more elegantly?
Something like this?
http://www.vmware.com/uk/products/vsphere/
Spontaneously, I can imagine a workflow similar to Rez and package managers overall; user picks what he wants and the VM gets built with only these requirements.
The problem still remains to resolve the dependencies of course, but storage is currently also a mystery to me. Directory tree with or without VCS? Which VM system?
I take it this isn’t too common.
Is there anyone besides me and Thorsten with an interest in software configuration, ensuring access to software @ certain versions? Perhaps for task-distribution, like renders, ensuring the remote worker has access to the required software?
I think this is usally more of an IT Task than a dev/ta task. I regards to virtualization we are currently using a proxmox cluster. This is pretty convenient to use.
Cheers,
Thorsten
Makes sense. Were do IT folks hang out usually, if not at TA.org?
This is the only place i know: http://www.studiosysadmins.com/
(That is not to say that there are no other places. Go ahead and share them if you know any 
)
Cheers,
Thorsten
[QUOTE=instinct-vfx;24595]This is the only place i know: http://www.studiosysadmins.com/[/QUOTE]
This really is the only place for IT peeps in the vfx and animation industries.
Would something like Environment Modules do what you’re looking for?
http://nickgeoghegan.net/linux/installing-environment-modules
I seem to recall there being a similar Linux tool included with most distributions, I’ll try to jog my memory and dig it up.
-Kevin
edit: CDE is another great option, but I think there’s still another command line tool I’m forgetting.
@instinct-vfx: Thanks! Didn’t know about that one.
@kmacphail: That looks very much like what I was looking for, I’ll have a closer look. Ultimately whatever solution I end up using will have to run on all three platforms (win, lin, osx), but seeing something like this in action goes a long way.
Thanks, and let me know if you find the linux tool.
Fingers crossed for Rez. Save us!