Create one combining python library for all apps, good idea?

[svennovich]

Hello.
in our company we are having some huge discussions about The construction of our library.

we were thinking of combining all tools In the same library.

so if U start a script you will import like the following.
import projectname.maya.rigging
import projectname.modo.rendering

this means that all Python scripts will be centralised in one Directory.

disadvantage that i see is if U have A non python Lib Script u ll have to place it somewhere else then In this pythonTree.

also we were thinking about placing all tools and Apps also in this tree so that u can call ur information like this:
Import project.maya.rigging.tools.rigmaker

this Would Launch the tool.

any toughts about This?

[Adam_Pletcher]

Effective structure of code libraries is a subjective thing, specific to studios. In general I would say if you have more than one tools developer they are absolutely crucial. I can’t imagine developing anything without having a common library of useful code we’ve developed in the past, and refined over time.

Offhand I would recommend against putting project names in your library structure. You hope that code will survive to be used more than one project, right? Projects come and go, and having their names baked into your code will be quickly inaccurate, and irritating.

I’m unclear what you mean by “…centralized in one directory.” The example you list already shows at least three levels of subdirectories, “projectname”, “maya”, “rigging”, etc. In any case, subpackages (subdirectories) are a vital part of any Python code library, so absolutely use those to partition things into more specific functionality as the tree gets deeper. How granular you get is up to you, really. We sometimes start with a module, watch it grow over time, then split it up into multiple modules and/or subpackages later once it gets unwieldy. This requires some changes to client code, so you want a structure that minimizes that kind of thing. A small example from our studio-wide Python library (off the top of my head):

vlib
vlib.com
vlib.devkit
vlib.devkit.xbox
vlib.devkit.ps3
vlib.log
vlib.os
vlib.perforce
vlib.ui
vlib.ui.widgets
vlib.ui.widgets.spin
vlib.ui.frame
… etc.

I’m also unclear what you mean by “non python lib script”. Do you mean compiled C modules, like “spam.pyd”? If so, there’s no reason you can’t put those alongside Python modules, since the import syntax is the same, “import whatever.spam”.

To your idea of putting all tool-specific code in the same tree/package structure… I would be very wary of doing that. How to build studio-wide code libraries and how tools are distributed are two different beasts, and it’s hard for me to see how the same solution would work effectively for both. Think of it this way… as you’re writing code for a tool, you start by writing some general code that could be useful in several tools. That could go into the shared library, sure. But eventually you’re going to be writing code that’s very specific to your particular tool. If you also put that code in the same library (even tucked away in some low subpackage) you’re expanding that library and making it harder to find truly common code later. That’s a crucial feature of any library… being able to find existing code in it very easily and quickly. Filling it up with tool-specific code seems very risky to me, and if your tools developers can’t find the common code quickly they will just write it themselves, and you’re back to having similar but non-shared code everywhere.

The key question when writing code, as far as where to put it, is “Is this code potentially usable by another programmer for another tool?” If the answer is yes, then put it in your shared library. If not, and it’s specific to your tool, put it with the tool. If you have a larger tool that needs many Python modules, give it its own little library, but distribute it with that tool.

On the flipside, I would also be skeptical that one tool distribution method will be effective for every single tool. Whether you distribute it via a library like you propose, or some other single method, odds are you’re going to cook up some tools later that don’t fit well with that, and need to be distributed another way. Especially if you deal with outsourcers or other sites.

Again, these are pretty subjective things. Your studio has to find something that fits. Hope this helps.

[Theodox]

Host packages are one logical ‘top level’ grouping; in your example you had

import projectname.maya.rigging
import projectname.modo.rendering

but you might find

import maya.rigging
import modo.rendering
import common.pathnames
import common.db

to be easier to maintain over time. It’s particularly useful because you know which code is reusable for ‘all apps’ and which isn’t without having to know much about the individual modules.

Projects stuff is a bit harder to judge – it’s especially intertwined with the way your source control is set up. The long term issue is making sure that people manage the shared libraries effectively; every project has its own culture, and the different teams will want to edit the shared libraries in different ways. This can be awkward when different projects have different needs. Some teams create branches in their source control setup for each project – this gives a clear record of what the differences between the original and project specific code are. Others spend a lot of time negotiating the shared APIs and then lock them in stone, so they change rarely. And some do the way game engine vendors do: the main libraries evolve on their own and project teams take ‘drops’ when they feel they need them.

One really important way to keep things from going crazy is to have a strong set of unit tests on the shared libraries. That way anybody changing the library will at least be pretty sure not to break the existing code. Strong code standards help too.

[awforsythe]

What I’ve done in the past is to have a central companyname module that contains the more stable, non-project-specific codebase. This module contains individual submodules which address a certain area of functionality. For example, extensions for various applications’ python APIs are implemented in the extension submodule:

[ul]
[li]companyname.extension.maya
[/li][li]companyname.extension.motionbuilder
[/li][li]companyname.extension.fusion
[/li][/ul]
Other library code fits in here in the form of other submodules: automation provides classes to execute custom functionality in third-party apps either through ephemeral script files or a network interface, project encapsulates project/shot metadata and directory structure, footage helps in working with numbered stills as individual render passes, and so on. Each of these submodules is a subrepository, and the companyname module is the master repo.

For individual projects, I use a separate repository/module. This typically includes project-specific library code and scripts…

[ul]
[li]projectname
[/li][li]projectname.maya
[/li][li]projectname.motionbuilder
[/li][li]projectname.fusion
[/li][/ul]
…as well as tools:

[ul]
[li]projectname.toolnameone
[/li][li]projectname.toolnametwo
[/li][li]projectname.toolnamethree
[/li][/ul]
This is all contained in one repository, and I typically work on a development mirror (which I substitute into the PATH on my machine for testing) before pushing changes to the live version, so that I’m not inadvertently laying land mines in front of animators who are using the tools.

Each projectname module depends on the code in companyname, but nothing depends on projectname. Because I’ve historically worked on short projects (a few months) with only one or two programmers, this setup serves to allow rapid changes to the projectname module without fear of screwing up anything else. Where reckless haste is unfortunately needed, it can be employed, but more often than not the benefit comes in simply being psychologically unencumbered by the notion that rapid, sometimes experimental changes will have an undesired ripple effect.

Once the project is over, useful stuff from projectname can be cleaned up, refactored, and integrated into the companyname module, where a usual high quality standard and expectation for stability applies. Code that’s only relevant to the project, as well as what-the-hell-was-I-smoking features, can be left where they are to rot.