File specification rules system for assets deployment

Pipelines
1

Hi everyone,

I’m looking for some ideas and references to help us design a solution for a problem in our pipeline that most of you probably already solved :slight_smile:
We already know about Object Model Pipeline, and we chose to design some parts of our pipeline using those concepts :slight_smile:

We have a 3dsMax project containing the max scenes in a directory and in another one the meshes, materials and textures (tga) exported for a real-time 3D framework.
This enables artists to preview the exported meshes using a lightweight viewer just after exporting.
This project is under version control (a Perforce depot).

Then we have another project for the 3D real-time application, which contains the same meshes, but put in a different directory structure, with compressed DDS textures only,
and maybe with a different version of materials (using shaders instead of base OpenGL materials).
This project is under version control too (another Perforce depot).

We need to write a piece of software that “bridges” these two projects.
So we thought of some sort of very generic system that, using a set of rules written by hand by our tech-artists and developers,
copies the files following those rules.
Example rules:

  • all files *.meshes must go in /meshes
  • all files *.tga must be converted to DDS and copied to /textures
  • etc

Have you any suggestion on how to express, parse and execute those rules? Languages, libraries?
We’d like to setup this “bridging/deployment system” as a Python class instance and then dumping/saving it using YAML
(we already use YAML to dump objects representing our pipeline configuration)

Thank you!

2

Couple of observations:

  1. you’ll probably need to support arbitrary placement as well as rule based placement - I’ve never shipped a game where the original rules about what-lives-where where completely respected throughout the project: between people disagreeing about how to classify stuff and last-minute changes of technical direction, things always end up in wierd locations that don’t make any sense. So, be ready for it before it happens.

  2. It’s a really nice idea to make the relationship between source and output files explicit - that lets you do analysis or integrity checking without lots of hunting all over the disk. You could store the relationships in a database, or inside metadata in files, using extra files (for example, a yaml file alongside every file pointing out the source <> output relationships, etc), or using Perforce attributes. All of these methods have side-issues: you’ve got to make sure that all the records of what-goes-where are up to date, which can be complicated if you have multiple users working on the same file at the same time and so on. However it’s a big improvement over a purely rule-based system because it’s a lot easier to maintain over time - the rules for even a medium size project get very complex and conditional as time goes on and edge cases accumulate. It’s a lot easier to say “hey database - where’s that file?” than "if this is a vehicle, look first in vehicles/models/… for a .veh file, and then try props/static/static_vehicles/… and /player/drivable/vehicles/cars/… for a file with the .dvb extension " and so one

  3. use the same system to place the files: if naming and location conventions are great ideas even when they are not 100% reliable - so use the same rules to place your output files instead of making users manually find the right place and type the right name.

  4. Don’t store the data as executable code (ie, use a data format like yaml or xml) instead of a maxscript or python file. If you’re completely, completely sure you can trust every machine/user that will ever use your tools you could relax this - but if, for example, you need to let outsourcers use your tools you have to be careful not to execute arbitrary code that’s just lying around in your project.

  5. Human readable files FTW, if you go with file-based metadata

  6. Make sure that the code that manages the system is not too max-specific: you want to be able to use it with all your tools and not just max. If it works well you’ll get people wanting to use it for collecting stats, automating builds, etc - so don’t make it dependent on the max GUI.

3

Thanks Theodox,
we’ll keep these advices in mind.
Have you any suggestion about languages/system for these rules?
Otherwise we’ll go for something between regex and rule tables…

4

If you view the whole system as primarily a naming problem, you could create a domain-specific language for it. It sounds like you guys are using max, so you might want make it as a dll in C# or Boo:

However I tend to look at the naming part as secondary. I prefer to match my library of asset types to a class hierarchy, where the classes and subclasses know how to generate names etc for themselves based on higher level knowledge about what kind of things they are:

Asset
– DCC file
— Max
— Photoshop
— etc
– Model
— AnimatedCharacter
---- PlayerCharacter
---- NPC
---- Ridable
— Prop
---- AnimatedProp
---- GameplayObject
— Environment

(generic examples, you get the idea)

Since files are typically organized hierarchically I find this usually makes for a good match between the code and the subject matter.

If I were you I would avoid using regexes as a primary tool for this sort of thing: they are beastly to maintain and debug, and if you end up having to refactor your setup re-writing them will be pretty nightmarish… which brings up the most important point: no matter what you think the right rule set is today, in a year you’ll hate it and want to change it :slight_smile: Plan accordingly.

5

[QUOTE=Theodox;21544]…in a year you’ll hate it and want to change it :slight_smile: Plan accordingly.[/QUOTE]

That’s the best advice I’ve ever had :smiley:
Thank you, we were thinking the same too, we have a hierarchy of classes to describe the assets, similar to the one you wrote here, and we are developing a regex based system to handle the “bridging”.

However, in your idea of class hierarchy, what is the relation between a DCC file and a model? Does a DCC File instance has got some sort of reference to a generated model?
Are all those classes assets that are represented by some file on a file system?
Thank you!

6

I would generally have subclasses for every distinct set of rules, and source files usually have different rues than content files. If they don’t have different rules right now, they may have them in future, so think ahead :slight_smile: Ultimately the relationship you’re describing is a large collection of dependency trees: this asset in the game requires these asset files; the asset files come from these source files. It’s extremely useful to have that view of your data so you can debug things and estimate the scope of ripple effects.

The source file <> game file(s) relationship can be done a couple of ways:

  1. if you control the game file format completely, just embed the name of the exporting file in the game file’s metadata. That way you have a perfect record of where stuff came from - and you can even expose the debug info in the game (“what model is that with the bad normals? Oh, it came from /models/test/mynormalsarebad.ma”)

  2. If you can’t stick information into the exported files, you can record it in some kind of file (xml, or better yaml/json). Typically this would live with the exported file so you can backtrack (it’s nice if there are naming conventions or folder-location conventions that make the back-tracking easy - but that is not always survivable: sometimes there are external constraints (“this stuff has to go here because it’s part of the streaming system” ) and often there are organizational changes that break the constraints. As long as the original file relationships are recorded you can always figure out where to look. This also handles 1 > many or many > many relationships - if a game character is assembled out of a skeleton file, a mesh file, and a physics file you want to have all of those relationships stored. The other nice thing about storing the source <> target relationship in a file is that it’s way easier to standarize what your tools do without tortuous “if it’ has X in the name go up 3 folders and look for Y, but if it has Z in the name go up 2 folders and look for A”, etc code.

If you go with files, have them under source control - and have the exporter update them automatically as the relationships are created or changed.

  1. You can also store the relationships in a database. I like that for two reasons: It’s really easy to iterate over when looking for patterns or making reports, and it keeps you from having to manage thousands of little files. The big drawback to databases is that you have to work to keep them up to date - if they don’t reflect the real state of the data they don’t help.

In all 3 cases the software that enforces the rules should also manage the data recording - if it’s got any manual component at all, people will hate it and won’t use it :wink: As I said, I’m skeptical of regexes because they’re so hard to debug and update - I’d prefer any of these to going with a name-convention based solution – even if you do want to have name conventions to. Names are out there where anybody can edit them, and they are usually typed by people with the typing skills of artists.

You might want to look at off-the-shelf products like http://www.southpawtech.com/ which are designed around this kind of file-relationship maintenance. I don’t use it myself since it doesn’t integrate easily with our perforce system, but it might work for you.