Asset Tracking Techniques

Pipelines
1

Forked this thread from the Creating a Pipleine for a Small Studio due to some responses being more detailed than necessary.

How do you, technically, deal with keeping track of files and assets in a project?

I brought up the idea of using a tagging approach. Having each file tagged with relevant information so that you search for specific files easily, and also maintain your directory structure more efficiently.

E.g. You’d like to find all Character rigs from your all your projects put toghether, in case one of them suits the need for the next one. You’d simply filter all files with a “Project*” tag, that also had tags like “Character, Biped, Final” to list all of the relevant characters that are ready for re-use.

My post from the previous thread:

I’ve been giving the metadata idea some more thought tho, and I think it could be pretty transparent and, in the end, very helpful.

Like, if files were tagged with information about to which project the belong, which sequence/shot/version and date etc. etc., then files that somehow manage to strain from the directory structure could be searched for and automatically be put back into place. Adding an extra layer of security if nothing else.

Also, dunno if you’ve ever tried the asset tracking software Shotgun, but it has some interesting and very helpful filtering techniques that are based on tagging that could also be facilitated if all your project files were tagged in a similar fashion. Like, you could search for files belonging to a certain user, created between this and this time interval and was currently Not Finished, or something. It would filter the metadata of all your files, and return only the files that matches all the tags.

And, for transparency, you wouldn’t want to have to go though an individual tagging process, right? But that could instead be easily be integrated into the asset tracking software. So that for every file you submit (via right click menu, for instance) would, after user specification, not only be put into the right folder in the right location, but also get tagged with relevant information.

Seamless and harmless. Harmless cause, to my understanding, the metadata would be put into a separate “stream” of the file (still reading up on the subject), thus not having the ability to cause any damage like corruption or what not.

Thoughts?

2

One of our asset management tools is a house-made “resource browser”.

It offers use of arbitrary metadata on files (name/value pairs), either core/shared or specific to the game projects. Tools can easily read/write the metadata from C or Python, use it to create new workflows like review/approval states, or track what things are used where in the game worlds, etc.

One piece of metadata on every file is “categories”. These are basically keywords, but are presented in the UI in a tree hierarchy. Like a disk folder tree, but each file can appear in > 1 category folder. So an old brick texture can be in “age->old” as well as “color->red”, etc.

The metadata is actually stored via Perforce attributes on the back-end, so everything is revisioned along with the file. The attributes are also auto-indexed locally in a little database, making searches on metadata with the browser crazy fast.

3

Me and my technical supervisor created a asset checking/managing script. So you run a script in 3ds Max. You would select a folder and all the max files in that folder and its subfolders will open in max take 2 screen shots, front and back, and then generate an HTML file that had all the informations of the files. So you end up having an HTML browser that had a big list of all the files that had 2 images to show wat’s inside the file and at its side it would give you scene info,layer info and material info. There was an option of running more script to each file as well. As in you can select a script that will run each time every file opens. This was for checking the assets, for instance, freeze transform, pivot point location, naming convention etc. The result would show up under the image. There was a filter system as well where you can check for 3ds Max version,polycount etc. which were highlighted in red in the HTML. It was very helpful when we had to deliver a huge number of assets within a very short period of time. Leads would just run the script and check it in HTML instead of opening each file n checking stuff. If there was any issues you could click on the file name which would open up the file in max itself. I guess that should kind of solve the tagging issue as well.

4

[QUOTE=Adam Pletcher;8284]One piece of metadata on every file is “categories”. These are basically keywords, but are presented in the UI in a tree hierarchy. Like a disk folder tree, but each file can appear in > 1 category folder. So an old brick texture can be in “age->old” as well as “color->red”, etc.
[/QUOTE]

Thats exactly the kind of behaviour I’m looking for. You wouldn’t happen to have any hints as to how exactly this metadata is applied, technically? I mean, specific C or Python libs/commands?

Knowing that it is used is of great relief though. Think that this is the way to go.

5

This is a really interesting thread, for part of my fyp I plan on creating a check in check out system and I am also interested greatly in metadata.

I was planning on using a database to store the metedata for each file.

Will keep an eye on this thread for sure.

6

Perforce Attributes were an undocumented feature until one of the fairly recent releases. We’re using Server 2010.1/260003. Doing “p4.exe help attribute” on the command-line gives this help:

attribute -- Set per-revision attributes on revisions (unsupported)
p4 attribute [ -e -f -p ] -n name [ -v value ] files...

Sets a user-managed attribute ‘name’ to ‘value’ on opened files.
If -v isn’t given the attribute is cleared (if set).
Attributes can only be viewed with ‘p4 fstat’.
The -e flag indicates the ‘value’ is in hex.
The -f flag sets the attribute on submitted files.
The -p flag creates an attribute whose value will be propagated
when the file is opened with ‘p4 add’, ‘p4 edit’, or ‘p4 delete’.
Multiple attributes can be set or cleared by specifying multiple
‘-n name’ flags and an equal number of corresponding ‘-v value’
flags (to set) or no ‘-v’ flags (to clear).

Like it says, you use “fstat” to retrieve attribute data. The C and Python Perforce APIs both have this stuff exposed as well.

As for categories, our tool simply works from a config file that contains a list of all category names, and what the parent category is for each (root cats have no parent). When the user assigns something to a category, that file is given an attribute named “categories” with the value “<category name>”. The UI looks at all attributes on a file named “categories” to get the full list of categories it’s assigned to.

That’s really all there is to it.

7

Yeah, if you use Perforce attributes for metadata and want to search on them quickly, using a database is really a must. Perforce fstat operations are slow, so you can’t really fire tons of those off whenever a user searches on metadata.

Still, as long as you have some sort of UI actually setting the metadata, having it also index into a searchable database is not difficult.

Semi-related… As part of the Tech Art Boot Camp Tutorial at GDC on 3/1/2011 I’ll be showing how to effectively use databases for things like this, specifically from Python.

8

I installed a SQL database into the studio, and then all kinds of requests came through about wanting to re-assign jobs, and mark them as done etc… so I installed Bugzilla (Free) and built my own mySQL interface to the Database, with the help of, i believe this site SQL and started with making example entries into each department’s Bug Database, and then started looking at the fields it created and the schema, and started writing transactions in python to fill those tables, once that was in, I hooked the ability to “Publish” a model once it’s been cleaned and checked for errors, and the username came from the windows environment variable user, and the folder and whatnot were all logged, including the perception of completeness. And an e-mail was generated and sent to the relevant department heads in case something was marked for approval or final.

It sounds like a lot, but took me maybe 2 Weeks to get up and running, but I also had a publishing system that relied on an .csv file that got merged into an excel data sheet, and I left it in for compatibility sake, but re-directed the functionality with checking models, rigs, animations to the new database for sending it’s info.

So 2 weeks to get the Bugzilla up, asking the department to nail down the requirements which fluctuated for months after, but if the system is robust and addresses each entry specifically and has default parameters in the database it’s easy to add new stuff, and merge in the old…