Pipeline asset tracking - reviews, feedback etc

Pipelines
1

Hi All,

I wanted to start a discussion to bump some ideas around re. asset tracking, and how it can be stored/modified/viewed in an intuitive and composable manner.

To set the tone of where I am coming from, Ill explain briefly what I am doing.

Most people are familliar with tracking systems such as Shotgun, alienware etc. At the studio I am employed with, Creative Directors/Managers have found those systems too difficult/un-intuitive to use and so they inevitably stopped getting used - even though, generally our artists havent had too much difficulty.

With this in mind, the last couple of years I have developed my own software to deal with this. I did a rough prototype last year using wpf framework and earlier this year re-wrote the systems into asp.net so our external/out-sourced artists could use the systems cross platform.

One such issue I am currently trying to extend/develop an intuitive solution for, is to track reviews of assets, data and store them in an easy to access manner - in and outside of the tracking database.

For this to work, I am considering a solution where data (movies/images/documents):

Exists on the main file server, independently outside of any tracking software and can therefore be viewed independently
Conforms to existing project/studio folder conventions
Can be tagged with metadata for intuitive get/set property accessors

Most importantly, it needs to be linked directly into the file. ie. If a user moves a file, copies a file etc elsewhere on the file system, this information is transferred with it.

The metadata per se would allow for:

Direct linking to information on the database ( pretty simple to store an id/key somewhere on the file )
Direct linking to director revisions/drawovers
Direct linking to feedback (usually this is generated from a form somewhere on the UI )

The webserver is running on ii7.0 and all the studio machines ( and webserver ) runs windows7.

So with that all in mind, my question is - Has anyone here built\worked with a system having similar requirements? Ive found the design to be fairly challenging, and whilst I have a working prototype using office document tagging ( which can be stored in any file type ) - ive found it be a little slow and cumbersome and therefore was just looking for a sounding board to bounce ideas off of share experiences.

2

We’re about to launch a RESTful and service oriented system where entities can be referenced via URLs - many data items in the pipeline have URLs/URIs. We think it’s a good system because many external tools support hyperlinks by default - e.g. in Hansoft you can add links, in Shotgun, Excel, etc. you click them and you’re taken directly to the info you need. Of course, since you just need a HTTP lib, you’re pretty flexible in writing your own code around it in e.g. Python, C# or in batch files using CURL. When resources move, you can use HTTP redirection. If you need different representations of a resource - e.g. you can let the user choose if he wants to receive meta-data about an asset in JSON, XML or plain text - you can leverage HTTP’s media types and content negotiation.
URLs and URIs are standardized. This means you’re also independent for storage / vendor specific IDs. URLs also include location info, which can be useful if your servers don’t change too often, as it not only allows you to ID data, but also to locate it on a specific storage device or server.
Using URLs makes most sense if you serve HTTP; Doing so would also allow you to benefit from HTTPs infrastructure tech, such as proxies, caching, and load-balancing, which can really help you to scale up and distribute your pipeline.

But yes, getting the architectural design right isn’t that easy. I had to do some heavy reading into REST and service orientation. But also looking at the HTTP related tech behind Dropbox, Google drive and Amazon cloud services is very useful!

3

Cheers for the working Info / Summation of your system RobertKist.

I’m using URLS To pin some data to my db at the moment, but the following existing workflow might explain better why ive been experimenting with document tagging ( or more specifcally the DSOFile library in c# ).

Generally the process that is used here is that artists move data ( QT, images etc ) from their playblasts/images folder, to a review folder on the network.

ie. PROJECT/review/assets/DATE/

Then when something gets delivered to a client it gets copied to a separate folder, ie.

PROJECT/delivery/assets/DATE/

What im trying to implement is that the data from the review asset location is also transferred to delivery asset location and in the artists playblasts/images etc

Perhaps this is overcooking/complicating the whole setup… One of the reasons why I am doing this though is so that on a resource based view we can examine an asset/image/QT and see

delivered DATE
reviewed DATE
- feedback
- draw over

revisions DATE

I suppose I could ask the webserver to track filesystem changes via a watcher object. That may end up being fairly resource heavy though, given we may have up to 80 to 100 artists using the file server at any one time.

I should also note, that i experimented with people logging delivery/review updates into the webserver manually. IE. pinning urls with specific data about when/how an assets state has changed ( reviewed/delivered etc ). But ultimately, im trying to minimize artist involvement as much as possible.

4

I looked into the DSOFile mechanism a few years ago, and it’s actually a quite clever thing. But I remember it’s limited to the NTFS file system for formats which do not natively support meta data. You really have to be very careful and identify possible exit points where files leave your system and where metadata gets stripped. e.g. files moving to FAT32 storage, files being uploaded via FTP, or files being opened and re-saved by applications (e.g. file format conversion, or batch operations).
Some files have no internal meta data storage (e.g. ZBrush), some have it (e.g. MS Office docs), and with some it’s a mess: I could never really get it to work reliably with 3ds max.

For our system the only entry-point to the pipeline is via the webserver - i.e. the artist has to submit a file. There is no other access. But that’s not different from P4. However if an artist makes an exact copy of a file and uploads it with a different name, the system cannot tell that it’s the same file, and the metadata is lost. But it really depends on your requirements how tightly attached your metadata needs to be.
In our system metadata is not stored with the file, and there’s no central metadata storage, instead each service can store whatever metadata it needs for a file it encounters. i.e. metadata is owned by services and processes rather than the file entity. Services can thus evolve independently - i.e. change how they process metadata and what they actually want to store and in which formats.

Personally, I would aim having a balance of freedom / restrictions. Absolute freedom - i.e. the system will make sure you can do anything you want and your metadata is guaranteed to stay attached - will be very difficult to achieve. Best check your use-cases and then find something that’s reasonably easy to implement.

5

Reading your list of requirements makes me so happy, too few appreciate the inherent simplicity of associating metadata directly to the files on the filesystem.

I’ve been working on a solution to exactly this problem and have been waiting for an opportunity to take it further, it’s possible your requirements are a good fit in which case we should grab a virtual coffee and talk about the future.

I’ll put something more extended together for you here tomorrow as it’s getting late on my end, but have google on Open Metadata, it’s under a GitHub account for the Abstract Factory organisation, and let me know if it’s along the lines of your thinking.

6

As promised, here’s an extended version of the aims of the Open Metadata (OM) project.

OM is meant to augment the database in terms of metadata relative content, where the traditional database then is assigned the responsibility of pure social matters such as who does what and when along with transient information such as comments.

The content-bound information is directly coupled to folders on a file-system, where each folder represents either the lowest level of content or a group of other folders. The internals of each folder is then treated as though it were binary, a file. In short, it means metadata is stored alongside the internal files as what is known as “sidecar files”.

As sidecar files, as you seem to have experienced and aimed for, the metadata relevant to the content itself comes along with any copy/move action and remains synchronised as files get sent across to remote studio/freelancer work, whereas tasks, scheduling and whatever else is relevant to social matters remains centralised and away from the physical files.

With this technique, it is then possible to build visualisation and management tools which can then coincide with those built for files in general, to inspect a folder as though it were an abstract piece of content such as a shot and display/manipulate it’s properties.

Taken further, each folder can be assigned one or more unique identities that could be used for queries and search. Such as querying a directory for it’s parent shot or project. Or listing blue assets in a particular project.

Building upon this it’s possible and possibly even beneficial to completely eliminate the use of hierarchy templates/schemas and rely fully on any given directory knowing who it is and what it’s role is in a project.

I’ve written a summary about that in relation to path parsing here.

The interface for OM has come along well, it does automatic tracking of changes such that any change can be reverted back to or browsed after the fact. It also handles cascading of data, such that a parent directory can define a series of properties that children then inherit. Overall it provides an intuitive method of appending any type of data, including binary such as images or point-caches, along with more abstract types such as likes, color and kwarg.

More details about it’s past, present and future can be found here.

The challenge thus far has been performance, and this is where things would get picked up when diving back into the project. It’s well enough for 10-20 people teams, but the goal is to encompass the largest of needs. The current roadmap is heading towards a distributed indexing service, a database, which is kept fully optional to any relevant data; that is, the database should be breakable and only affect speed, not data. It remains synchronised through the use of calls to OM being made through an intermediary, such as RPC calls to the file-server, who can then trigger corresponding calls to the indexing service and at time same time provide the interface for queries occurring either locally or off-site.

With that, one should have the performance benefit of databases alongside the comfort, security and (underestimated) strength of filesystems and information stored hierarchically.

I look forward to hearing your thoughts!

7

Hi Marcus,
Thanks for joining in the discussion. The open metadata library looks very promising. Ive only had time to skim over the details thus far, but I think it could be very useful for integration into my application speciific pipelines. By that I mean the metadata that im using to share information between 3dsmax, maya, nuke and after fx. The afterfx component im not sure about yet without investigating your systems in more detail. At the moment im using xml files to share data between these apps, but it would be alot more convenient if i can use python to leverage this data. With the xml files, i obviously dont get the sidecar benefits which your system has.

I did briefly experiment with sidecar systems using XMP, but I ran into alot of trouble with the varying OS that our different departments use and the modification/copying of data. We have artists on linux, mac and windows ( predominantly windows, with the majority of editing departments using mac ).

In terms of the delivery/review tracking ( where my current systems have all been written in c# ) I could still get it to work if I dont run into the same issues that i had with XMP and the different operating systems. Overall though I think its very impressive what you’ve done so far, it looks extremely intuitive and well documented. I think alot of artists/pipelines could benefit from the great work you’ve been doing and utilizing python to leverage your ideas should be a good platform for easy adoption.