Creating a Pipleine for a Small Studio

Pipelines
1

Hi everyone.
I’m a fairly new TA that’s been tasked with creating an Art Pipeline for a small studio.
My background is mainly in art and animation. I’ve taken a programming class or two, but that was years ago.
Basically, I’m not an experienced TA. I’m mainly in this position because I tend to understand technical things more than most artists do. I have a high interest in learning programming and I actually enjoy the idea of getting into the pipeline creation process.
When it comes to art and programmers, we probably have fewer than 15 people. Up until now, the “do it whatever way you want” method has worked. Albeit sloppily… but it’s been easy enough to just walk down the hall and ask where the heck the file is. Or, “Hey, can you put that file in my folder?” Or, “Hmm… we lost those assets, so you’ll need to make them again.” I’m sure you get the picture.
Ideally, I know that we should create something where we don’t have to depend on the artists doing anything other than pressing an extra button here or there. We’re expecting growth (hopefully), so we need to create something that is scalable. And I know that asking artists to follow a list of procedures is just asking for problems.

The main goals right now are;
[ul]
[li]Keeping track of assets
[/li][li]Making efficient use of existing assets
[/li][li]Making it easy for the content department to know what assets we have so they can plan their budgets and balance requests for new assets with requests for already existing assets
[/li][li]Making sure assets are properly exported
[/li][/ul]

So my question is; What am I missing? I’m still in the beginning of this process, and I figure that all of you have gone through this before and see something that is obviously missing in my list of tasks. What should I be focusing on instead? What things are realistic for small budgets, and what things just aren’t feasible (but will be very very important as we grow)?
Any advice to point me in the right direction is greatly appreciated. This is an assignment that I will relish. But if I could learn from the obstacles that others have overcome before me, all the better.

2

Well from the list you mentioned there are two areas of concern, asset management and export pipeline.

When it comes to asset management there are a few considerations you need to make.
Firstly if have to realize that for each asset you will have a source file(s) and an export file(s). To have an efficient pipeline you really want to make sure that you keep a link between these somehow.
As an example lets say you have a Crate thats modeled in 3ds max (.max) and exported into unreal (.ase). The texture was done in Photoshop (.psd) and exported diffuse, normal, spec (.tga) Add to that an associated shader thats done in UnrealEd but is also has a preview shader in 3ds max (.fx for example).
An efficient asset pipeline would keep track of all those relationships.

The next consideration is getting a good revision control software. I recommend Perforce. It’s a matured and production proven software with easy to use gui as well as an api you could tap into to hook it into your pipeline.

When it comes to export pipelines I recommend starting with creating file templates for artists. These templates would come ready with the proper export hierarchy, any and all tags necessary and etc.
Further I would invest time in writing a robust and extendable validation tool that you could run on export as well as through a gui.

If you have any other more specific questions ask away.

3

Great post, WG. I’ve been basically in the same position between juggling lead design or artist on various projects. I’ve always been the tech nut and obsessed over anything related to shaders or utilizing what little powers our engines usually had and showing them to the art team. I lack a code background as well, though I’ve picked up bits of plenty of languages and posses a lot of the reading material discussed here in the past. Now that we have a stronger foundation of experienced people here I’d like to get out of just lead art/design and take control of anything tech-artsy.

Hoping we’ll fully jump onto the UE3 project bandwagon, any general tool ideas that have helped you guys would help a ton. WG you should let others know what your team is planning on working with for engine specific ideas as well… unless it’s proprietary of course but I’m sure some good would come out of knowing that as well.

4

Thanks ShadowM8. This is really helpful stuff to get in the pre-planning stage. I’m not sure if I will be able to get the okay for any 3rd party applications yet, but once we have a decent workflow here I might be able to show the need for it. Up to this point the idea of just organizing the pipeline with the assets we currently have would be a huge improvement.

And you make a good point Kovac. We’re currently using UE2, but are making a transition to Unity. Our artists are creating assets in Maya and Photoshop. Of course I can’t really say much about our content.

5

Unity does have an asset server addon you could purchase and use for source control! I would highly recommend looking at doing that at the very least.

6

[QUOTE=ShadowM8;4653]Unity does have an asset server addon you could purchase and use for source control! I would highly recommend looking at doing that at the very least.[/QUOTE]

We actually do have that. So far there are only 3 coders and 1 artist (myself) using it. Nobody else has migrated to Unity yet. So we figured that this would be the best time to completely overhaul the process.

7

Hmm…

So, Unity is not the only engine that we use. Have any of you had any success getting Unity to work with other Asset Management systems? It’s my understanding that the reason Unity uses it’s own is because of metadata that it keeps with its files.

We use the same assets for Unity and other delivery methods. Right now it’s looking like we’ll be having 2 sets of Asset Management systems for the same assets.

8

Hey Winning guy,

Good luck getting that pipeline going, sounds like a cool opportunity. I wanted to add a non-technical point to the other guys’ excellent tips: Don’t neglect the power of teaching your artists what to do. They may not like it, but you can save an incredible amount of time and frustration for everyone by simply laying down some ground rules.

Saving time is good for everyone because it means the tools you do write will be so much more stable.

It’s been my experience that artists do become comfortable with the core versioning tools, such as Open For Edit, Submit and Renaming. They may never love it, but it can save you an incredble amount of time right now which may be worth it in the long run since your team will get going so much faster.

IMO just validating source files is a plenty big task, there are a lot of ways to break a Maya scene :wink: Personally I wouldn’t try to solve every problem all at once, but rather flesh out the pipeline and tools involved and make sure to communicate it clearly to your team.

I can’t emphasize that enough, help them understand the process by drawing diagrams and making checklists for them (check out how simple this guy explains google wave).

Then I’d focus my efforts on validation, to ensure incorrect data doesn’t flow between departments. That way each department can exchange knowledge to get their models through the validator, and everyone slowly gets up to speed. Hopefully this starts to happen without your constant supervision, since some artists invariably ends up “getting it” and will teach their colleagues.

Let them help themselves is basically the point of my post =)

I say this particularly because you then mention Unity. For the time being you have to use their asset server, and it is not (in my experience) 100% stable. If your team knows the basic concepts of asset management they’ll be able to solve some problems on their own instead of you constantly having to firefight everything.

Sorry for the long post. It’s my first here, though I’ve lurked for a while and have wanted to jump in. I’ve just started a position myself where I’m having to learn a lot, and find it useful to focus on making as few tools as possible but making those that truly matter and making them stable.

9

…just giving an update.

What I’ve found interesting is that that in general, people don’t like change.

I expected a little push back from the artists. But for the most part they have been saying, “Ughh… change? Oh well, we know that the way we do things isn’t scalable. So we’re open to change.”

But the main push back I’m getting is from programmers. I’ve only been here a few months, and apparently the process of fixing the pipeline is a project that’s been going on for quite a while.

There are several different solutions and projects that are all in different stages of development. And in general, I don’t think anyone wants to think that all the work they’ve been putting into something is going to go to waste. But we really can’t have 3 different solutions implemented for the same problem.

We also have a content authoring department that is somewhat involved. They would be the equivelent of the Producer and the rest of the Production staff. But this department consists of about 20 people. And their tools are taking a lot of time from programmers.

It’s definitely an adventure. I’m learning a ton though. But the more I learn, the less I feel I know regarding the whole world of IT! It’s both enlightening and scary at the same time.

10

friends,

This website has exactly what you guys are talking about, worth visiting for

www.pipelinetd.com

11

I think I may be going about things a little wrong right now.

I keep looking at the ‘big picture’ here, and I keep coming to the conclusion that the whole company’s workflow needs to be revamped. I suppose I’m looking for some consistency from all departments. Processes are constantly being changed or refined in other departments. So sometime I feel like trying to integrate with them is like trying to shoot at a moving target.

This company has never had a Technical Artist before. So everything that I need to have my hand in is essentially controlled by other departments. They’ve devoted a lot of time and resources into their own solutions, so I suppose I have to work with what they have… although it’s constantly changing.

So maybe I need to be more simple. If I’m only thinking about the Art Department, the things they need are fairly straight forward.

An asset request system that is clear and trackable. We’re currently using TRAC’s bug tracking system. We’re also using that for our bugs, and just about everything else that has request tracking. As far as I know, in TRAC you won’t have the option of tickets being completely separate depending on the project. It’s clunkier than I’d like, but it’s free.

The actual storage location of art assets is up in the air. SVN? Fedora image repository? It may change. I suppose art just needs a UI that exports assets and automatically places it in the correct place, with metadata for searches and versions.

I’ve been talking to other departments about moving us off of TRAC and into a better project management system. We can’t really get the okay to spend the money for something like Alien Brain since it’s overkill for us. I’ve looked at things like activeCollab, JIRA with Confluence with the idea that their support of multiple projects could make a decent art asset request system.

Any input from you experts out there? I’m almost feeling like just giving the artists a UI to export art is ‘too small’. But changing the entire business’s project workflow is too big.

12

[QUOTE=Winning Guy;5253]
Any input from you experts out there? I’m almost feeling like just giving the artists a UI to export art is ‘too small’. But changing the entire business’s project workflow is too big.[/QUOTE]

Disclaimer: this is not input from an expert :slight_smile:

But I can relate a lot to what you’re saying. I think that getting from the point where someone ‘surfaces’ as TA in a small studio, to the point where everyone understands what you’re trying to accomplish it does take some time and effort. I think you’re in the right track there.

You’re right that changing the whole workflow is too much. I like the idea of setting a long term goal, and then working towards it in smaller tasks that you can accomplish fast and that will show most results (like the export UI you mentioned). This might not lead to the perfect pipeline, but it will definitely lead to a better pipeline.

And that your artists are that open to change… you lucky bastard! :laugh:

13

As far as version control goes I recommend SVN subversion. I’ve used perforce and found it a pain in the a$$. All the artists I worked with were able to “Get it” after a 10 minute explanation. SVN is free too (I think).

Complaints about perforce.
Open for edit.
Read only can clash with some pipes.
Hopeless explorer plugin.
Double your browsing time due to inability to open for edit in normal windows explorer.

14

Really? The Perforce shell extensions work fine for us. Right-click/Perforce/Open for Edit. Also works in any standard file open/save dialog. They were a bit slow to add 64-bit OS support to the shell extensions, but that’s there now too.

To the OP, I would also agree with the general sentiment of “don’t start too big”. Focus on the biggest 1-3 problems and build up from there.

15

Only the fairly recent Perforce versions have proper Explorer support for 64bit Windows platforms. Previously you were either stuck with 32bit Explorer extensions (E.g, in Photoshop and all other 32bit native applications) in the File Open/Save as Dialogs…

I have no idea how many people Perforce has working on “usability” features, but I guess it’s none. While I love the server backend, the randomness of the UI’s leaves a lot to be desired…

And obviously the most recent Perforce Explorer integrations only support P4V and not P4Win. F*ckers…

SamiV.

16

As far as asset tracking goes, i def have some suggestions:

-hook up your asset tracking to your source-control submit event. Extract the data you want to extract about your assets ‘after’ or at the time of check-in.

-make sure the asset-tracking data can archived and that the data is easily searchable.

-wrap your asset-tracking read/write/check-in/check-out in your DCC’s scripting language early on. This is a life-saver with implementation for ideas on-the-fly.

-if you decide to use xml, i would highly recommend you explore xml serialization via C#. It makes the whole process cleaner and easier. The down-side is that it’s not as fluid as other methods.

I’m sure there are other things im forgetting or mistaken about, but the guys here are really smart and will shoot down the ideas I’ve suggested that are ill-advised.

Good Luck!

-R

17

At the Blender Institute they use SVN (subversion), there are a huge number of DAM (digital asset management) projects out there.

There is the open source HELGA (login is guest, pw is blank)

http://anim.hampshire.edu/home/

I recently saw recommendations for shotgun

http://www.shotgunsoftware.com/

LetterRip

18

See if your library stocks a book called “The Checklist Manifesto: How to Get Things Right” by surgeon and New Yorker author Atul Gawande

He makes a convincing case for using simple low-tech methods to avoid the problem that plagues medicine and many other fields (like digital production): complexity

19

[QUOTE=ShadowM8;4650]Firstly if have to realize that for each asset you will have a source file(s) and an export file(s). To have an efficient pipeline you really want to make sure that you keep a link between these somehow.[/QUOTE]

This caught my interest. Not sure you’re still reading here ShadowM8 but if you do, would you, or anyone else for that matter, happen do have some insight on, from a technical standpoint, how that “somehow” might be done?

Using Python, I’m thinking: appending tags to files belonging to a certain project, with info on what part it plays in it.

E.g. xyz_shot01_rev001_trackBase.py might belong to a Project A, shot N, dynamics scene and might have been originally named point_cache.mdd but was renamed somewhere along the way, but due to the metadata was still able to be included by the asset browser.

Would this be an efficient route to take, or would you rather keep some sort of separate file with a list of all the filepaths? Is there another way to deal with this thing?

Never really dealt with tracking files before, so please bear with me. :slight_smile:

20

The easiest way to deal with source->export is to simply pick a file structure and stick to it.

Some people like putting all source files under a source directory tree, and exporting to a deliverable tree that has identical structure, e.g. source/characters/ted/model/shoe.ma corresponds to export/characters/ted/model/shoe.obj

(or whatever formats are appropriate)

Some prefer a different variation, like keeping “export” as a subdir of the model dirs.

Others also exist.

The key thing here is – pick a simple flavor, and stick with it – mistakes will occur so having something simple will make it easier to both avoid those mistakes, and to sort things out when mistakes need to be corrected.

And use revision control on everything. Not just source files. Not models. Everything. Design documents, textures, you name it. Mistakes will occur.