Node based pipeline automation and workflows

Hi everyone,

I am making some research on node based tools and workflows to automate pipeline tasks and processes for vfx and animation.

This is currently the different software and tools that I found :

  • Gaffer (open source, look development, automation)

  • Houdini’s PDG (procedural dependency graph for task automation)

  • Texel’s Kurtis (visual programming processing graphs)

Libraries :

  • qtnodes (Qt library for node graph visualization)

  • nodeeditor (Dataflow programming framework)

  • Rete.js (visual programming framework in JS)

  • Node-RED (flow-based programming tool)

So there is a lot to study! My goal is to explore ways to integrate a node based approach into an existing pipeline in my VFX school.

I wanted to have some feedback on the software I listed above, do anyone use them? In which context? What are the pros and cons over one solution? How do you integrate those tools into an existing architecture?

If you know other tools feel free to talk about them! :wink:

(Sorry I could only post two links, I’m new in the community)

Thanks

1 Like

There’s also

https://docs.unity3d.com/Packages/com.unity.assetgraph@1.7/manual/index.html

2 Likes

Hi,

Thanks for the info! Do you think it can interact with non Unity assets or export and import files into other software?

I haven’t used it – in fact, I just learned about it the other day. I’ll try to find out…

From the documentation it seems pretty focused on Unity related entities.

Hi again,

Just discovered NXT, an open source node execution tree system written in Python that looks pretty nice.

Has anyone heard of it or used it? Maybe in production?

UPDATE :

Oops there’s already a thread here : https://discourse.techart.online/t/studio-rigging-and-pipeline-tool-nxt-released-as-open-source/13447

1 Like

Am currently looking into a decent toolset basically lkike this (asked on Slack) and got pointed here.

This was my ask:

Does anyone have any recommendations on a neat little free (preferably Python and preferably open-source) library that can make a flow of tasks/steps to perform with configurable options for an artist to be configured. So, something like a simple node graph of ordered list of tasks to perform where its trivial to implement the “steps” that are available (e.g. simple classes, or simple entry points) but something that when given to an artist they quickly have an idea on how the tool itself works.
The artists are familiar with node graphs (like Fusion, Houdini, Nuke, Maya).
Basically I want to build a few simple steps they can put after each other as a ‘sequence of actions’ to perform in e.g. a certain workfile and then let them batch run that on say 100s of workfiles. And I just want to provide them some simple tasks with configurable options they can put in that sequence to run.
I feel like I don’t need a crazy Houdini PDG or Gaffer tool at this stage since there’s lots of overhead to learn the tooling itself.
Any recommendations are welcome! Feel free to also steer me into a completely different direction. Just curious to hear thoughts on how others might deploy (or dream to deploy) something like to artists.
The idea is that the stacks they build would get submitted to the farm per e.g. file to run it on so that one could easily queue: do these hundred steps on these hundred shots with this “context”
Even just typing this here makes me think that no matter what I end up what this will end up being way bigger than just a few tasks to expose to an artist. :') Anxious to hear everyone’s inputs/links/references/ideas!

Given that, I wonder if anyone in this topic ended up looking into any of these mentioned options or would nowadays recommend another?

I also had flowpipe on my list which got mentioned in a Rise VFX studio presentation but I’m not sure if there’s a visual toolset around that.

Some others:

Somehow i would also add Bifrost if you are in a USD pipeline.
In theory you should be able todo everything you can do with the USD SDK.

Hi everyone! Just yesterday I shared a post on LinkedIn about a project that I’m developing and Roy mentioned this post. So I thought I chime in and share it :slight_smile:

Basically, I’ve been developing a visual node system aimed at simplifying build pipelines for software developers. It is similar to node systems in Unreal Engine, Houdini, C4D, Blender, etc but platform and software independent. What started as a hobby project has evolved into a sophisticated toolset with native support for Linux, Win, macOS (editor included).

After some initial feedback from fellows in the industry, I am exploring product-market fit beyond its initial field, particularly in the realm of CG, AEC and VFX - from render pipelines, asset management pipelines to other automation processes.

From a usability perspective, my focus is to make this as artist-friendly as possible, while keeping it as performant as it currently is.

To give you some visuals on how this looks like:

I’m looking for studios in the CG, VFX or AEC sector that would be interested in adopting a system like this in their workflow and/or pipeline and help shape its roadmap. I haven’t thought about the details yet and leave this up for conversations on how to proceed.

Thanks a lot!

Sebastian

For the tech wizards amongst you, the engine is a native command-line tool and works on Linux, MacOS and Win. Nodes can be written in Python, C++, JS and Go. It is heavily multithreaded and with a binary size of 10 MB quite compact. Nodes can optionally interact with Docker and Kubernetes clusters. The startup of the executable takes around 8ms.

3 Likes

Does this double as a jobs system like Houdini TOPS or is something where parallelizartion and scheduling are implicit in the graph layout?

@Theodox There is a Parallel Node which allows multiple executions. The number of executions is customizable.

Screenshot 2024-01-27 at 12.00.01 AM

1 Like