I was wondering if anyone here has experience with Maya and getting real time access from within the shader/material editor to individual vertex skinning weights (of the bones affecting the skin).
I want to render in the viewport an animated skinned mesh (human character) with three different normal map targets that change dynamically based on animation keys.
To do that, each animated bone of would have (beside their usual data) a custom animation channel that holds 3 floats (or a float3 if you prefer) with the desired mix of the three normal map targets.
I want to have per vertex (or if that is impossible, than per pixel) access from within the material/shader to the weighted sum (based on skinning weights) of that float3 custom animation channel.
I have looked into using custom GLSL shaders for the Maya view-port, but as far as I can tell the skinning is done on the CPU not on the GPU, so the GLSL/HLSL shader receives the vertices already skinned, and just renders them as a rigid mesh (is my finding correct or erroneous?).
I fear if that is the case than information I need is not available for the cgfx or hlsl shader, so that’s a dead end…
Now, as a work-around, since the mesh is uniquely mapped (no overlaps), in theory the need for a cgfx shader could be circumvented if I could somehow get Maya to render in real-time in a RGB render target with the weighted sum of the custom animation channels as colors, (using the UV’s instead of screen space position). Then i could use this dynamic texture as an input to the regular maya material and do the normal map mix math based on the values of those 3 channels.
It is crucial for this setup to work in real-time the viewport at interactive framerates, for the artist to be able to animate properly. Any information on how could this be achieved in Maya?
What if you just put your weighting info into an animated vertex color channel? You won’t get real time preview of your actual normal maps but you could see the weighting changing over time in the viewport without custom shader work. Perhaps you could generate cpv color channel animations using on animated colors on the bones and an expression that drives the polytColorPerVertex node’s vert channels based on weights.
This precisely what we are doing in the game engine (and in our proprietary toolset), the 3 normal map weights (the custom float 3) ends up on the gpu in Color1 in the vertex data.
The custom shader work to blend between those normal maps in the Maya viewport is not the problem, I’d happily do it if the data i need would just reach the GPU, per vertex
Taking your advice we tried writing a python script for Maya that picks the animated custom float3 from all the bones in the rig, and updates all the color mesh vertices with the wanted colors, taking into account the skin weights. It works, the result is what it is supposed to be.
Unfortunately running the python script and updating all the vertex colors even on a lightweight 2000 vertices mesh takes about 14 (omg!) seconds (for one animation key!).
A brief profiling showed that 12 seconds of the 14 is just the “SetColor” calls without anything else (this seems like an awful lot tbh, this is nowhere near interactive framerates).
So doing this via the script seems a dead end, unless we cand find faster methods to write the colors in the vertices. ( i had hopes the script to would in 0.1 seconds or less, so we could set the script as as an ongoing pyJob, triggered by any time-cursor changes)
So the perspectives right now are either:
figure out different/faster script methods ( need ~ 100 times faster )
write a plugin for maya in c++ that does this vertex color update faster
go back to the proprietary tools for animation (this will be highly unpopular with the animators which would like to switch to Maya)
dig more though docs and ping maya support to see if Maya does not have functionality for this that we haven’t been able to find…
It still boggles my mind that Maya seems not to have support for animated color keys for the bones, that would “radiate” to the vertices based on the skinning weights, or if it does we haven’t been able to find it. I guess this is the price for working with proprietary tools for many years :).
Again many thanks, and if you have any other thoughts I’d love to hear them
I should have been clearer that this is a good application for an expression - a script that alters the mesh data will definitely be slow.
Have you tried the same thing using an expression rather than a script or scriptJob? It should be a lot faster. You’d probably want to create the expression via a script since it will be tons of this:
float $bone1R = rUpperArm.NormalWeights.r;
float $bone1G = rUpperArm.NormalWeights.g;
float $bone1B = rUpperArm.NormalWeights.b;
float $bone2R = rForeArm.NormalWeights.r;
float $bone2G = rForeArm.NormalWeights.g;
float $bone2B = rForeArm.NormalWeights.b;
// the expression will need to have these lines for every vert with appopriate weighting
polyColorPerVertex1.vertexColor[0].VertexColorR = (.75 * $bone1R) + (.25 * $bone2R);
polyColorPerVertex1.vertexColor[0].VertexColorR = (.75 * $bone1G) + (.25 * $bone2G);
polyColorPerVertex1.vertexColor[0].VertexColorR = (.75 * $bone1B) + (.25 * $bone2B);
You’d probably want to generate a big expression (just on demand, NOT every frame) using a script that creates the weightings based vert weights and colors based on animated attribs on the bones. I doubt it will run at 30fps ( what does in maya ?!!) but it shold be faster than 14 million milliseconds.
[QUOTE=Theodox;18724]I should have been clearer that this is a good application for an expression - a script that alters the mesh data will definitely be slow.
Have you tried the same thing using an expression rather than a script or scriptJob? It should be a lot faster. You’d probably want to create the expression via a script since it will be tons of this:
You’d probably want to generate a big expression (just on demand, NOT every frame) using a script that creates the weightings based vert weights and colors based on animated attribs on the bones. I doubt it will run at 30fps ( what does in maya ?!!) but it shold be faster than 14 million milliseconds.[/QUOTE]
Thank you again for having the patience to bear with me
I have went and read up on building proper driving expressions also read about how are they created using the Maya visual interface; at first I had assumed everything would be done via the scripting language, and a driving expression would be simply be an expression within a script that runs as a job.
Now that I know about writing the driving expressions the proper way, and how they are different from regular scripts (and I also hope much faster ), first thing Monday morning will write a python script that saves to disk a file with the (huge)driving expression together with the variable initializations for a selected skinned mesh.
Then will test the created driving expression to drive the polyColorPerVertex of that mesh and will see how it goes :).
Wrote python script to generate the driven expression.
The good:
Worked like a charm for a few vertices. Seems really speedy :).
The bad:
Unfortunately Maya freezes whenever i hit “Create” or “Edit” on a driven expression is longer than what we need for ~200 vertices.
Mitigation:
Have vectorized the computations so that the RGB is done all at once to reduce the line number in the driven expression. Also used short names instead of nice names (figured it might be an unhandled buffer overflow somewhere) but no cigar… still freezes.
Will probably end up setting up 10 different driven expressions for 200 vertices each rather than one expression for 2000 vertices, it should do the trick (didn’t do that today because new tasks needed urgent attention :P).
Ideally, will try to do all this multiple-driven-expression set up through a python script that needs to be called just once when setting up the scene for the animators.
Will update tomorrow with results :P.
Glad you made some progress. I don’t recall hitting the 200 line limit before, I don’t know if it’s a problem with the UI code or an inherent limitation on expressions that I’ve never hit before.
You should be able to get the expression text wuthout going through the UI – the ‘expression’ property on the expression node contains the text of the expression and you can get/set it via script.
As a fallback plan, it might be possible to do this with particles and a particle expression – but that’s an area that I know very little about. Unfortunately maya was done before large-scale parallelism was really a big thing in computing, so it doesn’t provide good tools for this sort of thing; but particles are the closest thing Maya has to a primitive for big arrays of vector calculation. XSI or Houdini would be a lot better at this sort of thing.
The good:
We have a python script that takes an input skinned mesh, and sets up however many driving expressions are needed to control the color of all the vertices in that mesh (sets them up automatically, avoiding the UI); it does this in batches of 200 vertices per expression. It works.
The bad:
For a mesh with 2000 vertices there are 10 driven expressions being generated and active as a given time. This works BUT makes the scene too sluggish to be comfortable for manipulation (as reported by the animators). There’s a 1-2 second delays on simple operations like selecting a bone or keying a value.
Mitigation:
We will update the script to work on a selection of vertices intead of the entire mesh, generating new driving expressions and clearing any pre-existing generated driven expression each time it is invoked.
This way the animators can work on smaller areas but at interactive frame-rates by selecting the area of interest and running the script to update the area they wish to preview, (and can do a fast preview render to see the entire animation in the end, or they can just export it to the game engine and view it there).
Also, with that temporary solution in place, more research will be going into a more robust solution, such as writing this task by using only the Maya API classes and methods directly instead of MEL or pyMel. Have reason to believe this could run more than 100 faster than the initial python script.
You can hook the nodeState attribute on your expressions to an attribute on the character and use that to toggle the behavior on and off (a hotkey would also be easy to do). That way you can avoid the interactivity problems until you want a full animated preview. You might try batching your verts by major bone influences so the animators could toggle display on the areas they are working on and ignore the rest, then turn everything on for a playbast preview once in a while.
If you’re comfortable doing API programming, you really just need a single node that takes inputs from a the animated RGB colors on the bones and the skin weights and outputs a list of vert colors as a compound of RGB colors attributes which you can then connect to the .vertexColor of the PolyColorPerVertex.
), first thing Monday morning will write a python script that saves to disk a file with the (huge)driving expression together with the variable initializations for a selected skinned mesh.