PyMel standalone - vertex color not appearing on mesh in scene

Coding
, , ,
1

Hi fellow tech artists, :slightly_smiling_face:

This is my first post to the forum. While I don’t consider myself an artist, I do consider myself an (aspiring) Python developer.

I’m creating my solution in PyCharm which, so far, has been working great to analyse meshes in a scene. I’m configuring PyMel standalone so that I can do this.
My PyCharm is setup to use the same mayapy interpreter from my Maya installation

However, when I try to colour vertices in my script, the vertices on the mesh in the scene aren’t updated with colour.

for record in self.records:
    error = record['error']
    vtx_id = record['scan_vtx_id']
    norm_err = (error - min_err) / (max_err - min_err)

    # map normalised error to an RGB tuple, starting from black (0,0,1)
    # TODO - consider using a PyMel dt.Color for black
    colour = (norm_err, 0, 1-norm_err)
    # trying another method that i saw in a code example in the pymel docs
    colour_curr =  self.scan_mesh.vtx[vtx_id].getColor(0)
    colour_curr.r, colour_curr.g, colour_curr.b = colour
    # preferred method
    # self.scan_mesh.vtx[vtx_id].setColor(colour)
         
    print(f"setting colour, {self.scan_mesh.vtx[vtx_id].getColor()} for {self.scan_mesh.vtx[vtx_id]}")
    # stackoverflow.com/questions/47436562/maya-color-per-vertex-not-working
    self.scan_mesh.setDisplayColors(True)
    # stackoverflow.com/questions/51365672/pymel-setcolor-not-working-on-vertices
    self.scan_mesh.updateSurface()
print(f'{self.scan_mesh.getDisplayColors()}') # this returns True
# pmc is an alias for pymel.core
pmc.refresh()

The print statement shows that the colour indices have been changed as I want.
Plus, if take some of these lines and run them in Maya’s script editor, I can see the vertex colour change.

What am I doing wrong here?
Many thanks,
Dan

For reference, I’ve checked out this on the forum, so far:

https://www.tech-artists.org/t/polycolorpervertex-with-pymel/12578

2

Did you set the viewport to display vertex colors?

https://help.autodesk.com/view/MAYAUL/2024/ENU/?guid=GUID-789B35EA-BDBE-4B82-81E3-75FB611FC96C

I’d strongly advise you against using Pymel to deal with geometry, it’s really slow as everything is an object in Pymel. Try using cmds instead.

3

Thanks Ben,

I have Smooth Shade All enabled in the view in Maya. I have done Toggle Display Colors Attribute and Color Material Channel set to ambient + diffuse. I’ve also set the same in the attribute editor with R, G, B channels enabled for the mesh (display colours, also enabled).

However, i have not set these programatically in my script, just inside Maya, in the project I’m working with.

Agreed about the performance. I’d like to get something basic working either in pymel or cmds and determine if there might be something wrong with my (standalone) setup.

Thanks again

4

Here’s an output of the print statement that makes a call to getColour() on the mesh:

setting colour, [0.36799606680870056, 0.0, 0.632003903388977, 1.0] for thumbsFBXASC032down|thumbsFBXASC032downShape.vtx[0]
setting colour, [0.32210057973861694, 0.0, 0.6778994202613831, 1.0] for thumbsFBXASC032down|thumbsFBXASC032downShape.vtx[1]
setting colour, [0.3092426061630249, 0.0, 0.6907574534416199, 1.0] for thumbsFBXASC032down|thumbsFBXASC032downShape.vtx[2]
setting colour, [0.33904173970222473, 0.0, 0.6609582304954529, 1.0] for thumbsFBXASC032down|thumbsFBXASC032downShape.vtx[3]
setting colour, [0.34133875370025635, 0.0, 0.6586612462997437, 1.0] for thumbsFBXASC032down|thumbsFBXASC032downShape.vtx[4]
setting colour, [0.35946738719940186, 0.0, 0.6405326128005981, 1.0] for thumbsFBXASC032down|thumbsFBXASC032downShape.vtx[5]
# and so on...

Since this looks like it has successfully set the colour attribute for the MeshVertex, i haven’t yet done anything with polyColorPerVertex. Is this a requirement for the vertex colour changes to appear on the mesh?

5

I’ve rewritten this block of code using cmds:

norm_err = (error - min_err) / (max_err - min_err)

# map normalised error to an RGB tuple, starting from black (0,0,1)
colour = (norm_err, 0, 1-norm_err)
cmds.select(f'{self.scan_mesh}.vtx[{vtx_id}]')
scan_mesh = cmds.ls(selection=True, objectsOnly=True)[0]
cmds.polyColorPerVertex(colorRGB=colour)
# print(f"setting colour, {self.scan_mesh.vtx[vtx_id].getColor()} for {self.scan_mesh.vtx[vtx_id]}")
cmds.setAttr(scan_mesh+".displayColors", 1)
print(f"setting colour for {scan_mesh}.vtx[{vtx_id}], result{cmds.polyColorPerVertex( query=True, colorRGB=True )}")
cmds.refresh()

I’ve pasted some of these lines into Maya’s script editor and they appear to work. I dont need the setAttr command.

having said that, i honestly would have thought the display colours command for the mesh would have worked.

This is leading my to suspect that there is something wrong with how my standalone is configured… do i need to do something with the Scene?

Currently, i’m opening the maya .mb file so that i can get the mesh objects from it… i’m not doing anything else with the file or the scene.

6

I have found a fix… which i think is perhaps more of a workaround.

I noticed in the logging that PyMel outputs when the script is run:

Warning: line 1: displayRGBColor is unavailable in batch mode

I didn’t realise I was running in batch mode. I guess that, if I’m not running interactive, then i must be running batch mode.

I’m still running the same code as above, same results in either PyMel or cmds, but i’m saving the vertex modifications to a new .ma file.

On opening the .ma file, the mesh is displayed with all coloured vertices.

If anyone has any idea on how to get around displaying the colours on the currently opened file/project, that’d be great. But for now, this is a way that works when creating a standalone solution.

Cheers!