Maybe you already figured this out (in which case post your solution!), or:
It looks like this isn’t working because nAttr.setNiceNameOverride operates on a single attribute, which I’m guessing in the case of a k3Double is the compound attribute name (which won’t be visible in the channel box):
rgkMyScale (this is the attr that I'm guessing setNiceNameOverride is affecting)
rgkMyScale0
rgkMyScale1
rgkMyScale2
You can accomplish this by building each of the sub attributes individually and then creating the compound attribute (and I’ve seen this method of adding a compound attribute in some of the plugin examples):
x = nAttr.create('rgkMyScaleX', 'msx', OpenMaya.MFnNumericData.kDouble, 1.0)
nAttr.setNiceNameOverride('My Scale X')
y = nAttr.create('rgkMyScaleY', 'msy',OpenMaya.MFnNumericData.kDouble, 1.0)
nAttr.setNiceNameOverride('My Scale Y')
z = nAttr.create('rgkMyScaleZ', 'msz', OpenMaya.MFnNumericData.kDouble, 1.0)
nAttr.setNiceNameOverride('My Scale Z')
MyNode.myScale = nAttr.create('rgkMyScale', 'ms', x, y, z)
nAttr.setChannelBox(True)
MyNode.addAttribute(MyNode.myScale)
Also, MFnNumericAttribute.createPoint will create a compound attribute and automatically append X Y and Z to the child attribute names. Autodesk Maya 2013 API Documentation
Also there are more guys with API experience both over at the Python In Maya google group and cgtalk (though if you do use 'em keep posting questions here too!)