Yeah I hear you Sune, and this is something I have been talking with the RD crew again lately.
There are a few systems out there but one that might be the closest jumping off point for you would be to look at. Not a perfect solution out there but starting with the most complete to the most work or not open.
and of course the one I think has the most promise but hasn’t seen an update in a while is GEAR
http://jeremiepasserin.com/gear/
For something that you might want to review if you haven’t is http://red9-consultancy.blogspot.com/ the latest pack allows for meta connections, has support for pose/animation transfer, library etc.
I also think that the CG Monks are working on some new tools and have hinted at working on a custom rigging solution http://cgmonks.com/
And there is always the zooTools that have started to be ported to python that could be extended adjusted as needed for your own situations,
Not opensource that I am aware, but if you haven’t seen the work on Ultimate Rig from Neo Reel you might be interested in talking with Marc-André Guindon about it since it is designed for mocap work and he knows what he is doing in that area. MoClip Help
The problem is that almost none of this stuff takes full advantage of the newer systems in Maya, HIK, Layers, ATOM etc, some have updated parts or are not entirely moved to python etc.
If I had to do this from scratch and it was a mixed mocap situation in 2013, I would lean heavy on the systems in place for HIK and ATOM, they have an auto skeleton builder, the character GUI is customizable now, the animation tools are more robust etc… and as much as I have been beyond frustrated with how badly done the 2013 releases were from Autodesk, the Idea of having something so close to working and just throwing it out to build it back up is not to be taken lightly.
Animation layers + red9 tools + HIK and mobu interop should be enough for 98% of uses. Just something to think about… and you can always layer over the top of HIK some custom controls and systems and still use Trax if you need the clip matching.
Good luck,
Brad