Method for using organic models?

[gvittrup]

I am looking for a workflow that would allow the use of organic models in Unity for increased accuracy with motion capturing data and environmental game objects. A prime example I found for this was this video under the dm_control domain, though I am not sure if this was achieved by converting a file to MJCF, if it was an OBJ file, or if the mesh was rendered onto the frame displayed at 00:00:30 in the video linked above.

Additionally, I noticed on the documentation its mention of skinning for visualizations, though I am not exactly confident this is what I am looking for nor does the documentation explain how to generate these files and if MuJoCo is dependent on a certain procedure.

Any point of reference or direction would be really appreciated!

[yuvaltassa]

This model lives here, assets in the dog_assets directory, and makes use of MuJoCo's skin asset type. Feel free to load it in simulate and play with it.

Import/export of skinned meshes to and from Unity is also possible, but currently not implemented.

[gvittrup]

What would the proper workflow be to generate SKN / MSH files? As per documentation, it says you can load custom binary files, though my test cases failed. I used the Godot engine, which exported my 3D model from the engine into a .msh, though despite other programs being able to read this file MuJoCo was unable to.

Additionally, when talking about mapping vertices to bone subelements, does this require my model/armature to be rigged? I noticed in the dog_assets and the XML file that it referenced all of these bones but I am unsure if this is relative to the skinning of the mesh.

[gvittrup]

Thank you for your continued support!

My approach will be writing a parser that converts the text document containing vertex and texture coordinate information of my model into a skin asset specified in an XML file and then using the above code to convert that into a binary skin file. I will be using the vertex coordinates to create the bodies, geom's, and joints.

Regarding the skin asset type, as it was created in the soft box composite object that you linked, what do the vertex and face components reference? I am confused as to why vertex only has a value of 0, and why face contains numbers up to the maximum vertices count of the composite object (293), but lists them in varying order and repetition.

Additionally, any thoughts regarding my approach would be greatly appreciated.

Thanks again.

[yuvaltassa]

Perhaps you didn't find the relevant section explaining how skins work?

Please read that and let me know what questions remain?

[gvittrup]

My apologies - the slight discrepancy in naming derailed me.

The issue I have been struggling with since then is regarding vertex weights. Under the documentation, it lists that for each bone, it has vertex_index[vertex_count] and vertex_weight[vertex_count]. I recognized this in the composite soft_box model, as it has 294 vertices, and 294 bones, each with a vertex weight of 1. My model has 372 vertices, though I am receiving an error in the engine that my vertices must have positive total weight in skin. I can only break this error in the engine if I change the vertex weight in my skin to a ridiculously high number (i.e. 999999999999), to which it then creates an error saying that my vertex count is too high. I can replicate these errors on the soft_box model if I add or remove a single digit from its vertex weights, to which I understand the cause being the disparity in vertex count. What could be causing the engine to assume my vertex weight/vertex count is so ridiculously high?

[yuvaltassa]

Hi @gvittrup,

Could you try to make a minimal model that we could load and look at? it's difficult to diagnose your problem like this. Perhaps look at this model with an explicit XML skin, and it will clarify things for you?

[gvittrup]

Yuval,

Thanks for that example. I broke my model down to 32 vertices for testing and followed that syntax exactly, and I was able to get past that error and finally render my model as a skin.