Wrong joint ids mapping when there are non-actuated ball joints

[NeoZng]

In mjlab\src\mjlab\entity\entity.py line 892, the initialization process treat all joints as hinge type. Though in line 924 it correctly expand the joint adress (3 for ball vel and 4 for ball quat) , the joint_ids is not modified.
Then when write_joint_state is called, the map between joint_ids and self.indexing.joint_q_adr & self.indexing.joint_v_adr are wrong.

[NeoZng]

i am training a closed-loop mechanism with spatial link connection. or do u recommend to use <connect> in mjcf to replace all non-actuated joint in the kinematic loop?
one potential problem is, the configuration is uniquely determined by the actuated joints, set a random value for the ball could result in constraint violation. maybe it is not a good idea to randomize the init state for closed-loop mechanism

[NeoZng]

After traversing the repo, i found obs, reward and event related impl are all affected by ball joint.
IsaacLab creates additional name placeholders for joints with more than 1 dof.
For a ball joint, it will be "divided" into 3 joints named "ball_joint:1", "ball_joint:2", "ball_joint:3" (possibly using euler or RPY), which is probably the minimum modification for mjlab to support ball joints.
But mujoco uses quat to represent the posture of the ball joint, and qvel has 3 elements, which will cause inconsistencies:(

[kevinzakka]

Hi @NeoZng, can you test #476 and tell me if it solves your problem?

[NeoZng]

@kevinzakka No.

New problems emerged. Since the Entity.data.joint_pos now includes all joints, when calc obs term joint_pos_rel and joint_vel_rel, there is a dim mismatch between default_joint_pos and joint_pose.

The "dynamics ids expansion" impl in cur pr is not that elegant compared to creating extra joint name&ids placeholder (allows 1-1 mapping, which maximinizes compatibility with existing code). while the quat rep for ball joint is really annoying.

I suggest changing the rep for ball joint pos with pure imaginary quat, the lie algebra of unit quat, it is a 3-dim vec (or simply RPY fixed angle rep) . Then the modification will only be made to Entity & Data. Just transform between unit quat and 3dof rep by re-impl write_joint_position() and property joint_pos. Another pro for 3dof minimal parameterization is that when the 3-dim vec is zero, both RPY & pure img quat represent identity transformation, that is much more intuitive.

[kevinzakka]

Indeed, I closed the PR realizing it wasn’t a good implementation. Working on a newer version.

[NeoZng]

@kevinzakka No.

New problems emerged. Since the Entity.data.joint_pos now includes all joints, when calc obs term joint_pos_rel and joint_vel_rel, there is a dim mismatch between default_joint_pos and joint_pose.

The "dynamics ids expansion" impl in cur pr is not that elegant compared to creating extra joint name&ids placeholder (allows 1-1 mapping, which maximinizes compatibility with existing code). while the quat rep for ball joint is really annoying.

I suggest changing the rep for ball joint pos with pure imaginary quat, the lie algebra of unit quat, it is a 3-dim vec (or simply RPY fixed angle rep) . Then the modification will only be made to Entity & Data. Just transform between unit quat and 3dof rep by re-impl write_joint_position() and property joint_pos. Another pro for 3dof minimal parameterization is that when the 3-dim vec is zero, both RPY & pure img quat represent identity transformation, that is much more intuitive.

i can write a pr for my suggestion later. but now i will just replace all ball joints with connect constraint for convenience, it is an easy expedient. Maybe the performance is acceptable

[NeoZng]

Indeed, I closed the PR realizing it wasn’t a good implementation. Working on a newer version.

really appreciate. hope to get your new impl ASAP ;)

[RumblingTurtle]

Next issue will arise during initial joint state initialization since spherical joints are not normalized during the addition and scaling. Moreover simply perturbing connect constraints may result in simulation instability (that's what I regularly observe in IsaacSim), and simple normalization won't fix the fact that the resulting quaternion may not be feasible at all on the real robot because the endpoint of the body is welded to anchor. This imo makes the sim data less representative for training. And you'd probably somehow need to pass the entire qpos through joint_pos of the InitialStateCfg for the same initial state stability reasons which brings the question that @NeoZng brought up.

[kevinzakka]

Hi all, aware of all the side effects. This is going to require a very careful PR as it will have some breaking changes.

[kevinzakka]

@NeoZng and @RumblingTurtle, OOC what are you using a ball joint for out of curiosity? If this is niche, I'm thinking of not fixing it.

[RumblingTurtle]

@NeoZng and @RumblingTurtle, OOC what are you using a ball joint for out of curiosity? If this is niche, I'm thinking of not fixing it.

Humanoid locomotion and imitation. Our custom robot has closed loops in the knees and ankles similar to Cassie. We train our policies in motorspace so spherical joints are not an issue on the policy side but IsaacSim's recent friction modeling update and constant regression issues got me really frustrated so I decided to try out mj_warp. I feel like I've already fixed both Entity and Indexing class for spherical joints anyways so no worries. If I manage to get a fully trainable example I'll post a PR.