rtmw3d: keypoints z prediction

[philchaub]

i've tried rtmw3d in inference, and i'm very surprised in Z prediction results.
i've tried on several images, with classical poses (only one character, walking) .
As explained in the documentation, I thought the values ​​on the Z axis would be distributed around the root, and therefore the pelvis.
So I expected positive and negative values ​​whose amplitude would be consistent with the scale of the x and y values.
But what I get are only positive values ​​distributed over a very short scale. For example, left hip = [219.62266540527344, 381.46295166015625, 6.810750484466553] and right hip = [135.74488830566406, 385.10980224609375, 6.794205188751221]

how it's possible ?

[michele-akina]

@philchaub My understanding of the output of RTMPose3D is that xy are in pixel values, but the z is in the range of the training dataset (should be something like [-2.17, 2.17]). This is expected from the model. If you visualize or use the landmarks in this form, they will obvioulsy appear squashed on the xy plane, so you have two options, if you want a human skeleton with reasonable proportions:

• scale down the xy coordiantes to the scale of z, for example by clipping thme in a range of [-1, 1]. A tip here is to use the same scale for x and y, when normalizing (e.g. 384, hte model input height)

let x = to_clip_range(lm.x, config.input_height) 
let y = to_clip_range(lm.x, config.input_height) 

fn to_clip_range(lm: f32, scale: f32) -> f32 {
    // from scale to clip range [-1, 1]
    (lm / scale - 0.5) * 2.0
}

• scale up the z coordinates to the xy scale. I would NOT recommend this because z is much noisier, and you will amplify this.

This is a bit of a heuristic approach. Once you have xy in clip space you can "tune" their range more easily to have realistic proportions.
In the mmpose repo, there is also another option that includes using the camera pinhole model to bring everything in metric space. I tried this but it does not work very well for unseen data because it makes big assumptions about the z_offset, I think this is an average taken from the trainining dataset. It also requires camera intrinsics, which are troublesome to get for most users.