Visualize Ham2Pose predictions

[m7mdhka]

Hello,

How I can save the Ham2Pose model predictions as .pose format for use it in pose_to_video?

def pred(model, dataset, output_dir, gen_k=30, vis=True, subset=None):
    os.makedirs(output_dir, exist_ok=True)
    _, num_pose_joints, num_pose_dims = dataset[0]["pose"]["data"].shape
    pose_header = dataset[0]["pose"]["obj"].header
    preds = []

    model.eval()
    with torch.no_grad():
        for i, datum in enumerate(dataset):
            if subset is not None and datum["id"] not in subset:
                continue
            if i >= gen_k and subset is None:
                break
            first_pose = datum["pose"]["data"][0]
            seq_iter = model.forward(text=datum["text"], first_pose=first_pose.cuda())
            for j in range(model.num_steps):
                seq = next(seq_iter)
                    
            if vis:
                visualize_seq(seq, 
                  pose_header, 
                  output_dir, 
                  datum["id"], 
                  #datum["pose"]["obj"]
                )
            else:
                data = torch.unsqueeze(seq, 1).cpu()
                conf = torch.ones_like(data[:, :, :, 0])
                pose_body = NumPyPoseBody(25, data.numpy(), conf.numpy())
                predicted_pose = Pose(pose_header, pose_body)
                pose_hide_legs(predicted_pose)
                preds.append(predicted_pose)
    return preds

these are the predictions of Ham2Pose.

[AmitMY]

preds contains a list of Pose objects.
The pose library specifies a way to write that into a file.
You can therefore do something like:

for i, pose in enumerate(preds):
  with open(f"output{i}.pose", "wb") as f:
    pose.write(f)

[m7mdhka]

preds contains a list of Pose objects. The pose library specifies a way to write that into a file. You can therefore do something like:

for i, pose in enumerate(preds):
  with open(f"output{i}.pose", "wb") as f:
    pose.write(f)

I'm trying that, it's just saved 1 file "output0.pose". However, I tried to use it in pose_to_video but I got this error:

Loading input pose ...
Traceback (most recent call last):
  File "/usr/local/bin/pose_to_video", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.10/dist-packages/pose_to_video/bin.py", line 39, in main
    correct_wrists(pose)
  File "/usr/local/lib/python3.10/dist-packages/pose_format/utils/generic.py", line 134, in correct_wrists
    pose = correct_wrist(pose, 'LEFT')
  File "/usr/local/lib/python3.10/dist-packages/pose_format/utils/generic.py", line 117, in correct_wrist
    wrist_index = pose.header._get_point_index(f'{hand}_HAND_LANDMARKS', 'WRIST')
  File "/usr/local/lib/python3.10/dist-packages/pose_format/pose_header.py", line 318, in _get_point_index
    raise ValueError("Couldn't find component")
ValueError: Couldn't find component

[AmitMY]

pose_to_video is designed with Mediapipe poses, while Ham2Pose is OpenPose.
You should still be able to generate videos, but they would be of lesser quality.
To do so, there would need to be a fix here, supporting openpose as well:
https://github.com/sign-language-processing/pose/blob/master/src/python/pose_format/utils/generic.py#L116-L130

Please upload the .pose file you are using here, for future reference

[m7mdhka]

I'm sorry, I can't upload it as .pose due to github limitations.
.pose as a .zip file:
output0.zip

[m7mdhka]

I've tried to edit it, for generic.py
I've updated it:

def correct_wrist(pose: Pose, hand: str) -> Pose:
    wrist_index = pose.header._get_point_index(hand, 'BASE')
    wrist = pose.body.data[:, :, wrist_index, :]
    wrist_conf = pose.body.confidence[:, :, wrist_index]

    body_wrist_index = 135
    body_wrist = pose.body.data[:, :, body_wrist_index, :]
    body_wrist_conf = pose.body.confidence[:, :, body_wrist_index]

    new_wrist_data = ma.where(wrist.data == 0, body_wrist, wrist)
    new_wrist_conf = ma.where(wrist_conf == 0, body_wrist_conf, wrist_conf)

    pose.body.data[:, :, body_wrist_index, :] = ma.masked_equal(new_wrist_data, 0)
    pose.body.confidence[:, :, body_wrist_index] = new_wrist_conf
    return pose


def correct_wrists(pose: Pose) -> Pose:
    pose = correct_wrist(pose, 'hand_right_keypoints_2d')
    pose = correct_wrist(pose, 'hand_left_keypoints_2d')
    return pose

Then I'm faced an error in this line pose_to_video/conditional/controlnet/init.py

which is this:
ValueError: operands could not be broadcast together with shapes (44,1,137,2) (3,)

I reshaped it:

scaling_array = np.array([scale_w, scale_h]).reshape((1, 1, 1, 2))

but this is the result of the generated image:

controlnet-animatediff-upscaled-3.mp4

[m7mdhka]

I've also using pose_converter

The result is the same.

[AmitMY]

You are getting this result, because your pose data is not correct.

If you run:

visualize_pose -i output0.pose -o output.mp4

You get an empty video.

But if you use the new --normalize option, you will see your pose: (new as of sign-language-processing/pose@be400fa)

(also, you will see your pose does not move, but that's on you)

So my recommendation for you is:

1. Normalize your pose (see code above)
2. Scale it to a good size (preferably even 512x512, for best performance with the model)