keypoints format

[juanbratti]

Hola!! Estuve explorando el dataset, y quería consultar una cierta discrepancia entre el paper y el contenido del archivo keypoints.h5. En el paper se menciona que para obtener los keypoints se utiliza AlphaPose con el Hape fullbody format, teniendo así 136 keypoints.
Analizando el archivo h5 con Python, obtengo que cada dataset de cada signer identificado por clip, tiene (por ejemplo) shape (225, 2172). Entiendo que el 225 es por la cantidad de frames, pero no me queda claro como se interpretan las 2172 columnas, ya que dividiendo por 4 (x,y,z,confidence) se obtiene 543 (keypoints?). Quería preguntar si se cambió el formato de los keypoints o es una mala interpretación mía.

Desde ya gracias!

[juanbratti]

For future readers, it seems the new keypoints file was recreated using MediaPipe Holistic, where the amount of keypoints is 543 and the mapping of them is the following:

Let the following array be the keypoints of the first frame of a specific signer in keypoints.h5:

[1.294e+03, 3.805e+02, nan, ..., 6.750e+02, -3.030e-02, nan] with shape (1, 2172)

if we reshape this to (543, 4):

columns: x, y, z, k
[ row0: [ 1.294e+03, 3.805e+02, nan, 1.000e+00]
row1: [ 1.314e+03, 3.422e+02, nan, 1.000e+00]
row2: [ 1.329e+03, 3.430e+02, nan, 1.000e+00]
...
row540: [ 1.462e+03, 6.655e+02, -3.778e-02, nan]
row541: [ 1.475e+03, 6.625e+02, -3.354e-02, nan]
row542: [ 1.486e+03, 6.750e+02, -3.030e-02, nan]]

what each group of rows represent is:

• Rows 0–32: Pose – 33 points. Here, z is the depth (not calculated), setted to nan for all values. k represents visibility of the point (either 0 or 1)
• Rows 33–500: Face – 468 points. Here, z represents the landmark depth with the depth at center of the head being the origin. k is nothing, setted to nan for all values.
• Rows 501–521: Left hand – 21 points. Here z represents the landmark depth with the depth at the wrist being the origin. k is nothing, setted to nan for all values.
• Rows 522–542: Right hand – 21 points. Same as left hand.

Reference paper: https://sedici.unlp.edu.ar/bitstream/handle/10915/176192/Documento_completo.pdf?sequence=1
mediapipe docs: https://github.com/google-ai-edge/mediapipe/blob/master/docs/solutions/holistic.md