Running with Multi Nodes

[audedeThezy]

Hi,

I'm trying to run the vocalocator (Version: 1.1.0) using the provided multi_node_environment_1.json5 config, and I’d like to include orientation as part of the prediction (in addition to X, Y, Z).

The README states:

"If using multiple nodes (e.g. for inference on orientation in addition to position), use the DATA/NODES_TO_LOAD entry in the config JSON file to list all nodes in the dataset that will be used for training."

I've tried the following two dataset/config combinations, but both result in the same runtime error:

---

HDF5:

• /locations shape: (n, 4, 1)
• /node_names: ["X", "Y", "Z", "ORIENTATION"]

Config:
"NODES_TO_LOAD": ["X", "Y", "Z", "ORIENTATION"],
"OUTPUT_TYPE": "GAUSSIAN_4D_ORIENTED"
Error
RuntimeError: shape '[32, 4]' is invalid for input of size 64

---

HDF5:

• /locations shape: (n, 1, 4)
• /node_names: ["X", "Y", "Z", "ORIENTATION"]

Config:
"NODES_TO_LOAD": [0],
"OUTPUT_TYPE": "GAUSSIAN_4D_ORIENTED"
Error
RuntimeError: shape '[32, 4]' is invalid for input of size 64
(But is running correclty with the xcorr_config_environment_1.json5)

---

I think I may be misunderstanding how nodes and dimensions are treated differently by the model - and how data are supposed to be shaped for gaussian_4D_oriented output type

Many thanks for your help,
Aude

[ralphpeterson]

@Aramist do some of the patches you worked on for the mouse data work here?

[Aramist]

Hi Aude,

Thanks for reaching out! It seems there's been some confusion about how the node dimension is used in the locations dataset.

In our system, each "node" represents a specific point, or joint, on the animal's body. For output types that make use of orientation, we typically provide the labels via two nodes: the first being the nose or mouth, and the second being the center of the head or another point along the body which reliably establishes orientation. For each vocalization, our system internally takes the difference between these two points to compute the orientation.

In your setup, since the orientation is already precomputed as a scalar, there are two ways you could restructure your data to be compatible with the expected input format:

1. Synthesize a second node:
   You can artificially construct a second point by adding a small vector in the direction of your orientation angle to your existing locations. This will yield a (n, 2, 3) locations dataset compatible with the GAUSSIAN_3D_ORIENTED output type.
2. Use the original tracking data (if available):
   If you computed the orientation from a second tracked point on the animal and still have this available, you can stack this second point with your current locations to obtain a (n, 2, 3)-shaped locations array. This approach is compatible with both the GAUSSIAN_3D_ORIENTED and GAUSSIAN_6D_ORIENTED output types. The primary distinction between these is that the 6D output type models both the pitch and the yaw of the animal whereas the 3D output type only considers yaw.

Please let us know if you have additional questions