Handling missing HR motion vectors when converting custom NSS data to safetensors

[zhaozunjin]

Handling missing HR motion vectors when converting custom NSS data to safetensors

Hi, thanks for the excellent NSS implementation in neural-graphics-model-gym.
I’m currently integrating a custom dataset into the NSS pipeline and ran into a question regarding motion vector resolution requirements, especially when converting data into the official NSS safetensors format.

---

Available data

From the renderer, I only have the following per-frame .npy files:

• input_color_jittered.npy
• jitter.npy
• motion_vectors.npy ← low-resolution motion only
• depth.npy

In other words, I do not have native high-resolution motion vectors.

---

Current conversion to NSS safetensors

To satisfy the NSS loader and avoid shape-related errors, I currently generate a fake HR motion tensor by upsampling the LR motion:

motion_hr_tensor = F.interpolate(
    motion_tensor, scale_factor=2.0, mode="nearest"
) * 2.0

tensors = {
    "colour_linear": colour_tensor,
    # dummy GT to satisfy training pipeline
    "ground_truth_linear": colour_tensor.clone(),

    "motion_lr": motion_tensor.clone(),
    "motion": motion_hr_tensor,  # interpolated HR motion

    "depth": depth_tensor,
    "jitter": jitter_tensor,

    "depth_params": torch.tensor([[1.0, 0.0, 0.0, 0.0]] * T).float(),
    "outDims": torch.tensor([[H * 2, W * 2]] * T),
    "render_size": torch.tensor([[H, W]] * T),
    "scale": torch.ones((T, 1)) * 2.0,
    "seq": torch.ones((T, 1)) * 8897243125409831936,
}

NSS data format

My converted data format

This allows the model to run, but raises some concerns.

Summary of questions

• Is HR motion semantically required by NSS, or mainly a structural requirement?
• Are there any recommended practices for datasets that only provide LR motion?

Thanks a lot for your time, and for making NSS available — it’s been very helpful for understanding temporal super-resolution pipelines.