4dgs discussion

[slimbuck]

Introduction

Supersplat editor currently supports "ply sequences" which are just multiple named PLY files. The files can be loaded and displayed one after the other, like a flipbook animation. The editor can currently play the sequences only, editing is not possible.

We consider what might be required to properly support "animated gaussian" content (also referred to as 4dgs) in terms of tooling and runtime.

We do not consider which particular trainer or technique should be used to generate the data, as there are multiple options and currently no specific obvious best toolset or approach.

Challenges

The extra (temporal) dimension in 4dgs means:

• source data is orders of magnitude larger than before
• tooling must become time-aware to be useful
  • if the user edits a gaussian in one frame, the edit most likely should be applied to all frames
  • this accounts for almost any operation, deleting, moving, coloring ,etc

Data pipeline

We imagine a 4dgs pipeline will look something like:

1. Capture & Train
 ↓
2. PLY sequence (📁)
 ↓
3. Edit
 ↓
4. Compress
 ↓
5. Runtime format (📁)
 ↓
6. Deliver & Consume

Intermediate file format (PLY sequence)

The intermediate representation of 4dgs content will be output by training tools and used for further editing and processing, including being used as input to the final 4dgs compression step. We hope the various training tools and approaches can output this format for general processing by tools.

The intermediate representation is just a ply sequence (as it is now), but with certain assumptions about the data layout.

Each frame of the sequence is a fully-specified ply model, but is implicitly grouped:

• the first frame in the group is an "i-frame" which defines the starting point for all gaussians in a group of related frames
• subsequent "p-frames" only modify a subset of the values from the base i-frame
• all gaussians in a group are co-located: the data for a gaussian will always appear at the same index n in every frame of the group

Note

It is also possible for p-frames to only include the animating/changing properties. Fully specifying all PLY frame data is useful for viewers and existing PLY tools, however the resulting data could prove to be prohibitively large. Since 4dgs tools will have to be i-frame aware anyway, it may be preferable for i-frames to store only deltas.

Runtime format

The runtime compressor should be able to inspect the intermediate frames and automatically infer the group size and animating attributes.

The exact format would be:
TODO :)

Conclusion

A standard intermediate format is a simple extension to PLY which would form the bridge between sources of 4dgs data and all the downstream tooling for editing and compressing.

Notes / Questions

• should we already be considering LOD with 4dgs? cc @mvaligursky
• which trainers & papers currently look the most compelling?