RRD footers 3: encoding/decoding manifests #12048

teh-cmc · 2025-12-02T13:03:06Z

LLM summary:

This PR implements encoding and decoding of RRD manifests in footers for the Rerun RRD file format. The changes enable random access to chunks within RRD files by storing metadata about chunk locations and properties in footer manifests.

Key changes:

Adds manifest building during encoding, tracking chunk metadata (offsets, sizes, entity paths, etc.)

Implements manifest parsing during decoding with transport-to-application conversion

Adds CLI support for displaying parsed footers (--footers flag) and recomputing manifests during routing (--recompute-manifests flag)

To which I actually don't have all that much to add.

This PR is basically all the remaining glue so that, whenever one uses our Encoder or one of our Decoder variants, RRD footers and manifests will automagically be computed, injected and serialized/deserialized.

The most important part of this PR is arguably the addition of a footer_roundtrip test, that encodes a recording and then manually decodes all of its chunks directly using the generated RRD manifest, instead of using a Decoder.

Part of RRD footers series of PRs:

github-actions · 2025-12-02T13:03:32Z

Web viewer built successfully.

Result	Commit	Link	Manifest
✅	`92b97b9`	https://rerun.io/viewer/pr/12048	`+nightly` `+main`

View image diff on kitdiff.

^{Note: This comment is updated whenever you push a commit.}

github-actions · 2025-12-02T13:03:40Z

Latest documentation preview deployed successfully.

Result	Commit	Link
✅	`92b97b9`	https://landing-7t8x3cba1-rerun.vercel.app/docs

^{Note: This comment is updated whenever you push a commit.}

Copilot

Pull request overview

This PR implements encoding and decoding of RRD manifests in footers for the Rerun RRD file format. The changes enable random access to chunks within RRD files by storing metadata about chunk locations and properties in footer manifests.

Key changes:

Adds manifest building during encoding, tracking chunk metadata (offsets, sizes, entity paths, etc.)
Implements manifest parsing during decoding with transport-to-application conversion
Adds CLI support for displaying parsed footers (--footers flag) and recomputing manifests during routing (--recompute-manifests flag)

Reviewed changes

Copilot reviewed 17 out of 18 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
docs/content/reference/cli.md	Documents new CLI flags for footer display and manifest recomputation
crates/top/rerun/src/commands/stdio.rs	Updates stream reading to return RRD manifests alongside messages
crates/top/rerun/src/commands/rrd/route.rs	Adds `--recompute-manifests` flag to enable footer generation during routing
crates/top/rerun/src/commands/rrd/print.rs	Implements `--footers` flag to display parsed footer manifests
crates/top/rerun/src/commands/rrd/merge_compact.rs	Updates to handle new return signature with footer metadata
crates/top/rerun/src/commands/rrd/filter.rs	Updates to handle new return signature with footer metadata
crates/store/re_log_encoding/tests/snapshots/*	Adds test snapshots for manifest data and schema validation
crates/store/re_log_encoding/tests/footer_roundtrip.rs	Comprehensive roundtrip test for footer encoding/decoding
crates/store/re_log_encoding/src/rrd/log_msg.rs	Implements Encodable/Decodable traits for RrdManifest
crates/store/re_log_encoding/src/rrd/encoder.rs	Implements manifest building during message encoding
crates/store/re_log_encoding/src/rrd/decoder/stream.rs	Adds rrd_manifests() method to async decoder
crates/store/re_log_encoding/src/rrd/decoder/state_machine.rs	Accumulates manifests during decoding and implements manifest extraction
crates/store/re_log_encoding/src/rrd/decoder/iterator.rs	Adds rrd_manifests() method to sync decoder iterator
crates/store/re_log_encoding/Cargo.toml	Adds insta test dependency

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

crates/store/re_log_encoding/src/rrd/encoder.rs

crates/store/re_log_encoding/tests/footers_and_manifests.rs

teh-cmc · 2025-12-02T17:21:53Z

crates/store/re_log_encoding/tests/footers_and_manifests.rs

 }

+#[test]
+fn footer_roundtrip() {


Arguably the most important part of this PR.

(really nice test!)

I'll actually apply it once these PRs are merged: * #12044 * #12047 * #12048

zehiko

went through all 3 PRs, it was really nicely documented and easy to follow, from the state machine updated, through rrd manifest builder (which I luckily knew bit about). nice work!

zehiko · 2025-12-03T07:58:04Z

crates/store/re_log_encoding/tests/footers_and_manifests.rs

 }

+#[test]
+fn footer_roundtrip() {


(really nice test!)

emilk

Yay!

emilk · 2025-12-03T08:02:35Z

crates/store/re_log_encoding/src/rrd/decoder/state_machine.rs

+    /// RRD manifests are parsed from footers, of which there might be more than one e.g. in the
+    /// case of concatenated streams.
+    ///
+    /// This is not cheap: it automatically performs the transport to app level conversion.


Maybe add a re_tracing::profile_function!(); to it then

emilk · 2025-12-03T08:02:38Z

crates/store/re_log_encoding/src/rrd/decoder/iterator.rs

+    /// RRD manifests are parsed from footers, of which there might be more than one e.g. in the
+    /// case of concatenated streams.
+    ///
+    /// This is not cheap: it automatically performs the transport to app level conversion.


Maybe add a re_tracing::profile_function!(); to it then

emilk · 2025-12-03T08:02:57Z

crates/store/re_log_encoding/src/rrd/decoder/state_machine.rs

                        if !bytes.is_empty() {
                            let rrd_footer =
                                re_protos::log_msg::v1alpha1::RrdFooter::from_rrd_bytes(&bytes)?;
-                            _ = rrd_footer; // TODO(cmc): we'll use that in the next PR, promise.


a man of your word 👍

emilk · 2025-12-03T08:04:31Z

crates/store/re_log_encoding/src/rrd/encoder.rs

+struct ManifestState {
+    /// The accumulated recording IDs of each individual chunk, extracted from their `LogMsg`.
+    ///
+    /// This will only be used if [`FooterState::recording_id_scope`] is empty.
+    recording_ids: Vec<re_log_types::StoreId>,


This will in most cases be the same store id repeated, correct?

emilk · 2025-12-03T08:04:56Z

crates/store/re_log_encoding/src/rrd/encoder.rs

+        byte_offset: u64,
+        byte_size: u64,


Can use re_span::Span here

Suggested change

byte_offset: u64,

byte_size: u64,

bytes: Span<u64>,

emilk · 2025-12-03T08:05:35Z

crates/store/re_log_encoding/src/rrd/encoder.rs

+                // bit weirder on the other hand, but then again this is generally not a new a
+                // problem: we tend to perform Sorbet migrations a bit too aggressively all over
+                // the place. We really need a layer that sits between the transport and
+                // application layer where one can accessed the parsed, unmigrated data.


emilk · 2025-12-03T08:08:43Z

crates/store/re_log_encoding/tests/footers_and_manifests.rs

+    // Right now, the implemented behavior is that we end up with an empty footer, i.e. there are
+    // no manifests in it.
+    // Whether that's the correct behavior is another question, but at least it is defined for now
+    // and can be changed.


I love these kind of comments ⭐⭐⭐

teh-cmc · 2025-12-03T08:19:42Z

@rerun-bot full-check

github-actions · 2025-12-03T08:20:14Z

Started a full build: https://github.com/rerun-io/rerun/actions/runs/19887096962

teh-cmc added 📉 performance Optimization, memory use, etc do-not-merge Do not merge this PR include in changelog 🔩 data model Sorbet 🪵 Log & send APIs Affects the user-facing API for all languages dataplatform Rerun Data Platform integration labels Dec 2, 2025

teh-cmc changed the title ~~RRD footers 2: encoding/decoding manifests~~ RRD footers 3: encoding/decoding manifests Dec 2, 2025

This was referenced Dec 2, 2025

RRD footers 1: framing #12044

Open

RRD footers 2: RRD manifests #12047

Open

teh-cmc requested a review from Copilot December 2, 2025 13:41

Copilot AI reviewed Dec 2, 2025

View reviewed changes

crates/store/re_log_encoding/src/rrd/encoder.rs Show resolved Hide resolved

crates/store/re_log_encoding/tests/footers_and_manifests.rs Show resolved Hide resolved

teh-cmc force-pushed the cmc/rrd_footers_2_rrd_manifests branch 2 times, most recently from 964c086 to c1e6776 Compare December 2, 2025 15:51

teh-cmc added 6 commits December 2, 2025 18:10

encoding RRD manifests into footers

40874db

decoding RRD manifests and pulling them out of the state machine

8d3eb2e

add footer roundtrip test suite

a090291

snapshots

1cd77a0

expose RRD manifests all through the CLI

99a9e68

option to recompute manifests in rrd route

92b97b9

teh-cmc force-pushed the cmc/rrd_footers_3_encoding_decoding branch from f1ad3fb to 92b97b9 Compare December 2, 2025 17:14

teh-cmc commented Dec 2, 2025

View reviewed changes

teh-cmc marked this pull request as ready for review December 2, 2025 17:26

emilk mentioned this pull request Dec 2, 2025

Codify how to group our imports #12063

Merged

emilk added a commit that referenced this pull request Dec 2, 2025

Codify how to group our imports (#12063)

1e44c8e

I'll actually apply it once these PRs are merged: * #12044 * #12047 * #12048

zehiko approved these changes Dec 3, 2025

View reviewed changes

crates/store/re_log_encoding/tests/footers_and_manifests.rs

}

#[test]

fn footer_roundtrip() {

Copy link

Member

zehiko Dec 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(really nice test!)

emilk approved these changes Dec 3, 2025

View reviewed changes

RRD footers 3: encoding/decoding manifests #12048

Are you sure you want to change the base?

RRD footers 3: encoding/decoding manifests #12048

Conversation

teh-cmc commented Dec 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Dec 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Dec 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zehiko left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

emilk left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

teh-cmc commented Dec 3, 2025

Uh oh!

github-actions bot commented Dec 3, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

teh-cmc commented Dec 2, 2025 •

edited

Loading

github-actions bot commented Dec 2, 2025 •

edited

Loading

github-actions bot commented Dec 2, 2025 •

edited

Loading