Skip to content

Commit ae8b0c4

Browse files
committed
doc: update CHUNKS.md with extended layer info
Signed-off-by: Eduardo Silva <[email protected]>
1 parent 3748d54 commit ae8b0c4

File tree

1 file changed

+44
-0
lines changed

1 file changed

+44
-0
lines changed

CHUNKS.md

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -91,6 +91,50 @@ Content Data < | | records | |
9191
Fluent Bit API provides backward compatibility with the previous metadata and content
9292
format found on series v1.8.
9393

94+
Starting with the Fluent Bit release that introduces direct route persistence, the
95+
fourth metadata byte now carries feature flags. A zero value preserves the legacy
96+
layout, while a non-zero value indicates that additional structures follow the tag.
97+
When the ``FLB_CHUNK_FLAG_DIRECT_ROUTES`` bit is set the tag is terminated with a
98+
single ``\0`` byte and a routing payload is appended. Fluent Bit v3.0 and later
99+
also set the ``FLB_CHUNK_FLAG_DIRECT_ROUTE_LABELS`` bit to store each destination's
100+
alias (or generated name) alongside its numeric identifier so routes can survive
101+
configuration changes that renumber outputs. If any stored identifier exceeds
102+
65535 the ``FLB_CHUNK_FLAG_DIRECT_ROUTE_WIDE_IDS`` bit is enabled and each ID is
103+
encoded using four bytes so large configurations remain routable after a restart:
104+
105+
```
106+
Metadata
107+
-- +---------+-------+
108+
/ | 0xF1 | 0x77 | <- Magic Bytes
109+
/ +---------+-------+
110+
< | Type | Flags | <- Chunk type and flag bits
111+
\ +---------+-------+
112+
\ | Tag | <- Tag string (no size prefix)
113+
+---------+-------+
114+
| 0x00 | | <- Tag terminator (present when flags != 0)
115+
+---------+-------+
116+
| Routing Length | <- uint16_t big endian
117+
+---------+-------+
118+
| Route Count | <- uint16_t big endian
119+
+---------+-------+
120+
| Output IDs ... | <- Each stored as uint16_t (big endian)
121+
| | or uint32_t when FLB_CHUNK_FLAG_DIRECT_ROUTE_WIDE_IDS
122+
+---------+-------+
123+
| Label Lens ... | <- Present when FLB_CHUNK_FLAG_DIRECT_ROUTE_LABELS
124+
+---------+-------+
125+
| Label Bytes ... | <- Concatenated label strings (no terminators)
126+
-- +-----------------+
127+
```
128+
129+
The routing payload captures the direct route mapping so that filesystem chunks
130+
loaded by the storage backlog re-use the same outputs after a restart. Chunks
131+
without direct routes keep the legacy layout (flags byte set to zero) and remain
132+
fully backwards compatible across Fluent Bit versions. When labels are stored the
133+
reader first reconstructs routes by matching aliases or numbered names and only
134+
falls back to numeric identifiers if the textual metadata cannot be matched. This
135+
ensures that chunks continue to flow to the intended destinations even when the
136+
output configuration is re-ordered.
137+
94138
### Fluent Bit <= v1.8
95139

96140
Up to Fluent Bit <= 1.8.x, the metadata and content data is simple, where metadata

0 commit comments

Comments
 (0)