[refactor] Simplify block structure by removing unnecessary serialization/deserialization

[cendhu]

Simplify block structure by removing unnecessary serialization/deserialization

Summary

The current block structure carries several fields as serialized byte arrays that are then deserialized on every access. This introduces unnecessary CPU overhead and code complexity. We should simplify the block structure by using typed fields directly and removing unused fields.

Current Block Structure in Fabric

message Block {
  BlockHeader header = 1;
  BlockData data = 2;
  BlockMetadata metadata = 3;
}
message BlockHeader {
  uint64 number = 1;
  bytes previous_hash = 2;
  bytes data_hash = 3;
}
message BlockData {
  repeated bytes data = 1;  // each entry is a serialized Envelope
}
message Envelope {
  bytes payload = 1;    // serialized Payload
  bytes signature = 2;
}
message Payload {
  Header header = 1;
  bytes data = 2;
}
message Header {
  bytes channel_header = 1;    // serialized ChannelHeader
  bytes signature_header = 2;  // serialized SignatureHeader
}
message ChannelHeader {
  int32 type = 1;
  int32 version = 2;
  google.protobuf.Timestamp timestamp = 3;
  string channel_id = 4;
  string tx_id = 5;
  uint64 epoch = 6;
  bytes extension = 7;
  bytes tls_cert_hash = 8;
}
message SignatureHeader {
  bytes creator = 1;
  bytes nonce = 2;
}

Note how the structure is deeply nested with serialized bytes at every level: Block.data → Envelope.payload → Payload.header.channel_header / Payload.header.signature_header. Each layer requires a separate proto.Unmarshal call to access the typed fields within.

Problem

In the current implementation, various components of the block are stored as serialized []byte within the protobuf message and must be deserialized each time they are accessed. For example:

• Signature Header: The SignatureHeader is embedded as raw bytes inside the payload. Every consumer of this field must unmarshal it before use, even though it could be stored as a typed struct directly.
• Common Header: Similarly, the CommonHeader (channel header + signature header) is serialized into bytes within the Header field of the payload. Accessing channel ID, tx type, or timestamp requires repeated deserialization.
• Envelope payload: The Payload inside each Envelope is carried as bytes, requiring deserialization at each processing stage (validation, commit, indexing, etc.).

This pattern leads to:

1. Redundant CPU work — the same fields are deserialized multiple times across the transaction lifecycle (endorsement, ordering, validation, commit).
2. Verbose boilerplate — every call site needs error-handling logic around proto.Unmarshal for what are conceptually direct field accesses.
3. Increased GC pressure — repeated deserialization creates short-lived objects that add to garbage collection overhead.

Proposal

1. Use typed structs instead of byte slices — Replace serialized byte fields (e.g., SignatureHeader, ChannelHeader) with their corresponding typed protobuf message fields in the internal block representation.
2. Deserialize once at ingress — Parse the block fully when it is first received (e.g., at the orderer or committer ingress) and pass the fully typed structure through the pipeline.
3. Remove unused fields — Audit the block and envelope structures for fields that are never read or are redundant, and remove them from the internal representation.
4. Serialize only at egress — Re-serialize to the wire format only when the block needs to be persisted or transmitted over the network.

Example

Before:

// Every call site repeats this pattern
payload := &cb.Payload{}
err := proto.Unmarshal(envelope.Payload, payload)
// ...
header := &cb.SignatureHeader{}
err = proto.Unmarshal(payload.Header.SignatureHeader, header)
// ... finally use header.Creator, header.Nonce

After:

// Direct typed access — deserialized once at ingress
creator := envelope.Payload.Header.SignatureHeader.Creator
nonce := envelope.Payload.Header.SignatureHeader.Nonce

Expected Benefits

• Reduced CPU usage from eliminating redundant proto.Unmarshal calls across the transaction pipeline
• Simpler, more readable code with direct field access
• Lower GC pressure from fewer intermediate allocations
• Smaller internal block representation after removing unused fields

Backward-Compatible Evolution via oneof

If we need to introduce a new SignatureHeader type (e.g., one with typed fields or additional metadata), we could potentially leverage protobuf's oneof to do so without breaking backward compatibility:

// Current Header definition (see above)
message Header {
  bytes channel_header = 1;    // serialized ChannelHeader
  bytes signature_header = 2;  // serialized SignatureHeader
}
// Proposed Header with oneof for backward-compatible evolution
// Since Header only has fields 1 and 2, field 3 is safe to use.
// ⚠️  For messages with more fields, always verify the next unused field number.
message Header {
  bytes channel_header = 1;    // existing: serialized ChannelHeader
  oneof signature_header_type {
    bytes signature_header = 2;                // existing: serialized SignatureHeader
    SignatureHeaderV2 signature_header_v2 = 3; // new: typed, uses next available field number
  }
}

How this could work:

• Existing nodes that only understand the old format would continue reading field 2 (signature_header bytes) and deserializing as before.
• Newer nodes would populate and read field 3 (signature_header_v2) directly, avoiding the serialize/deserialize overhead.
• Since oneof fields are mutually exclusive, only one representation is present on the wire at a time — no storage overhead from carrying both.
• The same oneof pattern could also be applied to channel_header (field 1) in the future if needed.

⚠️ Critical: New oneof members must use unused field numbers. If the existing message has additional fields beyond the oneof candidates, the new oneof member cannot reuse an occupied field number. Protobuf will either reject the descriptor outright ("duplicate field number") or, when old and new nodes run different schema versions, cause silent data corruption — the old node would misinterpret the serialized embedded message bytes as the original field's data. The safe approach is to always use the next available unused field number.

Considerations:

• Wire compatibility is confirmed for old → new direction: Old nodes can produce data using field 2 (bytes) and new nodes correctly identify it via the oneof discriminator. New nodes can also continue writing the old bytes format for full backward compatibility.
• New V2 → old nodes requires gating: When a new node writes using signature_header_v2 (field 3), old nodes see an empty signature_header — the new field is unknown to them and silently ignored. This means a capability flag or channel configuration update is required to gate when the new format is used during rolling upgrades.
• Field number safety varies by message: For Header specifically, field 3 is safe since only fields 1 and 2 exist today. When applying this pattern to other messages (e.g., ChannelHeader which has fields up to 8), care must be taken to use a truly unused field number.
• Ledger persistence: If the block is persisted to the ledger in the new format, older binaries reading historical blocks post-upgrade must handle both variants. This may require a migration strategy or dual-write period.

This approach is viable but requires a clear upgrade path and capability gating before it can be adopted.

Compatibility Matrix

Scenario	Result
Old node writes → New node reads	✅ Fully compatible — new node reads bytes via oneof
New node writes V2 → Old node reads	⚠️ Old node sees empty signature_header — V2 field is unknown
New node writes old format → Old node reads	✅ Fully compatible — new node can still produce old wire format
Oneof mutual exclusivity	✅ Setting one field correctly clears the other
Field number collision (reusing existing field number for oneof)	❌ Silent data corruption — old node misinterprets V2 bytes as the original field
Safe approach (unused field number for oneof)	✅ All existing fields preserved, V2 invisible to old nodes

[cendhu]

@ale-linux @adecaro @tock-ibm @liran-funaro @mbrandenburger @HagarMeir

Thanks to Bob for the layout and text improvements. We need to lock down the block structure now as we near production. Modifying this later will be painful due to backward compatibility requirements.

For instance, a peer joining at v5 must be able to validate blocks from v1. Even with snapshots, full history compatibility is mandatory, so we should finalize this spec immediately.

If we decide to modify this block structure, the impact on the block store will be significant. I can take care of that, as I am familiar with the code.

[adecaro]

I think the problem here can be the verification of the signature carried by Envelope. If the payload field is not bytes anymore, protobuf does not guarantee that it will be serialized in the same way the producer of the signature did.

The same consideration must be applied to the payload signed by the endorsers.

[cendhu]

@adecaro Protobuf does not guarantee deterministic serialization across versions. They specify this explicitly. Hence, in the committer, we have already moved to an ASN1 schema and marshaling for computing the bytes and verifying the signatures. We can do the same for the envelope payload too.

https://github.com/hyperledger/fabric-x-common/blob/main/api/applicationpb/asn1_tx_schema.asn and https://github.com/hyperledger/fabric-x-common/blob/main/api/applicationpb/asn1.go

We can make changes only to BlockData and Payload.Header and retain the Payload as bytes (as shown below). Another option is to make payload a Payload object itself rather than bytes.

message BlockData {
  repeated Envelope data = 1;
}

message Envelope {
  bytes payload = 1;
  bytes signature = 2;
}

message Payload {
  Header header = 1;
  bytes data = 2;
}

message Header {
  ChannelHeader channel_header = 1;
  SignatureHeader signature_header = 2;
}

[cendhu]

For the following two messages, we can certainly use message name directly instead of bytes. This itself would save three unnecessary unmarshaling.

message BlockData {
  repeated bytes data = 1;  // each entry is a serialized Envelope. -- can be replaced with repeated Envelope data = 1;
}

message Header {
  bytes channel_header = 1;    // serialized ChannelHeader -- can be replaced with ChannelHeader channel_header = 1;
  bytes signature_header = 2;  // serialized SignatureHeader -- can be replaced with SignatureHeader signature_header = 2;
}

[tock-ibm]

What is the performance benefit of doing this, in numbers? How much performance gain do we hope to gain, relative to the performance of a full system without this change? i.e. even if this speeds up block data access by a lot, this may be a fraction of the overall performance of a system because the bottlenecks are elsewhere (e.g. sig verification, communication costs, storage IOPS, etc).
What are the alternative approaches? i.e. in memory - in process - optimizations that avoid unnecessary marshalling & unmarshalling?

[cendhu]

@tock-ibm The main reason for creating this issue is based on observations made during benchmarking of the committer on an AWS machine. Initially, the sidecar was the bottleneck. In the first run, we achieved only 83k TPS. The primary bottleneck was synchronous writing to the block store at the sidecar. Writing each block (500 transactions) took around 6ms. Thus, 1000/6 ≈ 166 blocks/sec, which translates to 83k TPS. We changed this to periodic syncing at a configured interval.

Subsequently, the bottleneck shifted to the mapBlock() method in the sidecar, which performs envelope and transaction unmarshaling. While the latter is unavoidable, the former is not. urrently, instead of unmarshaling the envelope, header, etc., we use the protowire library to parse the bytes and fetch the required data. The PR is under review here: hyperledger/fabric-x-committer#316

IMO, such optimization may be fragile and adds unnecessary complexity to the code. The current design utilizes unnecessary CPU, allocates extra memory, and increases GC pressure, whereas this could be easily fixed by adjusting the block proto.
However, as shown in the PR, the performance gain is real: the sidecar was able to process 1.7x more blocks. Without addressing this bottleneck, we would not have achieved more than 200k TPS; now, we are at 440k TPS (there were few other optimizations too).

If we do not fix the proto in the first release, it will be almost impossible to fix later without adding significant complexity to the code.

[mbrandenburger]

@cendhu Thank you for this proposal; It would be great if we can simplify the block/envelope de-serialization.

While I understand the perf boost on the committer (sidecar), do you think it will make an impact on the orderer (router)? If I understand correctly, the router does not need to "process" the envelop payload; with this change, the router would unmarshal the entire envelope, instead of stopping at the envelope.Payload layer.

Moreover, payload fields as type []byte also give great flexibility for testing/developing, e.g., the orderer workload generator can test just send random bytes of different sizes via the payload field. I am wondering if something like that would be still possible.

[cendhu]

@mbrandenburger We can keep the data field in the payload as bytes but some extra work for arma load-generator. I don't know what fields are used in the orderer. @tock-ibm

[tock-ibm]

message BlockData {
  repeated bytes data = 1;  // each entry is a serialized Envelope. -- can be replaced with repeated Envelope data = 1;
}

In the orderer, there is a big advantage to this structure. When we replicate blocks among parties, we have to verify the consenter signatures on the block. To do this, one of the steps is to recalculate the block data hash, and verify it is the same as stated in the header. The we calculate the header hash and verify the sig against it. If we get a slice of Envelopes instead of a a slice of bytes, we then have to marshal each envelope again to bytes in order to calculate the data hash. This is also true for peers pulling form the ordering service, having Envelopes there means you will have to marshal each one of them.