Block Aggregator Protobuf API #3100

MitchTurner · 2025-09-16T17:22:19Z

Linked Issues/PRs

#3089

Description

Checklist

Breaking changes are clearly marked as such in the PR description and changelog
New behavior is reflected in tests
The specification matches the implemented behavior (link update PR if changes are needed)

Before requesting review

I have reviewed the code myself
I have created follow-up issues caused by this PR and linked them here

After merging, notify other teams

[Add or remove entries as needed]

Rust SDK
Sway compiler
Platform documentation (for out-of-organization contributors, the person merging the PR will do this)
Someone else?

Note

Introduces a gRPC Protobuf Block Aggregator API with storage and serialization, integrates it into Fuel Core as an RPC service with CLI config, and adds tests and build support.

RPC/API:
- Add new crates/services/block_aggregator_api with Protobuf definitions (proto/api.proto), tonic server, and adapters (ProtobufAPI).
- Implement block serialization/deserialization to/from Protobuf (full coverage of block/tx variants).
Node Integration:
- Wire Block Aggregator service into fuel-core sub-services; add CLI args --rpc_ip/--rpc_port and rpc feature.
- Extend Config with rpc_config; start service alongside GraphQL/others.
Storage:
- Add block_aggregation DB to CombinedDatabase; define BlockAggregatorDatabase and storage tables for Protobuf Blocks.
- Implement BlockAggregatorDB with range streaming and height tracking.
Serialization/Types:
- Use Protobuf types instead of opaque bytes; add serializer adapter and header helpers in fuel-core-types.
Tests:
- Add RPC integration tests and proptests for (de)serialization; extend test helpers.
Build/CI:
- Install protoc in CI; add prost/tonic deps; ignore RUSTSEC-2025-0118; update .gitignore.

^{Written by Cursor Bugbot for commit 0da8556. This will update automatically on new commits. Configure here.}

Co-authored-by: Aaryamann Challani <[email protected]>

…gregator-db-adapter

rymnc · 2025-10-02T12:11:19Z

crates/services/block_aggregator_api/Cargo.toml

 thiserror = { workspace = true }
 tokio = { workspace = true }
+tokio-stream = { workspace = true }
+tonic = { workspace = true }


think we want to support some form of compression. what you think between deflate and zstd ?

I add some naive serialization here:
#3101

But you're right, we could do some additional compression.

Maybe we need to version the serialization too....

this is compression at the rpc level, so it will be negotiated elsewhere, not talking about the serialized bytes of the block here :)

crates/services/block_aggregator_api/src/api/protobuf_adapter.rs

Co-authored-by: Aaryamann Challani <[email protected]>

MitchTurner · 2025-10-03T00:43:30Z

crates/services/block_aggregator_api/src/api/protobuf_adapter.rs

+        let _server_task_handle = tokio::spawn(async move {
+            tonic::transport::Server::builder()
+                .add_service(block_aggregator_server::BlockAggregatorServer::new(server))
+                .serve(addr)
+                .await
+                .unwrap();
+        });


I add handling for this error as well as adding a drop impl for this type so the thread doesn't get orphaned here:
7f917ba
(in the next PR)

xgreenx · 2025-10-03T00:48:13Z

.github/workflows/ci.yml

+      - name: Install Protoc
+        uses: arduino/setup-protoc@v3


Is there a way to not require additional dependency? For example, maybe tonic could use Rust-based implementation of the protobuf?

If not, I think we should follow @rymnc's suggestion and move Protobuf definition into a separate repository which will publish a Rust crate which we can use later in fuel-core, t avoid additional dependency for SDK and Sway

Yeah. That makes sense to me.

We need to remove dependency from protoc before we merge this change to the master=)

xgreenx · 2025-10-03T00:55:02Z

crates/services/block_aggregator_api/proto/api.proto

+  uint32 end = 2;
+}
+
+message Block {


The idea it to convert FuelBlock into a protobuf block, where protobuf block has all fields of FuelBlock. We don't plan to use any other serialization, so it is not clear to me what is data field here.

Oh. That's slightly different from what I understood. I'm just assuming it's getting serialized to bytes from an arbitrary serializer. The protobuf type is just accepting those bytes. I don't think it will be too hard to change it and still maintain dependency inversion.

xgreenx · 2025-10-03T00:55:53Z

crates/services/block_aggregator_api/proto/api.proto

@@ -0,0 +1,33 @@
+syntax = "proto3";
+
+package blockaggregator;


Is there a way to generate protobuf schema from the rust type?

It all seems to be .proto -> Rust. That's fine from a clean-architecture perspective.

tokio-rs/prost#1350

Let's see what they will suggest

…adatper

## Linked Issues/PRs  followup to feedback from #3100 ## Description  ## Checklist - [ ] Breaking changes are clearly marked as such in the PR description and changelog - [ ] New behavior is reflected in tests - [ ] [The specification](https://github.com/FuelLabs/fuel-specs/) matches the implemented behavior (link update PR if changes are needed) ### Before requesting review - [ ] I have reviewed the code myself - [ ] I have created follow-up issues caused by this PR and linked them here ### After merging, notify other teams [Add or remove entries as needed] - [ ] [Rust SDK](https://github.com/FuelLabs/fuels-rs/) - [ ] [Sway compiler](https://github.com/FuelLabs/sway/) - [ ] [Platform documentation](https://github.com/FuelLabs/devrel-requests/issues/new?assignees=&labels=new+request&projects=&template=NEW-REQUEST.yml&title=%5BRequest%5D%3A+) (for out-of-organization contributors, the person merging the PR will do this) - [ ] Someone else?  --- > [!NOTE] > <sup>[Cursor Bugbot](https://cursor.com/dashboard?tab=bugbot) is generating a summary for commit c0a6540. Configure [here](https://cursor.com/dashboard?tab=bugbot).</sup>

## Linked Issues/PRs  Closes #3090 ## Description  ## Checklist - [ ] Breaking changes are clearly marked as such in the PR description and changelog - [ ] New behavior is reflected in tests - [ ] [The specification](https://github.com/FuelLabs/fuel-specs/) matches the implemented behavior (link update PR if changes are needed) ### Before requesting review - [ ] I have reviewed the code myself - [ ] I have created follow-up issues caused by this PR and linked them here ### After merging, notify other teams [Add or remove entries as needed] - [ ] [Rust SDK](https://github.com/FuelLabs/fuels-rs/) - [ ] [Sway compiler](https://github.com/FuelLabs/sway/) - [ ] [Platform documentation](https://github.com/FuelLabs/devrel-requests/issues/new?assignees=&labels=new+request&projects=&template=NEW-REQUEST.yml&title=%5BRequest%5D%3A+) (for out-of-organization contributors, the person merging the PR will do this) - [ ] Someone else?  --- > [!NOTE] > <sup>[Cursor Bugbot](https://cursor.com/dashboard?tab=bugbot) is generating a summary for commit 4bb648d. Configure [here](https://cursor.com/dashboard?tab=bugbot).</sup>

cursor

This PR is being reviewed by Cursor Bugbot

Details

Your team is on the Bugbot Free tier. On this plan, Bugbot will review limited PRs each billing cycle for each member of your team.

To receive Bugbot reviews on all of your PRs, visit the Cursor dashboard to activate Pro and start your 14-day free trial.

Bug: Incomplete Backups Cause Data Loss

The backup_databases method doesn't back up the newly added block_aggregation database, causing data loss during backup operations.

crates/fuel-core/src/combined_database.rs#L116-L137

fuel-core/crates/fuel-core/src/combined_database.rs

Lines 116 to 137 in 55b3686

    
           #[cfg(feature = "backup")] 
        
           fn backup_databases( 
        
               db_dir: &std::path::Path, 
        
               temp_dir: &std::path::Path, 
        
           ) -> crate::database::Result<()> { 
        
               crate::state::rocks_db::RocksDb::<OnChain>::backup(db_dir, temp_dir) 
        
                   .trace_err("Failed to backup on-chain database")?; 
        
               crate::state::rocks_db::RocksDb::<OffChain>::backup(db_dir, temp_dir) 
        
                   .trace_err("Failed to backup off-chain database")?; 
        
               crate::state::rocks_db::RocksDb::<Relayer>::backup(db_dir, temp_dir) 
        
                   .trace_err("Failed to backup relayer database")?; 
        
               crate::state::rocks_db::RocksDb::<GasPriceDatabase>::backup(db_dir, temp_dir) 
        
                   .trace_err("Failed to backup gas-price database")?; 
        
               crate::state::rocks_db::RocksDb::<CompressionDatabase>::backup(db_dir, temp_dir) 
        
                   .trace_err("Failed to backup compression database")?; 
        
               Ok(()) 
        
           }

Bug: New Database Causes Shutdown Failure

The shutdown method doesn't call shutdown on the newly added block_aggregation database, potentially leaving resources uncleaned and file handles open.

crates/fuel-core/src/combined_database.rs#L611-L618

fuel-core/crates/fuel-core/src/combined_database.rs

Lines 611 to 618 in 55b3686

    
           pub fn shutdown(self) { 
        
               self.on_chain.shutdown(); 
        
               self.off_chain.shutdown(); 
        
               self.relayer.shutdown(); 
        
               self.gas_price.shutdown(); 
        
               self.compression.shutdown(); 
        
           }

cursor · 2025-11-12T15:04:52Z

crates/fuel-core/src/combined_database.rs

        &self.compression
    }

+    pub fn block_aggregation(&self) -> &Database<BlockAggregatorDatabase> {


Bug: Incomplete Database Version Validation Risks Corruption

The check_version method doesn't validate the newly added block_aggregation database, which could allow version mismatches to go undetected and cause data corruption or compatibility issues at runtime.

crates/fuel-core/src/combined_database.rs

crates/fuel-core/Cargo.toml

crates/services/block_aggregator_api/src/blocks/importer_and_db_source/serializer_adapter.rs

cursor

Bug: Block aggregator database not shut down properly

The shutdown() method does not call shutdown() on the block_aggregation database. This means the block aggregator database is not properly closed when the combined database shuts down, potentially leaving resources unfreed and causing data consistency issues if the service is restarted without proper cleanup.

crates/fuel-core/src/combined_database.rs#L490-L497

fuel-core/crates/fuel-core/src/combined_database.rs

Lines 490 to 497 in 7b49d6d

    
                               "compression database height({compression_db_height}) \ 
        
                               is less than target height({target_block_height})" 
        
                           )); 
        
                       } 
        
                       if on_chain_height > target_block_height { 
        
                           self.on_chain().rollback_last_block()?; 
        
                       }

cursor · 2025-11-19T00:58:41Z

crates/fuel-core/src/combined_database.rs

            relayer,
            gas_price,
            compression,
+            block_aggregation,


Bug: Block aggregator database not included in backup/restore operations

The newly added block_aggregation database field is not being backed up or restored in the backup_databases() and restore_database() methods. When backup/restore is called, the block aggregator database is silently skipped, which causes data loss if the backup is ever used to restore a node's state. This can result in missing block data that was previously stored.

cursor · 2025-11-19T18:15:49Z

bin/fuel-core/src/cli/run.rs

                    .echo_delegation_interval,
            };

+        let rpc_config = rpc_args.into_config();


Bug: Unconditional RPC config creation when feature disabled

The rpc_config variable is created unconditionally at line 460, but it's only used conditionally when the rpc feature is enabled (line 787-788). When compiling without the rpc feature, this creates an unused variable that will trigger a compiler warning or error. The line should be wrapped in #[cfg(feature = "rpc")] to match where it's consumed.

@xgreenx

## Linked Issues/PRs Complete feedback from: #3100 #3101 #3112 #3116 As they have all been merged into #3100 ### TODOs #### Completed - [x] Move proto types to new repo (or at least remove dep on protoc) FuelLabs/fuel-core-protobuf#1 - [x] Use ipv4 instead of ipv4 `let listener = TcpListener::bind("[::1]:0").unwrap();` - [x] make sure rpc is optional (may already be good with feature) - [x] "If we add a new database, we should include it in all places where we already interact with databases, like check_version, rollback_to, and so on(check where do we use databa liek relayer)" - [x] "The CombinedDatabase::check_version() method is missing a version check for the new block_aggregation database. This omission is inconsistent with other database checks and could lead to version compatibility issues." - [x] "I would like to mention the reason why this database doesn't force a monotonic increase in height." - [x] "Also, becuase we don't force monotonic height increase, we can't actually re-use rollback feature from historicat database. But we still need the ability to reset the state of the block aggregator to height X. So you can add another function, which we can call in rollback_to." - [x] "I know that ServiceRunner here is an overkill, but we have GraphQL that looks the same, and it was fiiiiine=D Maybe for consistency plus some logging we could reuse it here as well. It's just an internal service for your main service" - [x] `// TODO: Should this be owned to begin with?` Yeah, we should work with reference in proto_header_from_header and in proto_tx_from_tx - [x] maybe "If you used BoxStream, then you could avoid usage of the tokio::spawn( below. You can just do inner.map({...}).into_boxed()" in protobuf_adapater impl of protobuf trait: `type GetBlockRangeStream = BoxStream<Result<ProtoBlockResponse, Status>>;` - [x] `#[allow(unused)] fn arb_inputs()` - [x] in `serializer_adapter.rs`: "We should work with reference to everywhere in this file. And we should also avoid usage of unswap_or_default. All fields are always set, so it is strange why it can be None` - [x] `if let Some(value) = policies.get(PolicyType::Owner) { values[5] = value; }` - [x] in `serializer_adapter.rs`: "Why values for policies in an empty array? If the policy is not set, we shouldn't include them. If no policy is set, then it will be empty vector." - [x] in `crates/types/src/blockchain/header.rs`: "We don't need to expose the application header. You can find an example on how to create a header here: https://github.com/FuelLabs/fuel-rust-indexer/blob/main/crates/receipts_manager/adapters/graphql_event_adapter.rs#L250", "And the same for the block, you don't need manually calculate it, you can do that from the Block::new." - [x] in `types/src/test_helpers.rs`: "Block::new should be enough for you to create a header and a block. You don't need to implement all the logic by yourself.You can clean up getters and setters which you've added, after you udpated code here=) " - [x] Maybe remove `self.go_to_sleep_before_continuing().await;` #### Need Feedback - [ ] in `serializer_adapter.rs`: "If we use Rust definition to define variants inside of enums, then we can remove useless fields like data for MessageCoinPredicate and witness_index for MessageCoinPredicate" (@xgreenx I don't understand this. What "Rust definition"?) #### Noop - [ ] ~in `serializer_adapter.rs`: "I think instead of creating many procedure fucntions, we could implement TryFrom and From for types, plus, we could split them into its own folders by transactions and some common folder."~ **(Since this is in a separate repo now, the TryFrom/From impls wouldn't be valid due to the orphan rule)** - [ ] ~in `api.proto`: "I think all txs have these, maybe we could move them to the top-level Transaction?" DRY up chargeable txs?~**( I don't really care for this abstraction, but I could be convinced to DRY this up. Just doesn't seem like much gain)** - [ ] ~I think we can move fetching of the full blocks to be a part of default functionality provided by FuelClient along with old functionality (in `test-helpers/src/client_ext.rs`)~ **(I don't want to add new functionality in this work, we could do a followup)** - [ ] ~"[nit] https://github.com/FuelLabs/fuel-rust-indexer/blob/main/crates/receipts_manager/service.rs#L584 We can replace the whole service with stream of joined events(old + new)"~ **(WE could do this as a followup)** ## Description  ## Checklist - [ ] Breaking changes are clearly marked as such in the PR description and changelog - [ ] New behavior is reflected in tests - [ ] [The specification](https://github.com/FuelLabs/fuel-specs/) matches the implemented behavior (link update PR if changes are needed) ### Before requesting review - [ ] I have reviewed the code myself - [ ] I have created follow-up issues caused by this PR and linked them here ### After merging, notify other teams [Add or remove entries as needed] - [ ] [Rust SDK](https://github.com/FuelLabs/fuels-rs/) - [ ] [Sway compiler](https://github.com/FuelLabs/sway/) - [ ] [Platform documentation](https://github.com/FuelLabs/devrel-requests/issues/new?assignees=&labels=new+request&projects=&template=NEW-REQUEST.yml&title=%5BRequest%5D%3A+) (for out-of-organization contributors, the person merging the PR will do this) - [ ] Someone else?

MitchTurner and others added 30 commits September 1, 2025 11:11

Add naive service with test

9fb0368

Return a BoxStream instead of just sending over channel

d695469

Cleanup

8384238

Add failing test

db993bf

Add test for receiving new block

17ac53c

Cleanup

77d64ef

Lint

ad951b9

Add query for current height to allow checking the progress

267ce07

Remove commented code

78f1ee0

Update CHANGELOG

90867f8

Fix privacy of constructor

f6d2fdb

Improve tests, fix flake

e612ea2

Remove commented code

489e74c

Remove unnecessary task spawn

56f4eb0

Update crates/services/block_aggregator_api/src/lib.rs

7216d38

Co-authored-by: Aaryamann Challani <[email protected]>

Update crates/services/block_aggregator_api/Cargo.toml

5119ab9

Co-authored-by: Aaryamann Challani <[email protected]>

use futures instead of future-utils

dc4f1aa

Fix Debug impl and only use in test

720694a

Add assoc type for block response to allow for a variety of types

e1a7305

Lin toml

51b7268

Add example block range response type

389c46b

Add documentation

a12da43

Re-add debug impl for everyone

541b71b

WIP add bare adapter

5236bee

Fix compilation

146812e

Add failing test

995c6e1

Remove commented code

cedacfc

WIP

930b465

Use BlockHeight instead of u64

71b5d12

Merge branch 'chore/add-block-aggregator-service' into chore/block-ag…

b9c6ead

…gregator-db-adapter

rymnc reviewed Oct 2, 2025

View reviewed changes

crates/services/block_aggregator_api/src/api/protobuf_adapter.rs Outdated Show resolved Hide resolved

MitchTurner and others added 2 commits October 2, 2025 09:33

Update crates/services/block_aggregator_api/src/api/protobuf_adapter.rs

ad652dd

Co-authored-by: Aaryamann Challani <[email protected]>

include inner error in API error message

a0d99f2

MitchTurner commented Oct 3, 2025

View reviewed changes

xgreenx reviewed Oct 3, 2025

View reviewed changes

MitchTurner mentioned this pull request Oct 6, 2025

Use Proto Types in storage instead of bytes #3112

Merged

9 tasks

Merge branch 'master' into chore/protobuf-api-adatper

d821f9c

This comment was marked as outdated.

Sign in to view

Merge remote-tracking branch 'origin/master' into chore/protobuf-api-…

db15acc

…adatper

cursor bot reviewed Nov 12, 2025

View reviewed changes

MitchTurner added 2 commits November 12, 2025 08:05

Merge branch 'master' into chore/protobuf-api-adatper

a7c3e28

Fix cargo lock

b2b023b

MitchTurner mentioned this pull request Nov 14, 2025

Block RPC PR Followup #3145

Merged

31 tasks

Ignore rustsec with tracking issue

ab29ec6

MitchTurner changed the base branch from master to chore/rustsec-2025-0118 November 17, 2025 17:33

Merge branch 'chore/rustsec-2025-0118' into chore/protobuf-api-adatper

9dbd0e4

cursor bot reviewed Nov 17, 2025

View reviewed changes

crates/fuel-core/Cargo.toml Show resolved Hide resolved

MitchTurner added 2 commits November 17, 2025 10:41

Fix spellcheck, update changelog

412be8a

Merge branch 'chore/rustsec-2025-0118' into chore/protobuf-api-adatper

3db8c2f

MitchTurner changed the base branch from chore/rustsec-2025-0118 to master November 17, 2025 18:30

Merge branch 'master' into chore/protobuf-api-adatper

0ae1f71

cursor bot reviewed Nov 17, 2025

View reviewed changes

crates/services/block_aggregator_api/src/blocks/importer_and_db_source/serializer_adapter.rs Show resolved Hide resolved

Merge branch 'master' into chore/protobuf-api-adatper

7b49d6d

cursor bot reviewed Nov 19, 2025

View reviewed changes

Merge branch 'master' into chore/protobuf-api-adatper

0da8556

cursor bot reviewed Nov 19, 2025

View reviewed changes

	#[cfg(feature = "backup")]
	fn backup_databases(
	db_dir: &std::path::Path,
	temp_dir: &std::path::Path,
	) -> crate::database::Result<()> {
	crate::state::rocks_db::RocksDb::<OnChain>::backup(db_dir, temp_dir)
	.trace_err("Failed to backup on-chain database")?;

	crate::state::rocks_db::RocksDb::<OffChain>::backup(db_dir, temp_dir)
	.trace_err("Failed to backup off-chain database")?;

	crate::state::rocks_db::RocksDb::<Relayer>::backup(db_dir, temp_dir)
	.trace_err("Failed to backup relayer database")?;

	crate::state::rocks_db::RocksDb::<GasPriceDatabase>::backup(db_dir, temp_dir)
	.trace_err("Failed to backup gas-price database")?;

	crate::state::rocks_db::RocksDb::<CompressionDatabase>::backup(db_dir, temp_dir)
	.trace_err("Failed to backup compression database")?;

	Ok(())
	}


	pub fn shutdown(self) {
	self.on_chain.shutdown();
	self.off_chain.shutdown();
	self.relayer.shutdown();
	self.gas_price.shutdown();
	self.compression.shutdown();
	}

	"compression database height({compression_db_height}) \
	is less than target height({target_block_height})"
	));
	}

	if on_chain_height > target_block_height {
	self.on_chain().rollback_last_block()?;
	}

Block Aggregator Protobuf API #3100

Are you sure you want to change the base?

Block Aggregator Protobuf API #3100

Uh oh!

Conversation

MitchTurner commented Sep 16, 2025 • edited by cursor bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Linked Issues/PRs

Description

Checklist

Before requesting review

After merging, notify other teams

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MitchTurner Oct 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

MitchTurner Oct 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MitchTurner Oct 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This comment was marked as outdated.

Uh oh!

cursor bot left a comment

Choose a reason for hiding this comment

This PR is being reviewed by Cursor Bugbot

Bug: Incomplete Backups Cause Data Loss

Bug: New Database Causes Shutdown Failure

Uh oh!

cursor bot Nov 12, 2025

Choose a reason for hiding this comment

Bug: Incomplete Database Version Validation Risks Corruption

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cursor bot left a comment

Choose a reason for hiding this comment

Bug: Block aggregator database not shut down properly

Uh oh!

cursor bot Nov 19, 2025

Choose a reason for hiding this comment

Bug: Block aggregator database not included in backup/restore operations

Uh oh!

cursor bot Nov 19, 2025

Choose a reason for hiding this comment

Bug: Unconditional RPC config creation when feature disabled

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

MitchTurner commented Sep 16, 2025 •

edited by cursor bot

Loading

MitchTurner Oct 2, 2025 •

edited

Loading

MitchTurner Oct 3, 2025 •

edited

Loading

MitchTurner Oct 3, 2025 •

edited

Loading