feat(vc): add StateImporter for bulk state initialization from external sources#516
Conversation
…al sources Adds a StateImporter to the VC service's database layer that enables bulk-loading trusted state into the committer's database, bypassing the normal MVCC validation pipeline. This is the foundational component needed for the Fabric to Fabric-X ledger migration tool. The migration tool's bootstrap mode will use this importer to populate the state database from a Fabric peer snapshot. - Uses PostgreSQL COPY protocol (pgx.CopyFrom) for high-throughput bulk insert - StateIterator interface enables streaming import without loading entire snapshots into memory - ImportPolicy writes to __meta namespace so the coordinator's existing recoverPolicyManagerFromStateDB() works after import - VerifyImport provides post-import integrity checks (row counts + block height) - 14 unit tests covering all operations, edge cases, and a multi-namespace end-to-end scenario Signed-off-by: Shridhar Panigrahi <sridharpanigrahi2006@gmail.com>
|
@cendhu , please let me know your thoughts on this ! |
|
@sridhar-panigrahi Thank you for your interest in fabric-x-committer. Migration from Fabric to Fabric-X is a significant feature, we recommend starting with a detailed design document before proceeding with the implementation. Please submit your design to the Fabric-X RFCs repository for approval. We have a tentative list of tasks for this migration story hyperledger/fabric-x#21 The design should address all of them. |
|
Hi @cendhu, thanks for the feedback. I've submitted the RFC covering the full migration design to the Fabric-X RFCs repo: hyperledger/fabric-x-rfcs#5 It addresses all the tasks from hyperledger/fabric-x#21 — canonical data set format, channel-to-namespace mapping, exporter CLI, committer bootstrap mode, and verification process. Looking forward to your review and any feedback from the community. |
|
@cendhu — the Exporter CLI that pairs with this StateImporter is now up at hyperledger/fabric-x#97. Between the two PRs the full export → bootstrap pipeline from the RFC is taking shape. |
2 participants
Right now there's no way to initialize the committer's state database from an external source — state can only enter through the full block processing pipeline (orderer → sidecar → coordinator → vc). That pipeline does MVCC validation, endorsement verification, dependency tracking, etc., which makes no sense for trusted pre-validated state from a Fabric snapshot.
This PR adds a
StateImporterto the VC service that lets you bulk-load state directly into the database using PostgreSQL's COPY protocol, skipping the validation pipeline entirely. It's meant to be the building block for thecommitter --init-from-snapshotbootstrap mode that the Fabric → Fabric-X migration tool needs.What it does:
InitSchema/CreateNamespace— sets up system tables and per-namespace tablesImportNamespaceState— streams key-value-version triples into a namespace table via COPY (handles large datasets without loading everything into memory)ImportPolicy— writes namespace policies into__metaso the coordinator'srecoverPolicyManagerFromStateDB()picks them up on startupSetBlockHeight— sets the last committed block number so the committer resumes from the right pointVerifyImport— checks row counts and block height after import to catch integrity issuesNo existing code is modified (except adding one test helper method to
test_exports.go). The importer reuses the existingdatabasestruct, SQL templates, and table schema — imported data lands in the samens_<id>tables the normal pipeline uses.14 tests including a 10K-key bulk load and a multi-namespace end-to-end test that verifies imported policies are readable by
readNamespacePolicies().