Data Integrity Issues in DAC

[rachit77]

This discussion is meant for resolving Issue #120

Table: sync_tasks

Column	Type	Constraints	Description
task	TEXT	PRIMARY KEY	Name of the task being tracked i.e. L1Sync
block	BIGINT	NOT NULL	Last processed block number
processed	TIMESTAMP	DEFAULT NOW()	Timestamp of when the block was last processed

Table: unresolved_batches

composite key: (num, hash)

Column	Type	Constraints	Desc
num	BIGINT	NOT NULL	batch number
hash	TEXT	NOT NULL	hash of Batch L2Data that is stored on L1 smart contract

Table: offchain_data

Column	Type	Constraints	Description
key	TEXT	PRIMARY KEY	hash of Batch L2Data
value	TEXT	NOT NULL	Batch L2Data
batch_num	BIGINT	NOT NULL	batch number

---

Current Workflow:

1. Sequence Sending and Storage:
   • The sequence sender sends a sequence to the DAC for signing.
   • The DAC stores this sequence in the offchain_data table with a default batch number of 0, as the sequence provided by the sequence sender does not contain a batch number.
2. Processing Sequence Batch Events:
   • An L1Syncer in the DAC listens for sequence batch events emitted from L1.
   • These events are processed and stored in the unresolved_batches table.
3. Batch Number Update:
   • A separate Go routine picks entries from the unresolved_batches table, updates the batch number in the offchain_data table, and deletes the corresponding entries from the unresolved_batches table after processing.

---

Identified Issue:

• Data Integrity Problem:

  • The offchain_data table occasionally encounters primary key violations, leading to data integrity issues.
  • This happens because empty batches on L2 have identical batch data, resulting in the same hash for these batches. Consequently, certain batch numbers may be missing from the offchain_data table.

  Go-http-client/1.1" {"pid": 1, "version": "0.0.10"}
  2024-11-14T17:19:47.877Z    WARN    synchronizer/batches.go:501 detected offchain data gaps (current batch number => expected batch number): 330=>330
  392=>395
  534=>540
  628=>630
  650=>653
  654=>656
  591=>597
  657=>659
  318=>318
  336=>336
  346=>346
  363=>363
  368=>368

• Impact on the System:

  • Although the sequencer can recover fully from the DAC state by querying data based on batch hashes stored on L1, the system logs indicate batch gaps in the offchain_data table. This is because the logic for detecting batch gaps relies on the presence of batch numbers in the table.
  • Currently, the DAC merely logs these gaps and does not attempt to fill them by querying missing data from the sequencer or other DAC nodes.
  • The existing logic does not account for injected batch.

---

Discussion:

• Relevance of Batch Numbers in the DAC:
  • While sequencing, only batch hashes are stored on the L1 smart contract. This raises the question: is the storage of batch numbers in the DAC still necessary?
  • The only current dependency on batch numbers in the DAC is for batch gap detection logic in the offchain_data table.

---

Proposed Solution:

1. Remove Dependency on Batch Numbers:
   • Eliminate the batch_num column from the offchain_data and unresolved_batches tables.
   • Remove the existing batch gap detection logic that depends on batch numbers.
2. Revised Workflow:
   • Sequence Storage:
     • The sequence sender sends a sequence to the DAC for signing, which stores it directly.
   • Batch Event Processing:
     • The L1Syncer listens to L1 events and stores the corresponding batch hash in the unresolved_batches table.
   • Verification and Data Recovery:
     • A separate Go routine picks batch hashes from the unresolved_batches table and checks if the corresponding data exists in the offchain_data table.
     • If data is missing, it queries the sequencer or another DAC node to fetch the missing data and adds it to the offchain_data table.

[arnaubennassar]

Great job, @rachit77! The only suggestion I have is to check if the data already exists in offchain_data before adding an entry to unresolved_batches. In that context, it might be beneficial to rename unresolved_batches to missing_batches. Additionally, remember to remove the entry from the missing table once it has been added to offchain_data.

[arnaubennassar]

also, not sure how the syncer deals with progress: how it knows what's the last synced block. Would be nice to review this logic to ensure that:

• All the blocks are synced, and it's safe to restart the DAC node
• Adding entries to unresolved_batches or missing_batches if you agree with renaming is done in an atomic way (if we consider a block as synced it means that we've added any potential entry)

[rachit77]

The syncer keeps track of the last synced block in the sync_tasks table. This table contains a single row that indicates the L1 block number from which the syncer should begin processing sequence batch events.