fix: Handle blocks invalidated with an invalid block (#16836)

Fixes issue with invalidating blocks with invalid attestations. If a
block is invalidated, but immediately replaced with another invalid
block, then the next proposer will attempt to invalidate the original
one, which will fail since its attestations will have been replaced in
L1 storage with the ones from the second one. As an example:

1. Slot S:   Block N is proposed with invalid attestations
2. Slot S+1: Block N is invalidated, and block N' (same number) is
proposed instead, but also has invalid attestations
3. Slot S+2: Proposer tries to invalidate block N, when they should
invalidate block N' instead, and fails

Also added a new env var for the archiver
`skipValidateBlockAttestations` so we could force a proposer to keep
building on top of an invalid block, so we could write the test
`proposer invalidates multiple blocks`. This also required moving the
archiver config to stdlib, and adding an `updateConfig` method wired
through the node admin API so we could turn that config on or off.

Last, this PR uses a persistent storage for the
`pendingChainValidationStatus` in the archiver, so it is persisted
across restarts. Otherwise, if a node is restarted while the change is
in an invalid state, it will fail to invalidate such block.

Author: spalladino

yarn-project/archiver/README.md

@@ -1,16 +1,37 @@
 # Archiver
 
 Archiver is a service which is used to fetch data on-chain data and present them in a nice-to-consume form.
+
 The on-chain data specifically are the following events:
 
 1. `L2BlockProposed` event emitted on Rollup contract,
 2. `MessageAdded` event emitted on Inbox contract,
 
 The interfaces defining how the data can be consumed from the archiver are `L2BlockSource`, `L2LogsSource` and `ContractDataSource`.
 
-## Usage
+## Sync process
+
+The archiver sync process periodically checks its current state against the Rollup contract on L1 and updates its local state.
+
+### Handling invalid blocks
+
+After the implementation of [delayed attestation verification](https://github.com/AztecProtocol/engineering-designs/pull/69), the Rollup contract on L1 no longer validates committee attestations. Instead, these are posted in calldata, and L2 nodes are expected to verify them as they download blocks. The archiver handles this validation during its sync process.
+
+Whenever the archiver detects a block with invalid attestations, it skips it. These blocks are not meant to be part of the chain, so the archiver ignores them and continues processing the next blocks. It is expected that an honest proposer will eventually invalidate these blocks, removing them from the chain on L1, and then resume the sequence of valid blocks.
+
+> [!WARNING]  
+> If the committee for the epoch is also malicious and attests to a descendant of an invalid block, nodes should also ignore these descendants, unless they become proven. This is currently not implemented. Nodes assume that the majority of the committee is honest.
+
+When the current node is elected as proposer, the `sequencer` needs to know whether there is an invalid block in L1 that needs to be purged before posting their own block. To support this, the archiver exposes a `pendingChainValidationStatus`, which is the state of the tip of the pending chain. This status can be valid in the happy path, or invalid if the tip of the pending chain has invalid attestations. If invalid, this status also contains all the data needed for purging the block from L1 via an `invalidate` call to the Rollup contract. Note that, if the head of the chain has more than one invalid consecutive block, this status will reference the earliest one that needs to be purged, since a call to purge an invalid block will automatically purge all descendants. Refer to the [InvalidateLib.sol](`l1-contracts/src/core/libraries/rollup/InvalidateLib.sol`) for more info.
+
+> [!TIP]  
+> The archiver can be configured to `skipValidateBlockAttestations`, which will make it skip this validation. This cannot be set via environment variables, only via a call to `nodeAdmin_setConfig`. This setting is only meant for testing purposes.
 
-To install dependencies and build the package run `yarn install` followed by `yarn build`.
-To run test execute `yarn test`.
+As an example, let's say the chain has been progressing normally up until block 10, and then:
+1. Block 11 is posted with invalid attestations. The archiver will report 10 as the latest block, but the `pendingChainValidationStatus` will point to block 11.
+2. Block 11 is purged, but another block 11 with invalid attestations is posted in its place. The archiver will still report 10 as latest, and the `pendingChainValidationStatus` will point to the new block 11 that needs to be purged.
+3. Block 12 with invalid attestations is posted. No changes in the archiver.
+4. Block 13 is posted with valid attestations, due to a malicious committee. The archiver will try to sync it and fail, since 13 does not follow 10. This scenario is not gracefully handled yet.
+5. Blocks 11 to 13 are purged. The archiver status will not yet be changed: 10 will still be the latest block, and the `pendingChainValidationStatus` will point to 11. This is because the archiver does **not** follow `BlockInvalidated` events.
+6. Block 11 with valid attestations is posted. The archiver pending chain now reports 11 as latest, and its status is valid.
 
-To start the service export `ETHEREUM_HOSTS` (defaults to `http://127.0.0.1:8545/`), `ARCHIVER_POLLING_INTERVAL_MS` (defaults to `1000 ms`), `ROLLUP_CONTRACT_ADDRESS`, `INBOX_CONTRACT_ADDRESS` environmental variables and start the service with `yarn start`.

yarn-project/archiver/src/archiver/archiver.ts

@@ -10,7 +10,7 @@ import {
 } from '@aztec/ethereum';
 import { maxBigint } from '@aztec/foundation/bigint';
 import { Buffer16, Buffer32 } from '@aztec/foundation/buffer';
-import { pick } from '@aztec/foundation/collection';
+import { merge, pick } from '@aztec/foundation/collection';
 import type { EthAddress } from '@aztec/foundation/eth-address';
 import { Fr } from '@aztec/foundation/fields';
 import { type Logger, createLogger } from '@aztec/foundation/log';
@@ -102,6 +102,14 @@ export type ArchiverDeps = {
   dateProvider?: DateProvider;
 };
 
+function mapArchiverConfig(config: Partial<ArchiverConfig>) {
+  return {
+    pollingIntervalMs: config.archiverPollingIntervalMS,
+    batchSize: config.archiverBatchSize,
+    skipValidateBlockAttestations: config.skipValidateBlockAttestations,
+  };
+}
+
 /**
  * Pulls L2 blocks in a non-blocking manner and provides interface for their retrieval.
  * Responsible for handling robust L1 polling so that other components do not need to
@@ -138,7 +146,7 @@ export class Archiver extends (EventEmitter as new () => ArchiverEmitter) implem
     private readonly publicClient: ViemPublicClient,
     private readonly l1Addresses: { rollupAddress: EthAddress; inboxAddress: EthAddress; registryAddress: EthAddress },
     readonly dataStore: ArchiverDataStore,
-    private readonly config: { pollingIntervalMs: number; batchSize: number },
+    private config: { pollingIntervalMs: number; batchSize: number; skipValidateBlockAttestations?: boolean },
     private readonly blobSinkClient: BlobSinkClientInterface,
     private readonly epochCache: EpochCache,
     private readonly instrumentation: ArchiverInstrumentation,
@@ -198,10 +206,7 @@ export class Archiver extends (EventEmitter as new () => ArchiverEmitter) implem
       proofSubmissionEpochs: Number(proofSubmissionEpochs),
     };
 
-    const opts = {
-      pollingIntervalMs: config.archiverPollingIntervalMS ?? 10_000,
-      batchSize: config.archiverBatchSize ?? 100,
-    };
+    const opts = merge({ pollingIntervalMs: 10_000, batchSize: 100 }, mapArchiverConfig(config));
 
     const epochCache = deps.epochCache ?? (await EpochCache.create(config.l1Contracts.rollupAddress, config, deps));
     const telemetry = deps.telemetry ?? getTelemetryClient();
@@ -220,6 +225,11 @@ export class Archiver extends (EventEmitter as new () => ArchiverEmitter) implem
     return archiver;
   }
 
+  /** Updates archiver config */
+  public updateConfig(newConfig: Partial<ArchiverConfig>) {
+    this.config = merge(this.config, mapArchiverConfig(newConfig));
+  }
+
   /**
    * Starts sync process.
    * @param blockUntilSynced - If true, blocks until the archiver has fully synced.
@@ -796,7 +806,9 @@ export class Archiver extends (EventEmitter as new () => ArchiverEmitter) implem
       const validBlocks: PublishedL2Block[] = [];
 
       for (const block of publishedBlocks) {
-        const validationResult = await validateBlockAttestations(block, this.epochCache, this.l1constants, this.log);
+        const validationResult = this.config.skipValidateBlockAttestations
+          ? { valid: true as const }
+          : await validateBlockAttestations(block, this.epochCache, this.l1constants, this.log);
 
         // Only update the validation result if it has changed, so we can keep track of the first invalid block
         // in case there is a sequence of more than one invalid block, as we need to invalidate the first one.

yarn-project/archiver/src/archiver/config.ts

@@ -1,47 +1,28 @@
 import { type BlobSinkConfig, blobSinkConfigMapping } from '@aztec/blob-sink/client';
 import {
-  type L1ContractAddresses,
   type L1ContractsConfig,
   type L1ReaderConfig,
   l1ContractAddressesMapping,
   l1ContractsConfigMappings,
   l1ReaderConfigMappings,
 } from '@aztec/ethereum';
-import { type ConfigMappingsType, getConfigFromMappings, numberConfigHelper } from '@aztec/foundation/config';
+import {
+  type ConfigMappingsType,
+  booleanConfigHelper,
+  getConfigFromMappings,
+  numberConfigHelper,
+} from '@aztec/foundation/config';
 import { type ChainConfig, chainConfigMappings } from '@aztec/stdlib/config';
+import type { ArchiverSpecificConfig } from '@aztec/stdlib/interfaces/server';
 
 /**
+ * The archiver configuration.
  * There are 2 polling intervals used in this configuration. The first is the archiver polling interval, archiverPollingIntervalMS.
  * This is the interval between successive calls to eth_blockNumber via viem.
  * Results of calls to eth_blockNumber are cached by viem with this cache being updated periodically at the interval specified by viemPollingIntervalMS.
  * As a result the maximum observed polling time for new blocks will be viemPollingIntervalMS + archiverPollingIntervalMS.
  */
-
-/**
- * The archiver configuration.
- */
-export type ArchiverConfig = {
-  /** The polling interval in ms for retrieving new L2 blocks and encrypted logs. */
-  archiverPollingIntervalMS?: number;
-
-  /** The number of L2 blocks the archiver will attempt to download at a time. */
-  archiverBatchSize?: number;
-
-  /** The polling interval viem uses in ms */
-  viemPollingIntervalMS?: number;
-
-  /** The deployed L1 contract addresses */
-  l1Contracts: L1ContractAddresses;
-
-  /** The max number of logs that can be obtained in 1 "getPublicLogs" call. */
-  maxLogs?: number;
-
-  /** The maximum possible size of the archiver DB in KB. Overwrites the general dataStoreMapSizeKB. */
-  archiverStoreMapSizeKb?: number;
-} & L1ReaderConfig &
-  L1ContractsConfig &
-  BlobSinkConfig &
-  ChainConfig;
+export type ArchiverConfig = ArchiverSpecificConfig & L1ReaderConfig & L1ContractsConfig & BlobSinkConfig & ChainConfig;
 
 export const archiverConfigMappings: ConfigMappingsType<ArchiverConfig> = {
   ...blobSinkConfigMapping,
@@ -65,6 +46,10 @@ export const archiverConfigMappings: ConfigMappingsType<ArchiverConfig> = {
     parseEnv: (val: string | undefined) => (val ? +val : undefined),
     description: 'The maximum possible size of the archiver DB in KB. Overwrites the general dataStoreMapSizeKB.',
   },
+  skipValidateBlockAttestations: {
+    description: 'Whether to skip validating block attestations (use only for testing).',
+    ...booleanConfigHelper(false),
+  },
   ...chainConfigMappings,
   ...l1ReaderConfigMappings,
   viemPollingIntervalMS: {

yarn-project/aztec-node/src/aztec-node/server.ts

@@ -1126,7 +1126,10 @@ export class AztecNodeService implements AztecNode, AztecNodeAdmin, Traceable {
     this.slasherClient?.updateConfig(config);
     this.validatorsSentinel?.updateConfig(config);
     await this.p2pClient.updateP2PConfig(config);
-
+    const archiver = this.blockSource as Archiver;
+    if ('updateConfig' in archiver) {
+      archiver.updateConfig(config);
+    }
     if (newConfig.realProofs !== this.config.realProofs) {
       this.proofVerifier = config.realProofs ? await BBCircuitVerifier.new(newConfig) : new TestCircuitVerifier();
     }