diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 00000000000..e2d9c5e28cb --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,352 @@ +# AGENTS.md + +This file provides guidance to AI coding agents when working with code in this repository. + +## Overview + +Espresso Network is the **global confirmation layer** for Ethereum rollups. Rollups post their blocks to Espresso for +fast finality and cross-rollup composability. This repo contains: + +- **Sequencer node** (`sequencer/`): Rust binary for running consensus and serving APIs +- **HotShot** (`crates/hotshot/`): BFT consensus library +- **Contracts** (`contracts/`): Solidity contracts for L1 integration (light client, staking, fees, rewards) +- **Types** (`types/`): Domain types shared across crates + +## Critical Rules + +**MUST:** + +- Use `-p ` for all cargo commands during development (full workspace builds cause OOM) +- Update ALL THREE storage backends (PostgreSQL, SQLite, filesystem) when changing persistence +- Run `cargo test -p espresso-types reference` after modifying any serializable type +- Run `cargo fmt` and `cargo check -p ` after making changes +- Write tests that prove correctness, not just exercise code paths + +**NEVER:** + +- Modify V1 contract storage layout (V2 inherits V1; changing V1 breaks upgrades) +- Combine `embedded-db` feature with default features (sqlx feature conflict) +- Use `just test` during iteration (compiles everything; use `cargo test -p `) +- Read from non-finalized L1 blocks (reorg risk) + +## Commands + +```bash +# Rust - ALWAYS use package-specific commands +cargo fmt +cargo check -p --tests +cargo clippy -p --tests +cargo test -p -- + +# Full workspace (pre-commit only) +just check # Check postgres + embedded-db variants +just lint # Clippy with -D warnings +just test # Run nextest (skips slow tests) + +# Solidity contracts +forge fmt +forge test # Unit tests only +just sol-test # Full test suite +just gen-bindings # Regenerate Rust bindings + +# E2E integration tests +just test-demo base # Basic integration +just test-demo pos-base # PoS integration + +# Running locally +just demo-native # Full local network via process-compose +``` + +## Code Style + +**Rust:** + +- Imports: `use module::Type` for types, `module::func()` for function calls +- Errors: `anyhow` for binaries/applications, `thiserror` for libraries +- Async: Prefer `async_trait` for trait definitions with async methods +- Testing: `#[cfg(test)]` modules in same file, integration tests in `tests/` + +**Solidity:** + +- Use `forge fmt` before committing +- Upgradeable contracts: V2 extends V1, never modify V1 storage +- Events: Emit for all state changes that external systems need to track + +## Architecture Overview + +### HotShot vs Sequencer Separation + +The codebase separates **consensus** (HotShot) from **application logic** (Sequencer): + +- **HotShot** (`crates/hotshot/`): Generic BFT consensus library. Defines traits like `NodeType`, handles view-based + voting, leader election, certificates, and network communication. Application-agnostic. + +- **Sequencer** (`sequencer/`, `types/`): Espresso-specific application built on HotShot. Implements `NodeType` via + `SeqTypes` in `types/src/v0/mod.rs`, defining concrete types for headers, payloads, transactions, and validated state. + Handles L1 integration, namespaces, fees, and rollup-specific logic. + +### Sequencer Internal Architecture + +The sequencer is built around `SequencerContext` (`context.rs`), which wraps HotShot's `SystemContextHandle`. Key +components: + +- **Node struct** (`lib.rs`): Generic over network (`N: ConnectedNetwork`) and persistence (`P: SequencerPersistence`) +- **ValidatedState** (`state.rs`): Manages three merkle trees - fee accounts, block commitments, and validator rewards +- **API** (`api/`): Tide-disco HTTP server with modular endpoints (query, submit, status, catchup, light_client, + explorer) +- **Persistence** (`persistence/`): Pluggable storage backends implementing `SequencerPersistence` + +### Transaction and Block Flow + +**Transaction Submission:** + +1. Client submits via HTTP POST to `/submit/submit` +2. Sequencer validates size, broadcasts to DA committee via P2P network +3. Builders listen to the network and accumulate transactions + +**Block Proposal (Leader only):** + +1. Leader queries configured builder URLs for available blocks +2. Selects best block (by fee), creates `QuorumProposal`, broadcasts to validators + +**Block Validation (All validators):** + +1. `ValidatedState::validate_and_apply_header()` performs application-level validation +2. Computes state transition: fee charges, L1 deposits, block rewards +3. Validates: timestamps, builder signature, height, chain config, block size, fees +4. Validates merkle roots: fee tree, block tree, reward tree +5. Validates L1 references are non-decreasing +6. If valid, validator votes on the proposal + +### HotShot Internal Architecture + +`SystemContext` (`crates/hotshot/hotshot/src/lib.rs`) is the main entry point. Key abstractions: + +- **NodeType trait**: Defines types for View, Epoch, BlockHeader, BlockPayload, SignatureKey, Transaction, + ValidatedState, Membership +- **Tasks**: Spawned via `ConsensusTaskRegistry`, communicate through broadcast channels using `HotShotEvent` variants +- **View-based consensus**: Each view has a deterministic leader. Leader collects QC, fetches transactions via builder, + creates DA/Quorum proposals. Replicas validate and vote. +- **Epoch membership**: `EpochMembershipCoordinator` manages stake tables per epoch. Epoch transitions occur at block + boundaries. + +### L1 Integration + +The sequencer uses **only finalized L1 blocks** to avoid reorg issues: + +- **L1Client** (`types/src/v0/impls/l1.rs`): Tracks L1 `head` and `finalized` block numbers. Uses `BlockId::finalized()` + for all reads. +- **Block headers**: Every Espresso header contains `l1_finalized` referencing the latest finalized L1 block. Proposal + validation enforces this is non-decreasing. +- **Data read from L1**: Fee deposits (FeeContract), stake table events (ValidatorRegistered, Delegated, etc.) + +### Stake Table Events + +The StakeTable contract emits events that affect consensus membership: + +- `ValidatorRegistered/Exit` - Validator registration/deregistration +- `Delegated/Undelegated` - Stake delegation changes +- `ConsensusKeysUpdated` - Key rotation + +The `Fetcher` (`types/src/v0/impls/stake_table.rs`) polls finalized L1 blocks, fetches events, validates signatures. +Events transform into a `ValidatorMap`, then `select_active_validator_set()` picks top 100 validators by stake. Changes +affect consensus starting from the next epoch boundary. + +### Reward Claims + +Rewards accumulate in a `RewardMerkleTreeV2` (160-level binary tree keyed by Ethereum address). The tree root is +committed in each block header as part of `auth_root`. + +**Claim flow:** + +1. User queries API at `reward-state-v2/reward-claim-input/{block_height}/{address}` for merkle proof +2. Calls `RewardClaim.claimRewards(lifetimeRewards, authData)` on L1 +3. Contract verifies merkle proof matches `lightClient.authRoot()` +4. On success, mints `lifetimeRewards - alreadyClaimed` ESP tokens + +### Catchup + +Nodes use sparse merkle trees (storing only necessary paths). When validating blocks, missing proofs are fetched +on-demand. + +**Triggered during** `ValidatedState::apply_header()` when fee accounts or block frontier entries are missing. + +**Providers** (`sequencer/src/catchup.rs`): + +- `SqlStateCatchup` - Local database lookup +- `StatePeers` - Remote peer HTTP fetch with reliability scoring +- `ParallelStateCatchup` - Tries local first, falls back to peers + +**Data fetched:** Fee account proofs, reward account proofs, block merkle frontier, chain config, leaf chain for stake +table sync. + +**API:** Endpoints under `/catchup/` serve proof data to peers (schema in `sequencer/api/catchup.toml`). + +### Key Crates + +| Crate | Purpose | +| -------------------------- | ---------------------------------------------------------- | +| `sequencer` | Main node binary, API, persistence | +| `espresso-types` (types/) | Domain types: Header, Payload, Transaction, ValidatedState | +| `hotshot` | BFT consensus implementation | +| `hotshot-query-service` | Query APIs for blocks and availability data | +| `hotshot-state-prover` | ZK proof generation for light client updates | +| `hotshot-contract-adapter` | Rust-Solidity type bridge | +| `staking-cli` | CLI for stake table contract interaction | + +### Key Contracts (`contracts/src/`) + +| Contract | Purpose | +| ------------------------------------- | ------------------------------------------------------- | +| `LightClient.sol` | Verifies HotShot state proofs, stores block commitments | +| `StakeTable.sol` / `StakeTableV2.sol` | Validator staking, delegations, withdrawals | +| `FeeContract.sol` | Builder fee deposits | +| `EspToken.sol` | ESP token (ERC20) | +| `RewardClaim.sol` | Validator reward distribution | + +### Protocol Versions + +Versions in `types/src/v0/mod.rs`. `SequencerVersions` defines version pairs for network operation. + +| Version | Alias | Key Changes | +| ------- | ---------------------------- | -------------------------------------------------------------------------- | +| V0_1 | - | Base types: Header, ChainConfig, Transaction, ADVZ VID proofs | +| V0_2 | `FeeVersion` | Fee support (version marker) | +| V0_3 | `EpochVersion` | PoS: stake_table_contract, reward_merkle_tree, AvidM VID proofs | +| V0_4 | `DrbAndHeaderUpgradeVersion` | Header adds timestamp_millis, total_reward_distributed, RewardMerkleTreeV2 | +| V0_5 | `DaUpgradeVersion` | DA upgrade (version marker) | +| V0_6 | `Vid2UpgradeVersion` | VID2 (AvidmGf2) proofs | + +## Consensus Upgrades + +HotShot supports protocol upgrades via an `UpgradeProposal` mechanism. See `doc/upgrades.md` for full details. + +**How upgrades work:** + +1. An `UpgradeProposal` is broadcast several views before the upgrade +2. Validators vote on the proposal; once enough votes are collected, an `UpgradeCertificate` is formed +3. The certificate is attached to subsequent `QuorumProposal`s until the network upgrades + +**Configuration** (in genesis TOML): + +- View-based: `start_proposing_view`, `stop_proposing_view`, `start_voting_view`, `stop_voting_view` +- Time-based: Same parameters but with Unix timestamps + +## Feature Flags + +| Feature | Default | Purpose | +| ---------------- | ------- | ------------------------ | +| `fee` | Yes | Fee contract integration | +| `pos` | Yes | Proof of stake | +| `drb-and-header` | Yes | DRB and header upgrades | +| `da-upgrade` | Yes | DA committee upgrades | +| `embedded-db` | No | SQLite backend | + +**IMPORTANT:** `embedded-db` requires sqlx with different features than PostgreSQL. Since Rust features are additive and +global to compilation, use `sequencer-sqlite` crate for SQLite builds. + +## Testing + +**This is blockchain infrastructure. Bugs can cause irreversible financial losses.** + +### Testing Philosophy + +- **Correctness over coverage**: Tests must prove the code is correct, not just hit line counts +- **Requirements traceability**: Each requirement should have corresponding test(s) +- **Edge cases are mandatory**: Boundary conditions, error paths, adversarial inputs +- **Regression tests first**: When fixing bugs, write a failing test before the fix + +Agents make writing tests fast. There is no excuse for untested code. + +### Test Structure + +| Layer | Location | Purpose | Command | +| ----------------------- | ------------------------------- | ----------------------------------------- | ---------------------------------------- | +| Unit tests | Within crate modules | Test individual functions/modules | `cargo test -p ` | +| Reference/Serialization | `types/src/reference_tests.rs` | Verify serialization compatibility | `cargo test -p espresso-types reference` | +| HotShot tests | `crates/hotshot/testing/tests/` | Consensus task tests, network simulations | `just hotshot::test ` | +| Integration (E2E) | `tests/` | Full system tests | `cargo nextest run -p tests` | +| Slow tests | `slow-tests/` | Long-running tests | `just test-slow` | +| Contract tests | `contracts/test/` | Solidity unit/fuzz/invariant tests | `just sol-test` | + +### Serialization Compatibility Tests + +**IMPORTANT:** The `types/src/reference_tests.rs` module ensures backward compatibility. If you change a serializable +type: + +1. Run `cargo test -p espresso-types reference` +2. If tests fail and change is intentional, update reference files in `/data/` and commitment constants +3. If unintentional, revert your changes + +## Storage Layer + +For node operator details, see https://docs.espressosys.com/network/guides/node-operators/running-a-sequencer-node + +### Storage Backends + +| Backend | Module | Production Use | Merklized State | Pruning | +| ---------- | ------------------------ | ----------------------- | --------------- | ------- | +| PostgreSQL | `sql.rs` | Yes (DA/Archival) | Yes | Yes | +| Filesystem | `fs.rs` | Yes (non-DA validators) | No | Limited | +| SQLite | `sql.rs` + `embedded-db` | Not yet | Yes | Yes | + +### Storage Migrations + +**IMPORTANT:** When adding storage functionality, ALL THREE backends must be updated. + +**SQL Migrations (PostgreSQL and SQLite):** + +- Uses [Refinery](https://github.com/rust-db/refinery) migration framework +- Naming: `V{version}__{description}.sql` (e.g., `V501__epoch_tables.sql`) +- Locations: `sequencer/api/migrations/{postgres,sqlite}/`, `hotshot-query-service/migrations/{postgres,sqlite}/` +- Version numbering: hotshot-query-service uses multiples of 100 (V100, V200...) leaving gaps for applications + +**Filesystem Migrations (`sequencer/src/persistence/fs.rs`):** + +- Code-based migrations tracked via `migrated` HashSet +- Requirements: Must be recoverable, use atomic file operations, be tested + +**Adding storage functionality checklist:** + +1. Add PostgreSQL migration: `sequencer/api/migrations/postgres/V{next}__{name}.sql` +2. Add SQLite migration: `sequencer/api/migrations/sqlite/V{next}__{name}.sql` +3. Update filesystem persistence if data format changes +4. Update `SequencerPersistence` trait implementation for all backends +5. Test: `cargo test -p sequencer persistence` + +## API Development + +APIs use [tide-disco](https://github.com/EspressoSystems/tide-disco) with TOML schema definitions. + +**Schema files:** `sequencer/api/*.toml`, `hotshot-query-service/api/*.toml` + +**Adding a new endpoint:** + +1. Add route to `.toml` file with `PATH`, parameter types, `METHOD`, and `DOC` +2. Implement handler in corresponding Rust module (e.g., `sequencer/src/api/endpoints.rs`) +3. Register handler with `.get("route_name", handler)` or `.at("route_name", handler)` +4. Ensure data source trait has required method + +## Debugging + +**Compilation slow or OOM:** Use `-p ` for all cargo commands. For HotShot tests, use +`just hotshot::test `. + +**Tests fail after type changes:** Run `cargo test -p espresso-types reference`. Update `/data/` if change is +intentional. + +**Storage migration failures:** Verify all three backends updated. Check version numbers don't conflict. + +## Key Files + +- `justfile` - All build/test/deploy commands +- `Cargo.toml` - Workspace definition and default members +- `data/genesis/*.toml` - Genesis configurations +- `data/v1/`, `data/v2/`, etc. - Reference serialization test vectors +- `doc/upgrades.md` - Upgrade mechanism documentation +- `sequencer/api/*.toml` - API schema definitions + +## Maintaining This Document + +**Keep this file up to date.** When making changes that affect build/test commands, architecture, storage backends, +feature flags, or API definitions, update the relevant section. diff --git a/CLAUDE.md b/CLAUDE.md new file mode 120000 index 00000000000..47dc3e3d863 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1 @@ +AGENTS.md \ No newline at end of file