Skip to content

Commit 7239a4e

Browse files
mraljmralj
committed
added txpool readme
1 parent 0d8fbea commit 7239a4e

File tree

2 files changed

+252
-11
lines changed

2 files changed

+252
-11
lines changed

‎yarn-project/p2p/src/mem_pools/tx_pool/README.md‎

Lines changed: 241 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,241 @@
1+
# Transaction Pool (Mempool)
2+
3+
This module implements the transaction pool (mempool) for the Aztec P2P network. The mempool holds unconfirmed transactions awaiting inclusion in a block.
4+
5+
## Overview
6+
7+
The transaction pool serves as a staging area for transactions before they are included in blocks. It manages the lifecycle of transactions from initial submission through mining, handling duplicates, priority ordering, and eviction of invalid or low-priority transactions.
8+
9+
## Interface: `TxPool`
10+
11+
The [`TxPool`](tx_pool.ts) interface defines the contract that all transaction pool implementations must fulfill:
12+
13+
### Transaction Lifecycle
14+
15+
| Method | Description |
16+
|--------|-------------|
17+
| `addTxs(txs, opts?)` | Adds transactions to the pool. Duplicates are ignored. Returns count of newly added txs. |
18+
| `deleteTxs(txHashes, opts?)` | Removes transactions from the pool. Supports soft-delete for mined txs. |
19+
| `markAsMined(txHashes, blockHeader)` | Marks transactions as included in a block. |
20+
| `markMinedAsPending(txHashes, blockNumber)` | Reverts mined transactions to pending (used during reorgs). |
21+
| `getArchivedTxByHash(txHash)` | Retrieves archived (historical) transactions. |
22+
| `getTxStatus(txHash)` | Returns status: `'pending'`, `'mined'`, `'deleted'`, or `undefined`. |
23+
24+
### Transaction Fetching
25+
26+
| Method | Description |
27+
|--------|-------------|
28+
| `hasTx(txHash)` / `hasTxs(txHashes)` | Checks if transaction(s) exist in the pool. |
29+
| `getTxByHash(txHash)` | Retrieves a transaction by its hash. |
30+
| `getTxsByHash(txHashes)` | Batch retrieval of transactions by hash. |
31+
| `getAllTxs()` / `getAllTxHashes()` | Returns all transactions or their hashes. |
32+
| `getPendingTxHashes()` | Returns pending tx hashes **sorted by priority** (highest first). |
33+
| `getPendingTxCount()` | Returns count of pending transactions. |
34+
| `getMinedTxHashes()` | Returns mined tx hashes with their block numbers. |
35+
36+
### Pool Management
37+
38+
| Method | Description |
39+
|--------|-------------|
40+
| `updateConfig(config)` | Updates pool configuration (max size, archive limit). |
41+
| `markTxsAsNonEvictable(txHashes)` | Protects transactions from eviction. |
42+
| `clearNonEvictableTxs()` | Clears non-evictable flag from all transactions. |
43+
| `cleanupDeletedMinedTxs(blockNumber)` | Permanently removes soft-deleted txs from blocks ≤ blockNumber. |
44+
| `isEmpty()` | Checks if the pool has no transactions. |
45+
46+
### Events
47+
48+
The pool emits a `txs-added` event when new transactions are successfully added, allowing subscribers to react to pool changes.
49+
50+
## Implementations
51+
52+
### `InMemoryTxPool` (Testing Only)
53+
54+
The [`InMemoryTxPool`](memory_tx_pool.ts) is a simple in-memory implementation intended **exclusively for testing purposes**. It stores all data in JavaScript `Map` and `Set` structures without persistence. This implementation lacks the eviction rules and advanced indexing of the production implementation.
55+
56+
### `AztecKVTxPool` (Production)
57+
58+
The [`AztecKVTxPool`](aztec_kv_tx_pool.ts) is the production-grade implementation backed by a persistent key-value store. It provides:
59+
60+
- **Persistent storage** via `AztecAsyncKVStore`
61+
- **Multiple indexes** for efficient queries
62+
- **Automatic eviction** of invalid and low-priority transactions
63+
- **Transaction archival** for historical lookups
64+
- **Soft-delete semantics** for mined transactions
65+
66+
#### Storage Structure
67+
68+
The pool maintains several KV maps and indexes:
69+
70+
| Store | Purpose |
71+
|-------|---------|
72+
| `#txs` | Primary storage: tx hash → serialized tx buffer |
73+
| `#minedTxHashToBlock` | Index of mined txs: tx hash → block number |
74+
| `#pendingTxPriorityToHash` | Priority-ordered index of pending txs |
75+
| `#deletedMinedTxHashes` | Soft-deleted mined txs: tx hash → original block number |
76+
| `#blockToDeletedMinedTxHash` | Reverse index for cleanup: block → deleted tx hashes |
77+
| `#txHashToHistoricalBlockHeaderHash` | Anchor block reference for each tx |
78+
| `#historicalHeaderToTxHash` | Index from historical block → tx hashes |
79+
| `#feePayerToTxHash` | Index from fee payer address → tx hashes |
80+
| `#archivedTxs` | Archived transactions for historical lookup |
81+
82+
#### In-Memory Caches
83+
84+
| Cache | Purpose |
85+
|-------|---------|
86+
| `#pendingTxs` | Hydrated pending transactions for fast access |
87+
| `#nonEvictableTxs` | Set of tx hashes protected from eviction |
88+
89+
## Transaction Priority
90+
91+
Transactions are prioritized based on their **total priority fees** (see [`priority.ts`](priority.ts)):
92+
93+
```typescript
94+
priorityFee = maxPriorityFeesPerGas.feePerDaGas + maxPriorityFeesPerGas.feePerL2Gas
95+
```
96+
97+
The priority is stored as a hex string derived from a 32-byte buffer representation of the fee amount, enabling lexicographic ordering in the KV store. Pending transactions are returned in **descending priority order** (highest fees first).
98+
99+
## Transaction Lifecycle in AztecKVTxPool
100+
101+
### 1. Adding Transactions
102+
103+
When `addTxs()` is called:
104+
105+
1. Check for duplicates (skip if tx already exists)
106+
2. Store the serialized tx in `#txs`
107+
3. Index the tx by its anchor block hash
108+
4. If not already mined, add to pending indexes:
109+
- Priority-to-hash index (for ordering)
110+
- Historical header index (for reorg handling)
111+
- Fee payer index (for balance validation)
112+
5. Record metrics
113+
6. Trigger eviction rules for `TXS_ADDED` event
114+
7. Emit `txs-added` event
115+
116+
### 2. Marking as Mined
117+
118+
When a block is finalized, `markAsMined()`:
119+
120+
1. Move tx from pending to mined status
121+
2. If previously soft-deleted, restore to mined status
122+
3. Trigger eviction rules for `BLOCK_MINED` event
123+
124+
### 3. Handling Reorgs
125+
126+
When blocks are pruned, `markMinedAsPending()`:
127+
128+
1. Remove tx from mined index
129+
2. Rehydrate pending indexes
130+
3. Trigger eviction rules for `CHAIN_PRUNED` event
131+
132+
### 4. Deleting Transactions
133+
134+
The `deleteTxs()` method handles two cases:
135+
136+
- **Pending transactions**: Permanently deleted (transactions and all indexes to the transaction)
137+
- **Mined transactions**: Soft-deleted by default (moved to `#deletedMinedTxHashes`), with option for permanent deletion
138+
139+
Soft-deleted mined transactions are retained for potential future reference and can be permanently cleaned up later via `cleanupDeletedMinedTxs()`.
140+
141+
## Eviction System
142+
143+
The eviction system automatically removes invalid or low-priority transactions based on configurable rules. See the [`eviction/`](eviction/) subdirectory for implementation details.
144+
145+
### Architecture
146+
147+
```
148+
┌─────────────────────────────────────────────────────────────────┐
149+
│ EvictionManager │
150+
│ Orchestrates eviction rules based on pool events │
151+
├─────────────────────────────────────────────────────────────────┤
152+
│ │
153+
│ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ │
154+
│ │EvictionRule #1 │ │EvictionRule #2 │ │EvictionRule #N │ │
155+
│ └─────────────────┘ └─────────────────┘ └─────────────────┘ │
156+
│ │
157+
└─────────────────────────────────────────────────────────────────┘
158+
│
159+
▼
160+
┌─────────────────┐
161+
│ TxPoolOperations│
162+
│ (interface) │
163+
└─────────────────┘
164+
```
165+
166+
The [`EvictionManager`](eviction/eviction_manager.ts) coordinates eviction by:
167+
168+
1. Registering multiple `EvictionRule` implementations
169+
2. Calling each rule when tx pool events occur
170+
3. Propagating configuration updates to all rules
171+
172+
### Eviction Events
173+
174+
| Event | Trigger | Purpose |
175+
|-------|---------|---------|
176+
| `TXS_ADDED` | New transactions added | Enforce pool size limits |
177+
| `BLOCK_MINED` | Block finalized | Remove invalidated transactions |
178+
| `CHAIN_PRUNED` | Chain reorganization | Remove txs referencing pruned blocks |
179+
180+
### Eviction Rules
181+
182+
#### 1. `InvalidTxsAfterMiningRule`
183+
184+
**Triggers on:** `BLOCK_MINED`
185+
186+
Evicts transactions that become invalid after a block is mined:
187+
188+
- Duplicate nullifiers: Txs with nullifiers already included in the mined block
189+
- Expired transactions: Txs with `includeByTimestamp` ≤ mined block timestamp
190+
191+
#### 2. `InvalidTxsAfterReorgRule`
192+
193+
**Triggers on:** `CHAIN_PRUNED`
194+
195+
Evicts transactions that reference blocks no longer in the canonical chain:
196+
197+
- Checks each pending tx's anchor block hash against the archive tree
198+
- Removes txs whose anchor blocks are not found (pruned)
199+
200+
#### 3. `InsufficientFeePayerBalanceRule`
201+
202+
**Triggers on:** `BLOCK_MINED`, `CHAIN_PRUNED`
203+
204+
Evicts transactions whose fee payer no longer has sufficient balance:
205+
206+
- Uses `GasTxValidator` to check fee payer balances against current world state
207+
208+
#### 4. `LowPriorityEvictionRule`
209+
210+
**Triggers on:** `TXS_ADDED`
211+
212+
Enforces maximum pool size by evicting lowest-priority (by fee) transactions:
213+
214+
- Configured via `maxPendingTxCount` option (0 = disabled)
215+
- Uses `getLowestPriorityEvictable()` to find txs to evict
216+
217+
### Non-Evictable Transactions
218+
219+
Transactions can be marked as non-evictable via `markTxsAsNonEvictable()`. This protects them from all eviction rules, typically used during block building to ensure transactions being processed aren't evicted mid-operation. The flag is cleared after block processing via `clearNonEvictableTxs()`.
220+
The `clearNonEvictableTxs` is called upon getting new L2 block.
221+
222+
## Configuration
223+
224+
The pool accepts configuration via `TxPoolOptions`:
225+
226+
```typescript
227+
type TxPoolOptions = {
228+
maxPendingTxCount?: number; // Max pending txs (0 = unlimited)
229+
archivedTxLimit?: number; // Number of archived txs to retain
230+
};
231+
```
232+
233+
Configuration can be updated at runtime via `updateConfig()`.
234+
235+
## Telemetry
236+
237+
The pool integrates with the telemetry system to report:
238+
239+
- Transaction counts (pending vs mined)
240+
- Transaction sizes
241+
- Store size estimates

‎yarn-project/p2p/src/mem_pools/tx_pool/aztec_kv_tx_pool.ts‎

Lines changed: 11 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -174,16 +174,6 @@ export class AztecKVTxPool
174174
const key = hash.toString();
175175
await this.#minedTxHashToBlock.set(key, blockHeader.globalVariables.blockNumber);
176176

177-
// If this tx was previously soft-deleted, remove it from the deleted sets
178-
if (await this.#deletedMinedTxHashes.hasAsync(key)) {
179-
const originalBlock = await this.#deletedMinedTxHashes.getAsync(key);
180-
await this.#deletedMinedTxHashes.delete(key);
181-
// Remove from block-to-hash mapping
182-
if (originalBlock !== undefined) {
183-
await this.#blockToDeletedMinedTxHash.deleteValue(originalBlock, key);
184-
}
185-
}
186-
187177
const tx = await this.getPendingTxByHash(hash);
188178
if (tx) {
189179
const nullifiers = tx.data.getNonEmptyNullifiers();
@@ -193,6 +183,16 @@ export class AztecKVTxPool
193183

194184
await this.removePendingTxIndices(tx, key);
195185
}
186+
187+
// If this tx was previously soft-deleted, remove it from the deleted sets
188+
if (await this.#deletedMinedTxHashes.hasAsync(key)) {
189+
const originalBlock = await this.#deletedMinedTxHashes.getAsync(key);
190+
await this.#deletedMinedTxHashes.delete(key);
191+
// Remove from block-to-hash mapping
192+
if (originalBlock !== undefined) {
193+
await this.#blockToDeletedMinedTxHash.deleteValue(originalBlock, key);
194+
}
195+
}
196196
}
197197
});
198198

@@ -273,7 +273,7 @@ export class AztecKVTxPool
273273
/**
274274
* Adds a list of transactions to the pool. Duplicates are ignored.
275275
* @param txs - An array of txs to be added to the pool.
276-
* @returns Empty promise.
276+
* @returns count of added transactions
277277
*/
278278
public async addTxs(txs: Tx[], opts: { source?: string } = {}): Promise<number> {
279279
const addedTxs: Tx[] = [];

0 commit comments

Comments
 (0)