Add tx-granular ledger snapshot source using meta, RPC, and archives#1657
Add tx-granular ledger snapshot source using meta, RPC, and archives#1657leighmcculloch wants to merge 26 commits intomainfrom
Conversation
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub. |
leighmcculloch
force-pushed
the
snapshot-source-tx
branch
2 times, most recently
from
42f37c9 to
598ad4c
Compare
leighmcculloch
changed the title
leighmcculloch
force-pushed
the
snapshot-source-tx
branch
3 times, most recently
from
0a8ae20 to
19904da
Compare
leighmcculloch
force-pushed
the
snapshot-source-tx
branch
from
19904da to
cdc7199
Compare
|
is it possible to set it so the timestamp is also set after the snapshot has been loaded? Currently we need to manually set it after the |
|
This seems like an interesting way of handling a simulation locally. Or at least doing a read call locally. |
|
Looks awesome. Thanks, Leigh! ❤️ |
| let ledger_cache_dir = self.cache_path.join( | ||
| self.tx_hash | ||
| .map(|h| { | ||
| let tx_hash_str: String = h.iter().map(|b| format!("{b:02x}")).collect(); | ||
| format!("{}-{}-before", self.fetcher.ledger(), tx_hash_str) | ||
| }) | ||
| .unwrap_or_else(|| format!("{}-after", self.fetcher.ledger())), | ||
| ); | ||
|
|
||
| // Ensure cache directory exists | ||
| std::fs::create_dir_all(&ledger_cache_dir).expect("failed to create cache directory"); |
There was a problem hiding this comment.
This probably should be the cache crate's responsibility. If it is abstracted more than this crate will never need to use a fs. This way we could eventually use Wasm to run this in the browser.
|
Warning Review the following alerts detected in dependencies. According to your organization's Security Policy, it is recommended to resolve "Warn" alerts. Learn more about Socket for GitHub.
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Agent: Claude Code Agent-Model: claude-opus-4-6 Agent-Session-Id: 09a10129-7596-430d-979d-70a4c8fed82f
|
Fixed an issue that makes it much faster to run on subsequent runs. |
Agent: Claude Code Agent-Model: claude-opus-4-6 Agent-Session-Id: 09a10129-7596-430d-979d-70a4c8fed82f
Agent: Claude Code Agent-Model: claude-opus-4-6 Agent-Session-Id: 09a10129-7596-430d-979d-70a4c8fed82f
Agent: Claude Code Agent-Model: claude-opus-4-6 Agent-Session-Id: 09a10129-7596-430d-979d-70a4c8fed82f
What
Add new crates for fetching ledger entries from multiple data sources to enable more seamless and built-in fork testing in the sdk. Add
soroban-ledger-snapshot-source-txfor transaction-level snapshot sources.sequenceDiagram autonumber actor App as participant SelTxMeta as Query<br/>Ledger<br/>Tx Meta participant LedgerMeta as Query<br/>Ledger – 1..N<br/>Tx Meta participant Archive as History<br/>Archive participant RPC as RPC App->>SelTxMeta: Lookup with key Note over App,SelTxMeta: If found → use it.<br/>If not → continue. App->>SelTxMeta: Look in prior txs in same ledger Note over App,SelTxMeta: If found → use it.<br/>If not → continue. opt Optionally use RPC App->>RPC: getLedgerEntries([key]) Note over App,RPC: Use if (lastModified < queryLedger)<br/>AND (rpcLatestSeen >= queryLedger).<br/>Otherwise continue. end App->>LedgerMeta: Look in prior ledger meta Note over App,LedgerMeta: If found → use it.<br/>If checkpoint ledger not reached → continue to next ledger meta<br/>If checkpoint ledger is reached → continue to archive. App->>Archive: Download checkpoint from history archive and search Note over App,Archive: If found → use it.<br/>If not → does not exist.Why
The current fork testing experience utilising the stellar-cli has low granularity only at the boundaries of ledgers, and requires downloading full history archives and manually identifying footprints ahead of time, which is difficult to do well and a poor developer experience. This change enables the SDK to lazily fetch ledger entries on-demand from the most efficient source available, caching results locally for subsequent runs. Developers will be able to fork test against any ledger and transaction without pre-identifying the footprint.
The change uses a Ledger Meta Storage (SEP-54), an RPC, and a History Archive to collect ledger entries.
The change caches results in three layers. All raw files downloaded are cached in the system cache directory and reused across tests, across workspaces. All ledger entries found are cached in the system cache directory. All ledger entries found for the current workspace are cached in the
tests-snapshot-sourcedirectory intended to be committed so that CI runs reproducibly without needing to collect entries. Note that the format of that cache is not a ledger snapshot json file because for many tests running concurrently one file per ledger entry is easier to manage.Close #1448
Try it out
Add the following dependencies to your
Cargo.toml:Note: Requires
soroban-sdkv23.4.0or later.Example
Observing State Changes
The
TxSnapshotSourcelooks up state with transaction-level granularity when a transaction hash is provided allowing developers to debug a transaction by starting at the point just before the transaction. For example:Before a specific tx: Pass the tx hash to get state just before that tx executed:
End of ledger (after all txs): Use
Nonefor the tx_hash to get state at the end of the ledger:Different ledgers/txs: Update the test to investigate the state of the balance at each of the following ledgers and transactions to see how the transactions affected the balance. Check out the links to see the operations the transactions performed and how they align with the changes in balances observed.
Debugging with RUST_LOG
Enable logging to see which data sources are being queried and what entries are found:
Example output:
TODO
Thanks
Thanks @orbitlens for sharing the idea of using transaction meta as a way to collect recent state. Thanks to all the people who provided feedback to me about how they use the existing
stellar snapshot createfunctionality.