feat: store token metadata (name, symbol, decimals) on first encounter #10
Merged
glihm merged 19 commits intodojoengine:mainfrom Feb 23, 2026
Merged
feat: store token metadata (name, symbol, decimals) on first encounter #10glihm merged 19 commits intodojoengine:mainfrom
glihm merged 19 commits intodojoengine:mainfrom
Conversation
MartianGreed
force-pushed
the
feat/token-metadata
branch
from
63bbbdb to
699987d
Compare
Adds token metadata storage and fetching across all ERC sinks: Storage: - New token_metadata table in ERC20/ERC721/ERC1155 databases - ERC20: stores name, symbol, decimals - ERC721/ERC1155: stores name, symbol - UPSERT semantics (COALESCE to preserve existing values) Fetching (MetadataFetcher in torii-common): - Calls name(), symbol(), decimals() via starknet_call on first token encounter - Handles short strings (felt-packed), ByteArray, and legacy array formats - Handles both snake_case and camelCase selectors - token_uri() and uri() helpers for NFT metadata (not auto-fetched yet) gRPC: - New GetTokenMetadata RPC on all three services - Query single token or list all indexed tokens - Returns name, symbol, decimals (ERC20) or name, symbol (ERC721/ERC1155) Pipeline: - Sinks check has_token_metadata() before each batch - New tokens trigger async RPC fetch + storage - ERC20/ERC1155 reuse provider from balance tracking - ERC721 gets new with_metadata_fetching() builder method Note: Proto regeneration required (build.rs will handle it). tokenUri fetching on MetadataUpdate events is prepared but not wired yet.
- Add append_short_string() and append_fixed_string() that write directly to a pre-allocated buffer instead of creating intermediate Strings for each felt chunk - Pre-allocate String capacity based on expected size (data_len * 31) - ByteArray and legacy array decoding now use zero-intermediate-alloc append pattern
…nctions Replace all custom felt-to-string decoding with starknet crate utilities: - parse_cairo_short_string() for short strings (single felt) - ByteArray::decode() for Cairo ByteArray format - Remove felt_to_short_string, felt_to_fixed_string, append_short_string, append_fixed_string — all replaced by starknet crate equivalents Net -86 lines of custom decoding logic.
- Add fetch_total_supply() to MetadataFetcher (tries total_supply then totalSupply) - Add total_supply BLOB column to token_metadata table in ERC721/ERC1155 - Store total_supply on first token encounter alongside name/symbol - Expose via GetTokenMetadata gRPC (optional bytes total_supply field) - Add total_supply to TokenMetadata struct
- Add GetTokenMetadata schema and getErc20TokenMetadata() query - getErc20Balance() auto-fetches decimals and formats with 10^decimals - getErc20Transfers() auto-fetches decimals when filtering by contract - Add formatU256(bytes, decimals?) and formatBigIntWithDecimals() - Export TokenMetadataResult type and new functions
- Add ERC721 and ERC1155 TokenMetadataEntry, GetTokenMetadataRequest/Response schemas - Add getErc721TokenMetadata() and getErc1155TokenMetadata() query functions - Wire metadata loading and display into React, Svelte, and Vanilla apps - TokenPanel components now show a metadata table (contract, name, symbol, decimals for ERC20)
… refs - bytesToHex and formatU256 now properly decode base64 strings from protobuf instead of treating them as UTF-8 text (was using TextEncoder.encode) - Add messageType references to GetTokenMetadataResponse schemas so the SDK can properly decode nested TokenMetadataEntry messages
getErc20Transfers now fetches all token metadata and builds a decimals lookup map, so amounts are properly formatted even when no specific contract address filter is provided (e.g. the initial transfers list).
MartianGreed
force-pushed
the
feat/token-metadata
branch
2 times, most recently
from
e852510 to
48ba0d9
Compare
MartianGreed
force-pushed
the
feat/token-metadata
branch
from
48ba0d9 to
948dd1f
Compare
Architecture: - TokenUriService in torii-common: background service with tokio channel that processes (contract, token_id) URI fetch requests - Deduplicates in-flight tasks: new request for same key cancels the previous task, so we always apply the latest value - Configurable concurrency via semaphore (default: 8 concurrent fetches) - Resolves URIs to JSON metadata (supports https, ipfs, data: URIs) ERC721 changes: - Decoder: Added MetadataUpdate and BatchMetadataUpdate event decoding (EIP-4906) - Sink: Dispatches URI fetch on first token encounter + on MetadataUpdate events - Storage: Added token_uris table (token, token_id, uri, metadata_json) - Implements TokenUriStore trait for async result storage ERC1155 changes: - Sink: Dispatches URI fetch on first token encounter - Storage: Migrated token_uris table to new schema (added metadata_json, removed block_number/tx_hash, changed to composite PK) - Implements TokenUriStore trait main.rs: Spawns TokenUriService per standard and wires senders to sinks
Improvements ported from dojoengine/torii's battle-tested erc.rs:
- HTTP fetching with exponential backoff retries (5 retries, 100ms initial)
- ERC1155 {id} substitution: replaces {id} in URI with zero-padded hex token ID
- data: URI handling: supports base64, URL-encoded, and generic data URIs
with # character workaround (servo/rust-url#908)
- JSON sanitization: filters ASCII control characters, escapes unescaped
internal quotes (handles Loot Survivor-style broken metadata)
- Raw JSON fallback: if URI isn't a URL scheme, tries parsing as inline JSON
- Validates resolved content is actually valid JSON before storing
- Added unit tests for sanitization, data URIs, and {id} substitution
- Added token_attributes table to ERC721 and ERC1155 storage - Composite PK (token, token_id, key) with FK to token_uris - Auto-parses metadata_json on upsert: top-level fields, OpenSea attributes, properties - Added get_token_attributes() and get_tokens_by_attribute() queries - Updated generated proto files with GetTokenMetadata RPCs - Added missing serde_json dependency to torii-common - Fixed all clippy warnings and formatting
feat: async token URI fetching service with deduplication
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.