-
Notifications
You must be signed in to change notification settings - Fork 12
Process SAC balance entry changes and store SAC balances #454
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Replace GetOrInsertTrustlineAssets with EnsureTrustlineAssetsExist (no return) - Update ProcessTokenChanges to compute IDs directly via DeterministicAssetID - Add streaming batch processing (50K batches) for checkpoint trustlines - Remove TrustlineFrequency tracking and processTrustlineAssets (no longer needed) - Simplify storeAccountTokensInPostgres to storeContractsInPostgres (contracts only) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <[email protected]>
- Rename insertTrustlinesAndContractTokensWithRetry to ensureTokensExistWithRetry - Call EnsureTrustlineAssetsExist instead of GetOrInsertTrustlineAssets - Remove trustlineAssetIDMap passing (IDs computed via DeterministicAssetID) - Update ProcessTokenChanges call to remove assetIDMap parameter 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <[email protected]>
- Call EnsureTrustlineAssetsExist instead of GetOrInsertTrustlineAssets - Remove trustlineAssetIDMap passing (IDs computed via DeterministicAssetID) - Update ProcessTokenChanges call to remove assetIDMap parameter 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <[email protected]>
…icipation Changed ProcessTokenChanges signature to accept pgx.Tx parameter, removing the internal db.RunInPgxTransaction call. This allows the function to participate in an outer transaction for atomicity. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <[email protected]>
Changed EnsureTrustlineAssetsExist signature to accept pgx.Tx parameter, removing the internal db.RunInPgxTransaction call. This allows the function to participate in an outer transaction for atomicity with other operations. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <[email protected]>
Replaced in-memory knownContractIDs cache with a DB query to get existing contract IDs. For < 10k contracts, this DB query is fast enough and eliminates cache maintenance complexity. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <[email protected]>
- Merge ensureTokensExistWithRetry into ingestProcessedDataWithRetry - All operations (trustline assets, contracts, txs, ops, token changes, cursor update) now execute in a single transaction - Remove initializeKnownContractIDs (no longer needed with DB queries) - Remove identifyNewContractIDs (cache-related, now unused) - Remove unused set package import This ensures atomicity: if cursor update fails, no partial data is committed. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <[email protected]>
The cache is no longer needed as GetOrInsertContractTokens now queries the database directly for existing contract IDs. For < 10k contracts, this DB query is fast enough and eliminates cache maintenance complexity. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <[email protected]>
Updated ingest_backfill.go to use the new signatures: - EnsureTrustlineAssetsExist now accepts dbTx - GetOrInsertContractTokens no longer takes cache parameter - ProcessTokenChanges now accepts dbTx All three operations now execute in a single atomic transaction. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <[email protected]>
Updated mock methods to match interface changes: - EnsureTrustlineAssetsExist now accepts dbTx - GetOrInsertContractTokens no longer takes cache parameter - ProcessTokenChanges now accepts dbTx Removed unused set package import. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <[email protected]>
…ert in PopulateAccountTokens Part of simplification to remove duplicate code in ContractMetadataService. Use the same pattern as EnsureContractTokensExist for consistency. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <[email protected]>
…nterface Simplify interface to only include FetchMetadata and FetchSingleField methods. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <[email protected]>
This method is no longer needed as callers use FetchMetadata + BatchInsert directly. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <[email protected]>
This helper was only used by FetchAndStoreMetadata which is now removed. Also updated file header comment to reflect the service now only fetches metadata. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <[email protected]>
These tests are no longer needed as the functions were removed. Also removed unused imports (db, dbtest, metrics). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <[email protected]>
… types
- account_tokens_test.go: setupTrustlineAssets and setupContractTokens
now use DeterministicAssetID/DeterministicContractID to compute IDs
since tables require explicit IDs (not auto-generated)
- account_balances_test.go: Partial fix - updated GetAccountContracts
mock return types from []string{} to []*data.Contract{} and removed
obsolete BatchGetByIDs mock expectations (still in progress)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <[email protected]>
…urn type GetAccountContracts now returns []*data.Contract instead of []string. Removed all BatchGetByIDs mock expectations since the implementation no longer calls this method - contracts are returned directly from GetAccountContracts. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <[email protected]>
Fixed two additional tests that were still using []string{} instead of
[]*data.Contract{} for GetAccountContracts mock returns, and removed
unnecessary mockContract references.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <[email protected]>
…tion The implementation now: 1. Calls models.TrustlineAsset.BatchInsert (not EnsureTrustlineAssetsExist) 2. Calls contractMetadataService.FetchMetadata (not FetchAndStoreMetadata) 3. Calls ProcessTokenChanges(ctx, dbTx, trustlineChanges, contractChanges) Removed obsolete EnsureTrustlineAssetsExist mock expectations and updated tests to use the new ProcessTokenChanges signature. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <[email protected]>
Removed obsolete EnsureTrustlineAssetsExist and GetOrInsertContractTokens mock expectations. The implementation now only calls ProcessTokenChanges(ctx, dbTx, trustlineChanges, contractChanges) on tokenCacheWriter. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <[email protected]>
Part of migration from FNV-64 hash to UUID v5 for deterministic IDs to eliminate collision risk. Co-Authored-By: Claude Opus 4.5 <[email protected]>
Part of migration from FNV-64 hash to UUID v5 for deterministic IDs. Co-Authored-By: Claude Opus 4.5 <[email protected]>
ef82c45 to
6adcd48
Compare
e86ee31 to
49cad15
Compare
aristidesstaffieri
approved these changes
Jan 28, 2026
Base automatically changed from
store-native-balance
to
main_balances_postgres
January 28, 2026 22:10
Open
8 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What
Add SAC (Stellar Asset Contract) balance tracking for contract addresses (C...) by storing balances directly in PostgreSQL and reading them from the database instead of making RPC calls. This includes a dedicated
SACBalancesProcessorfor extracting SAC balance changes from ledger entries during live ingestion, and a newsac_balancestable for persistent storage.Why
balance,is_authorized,is_clawback_enabled,last_modified_ledger) enabling richer balance queriesKey Architectural Changes
BEFORE:
flowchart LR A[GraphQL Resolver] --> B[RPC getLedgerEntries] B --> C[Parse SAC Balance Map] C --> D[Return Balance]AFTER:
flowchart LR subgraph Ingestion I1[SACBalancesProcessor] --> I2[(sac_balances)] end subgraph Query A[GraphQL Resolver] --> B{Address Type?} B -->|C-address| C[SACBalanceModel.GetByAccount] C --> D[(PostgreSQL)] B -->|G-address| E[TrustlineBalanceModel] endPer-File Change Summary
internal/db/migrations/2026-01-16.0-account-sac-balances.sqlaccount_sac_balancestable withaccount_address,contract_id,balance,is_authorized,is_clawback_enabled,last_modified_ledgerinternal/data/sac_balances.goSACBalanceModelwithGetByAccount,BatchUpsert,BatchCopymethods for CRUD operationsinternal/data/mocks.goSACBalanceModelInterfacemock for testinginternal/data/models.goSACBalanceModelto models initializationinternal/indexer/types/types.goSACBalanceChangestruct andSACBalanceOpenum (ADD,UPDATE,REMOVE)internal/indexer/processors/sac_balances.goSACBalancesProcessorextracts SAC balance changes from contract data ledger entries usingsac.ContractBalanceFromContractDataSDK functioninternal/indexer/indexer_buffer.goSACBalanceChangeKeycomposite key,sacBalanceChangesByKeymap,PushSACBalanceChange/GetSACBalanceChangesmethods with deduplicationinternal/indexer/indexer.goSACBalancesProcessorinto indexer, calls processor inprocessTransactioninternal/services/token_ingestion.goprocessSACBalanceChangesmethod, updatedProcessTokenChangesinterface to accept SAC balance changes, streams SAC balances during checkpointinginternal/services/contract_metadata.gointernal/services/ingest.goProcessTokenChangesinternal/services/ingest_live.goBatchTokenChangesstructinternal/services/ingest_backfill.gointernal/serve/graphql/resolvers/queries.resolvers.gointernal/serve/graphql/resolvers/account_balances_utils.gobuildSACBalanceFromDBhelper, removedparseSACBalance(no longer needed), addedsacBalancesfield toaccountKeyInfointernal/serve/graphql/resolvers/balance_reader.goGetSACBalancesmethod toBalanceReaderinterfaceinternal/serve/graphql/resolvers/account_balances_test.gointernal/serve/serve.goSACBalanceModelto resolver dependenciesinternal/utils/utils.goIssue that this PR addresses
Closes #460