Remove redis dependency and store balances in postgres

[aditya1702]

What

This PR is a simple merge of all approved PRs listed here:

• Remove redis and store account's trustlines and contract tokens in postgres tables: Use normalized (accountID, tokenID) tables for storing account trustlines and contract tokens #451
• Process trustline entry changes and store account trustline balances: Process trustline entry changes and store trustline balances #452
• Process account entry changes and store native balances and base reserve balance: Process account entry changes and store XLM balances and minimum XLM reserves #453
• Process SAC balance entry changes and store SAC balances for C-addresses: Process SAC balance entry changes and store SAC balances #454

Why

Storing balances in wallet backend itself eliminates dependency on RPC and makes it efficient to support account balances feature

Known limitations

N/A

Issue that this PR addresses

Closes #457 #458 #459 #460

[Copilot]

Pull request overview

This PR consolidates multiple previously approved changes to remove Redis, move all token/balance state into PostgreSQL, and expose richer balance data (trustlines, native XLM, and SAC balances) via GraphQL and the wbclient SDK.

Changes:

• Remove the Redis dependency end-to-end (code, tests, config, integration infra) and replace it with Postgres-based models and migrations for trustline assets, contract tokens, account→token mappings, and balance tables (trustline, native, SAC).
• Introduce ingestion-time processing of ledger changes for trustlines, accounts, SEP-41 contracts, and SAC balances (new processors, indexer wiring, ingestion service refactor) to keep Postgres balance state in sync.
• Update the GraphQL schema, resolvers, wbclient types/queries, and integration tests to consume balances exclusively from Postgres, with new fields for native XLM (minimum balance, liabilities, last-modified ledger).

Reviewed changes

Copilot reviewed 74 out of 77 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
go.mod / go.sum	Remove Redis and related dependencies; add github.com/google/uuid and pgx usage for new models.
docker-compose.yaml	Drop Redis service, volumes, and environment variables; ingestion/serve now rely solely on Postgres and RPC.
cmd/utils/global_options.go	Remove Redis CLI options; only DB / ingest / GraphQL options remain.
cmd/serve.go	Drop Redis options from the serve command; handler deps now wire in DB-backed balance models instead of AccountTokenService.
cmd/ingest.go	Remove Redis CLI options from ingest command.
internal/store/redis.go / redis_test.go	Delete Redis store implementation and tests entirely.
internal/utils/utils.go	Remove RPC-oriented ledger-key helpers that are no longer used; keep IsContractAddress helper.
internal/db/migrations/2025-06-13.0-contract_tokens.sql	Change contract_tokens id from TEXT to UUID and add contract_id TEXT UNIQUE NOT NULL + index, enabling deterministic IDs.
internal/db/migrations/2025-12-14.0-trustline_assets.sql	Change trustline_assets id from BIGSERIAL to UUID to support deterministic asset IDs.
internal/db/migrations/2026-01-12.0-account-token-tables.sql	Create account_trustlines and account_contracts junction tables for account→asset/contract relationships.
internal/db/migrations/2026-01-12.0-trustline_balances.sql	New trustline_balances table storing balance, limit, liabilities, flags, last_modified_ledger keyed by (account_address, asset_id).
internal/db/migrations/2026-01-12.1-account_contract_tokens.sql	New account_contract_tokens table for account→contract token relationships used by GraphQL.
internal/db/migrations/2026-01-15.0-native_balances.sql	New native_balances table for XLM balance, minimum_balance, liabilities, last_modified_ledger.
internal/db/migrations/2026-01-16.0-sac-balances.sql	New sac_balances table for SAC balances keyed by (account_address, contract_id) with auth/clawback flags and last_modified_ledger.
internal/data/trustline_assets.go	Switch to deterministic UUIDv5 asset IDs (DeterministicAssetID), add BatchInsert using pgx; remove old BatchGetOrInsert/GetTopN APIs.
internal/data/contract_tokens.go	Switch to deterministic UUIDv5 contract IDs (DeterministicContractID), add contract_id column usage, new GetExisting + BatchInsert APIs.
internal/data/trustline_balances.go	New model for trustline balances with GetByAccount, BatchUpsert, and BatchCopy on trustline_balances joined with trustline_assets.
internal/data/native_balances.go	New model for native XLM balances with GetByAccount, BatchUpsert, and BatchCopy on native_balances.
internal/data/sac_balances.go	New model for SAC balances with GetByAccount (joining contract_tokens), BatchUpsert, and BatchCopy on sac_balances.
internal/data/account_tokens.go	New AccountTokensModel for account_trustlines/account_contracts (used in ingestion/backfill checkpointing paths).
internal/data/account_contract_tokens.go	New AccountContractTokensModel for account→contract mappings used by GraphQL (via join to contract_tokens).
internal/data/models.go	Extend Models struct to include TrustlineBalance, NativeBalance, SACBalance, and AccountContractTokens.
internal/data/mocks.go	Add mocks for new models (TrustlineBalanceModelMock, NativeBalanceModelMock, SACBalanceModelMock, AccountContractTokensModelMock) and update ContractModelMock for new signatures.
internal/services/mocks.go	Replace AccountTokenServiceMock with TokenIngestionServiceMock; update ContractMetadataServiceMock to support FetchSep41Metadata, FetchSACMetadata, and FetchMetadata.
internal/services/ingest.go	Switch ingest service config and struct to use TokenIngestionService; refactor ingestion to insert transactions/ops/state changes via insertIntoDB and delegate all token processing to TokenIngestionService.
internal/services/ingest_live.go	Introduce ingestProcessedDataWithRetry that, within a single DB transaction, inserts trustline assets, inserts new contracts (including SAC/SEP-41), calls filterParticipantData, writes core ledger data, unlocks channels, and calls TokenIngestionService.ProcessTokenChanges.
internal/services/ingest_backfill.go	Update backfill to use new token ingestion flow using maps of changes, and to work with the new models and contract metadata service.
internal/services/contract_metadata.go	Redesign ContractMetadataService to only fetch metadata via RPC: FetchSep41Metadata and FetchSACMetadata now return fully-formed data.Contract objects with deterministic IDs but do not store directly to DB.
internal/services/token_ingestion.go	New/renamed ingestion service implementing TokenIngestionService: streams checkpoint data into trustline_balances, native_balances, sac_balances, and contract tables; processes per-ledger trustline/account/SAC balance changes and SEP-41 contract changes using new models and deterministic IDs.
internal/ingest/ingest.go	Wire up TokenIngestionService instead of Redis-based AccountTokenService; remove Redis from ingest dependency graph.
internal/indexer/types/types.go	Extend TrustlineChange with full XDR fields, add TrustlineOpUpdate; introduce AccountChange and SACBalanceChange structs and associated enums for ingestion.
internal/indexer/processors/trustlines.go / _test.go	New TrustlinesProcessor to extract full trustline XDR state from ledger changes; tests for create/update/remove and field correctness.
internal/indexer/processors/accounts.go / _test.go	New AccountsProcessor to extract account balance, liabilities, and computed minimum balance from account ledger entries; tests for create/update/remove and signer-only updates.
internal/indexer/processors/sac_balances.go / _test.go	New SACBalancesProcessor to extract SAC balances from contract data via sac.ContractBalanceFromContractData (contract-only holders); tests for create/update/remove, Int128 balances, and non-balance entries.
internal/indexer/processors/sac_instances.go / _test.go	New SACInstanceProcessor to parse SAC instance contract data and emit data.Contract metadata (code, issuer, name, symbol, decimals).
internal/indexer/processors/effects.go	Simplify trustline effect handling to only manage state-change history; balance fields now come from TrustlinesProcessor.
internal/indexer/processors/token_transfer.go	Minor refactor to compute contract type once and reuse in debit/credit state changes.
internal/indexer/processors/contracts/sac.go	Harden isContractAddress using strkey decoding rather than checking first char.
internal/indexer/indexer_buffer.go / _test.go	Extend buffer to track maps of trustline, account, and SAC balance changes keyed by composite keys; add SAC balance merge/dedup behavior.
internal/indexer/mocks.go	Add mocks for new ledger change processors (MockTrustlinesProcessor, MockAccountsProcessor, MockSACBalancesProcessor, MockSACInstancesProcessor).
internal/indexer/indexer.go / _test.go	Wire in new ledger change processors (trustlines, accounts, SAC balances, SAC instances) into Indexer.processTransaction and update tests accordingly; use utils.IsContractAddress for distinguishing C/G-addresses when creating ContractChanges.
internal/serve/serve.go	Remove Redis from serve wiring; inject TrustlineBalanceModel, NativeBalanceModel, SACBalanceModel, AccountContractTokensModel, and ContractMetadataService into resolvers; pass sqlx DB into metrics.
internal/serve/graphql/schema/balances.graphqls	Extend NativeBalance type with minimumBalance, buyingLiabilities, sellingLiabilities, lastModifiedLedger and change TrustlineBalance.lastModifiedLedger to UInt32!.
internal/serve/graphql/generated/*	Regenerated GraphQL models and execution code for new balance fields and UInt32 ledger fields.
internal/serve/graphql/resolvers/resolver.go	Introduce BalanceReader interface; Resolver now depends on a BalanceReader and AccountContractTokensModel instead of an AccountTokenService.
internal/serve/graphql/resolvers/balance_reader.go	New adapter implementing BalanceReader on top of TrustlineBalanceModel, NativeBalanceModel, and SACBalanceModel.
internal/serve/graphql/resolvers/account_balances_utils.go	Replace RPC/XDR-based parsing of balances with DB-driven builders for native, trustline, and SAC balances; keep RPC only for SEP-41 balances via new metadata service interface.
internal/serve/graphql/resolvers/queries.resolvers.go	Multi-account balance resolver now uses BalanceReader to fetch DB-backed balances and parseAccountBalances to construct GraphQL types; SEP-41 balances still fetched via RPC.
internal/serve/graphql/resolvers/account_balances_test.go	Update tests to validate new native balance fields and DB-backed trustline/SAC behavior (including after checkpoint and live ingestion).
pkg/wbclient/types/types.go	Extend NativeBalance client type with new fields and change TrustlineBalance.LastModifiedLedger to uint32.
pkg/wbclient/queries.go	Update balance fragment to request new native balance fields.
internal/integrationtests/infrastructure/*	Remove Redis container from integration infra; adjust RPC config to use HISTORY_RETENTION_WINDOW instead of INGESTION_RANGE; ensure proper cleanup.
internal/integrationtests/main_test.go	Add early t.Fatal to stop subsequent suites if transaction-building suite fails.
internal/integrationtests/account_balances_test.go	Strengthen expectations for native balances after checkpoint and live ingestion, including assertions on minimum balance and lastModifiedLedger; now also asserts specific XLM balance value after live ingestion.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.