Skip to content

Improve cache refresh error handling for offline mirror nodes #2285

@jbair06

Description

@jbair06

Description:
Cache refreshing doesn't handle offline/unavailable mirror nodes gracefully. When a mirror node goes down, the current error handling blocks processing and produces verbose/less-useful logs. The system should treat mirror node failures as non-blocking, log them succinctly, and potentially implement backoff per offending network.

Current behavior:

Mirror node X offline → Cache refresh blocks → Processing stalls → Verbose error spam in logs

Expected behavior:

Mirror node X offline → Non-blocking error → Succinct log entry → Continue processing other nodes
→ Optional: Backoff retry for network X specifically

Scope:

  • Make mirror node failures non-blocking (continue processing other nodes)
  • Improve error logging to be more succinct/actionable
  • Track failures per network/mirror node for backoff logic
  • Ensure cache refresh failures don't halt transaction processing

Acceptance criteria:

  • Mirror node timeout/offline doesn't block cache refresh for other nodes
  • Cache refresh failures are logged succinctly (network + node + error type)
  • Processing continues immediately despite individual node failures
  • Per-network backoff tracking prevents log spam from repeated failures
  • Notification/messaging processing is never blocked by cache refresh errors

Nice to have:

  • Circuit breaker per mirror node/network
  • Metrics for mirror node health/failure rates

Metadata

Metadata

Assignees

No one assigned

    Labels

    BackendBugA error that causes the feature to behave differently than what was expected based on design docs

    Type

    Projects

    Status

    🏃‍♀ Sprint backlog

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions