CPU and Memory surge in Trickster Pod with Frequent Auto-Refresh Queries

[harshadparchandtheagilehub]

Hello team,
We are observing abnormal resource utilization behavior on our Trickster pods. The following details summarize the issue:

Version
Trickster v2.0.0-beta3
Commit: 2025/12/08 - 46c4ca5

Issue Description
We have a dashboard configured to auto-refresh every second. Each refresh request is routed through the Trickster ALB, which routes the query to our Mimir backends (~50) and receives approximately 108 series and ~70 kB of response data.
And we are using Redis for caching.

However, under this workload, we observe a significant and continuous increase in CPU and memory consumption within the Trickster pod. Specifically, the following metrics show sharp upward trends over time:

• go_memstats_heap_sys_bytes
• go_memstats_alloc_bytes
• go_goroutines

Resource usage continues to climb until the underlying Kubernetes node approaches CPU and memory saturation, ultimately causing the Trickster pod to crash or be evicted.

Impact

• Severe CPU and memory spikes on Trickster pods
• Progressive resource escalation leading to node-level saturation
• Pod crash / OOM kill

Evidence

[crandles]

Can you share more about your particular usage?

• Config example(s)
• Deployment details:
  • it looks like you're running many pods: how many? how many nodes are these spread across?
  • can you share your CPU / memory specs?
• Dashboard details:
  • how many panels?
  • what is the timeframe?
  • how many data points per panel?

[harshadparchandtheagilehub]

1. Below is how the config looks for us

backends:
  backend_1:
    healthcheck:
      expected_codes:
        - 200
        - 401
      failure_threshold: 3
      headers:
        Authorization: Basic password
      host: somehost.com.:12345
      interval: 10000ms
      path: /prometheus/api/v1/status/buildinfo
      recovery_threshold: 3
      scheme: https
      timeout: 10000ms
      verb: GET
    origin_url: https://prometheusURL:12345/prometheus
    prometheus:
      labels:
        region: backend_1
    provider: prometheus
    req_rewriter_name: backend_1_authentication

  ... 50 such backends


  trickster-alb:
    alb:
      healthy_floor: -1
      mechanism: tsm
      pool:
        - backend_1
        - backend_2
        ....
        - backend_50
    provider: alb

caches:
  default:
    provider: redis
    redis:
      client_type: cluster
      endpoints:
        - endpoints_1.com:6379
        - endpoints_2.com:6379
        - endpoints_3.com:6379
        - endpoints_4.com:6379
        - endpoints_5.com:6379
        - endpoints_6.com:6379
        - endpoints_7.com:6379
        - endpoints_8.com:6379
        - endpoints_9.com:6379
      protocol: tcp
      use_tls: true

frontend:
  listen_port: 8480

logging:
  log_level: info

metrics:
  listen_port: 8481

mgmt:
  config_handler_path: /trickster/config
  health_handler_path: /trickster/health
  listen_address: ""
  listen_port: 8484
  ping_handler_path: /trickster/ping
  reload_handler_path: /trickster/config/reload

request_rewriters:
  backend_1_authentication:
    instructions:
      - - header
        - set
        - Authorization
        - Basic password

  ... 50 such backends authentications

2. Deployment details:

• We are running 9 pods across 9 nodes, with one pod per node. The underlying node pool has a total capacity of 4 CPUs and 32 GB of memory, distributed across these nodes
• Each Trickster pod is configured with 100 mCPU and 128 MiB of memory to start with

3. Dashboard details :

• The dashboard has just one panel
• Time frame is last one hour
• Approximately 26,000 data points

[agoethals]

We are seeing the same issue with below config. Looks like the go_memstats_heap_sys_bytes stays very high, even when the pod is restarted by the OOM killer of Kubernetes. Our memory limit is already increased to 3 GiB but still running into OOM...

    frontend:
      listen_port: 9090

    caches:
      default:
        provider: 'filesystem'
        index:
          reap_interval: 3s
          flush_interval: 5s
          max_size_bytes: 1879048192
          max_size_backoff_bytes: 268435456
          max_size_objects: 0
          max_size_backoff_objects: 100
        filesystem:
          cache_path: '/trickster/cache'

    backends:
      default:
        provider: 'prometheus'
        origin_url: 'http://lmm-prometheus:9090'
        cache_name: 'default'
        timeout: 120s
        keep_alive_timeout: 300s
        max_idle_conns: 40
        timeseries_retention_factor: 2048
        timeseries_ttl: 21600s
        timeseries_eviction_method: 'oldest'
        fast_forward_disable: true
        fastforward_ttl: 15s

    metrics:
      listen_port: 8082