tools bucket web: ConcurrentLister still panics with "send on closed channel" after object-store cancellation in v0.42.4

[Amet13]

Thanos, Prometheus and Golang version used:

thanos, version 0.42.4 (branch: HEAD, revision: 45c4f39487e1653e606307d3952a6259fb96acae)
  build user:       root@cbb5ace98a03
  build date:       20260730-13:18:49
  go version:       go1.26.5
  platform:         linux/amd64
  tags:             netgo,slicelabels

Container image digest: sha256:9c84f99cb0ef16ee07e8d1132740b5c092618cf0d6e68d9794cf29996a0e2321

Prometheus version is not applicable. The affected component is thanos tools bucket web.

Object Storage Provider:

Google Cloud Storage. The bucket is regional in europe-west1. Authentication uses GKE Workload Identity.

What happened:

Bucket Web intermittently exits with panic: send on closed channel during metadata synchronization. Kubernetes then restarts the container with exit code 2.

One deployment recorded 329 restarts in approximately four days. Availability was approximately 85% during one six-hour observation window.

A second independent GKE cluster reproduced the same panic with the same image digest. That pod completed hundreds of successful refreshes before its first panic.

The failure is conditional. Another pod with the same v0.42.4 image ran for almost six days with zero restarts and zero object-store failures.

What you expected to happen:

Bucket Web must return and log the object-store error. It must retry during the next tick without terminating the process.

GetActiveAndPartialBlockIDs must not return while its internal workers can still send to the output channel.

How to reproduce it (as minimally and precisely as possible):

We do not yet have a deterministic standalone reproducer. Production reproduces the issue with these flags:

tools bucket web
--refresh=10m
--timeout=5m
--log.level=info
--log.format=json

A focused test could use an instrumented bucket with more than 64 block prefixes:

1. Let one Exists(<ULID>/meta.json) call return an error.
2. Keep another worker ready to send a successful block result.
3. Let the error cancel the errgroup context.
4. Let the top-level Iter return context canceled.
5. Verify that all workers stop before the output channel closes.

Full logs to relevant components:

The following trace came from the second independent cluster after many successful refreshes:

2026-08-28T12:11:17.111828179Z {"cached":272,"caller":"fetcher.go:697","component":"block.BaseFetcher","duration":"21.589784542s","duration_ms":21589,"level":"info","msg":"successfully synchronized block metadata","partial":0,"returned":229}
2026-08-28T12:21:08.716756941Z {"caller":"runutil.go:117","err":"BaseFetcher: iter bucket: context canceled","level":"error","msg":"function failed. Retrying in next tick"}
2026-08-28T12:21:10.425440322Z panic: send on closed channel

goroutine 1282793 [running]:
github.com/thanos-io/thanos/pkg/block.(*ConcurrentLister).GetActiveAndPartialBlockIDs.func1()
    /app/pkg/block/fetcher.go:294 +0x274
golang.org/x/sync/errgroup.(*Group).Go.func1()
    /go/pkg/mod/golang.org/x/sync@v0.21.0/errgroup/errgroup.go:93 +0x50
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 1282767
    /go/pkg/mod/golang.org/x/sync@v0.21.0/errgroup/errgroup.go:78 +0x95

The first cluster produced the same stack at /app/pkg/block/fetcher.go:294.

Anything else we need to know:

The ConcurrentLister error path appears to return before its internal workers finish:

1. An Exists worker returns an error and cancels gCtx.
2. The top-level Iter callback returns gCtx.Err().
3. GetActiveAndPartialBlockIDs returns before close(metaChan) and eg.Wait().
4. BaseFetcher.fetchMetadata closes activeBlocksCh.
5. A remaining worker sends to activeBlocksCh and panics.

Current source: https://github.com/thanos-io/thanos/blob/v0.42.4/pkg/block/fetcher.go#L266-L326

Supporting observations:

• Previous healthy v0.42.4 pod: average Exists duration 2.62s; average metadata sync 14.5s.
• Current affected pod: average Exists duration 10.3s; average metadata sync 99s.
• Store Gateway on the same node: average Exists duration 0.16s; average Get duration 0.41s; zero failures.
• Pod network counters show zero receive or transmit errors.
• Workload Identity token requests return HTTP 200 in 1–3ms.
• Memory remained below the limit. Kubernetes reported no OOM termination.
• A previous v0.39.2 pod recorded 217 Exists failures and 212 Iter failures with zero restarts.
• v0.42.4 release notes state that it contains no code changes from v0.42.3.

This appears to be a recurrence of the closed issue:

• thanos compact panic on GetActiveAndPartialBlockIDs #7491
• Fix GetActiveAndPartialBlockIDs panic #7621

Related open and merged work:

• Compactor: Deadlock on S3 error during meta sync #7514
• #7514 Fix channel deadlock in meta sync fetcher #7933
• New test case catching the ConcurrentLister channel deadlock #7966
• Fix deadlock in metadata fetcher #8092

Environment:

• Kubernetes: GKE v1.34.9-gke.1322001
• OS: Container-Optimized OS
• Kernel: 6.12.85+
• Runtime: containerd://2.1.7
• Resources: request 20m CPU / 50Mi; limit 40m CPU / 100Mi

[Amet13]

Can't reproduce it anymore, closing as stale.