rule: auto-reload --rule-file on filesystem changes

[yosshi825]

Fix: #8794

• I added CHANGELOG entry for this change.
• Change is not relevant to the end user.

Changes

thanos rule now auto-reloads rule files on filesystem changes when configured with a glob (e.g. --rule-file=rules/*.yaml), removing the need for configmap-reload sidecars or manual SIGHUP / POST /-/reload after a ConfigMap update. Mirrors thanos receive's --receive.hashrings-file and Prometheus' --enable-feature=auto-reload-config.

• New FileWatcher in pkg/rules/watcher.go: combines fsnotify for prompt detection with a 5-minute periodic refresh as a safety net, and a SHA-256 content hash to suppress spurious reloads from mtime-only changes. Structurally it mirrors pkg/receive/ConfigWatcher, but is intentionally not the same primitive — the two watchers sit at different responsibility layers:

  Responsibility	ConfigWatcher (receive)	FileWatcher (rule)
  File-change detection (fsnotify + tick + hash gate)	✅	✅
  File read	✅ (loadConfig → os.ReadFile)	✅ (for hashing only)
  Payload parsing	✅ json.Unmarshal → []HashringConfig	❌ — delegated to reloadRules()
  Per-payload metrics	✅ (hashring_nodes, hashring_tenants)	❌ — delegated to reloadRules()
  Pre-startup validation	✅ (ValidateConfig() rejects startup on error)	❌ — delegated to reloadRules()
  Channel payload	chan []HashringConfig (typed object)	chan struct{} (signal only)
  Watch granularity	single file (watcher.Add(path))	parent dirs of multiple globs
  Consumer's job	apply parsed payload (webHandler.Hashring(h) / dbs.SetHashringConfig(c))	call reloadRules()
  Layer	data-source layer	trigger layer

  Putting parsing, validation, and per-payload metrics in the rule watcher would duplicate the existing reloadRules() path that SIGHUP and POST /-/reload already drive — that path already handles file globbing, YAML + partial_response_strategy parsing, multi-file error aggregation, and metric updates. This is also what GiedriusS suggested in the issue discussion: "just call reloadRules() and that's all?". Sharing a common file-watching primitive would require first extracting receive's parser layer from ConfigWatcher; that refactor is out of scope here and is left as a follow-up. The watcher type doc records this rationale.

• Watches parent directories rather than individual files, so Kubernetes ConfigMap/Secret mounts (which atomically swap a ..data symlink) and glob patterns matching newly-added files are handled correctly.

• Wires the watcher into cmd/thanos/rule.go: the existing reload goroutine gets a third select case (<-fileWatcher.C()) alongside the existing SIGHUP and POST /-/reload paths. Both manual triggers continue to work unchanged. The case uses _, ok := so a closed channel during shutdown is handled cleanly, mirroring pkg/receive/ConfigFromWatcher.

• No new flags: per the issue discussion, --rule-file becomes auto-reloading without any opt-in.

• Help text and docs/components/rule.md updated to reflect the new behavior.

New metrics

• thanos_rule_config_files_changes_total — number of times the watcher detected a real (hash-differing) change.
• thanos_rule_config_files_errors_total — number of fsnotify watcher errors emitted while watching rule files. (Configuration-time errors such as malformed --rule-file glob patterns are reported as warning logs and are not counted by this metric.)

The existing thanos_rule_config_last_reload_successful and thanos_rule_config_last_reload_success_timestamp_seconds are updated automatically because the watcher invokes the same reloadRules() path used by SIGHUP and POST /-/reload.

Verification

• Unit tests: 6 new tests in pkg/rules/watcher_test.go covering content change, hash-gate suppression, glob-pattern new file, atomic rename (ConfigMap pattern), stop-closes-channel, and file deletion. All 60 invocations (6 tests × 10 iterations) pass under -race in ~12 s (go test -race -count=10 ./pkg/rules -run TestFileWatcher); the run is reproducible from a clean branch checkout with no external dependencies.
• Local end-to-end: ran the binary against a temp directory; verified reload on content change, atomic rename, and hash-gate (no reload on identical-content rewrite).
• Kubernetes (kind v0.31.0, K8s 1.35.0):
  • Single ConfigMap: edited via kubectl apply, observed reload triggered by kubelet's ..data symlink swap (~70 s after apply, sub-second from swap to reload). thanos_rule_loaded_rules reflected the new content; no SIGHUP/POST was needed.
  • Multiple ConfigMaps mounted at distinct paths: each parent directory is watched independently. Updating ConfigMap A reloaded A's rules and preserved B's; subsequently updating B had the inverse effect. thanos_rule_loaded_rules correctly reflects the union, and thanos_rule_config_files_changes_total increments exactly once per detected ConfigMap update.

kind cluster config and Kubernetes manifest used in this verification

kind cluster config

A single-node default cluster is sufficient. The verification used kind create cluster --name thanos-autoreload (no --config file). For explicit reproducibility:

# kind-config.yaml — equivalent to the default
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane

Kubernetes manifest used in the kind test

apiVersion: v1
kind: Namespace
metadata:
  name: thanos-test
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: rules-alerts
  namespace: thanos-test
data:
  alerts.yaml: |
    groups:
    - name: alerts-group
      rules:
      - alert: AlertA1
        expr: vector(1)
        for: 0s
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: rules-recording
  namespace: thanos-test
data:
  recording.yaml: |
    groups:
    - name: recording-group
      rules:
      - record: rec_r1
        expr: vector(1)
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: thanos-rule
  namespace: thanos-test
spec:
  replicas: 1
  selector:
    matchLabels:
      app: thanos-rule
  template:
    metadata:
      labels:
        app: thanos-rule
    spec:
      containers:
      - name: thanos-rule
        image: thanos:autoreload-test
        args:
        - rule
        - --rule-file=/etc/thanos/alerts/*.yaml
        - --rule-file=/etc/thanos/recording/*.yaml
        - --data-dir=/data
        - --query=localhost:9090
        - --label=replica="r0"
        # Bumped from default 1m to avoid eval-error log spam: the --query
        # endpoint above is a dummy, so every evaluation would log a failure.
        # The auto-reload behavior under test is independent of this interval.
        - --eval-interval=1h
        ports:
        - name: http
          containerPort: 10902
        volumeMounts:
        - name: rules-alerts
          mountPath: /etc/thanos/alerts
        - name: rules-recording
          mountPath: /etc/thanos/recording
        - name: data
          mountPath: /data
        readinessProbe:
          httpGet:
            path: /-/ready
            port: http
      volumes:
      - name: rules-alerts
        configMap:
          name: rules-alerts
      - name: rules-recording
        configMap:
          name: rules-recording
      - name: data
        emptyDir: {}

For the multi-ConfigMap test shown above, ConfigMap A was edited to 3 alerts (preserving B), then ConfigMap B was edited to 5 recording rules (preserving the updated A) to verify independent reload. The single-ConfigMap test (reported separately in the Verification bullets) used the same template with one ConfigMap and a single --rule-file=... arg.

[GiedriusS]

I would use a bit different language because right now this implies that we will ever remove those two:

Suggested change

	cmd.Flag("rule-file", "Rule files that should be used by rule manager. Can be in glob format (repeated). Changes are detected automatically via filesystem notifications; SIGHUP and HTTP POST /-/reload remain supported as manual triggers.").
	cmd.Flag("rule-file", "Rule files that should be used by rule manager. Can be in glob format (repeated). Changes are detected automatically via filesystem notifications; SIGHUP and HTTP POST /-/reload can be used to manually trigger.").

[yosshi825]

Thanks for the suggestion.
fix: 4be41e1

[GiedriusS]

I think we need to do this now because with this change we will have two reloaders. Not good in the long-term.

[yosshi825]

Agreed that two reloaders is the wrong long-term shape. I went ahead and built it out as three commits on this branch — happy to revert / redesign if any of the trade-offs below land badly.

Approach

Extracted the file-change-detection layer into a new package pkg/runutil/filewatch shared by both receive and rule.

• pkg/runutil/filewatch (7e8d96c): the primitive owns fsnotify wiring + safe Stop drain, the periodic safety-net tick, parent-directory watching, and the SHA-256 content-hash gate. It deliberately does not parse content or expose payload-specific metrics — counters are injected via Options. ChangesCounter is optional (nil-allowed) for cases where a wrapper layer already owns the "successful reload" semantics of a pre-existing metric (see receive below); the primitive then falls back to an internal unregistered counter so the hot path never has to nil-check.
• receive (b80472b): ConfigWatcher's public API (NewConfigWatcher, Run, C, ValidateConfig, Stop, ConfigFromWatcher) is unchanged. Parsing, hashring-specific metrics, and the typed channel stay in the wrapper. The wrapper passes nil for ChangesCounter so the pre-existing "successful reload" semantic of thanos_receive_hashrings_file_changes_total is preserved exactly (incremented only when loadConfig succeeds and the hash differs). ErrorsCounter is shared so fsnotify, read, and parse errors all land on thanos_receive_hashrings_file_errors_total. Stop is now guarded by sync.Once.
• rule (fdd0cba): cmd/thanos/rule.go constructs filewatch.Watcher inline with the same metric names as before. pkg/rules/watcher.go and its tests are removed — the equivalent coverage lives in the primitive's test suite.

Behavior changes worth flagging

1. thanos_receive_hashrings_file_refreshes_total becomes silent in steady state. The primitive's hash gate filters out no-op periodic ticks, so the counter only increments on actual content changes (and the initial load). This matches the metric's name more accurately than the previous "every tick increments" behavior, but operators using it as a heartbeat will need to migrate to up{}. Help text and CHANGELOG (under Changed) updated. Happy to add a TicksCounter to the primitive if you'd prefer to preserve the previous semantic verbatim.
2. receive switches from file-inode watching to parent-directory watching (matching rule). More robust for atomic-rename / ConfigMap symlink swaps; the only edge case I can think of is a hand-managed symlink whose target is edited without touching the symlink itself, which the 5-minute safety-net tick still catches.
3. NewConfigWatcher loses its watcher.Add(path) fail-fast (the primitive's directory watching is lazy). The fail-fast is preserved in practice because cmd/thanos/receive.go immediately calls cw.ValidateConfig(), which reads the file and surfaces missing-file errors at the same point in startup.
4. receive _errors_total help text broadened to "errors watching, reading, or parsing the hashrings configuration file" — same set of conditions are counted as before, just made explicit.

Verification

• 10 race tests for the primitive (pkg/runutil/filewatch), including ConfigMap atomic-rename, glob-pattern new file, unreadable-file (chmod 000) skips the reload, and nil-ChangesCounter does not panic.
• 5 race tests for receive (TestValidateConfig, TestUnmarshalEndpointSlice, plus three new: TestConfigWatcher_EmitsOnInitialAndChange, TestConfigWatcher_ChangesCounterNotDoubleIncremented, TestConfigWatcher_StopIsIdempotent).
• thanos rule exercised against a real binary: --rule-file='dir/*.yaml' → atomic rename → thanos_rule_loaded_rules and _changes_total move as expected; identical-content rewrite stays silent (hash gate); kill shuts down cleanly.

If any of the trade-offs above are problematic, the easiest dial-backs are (a) add a TicksCounter to restore _refreshes_total's old semantic, or (b) split the receive migration into a follow-up PR for review isolation.

[GiedriusS]

This resolves files then maybeNotify() also calls the same. There could be some more changes in between these calls. I would suggest calling w.resolveFiles() once and passing it as a param. Same with computeHash().

[yosshi825]

Thank you for comment.
fix:204f5fc

resolveFiles() now runs once per tick and the result is threaded through refreshWatchedDirs, maybeNotify, and computeHash so they share one snapshot.

[GiedriusS]

I think it's better to return this error here instead of silently failing. I think the users expect the reloading NOT to happen with partial data if some files are unreadable.

[yosshi825]

Thanks for catching this.

fix:1d1278a

computeHash now returns an error and maybeNotify skips the signal while preserving lastHash, so a real change during an outage is still detected after recovery. Constructor seeding only warns on read errors (one possibly spurious reload on first success is benign — reloadRules is idempotent). Added a chmod 0o000 test.

[GiedriusS]

Could we please generalize this watcher and use it in pkg/receive/config.go? 🙏

[yosshi825]

@GiedriusS — sorry, I missed your follow-up earlier; I'd been working on the refactor in response to your original comment at #discussion_r3225480791 and pushed the result earlier today.

The latest four commits extract the file-change-detection layer into pkg/runutil/filewatch and have both pkg/receive/ConfigWatcher and cmd/thanos/rule.go use it (pkg/rules/watcher.go is removed). I posted a detailed design walkthrough on that same thread, including the trade-offs and a few small behavior changes worth flagging.

Each of the four commits is self-contained, so they can be reviewed independently. Happy to split the receive migration into a follow-up PR if narrower scope would be easier. Thanks in advance for taking another look!

[yosshi825]

• Thanos unit tests Faild.

The Thanos unit tests failure on b32164d is a flaky goroutine leak in pkg/receive/handler.go (fanoutForward) detected by goleak.VerifyTestMain, unrelated to this PR.

[Goroutine in state chan receive, 2 minutes]
pkg/receive.(*Handler).fanoutForward.func3.1
    pkg/receive/handler.go:827
[Goroutine in state sync.WaitGroup.Wait, 2 minutes]
pkg/receive.(*Handler).fanoutForward.func2
    pkg/receive/handler.go:819

This code was introduced by #8803 (canReturnEarly change) merged into upstream main, and pulled in here via the recent merge from upstream/main. Evidence this is not caused by this PR:

• The previous commit on this branch (e6f5d63, before the upstream merge) passed Thanos unit tests.
• The same code on upstream/main passes its own Thanos unit tests CI.
• All files touched by this PR (pkg/runutil/filewatch, pkg/receive/config.go, cmd/thanos/rule.go, …) pass unit tests locally; pkg/receive also passes when re-run.