metrics: add slow store scheduler status graph to Grafana dashboard (#9941)

[ti-chi-bot]

This is an automated cherry-pick of #9941

What problem does this PR solve?

Issue Number: Close #9866

What is changed and how does it work?

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)

• No code

Code changes

• Has the configuration change
• Has HTTP API interfaces changed (Don't forget to add the declarative for the new API)
• Has persistent data change

Side effects

• Possible performance regression
• Increased code complexity
• Breaking backward compatibility

Related changes

• PR to update pingcap/docs/pingcap/docs-cn:
• PR to update pingcap/tiup:
• Need to cherry-pick to the release branch

Release note

None.

Summary by CodeRabbit

• New Features
  • Added a new "Slow Store Scheduler" monitoring panel to the PD dashboard, providing enhanced visibility into slow store scheduling behavior and status.

_{✏️ Tip: You can customize this high-level summary in your review settings.}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 78.57%. Comparing base (6521383) to head (dfb8567).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #9980      +/-   ##
==========================================
+ Coverage   78.54%   78.57%   +0.03%     
==========================================
  Files         520      520              
  Lines       69676    69676              
==========================================
+ Hits        54724    54750      +26     
+ Misses      10994    10980      -14     
+ Partials     3958     3946      -12     

Flag	Coverage Δ
unittests	78.57% <ø> (+0.03%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[okJiang]

/retest

[coderabbitai]

📝 Walkthrough

Walkthrough

A new Grafana dashboard panel was added to monitor slow store scheduler metrics. The panel displays status information using a PromQL query targeting the pd_scheduler_evicted_slow_store_status metric, configured within the existing Scheduler row on the PD dashboard.

Changes

Cohort / File(s)	Summary
Grafana Dashboard Panel metrics/grafana/pd.json	Added "Slow Store Scheduler" graph panel (id 114) with PromQL query for pd_scheduler_evicted_slow_store_status metric visualization, including legend formatting and graph renderer configuration

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~5 minutes

Poem

🐰 A scheduler once slow in its stores,
Now shines on the dashboard with graphs!
With metrics so bright,
And visibility's light,
We'll watch all those queues and their warps! 📊

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly describes the main change: adding a slow store scheduler status graph to the Grafana dashboard for the v7.5.6 release branch.
Description check	✅ Passed	The description includes required sections: issue number (Close #9866), what is changed, checklist with manual test evidence, and release note. However, the commit-message block is empty.
Linked Issues check	✅ Passed	The PR successfully addresses issue #9866 by cherry-picking the slow store scheduler metrics feature to v7.5.6 release branch, implementing the required enhancement.
Out of Scope Changes check	✅ Passed	All changes are in-scope: only the metrics/grafana/pd.json file was modified to add the slow store scheduler status graph as required.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 0

🧹 Nitpick comments (1)

metrics/grafana/pd.json (1)

7913-8004: Add store filter to align with dashboard patterns and avoid high-cardinality graphs.

The PromQL query omits the $store variable filter used consistently across 30+ other scheduler panels in this dashboard. This creates unnecessary cardinality on large clusters and deviates from established conventions. The metric pd_scheduler_evicted_slow_store_status exports store and slow_type labels, so the filter is supported.

Proposed fix

-              "expr": "pd_scheduler_evicted_slow_store_status{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\"}",
+              "expr": "pd_scheduler_evicted_slow_store_status{k8s_cluster=\"$k8s_cluster\", tidb_cluster=\"$tidb_cluster\", store=~\"$store\"}",

📜 Review details

Configuration used: defaults

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 6521383 and dfb8567.

📒 Files selected for processing (1)

• metrics/grafana/pd.json

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (13)

• GitHub Check: chunks
• GitHub Check: chunks (6, Tools Test)
• GitHub Check: chunks (7, Client Integration Test)
• GitHub Check: statics
• GitHub Check: chunks (10, Microservice Integration(TSO))
• GitHub Check: chunks (9, Microservice Integration(!TSO))
• GitHub Check: chunks (8, TSO Integration Test)
• GitHub Check: chunks (3, Unit Test(3))
• GitHub Check: chunks (4, Tests(1))
• GitHub Check: chunks (1, Unit Test(1))
• GitHub Check: chunks (5, Tests(2))
• GitHub Check: chunks (2, Unit Test(2))
• GitHub Check: tso-function-test

[okJiang]

/retest

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: okJiang, rleungx

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [rleungx]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ti-chi-bot]

[LGTM Timeline notifier]

Timeline:

• 2026-01-14 02:57:15.011713745 +0000 UTC m=+412679.073578654: ☑️ agreed by okJiang.
• 2026-01-14 03:00:55.765905207 +0000 UTC m=+412899.827770113: ☑️ agreed by rleungx.