From 7adcc703a6d1a0656396759d49b6e14724c2975a Mon Sep 17 00:00:00 2001 From: lhy1024 <19542290+lhy1024@users.noreply.github.com> Date: Wed, 24 Jun 2026 23:52:30 +0800 Subject: [PATCH 01/15] pd: document CPU-aware hot region scheduling --- grafana-pd-dashboard.md | 1 + pd-control.md | 27 +++++++++++++++++++++------ troubleshoot-hot-spot-issues.md | 4 ++++ 3 files changed, 26 insertions(+), 6 deletions(-) diff --git a/grafana-pd-dashboard.md b/grafana-pd-dashboard.md index c1e6029a3f466..30ebd3c33f2e2 100644 --- a/grafana-pd-dashboard.md +++ b/grafana-pd-dashboard.md @@ -88,6 +88,7 @@ The following is the description of PD Dashboard metrics items: - Total read bytes on hot peer Regions: The total read bytes of peers that have become read hotspots on each TiKV instance - Store read rate bytes: The total read bytes of each TiKV instance - Store read rate keys: The total read keys of each TiKV instance +- Store read cpu: The read CPU usage of each TiKV instance, which PD uses for CPU-aware read hotspot scheduling starting from v8.5.7 - Hot cache read entry number: The number of peers that are in the read hotspot statistics module on each TiKV instance ![PD Dashboard - Hot read metrics](/media/pd-dashboard-hotread-v4.png) diff --git a/pd-control.md b/pd-control.md index 1a0cb18fa60f8..01202fc272c66 100644 --- a/pd-control.md +++ b/pd-control.md @@ -514,6 +514,7 @@ Usage: "hot_region_type": "read", "hot_degree": 152, "flow_bytes": 0, + "flow_cpu": 32, "key_rate": 0, "query_rate": 305, "start_key": "7480000000000000FF5300000000000000F8", @@ -535,6 +536,7 @@ Usage: "hot_region_type": "read", "hot_degree": 152, "flow_bytes": 0, + "flow_cpu": 32, "key_rate": 0, "query_rate": 305, "start_key": "7480000000000000FF5300000000000000F8", @@ -545,6 +547,8 @@ Usage: } ``` +Starting from v8.5.7, `hot read` and `hot history` include `flow_cpu`, and `hot store` includes `cpu-read-rate`. These fields show read CPU usage for CPU-aware read hotspot scheduling. + ### `label [store ]` Use this command to view the label information of the cluster. @@ -1024,22 +1028,24 @@ Usage: "min-hot-byte-rate": 100, "min-hot-key-rate": 10, "min-hot-query-rate": 10, + "min-hot-cpu-rate": 10, "max-zombie-rounds": 3, "max-peer-number": 1000, "byte-rate-rank-step-ratio": 0.05, "key-rate-rank-step-ratio": 0.05, "query-rate-rank-step-ratio": 0.05, + "cpu-rate-rank-step-ratio": 0.05, "count-rank-step-ratio": 0.01, "great-dec-ratio": 0.95, "minor-dec-ratio": 0.99, "src-tolerance-ratio": 1.05, "dst-tolerance-ratio": 1.05, "read-priorities": [ - "query", + "cpu", "byte" ], "write-leader-priorities": [ - "key", + "query", "byte" ], "write-peer-priorities": [ @@ -1070,6 +1076,12 @@ Usage: scheduler config balance-hot-region-scheduler set min-hot-query-rate 10 ``` +- `min-hot-cpu-rate` means the smallest CPU usage of read requests to be counted, which is usually 10. + + ```bash + scheduler config balance-hot-region-scheduler set min-hot-cpu-rate 10 + ``` + - `max-zombie-rounds` means the maximum number of heartbeats with which an operator can be considered as the pending influence. If you set it to a larger value, more operators might be included in the pending influence. Usually, you do not need to adjust its value. Pending influence refers to the operator influence that is generated during scheduling but still has an effect. ```bash @@ -1082,7 +1094,7 @@ Usage: scheduler config balance-hot-region-scheduler set max-peer-number 1000 ``` -- `byte-rate-rank-step-ratio`, `key-rate-rank-step-ratio`, `query-rate-rank-step-ratio`, and `count-rank-step-ratio` respectively mean the step ranks of byte, key, query, and count. The rank-step-ratio decides the step when the rank is calculated. `great-dec-ratio` and `minor-dec-ratio` are used to determine the `dec` rank. Usually, you do not need to modify these items. +- `byte-rate-rank-step-ratio`, `key-rate-rank-step-ratio`, `query-rate-rank-step-ratio`, `cpu-rate-rank-step-ratio`, and `count-rank-step-ratio` respectively mean the step ranks of byte, key, query, CPU, and count. The rank-step-ratio decides the step when the rank is calculated. `great-dec-ratio` and `minor-dec-ratio` are used to determine the `dec` rank. Usually, you do not need to modify these items. ```bash scheduler config balance-hot-region-scheduler set byte-rate-rank-step-ratio 0.05 @@ -1096,15 +1108,18 @@ Usage: - `read-priorities`, `write-leader-priorities`, and `write-peer-priorities` control which dimension the scheduler prioritizes for hot Region scheduling. Two dimensions are supported for configuration. - - `read-priorities` and `write-leader-priorities` control which dimensions the scheduler prioritizes for scheduling hot Regions of the read and write-leader types. The dimension options are `query`, `byte`, and `key`. + - `read-priorities` controls which dimensions the scheduler prioritizes for scheduling hot Regions of the read type. The dimension options are `cpu`, `query`, `byte`, and `key`. + - `write-leader-priorities` controls which dimensions the scheduler prioritizes for scheduling hot Regions of the write-leader type. The dimension options are `query`, `byte`, and `key`. - `write-peer-priorities` controls which dimensions the scheduler prioritizes for scheduling hot Regions of the write-peer type. The dimension options are `byte` and `key`. > **Note:** > - > If a cluster component is earlier than v5.2, the configuration of `query` dimension does not take effect. If some components are upgraded to v5.2 or later, the `byte` and `key` dimensions still by default have the priority for hot Region scheduling. After all components of the cluster are upgraded to v5.2 or later, such a configuration still takes effect for compatibility. You can view the real-time configuration using the `pd-ctl` command. Usually, you do not need to modify these configurations. + > If a cluster component is earlier than v5.2, the configuration of the `query` dimension does not take effect. If some components are upgraded to v5.2 or later, the `byte` and `key` dimensions still by default have the priority for hot Region scheduling. After all components of the cluster are upgraded to v5.2 or later, such a configuration still takes effect for compatibility. + > + > Starting from v8.5.7, TiKV reports read CPU usage for hot Region scheduling. For clusters that support read CPU reporting, the default `read-priorities` value is `cpu,byte`. For clusters that do not support read CPU reporting, PD automatically falls back to `query,byte`, or to `byte,key` if the `query` dimension is also unsupported. You can view the real-time configuration using the `pd-ctl` command. Usually, you do not need to modify these configurations. ```bash - scheduler config balance-hot-region-scheduler set read-priorities query,byte + scheduler config balance-hot-region-scheduler set read-priorities cpu,byte ``` - `strict-picking-store` controls the search space of hot Region scheduling. Usually, it is enabled. This configuration item only affects the behavior when `rank-formula-version` is `v1`. When it is enabled, hot Region scheduling ensures hot Region balance on the two configured dimensions. When it is disabled, hot Region scheduling only ensures the balance on the dimension with the first priority, which might reduce balance on other dimensions. Usually, you do not need to modify this configuration. diff --git a/troubleshoot-hot-spot-issues.md b/troubleshoot-hot-spot-issues.md index 5f1cbd369f40f..716e0651db27d 100644 --- a/troubleshoot-hot-spot-issues.md +++ b/troubleshoot-hot-spot-issues.md @@ -185,6 +185,10 @@ For more details, see [Coprocessor Cache](/coprocessor-cache.md). In a read hotspot scenario, the hotspot TiKV node cannot process read requests in time, resulting in the read requests queuing. However, not all TiKV resources are exhausted at this time. To reduce latency, TiDB v7.1.0 introduces the load-based replica read feature, which allows TiDB to read data from other TiKV nodes without queuing on the hotspot TiKV node. You can control the queue length of read requests using the [`tidb_load_based_replica_read_threshold`](/system-variables.md#tidb_load_based_replica_read_threshold-new-in-v700) system variable. When the estimated queue time of the leader node exceeds this threshold, TiDB prioritizes reading data from follower nodes. This feature can improve read throughput by 70% to 200% in a read hotspot scenario compared to not scattering read hotspots. +Starting from v8.5.7, PD supports CPU-aware Hot Region scheduling for read hotspots. TiKV reports per-Region read CPU usage in store heartbeats, and PD can use CPU usage as a scheduling dimension. This helps PD identify read hotspots whose QPS or byte throughput looks balanced but whose TiKV CPU usage is still uneven, such as workloads that contain queries with different CPU costs or clusters that contain TiKV nodes with different performance profiles. + +For clusters whose TiKV version supports read CPU reporting, the default `read-priorities` value of `balance-hot-region-scheduler` is `cpu,byte`. For clusters that do not support read CPU reporting, PD automatically falls back to `query,byte`, or to `byte,key` if the `query` dimension is also unsupported. To view or adjust the scheduling dimensions, use [`pd-ctl scheduler config balance-hot-region-scheduler`](/pd-control.md#scheduler-config-balance-hot-region-scheduler). + ## Use TiKV MVCC in-memory engine to mitigate read hotspots caused by high MVCC read amplification When the retention time of historical MVCC data for GC is too long, or when the records are frequently updated or deleted, read hotspots might occur due to scanning a large number of MVCC versions. To alleviate this type of hotspot, you can enable the [TiKV MVCC In-Memory Engine](/tikv-in-memory-engine.md) feature. From 329b7497bca2a9166e820d6d61b545b6876fc56e Mon Sep 17 00:00:00 2001 From: lhy1024 <19542290+lhy1024@users.noreply.github.com> Date: Wed, 24 Jun 2026 23:57:31 +0800 Subject: [PATCH 02/15] pd: address CPU-aware scheduling doc comments --- grafana-pd-dashboard.md | 2 +- pd-control.md | 8 ++++---- troubleshoot-hot-spot-issues.md | 4 ++-- 3 files changed, 7 insertions(+), 7 deletions(-) diff --git a/grafana-pd-dashboard.md b/grafana-pd-dashboard.md index 30ebd3c33f2e2..4dc20d2b99739 100644 --- a/grafana-pd-dashboard.md +++ b/grafana-pd-dashboard.md @@ -88,7 +88,7 @@ The following is the description of PD Dashboard metrics items: - Total read bytes on hot peer Regions: The total read bytes of peers that have become read hotspots on each TiKV instance - Store read rate bytes: The total read bytes of each TiKV instance - Store read rate keys: The total read keys of each TiKV instance -- Store read cpu: The read CPU usage of each TiKV instance, which PD uses for CPU-aware read hotspot scheduling starting from v8.5.7 +- Store read cpu: The read CPU usage of each TiKV instance. PD uses this metric for CPU-aware read hotspot scheduling starting from v8.5.7. - Hot cache read entry number: The number of peers that are in the read hotspot statistics module on each TiKV instance ![PD Dashboard - Hot read metrics](/media/pd-dashboard-hotread-v4.png) diff --git a/pd-control.md b/pd-control.md index 01202fc272c66..2b4429bea438e 100644 --- a/pd-control.md +++ b/pd-control.md @@ -547,7 +547,7 @@ Usage: } ``` -Starting from v8.5.7, `hot read` and `hot history` include `flow_cpu`, and `hot store` includes `cpu-read-rate`. These fields show read CPU usage for CPU-aware read hotspot scheduling. +Starting from v8.5.7, the `hot read` and `hot history` commands include `flow_cpu`, and the `hot store` command includes `cpu-read-rate`. These fields show the read CPU usage for CPU-aware read hotspot scheduling. ### `label [store ]` @@ -1076,7 +1076,7 @@ Usage: scheduler config balance-hot-region-scheduler set min-hot-query-rate 10 ``` -- `min-hot-cpu-rate` means the smallest CPU usage of read requests to be counted, which is usually 10. +- `min-hot-cpu-rate` specifies the minimum CPU usage of read requests to count, which is usually 10. ```bash scheduler config balance-hot-region-scheduler set min-hot-cpu-rate 10 @@ -1094,7 +1094,7 @@ Usage: scheduler config balance-hot-region-scheduler set max-peer-number 1000 ``` -- `byte-rate-rank-step-ratio`, `key-rate-rank-step-ratio`, `query-rate-rank-step-ratio`, `cpu-rate-rank-step-ratio`, and `count-rank-step-ratio` respectively mean the step ranks of byte, key, query, CPU, and count. The rank-step-ratio decides the step when the rank is calculated. `great-dec-ratio` and `minor-dec-ratio` are used to determine the `dec` rank. Usually, you do not need to modify these items. +- `byte-rate-rank-step-ratio`, `key-rate-rank-step-ratio`, `query-rate-rank-step-ratio`, `cpu-rate-rank-step-ratio`, and `count-rank-step-ratio` represent the step ranks of byte, key, query, CPU, and count, respectively. The rank-step-ratio decides the step when calculating the rank. PD uses `great-dec-ratio` and `minor-dec-ratio` to determine the `dec` rank. Usually, you do not need to modify these items. ```bash scheduler config balance-hot-region-scheduler set byte-rate-rank-step-ratio 0.05 @@ -1116,7 +1116,7 @@ Usage: > > If a cluster component is earlier than v5.2, the configuration of the `query` dimension does not take effect. If some components are upgraded to v5.2 or later, the `byte` and `key` dimensions still by default have the priority for hot Region scheduling. After all components of the cluster are upgraded to v5.2 or later, such a configuration still takes effect for compatibility. > - > Starting from v8.5.7, TiKV reports read CPU usage for hot Region scheduling. For clusters that support read CPU reporting, the default `read-priorities` value is `cpu,byte`. For clusters that do not support read CPU reporting, PD automatically falls back to `query,byte`, or to `byte,key` if the `query` dimension is also unsupported. You can view the real-time configuration using the `pd-ctl` command. Usually, you do not need to modify these configurations. + > Starting from v8.5.7, TiKV reports read CPU usage for hot Region scheduling. For clusters that support read CPU reporting, the default `read-priorities` value is `cpu,byte`. For clusters that do not support read CPU reporting, PD automatically falls back to `query,byte`, or to `byte,key` if the cluster does not support the `query` dimension either. You can view the real-time configuration using the `pd-ctl` command. Usually, you do not need to modify these configurations. ```bash scheduler config balance-hot-region-scheduler set read-priorities cpu,byte diff --git a/troubleshoot-hot-spot-issues.md b/troubleshoot-hot-spot-issues.md index 716e0651db27d..44f4325970106 100644 --- a/troubleshoot-hot-spot-issues.md +++ b/troubleshoot-hot-spot-issues.md @@ -185,9 +185,9 @@ For more details, see [Coprocessor Cache](/coprocessor-cache.md). In a read hotspot scenario, the hotspot TiKV node cannot process read requests in time, resulting in the read requests queuing. However, not all TiKV resources are exhausted at this time. To reduce latency, TiDB v7.1.0 introduces the load-based replica read feature, which allows TiDB to read data from other TiKV nodes without queuing on the hotspot TiKV node. You can control the queue length of read requests using the [`tidb_load_based_replica_read_threshold`](/system-variables.md#tidb_load_based_replica_read_threshold-new-in-v700) system variable. When the estimated queue time of the leader node exceeds this threshold, TiDB prioritizes reading data from follower nodes. This feature can improve read throughput by 70% to 200% in a read hotspot scenario compared to not scattering read hotspots. -Starting from v8.5.7, PD supports CPU-aware Hot Region scheduling for read hotspots. TiKV reports per-Region read CPU usage in store heartbeats, and PD can use CPU usage as a scheduling dimension. This helps PD identify read hotspots whose QPS or byte throughput looks balanced but whose TiKV CPU usage is still uneven, such as workloads that contain queries with different CPU costs or clusters that contain TiKV nodes with different performance profiles. +Starting from v8.5.7, PD supports CPU-aware Hot Region scheduling for read hotspots. TiKV reports per-Region read CPU usage in store heartbeats, and PD can use CPU usage as a scheduling dimension. This helps PD identify read hotspots whose QPS or byte throughput looks balanced but whose TiKV CPU usage is still uneven, such as workloads with different CPU-cost queries or clusters with different TiKV performance profiles. -For clusters whose TiKV version supports read CPU reporting, the default `read-priorities` value of `balance-hot-region-scheduler` is `cpu,byte`. For clusters that do not support read CPU reporting, PD automatically falls back to `query,byte`, or to `byte,key` if the `query` dimension is also unsupported. To view or adjust the scheduling dimensions, use [`pd-ctl scheduler config balance-hot-region-scheduler`](/pd-control.md#scheduler-config-balance-hot-region-scheduler). +For clusters that support read CPU reporting, the default `read-priorities` value of `balance-hot-region-scheduler` is `cpu,byte`. For clusters that do not support read CPU reporting, PD automatically falls back to `query,byte`, or to `byte,key` if the cluster does not support the `query` dimension either. To view or adjust the scheduling dimensions, use [`pd-ctl scheduler config balance-hot-region-scheduler`](/pd-control.md#scheduler-config-balance-hot-region-scheduler). ## Use TiKV MVCC in-memory engine to mitigate read hotspots caused by high MVCC read amplification From 1e6bde9f2ccfc14d95b3e1d0c7ed4ca722db755b Mon Sep 17 00:00:00 2001 From: Grace Cai Date: Thu, 2 Jul 2026 13:31:48 +0800 Subject: [PATCH 03/15] Apply suggestions from code review --- grafana-pd-dashboard.md | 2 +- pd-control.md | 2 +- troubleshoot-hot-spot-issues.md | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/grafana-pd-dashboard.md b/grafana-pd-dashboard.md index 4dc20d2b99739..a33da055390e6 100644 --- a/grafana-pd-dashboard.md +++ b/grafana-pd-dashboard.md @@ -88,7 +88,7 @@ The following is the description of PD Dashboard metrics items: - Total read bytes on hot peer Regions: The total read bytes of peers that have become read hotspots on each TiKV instance - Store read rate bytes: The total read bytes of each TiKV instance - Store read rate keys: The total read keys of each TiKV instance -- Store read cpu: The read CPU usage of each TiKV instance. PD uses this metric for CPU-aware read hotspot scheduling starting from v8.5.7. +- Store read cpu: The read CPU usage of each TiKV instance. PD uses this metric for CPU-aware read hotspot scheduling starting from v8.5.7 and v9.0.0. - Hot cache read entry number: The number of peers that are in the read hotspot statistics module on each TiKV instance ![PD Dashboard - Hot read metrics](/media/pd-dashboard-hotread-v4.png) diff --git a/pd-control.md b/pd-control.md index 2b4429bea438e..e8be85ac28608 100644 --- a/pd-control.md +++ b/pd-control.md @@ -547,7 +547,7 @@ Usage: } ``` -Starting from v8.5.7, the `hot read` and `hot history` commands include `flow_cpu`, and the `hot store` command includes `cpu-read-rate`. These fields show the read CPU usage for CPU-aware read hotspot scheduling. +Starting from v8.5.7 and v9.0.0, the `hot read` and `hot history` commands include `flow_cpu`, and the `hot store` command includes `cpu-read-rate`. These fields show the read CPU usage for CPU-aware read hotspot scheduling. ### `label [store ]` diff --git a/troubleshoot-hot-spot-issues.md b/troubleshoot-hot-spot-issues.md index 44f4325970106..99e0bd8d13220 100644 --- a/troubleshoot-hot-spot-issues.md +++ b/troubleshoot-hot-spot-issues.md @@ -185,7 +185,7 @@ For more details, see [Coprocessor Cache](/coprocessor-cache.md). In a read hotspot scenario, the hotspot TiKV node cannot process read requests in time, resulting in the read requests queuing. However, not all TiKV resources are exhausted at this time. To reduce latency, TiDB v7.1.0 introduces the load-based replica read feature, which allows TiDB to read data from other TiKV nodes without queuing on the hotspot TiKV node. You can control the queue length of read requests using the [`tidb_load_based_replica_read_threshold`](/system-variables.md#tidb_load_based_replica_read_threshold-new-in-v700) system variable. When the estimated queue time of the leader node exceeds this threshold, TiDB prioritizes reading data from follower nodes. This feature can improve read throughput by 70% to 200% in a read hotspot scenario compared to not scattering read hotspots. -Starting from v8.5.7, PD supports CPU-aware Hot Region scheduling for read hotspots. TiKV reports per-Region read CPU usage in store heartbeats, and PD can use CPU usage as a scheduling dimension. This helps PD identify read hotspots whose QPS or byte throughput looks balanced but whose TiKV CPU usage is still uneven, such as workloads with different CPU-cost queries or clusters with different TiKV performance profiles. +Starting from v8.5.7 and v9.0.0, PD supports CPU-aware hot Region scheduling for read hotspots. TiKV reports per-Region read CPU usage in store heartbeats, and PD can use CPU usage as a scheduling dimension. This helps PD identify read hotspots whose QPS or byte throughput looks balanced but whose TiKV CPU usage is still uneven, such as workloads with queries that have different CPU costs or clusters with heterogeneous TiKV hardware. For clusters that support read CPU reporting, the default `read-priorities` value of `balance-hot-region-scheduler` is `cpu,byte`. For clusters that do not support read CPU reporting, PD automatically falls back to `query,byte`, or to `byte,key` if the cluster does not support the `query` dimension either. To view or adjust the scheduling dimensions, use [`pd-ctl scheduler config balance-hot-region-scheduler`](/pd-control.md#scheduler-config-balance-hot-region-scheduler). From a8eb3ace8d60bd6b7b785c8fea338ce639484267 Mon Sep 17 00:00:00 2001 From: Grace Cai Date: Thu, 2 Jul 2026 13:37:28 +0800 Subject: [PATCH 04/15] split one paragraph into two bullets and a smaller paragraph --- troubleshoot-hot-spot-issues.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/troubleshoot-hot-spot-issues.md b/troubleshoot-hot-spot-issues.md index 99e0bd8d13220..2eb8c1182c851 100644 --- a/troubleshoot-hot-spot-issues.md +++ b/troubleshoot-hot-spot-issues.md @@ -187,7 +187,10 @@ In a read hotspot scenario, the hotspot TiKV node cannot process read requests i Starting from v8.5.7 and v9.0.0, PD supports CPU-aware hot Region scheduling for read hotspots. TiKV reports per-Region read CPU usage in store heartbeats, and PD can use CPU usage as a scheduling dimension. This helps PD identify read hotspots whose QPS or byte throughput looks balanced but whose TiKV CPU usage is still uneven, such as workloads with queries that have different CPU costs or clusters with heterogeneous TiKV hardware. -For clusters that support read CPU reporting, the default `read-priorities` value of `balance-hot-region-scheduler` is `cpu,byte`. For clusters that do not support read CPU reporting, PD automatically falls back to `query,byte`, or to `byte,key` if the cluster does not support the `query` dimension either. To view or adjust the scheduling dimensions, use [`pd-ctl scheduler config balance-hot-region-scheduler`](/pd-control.md#scheduler-config-balance-hot-region-scheduler). +- For clusters that support read CPU reporting, the default `read-priorities` value of `balance-hot-region-scheduler` is `cpu,byte`. +- For clusters that do not support read CPU reporting, PD automatically falls back to `query,byte`, or to `byte,key` if the cluster does not support the `query` dimension either. + +To view or adjust the scheduling dimensions, use [`pd-ctl scheduler config balance-hot-region-scheduler`](/pd-control.md#scheduler-config-balance-hot-region-scheduler). ## Use TiKV MVCC in-memory engine to mitigate read hotspots caused by high MVCC read amplification From b581562ca1c34490eac9a86d73c751ed05727d85 Mon Sep 17 00:00:00 2001 From: Grace Cai Date: Thu, 2 Jul 2026 13:37:44 +0800 Subject: [PATCH 05/15] minor wording updates --- troubleshoot-hot-spot-issues.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/troubleshoot-hot-spot-issues.md b/troubleshoot-hot-spot-issues.md index 2eb8c1182c851..70766d4aa8a50 100644 --- a/troubleshoot-hot-spot-issues.md +++ b/troubleshoot-hot-spot-issues.md @@ -185,7 +185,7 @@ For more details, see [Coprocessor Cache](/coprocessor-cache.md). In a read hotspot scenario, the hotspot TiKV node cannot process read requests in time, resulting in the read requests queuing. However, not all TiKV resources are exhausted at this time. To reduce latency, TiDB v7.1.0 introduces the load-based replica read feature, which allows TiDB to read data from other TiKV nodes without queuing on the hotspot TiKV node. You can control the queue length of read requests using the [`tidb_load_based_replica_read_threshold`](/system-variables.md#tidb_load_based_replica_read_threshold-new-in-v700) system variable. When the estimated queue time of the leader node exceeds this threshold, TiDB prioritizes reading data from follower nodes. This feature can improve read throughput by 70% to 200% in a read hotspot scenario compared to not scattering read hotspots. -Starting from v8.5.7 and v9.0.0, PD supports CPU-aware hot Region scheduling for read hotspots. TiKV reports per-Region read CPU usage in store heartbeats, and PD can use CPU usage as a scheduling dimension. This helps PD identify read hotspots whose QPS or byte throughput looks balanced but whose TiKV CPU usage is still uneven, such as workloads with queries that have different CPU costs or clusters with heterogeneous TiKV hardware. +Starting from v8.5.7 and v9.0.0, PD supports CPU-aware hot Region scheduling for read hotspots. TiKV reports per-Region read CPU usage in store heartbeats, and PD can use CPU usage as a scheduling dimension. This mechanism helps PD identify read hotspots whose QPS or byte throughput looks balanced but whose TiKV CPU usage is still uneven, such as workloads with queries that have different CPU costs or clusters with heterogeneous TiKV hardware. - For clusters that support read CPU reporting, the default `read-priorities` value of `balance-hot-region-scheduler` is `cpu,byte`. - For clusters that do not support read CPU reporting, PD automatically falls back to `query,byte`, or to `byte,key` if the cluster does not support the `query` dimension either. From f9546cfef0e2f9a9d839bae50fa8be9b1ee0ce3a Mon Sep 17 00:00:00 2001 From: Grace Cai Date: Thu, 2 Jul 2026 14:31:11 +0800 Subject: [PATCH 06/15] Update pd-control.md --- pd-control.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pd-control.md b/pd-control.md index e8be85ac28608..6e16df90f4683 100644 --- a/pd-control.md +++ b/pd-control.md @@ -1076,7 +1076,7 @@ Usage: scheduler config balance-hot-region-scheduler set min-hot-query-rate 10 ``` -- `min-hot-cpu-rate` specifies the minimum CPU usage of read requests to count, which is usually 10. +- `min-hot-cpu-rate` specifies the minimum CPU usage of read requests to count. The value is measured as a percentage of one CPU core, and the default value is usually `10`, which means 10% of one CPU core. ```bash scheduler config balance-hot-region-scheduler set min-hot-cpu-rate 10 From c04cba2de863a860112b4ae7abba7978cc180020 Mon Sep 17 00:00:00 2001 From: Grace Cai Date: Thu, 2 Jul 2026 14:36:45 +0800 Subject: [PATCH 07/15] Update troubleshoot-hot-spot-issues.md --- troubleshoot-hot-spot-issues.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/troubleshoot-hot-spot-issues.md b/troubleshoot-hot-spot-issues.md index 70766d4aa8a50..a2a432c62cb3f 100644 --- a/troubleshoot-hot-spot-issues.md +++ b/troubleshoot-hot-spot-issues.md @@ -185,7 +185,7 @@ For more details, see [Coprocessor Cache](/coprocessor-cache.md). In a read hotspot scenario, the hotspot TiKV node cannot process read requests in time, resulting in the read requests queuing. However, not all TiKV resources are exhausted at this time. To reduce latency, TiDB v7.1.0 introduces the load-based replica read feature, which allows TiDB to read data from other TiKV nodes without queuing on the hotspot TiKV node. You can control the queue length of read requests using the [`tidb_load_based_replica_read_threshold`](/system-variables.md#tidb_load_based_replica_read_threshold-new-in-v700) system variable. When the estimated queue time of the leader node exceeds this threshold, TiDB prioritizes reading data from follower nodes. This feature can improve read throughput by 70% to 200% in a read hotspot scenario compared to not scattering read hotspots. -Starting from v8.5.7 and v9.0.0, PD supports CPU-aware hot Region scheduling for read hotspots. TiKV reports per-Region read CPU usage in store heartbeats, and PD can use CPU usage as a scheduling dimension. This mechanism helps PD identify read hotspots whose QPS or byte throughput looks balanced but whose TiKV CPU usage is still uneven, such as workloads with queries that have different CPU costs or clusters with heterogeneous TiKV hardware. +Starting from v8.5.7 and v9.0.0, PD supports CPU-aware hot Region scheduling for read hotspots. TiKV reports per-Region read CPU usage in store heartbeats, and PD can use CPU usage as a scheduling dimension. This mechanism helps PD identify read hotspots whose QPS or byte throughput appears balanced but whose TiKV CPU usage remains uneven, such as workloads with queries that have different CPU costs or clusters where TiKV nodes handle different workload patterns. - For clusters that support read CPU reporting, the default `read-priorities` value of `balance-hot-region-scheduler` is `cpu,byte`. - For clusters that do not support read CPU reporting, PD automatically falls back to `query,byte`, or to `byte,key` if the cluster does not support the `query` dimension either. From 7770d6368e41447d693601faf72e8f61e2ae3783 Mon Sep 17 00:00:00 2001 From: Grace Cai Date: Thu, 2 Jul 2026 15:03:45 +0800 Subject: [PATCH 08/15] Update pd-control.md --- pd-control.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pd-control.md b/pd-control.md index 6e16df90f4683..aa6d315d2e9fb 100644 --- a/pd-control.md +++ b/pd-control.md @@ -547,7 +547,7 @@ Usage: } ``` -Starting from v8.5.7 and v9.0.0, the `hot read` and `hot history` commands include `flow_cpu`, and the `hot store` command includes `cpu-read-rate`. These fields show the read CPU usage for CPU-aware read hotspot scheduling. +Starting from v8.5.7 and v9.0.0, the output of the `hot read` and `hot history` commands includes the `flow_cpu` field, and the output of the `hot store` command includes the `cpu-read-rate` field. These fields show the read CPU usage for CPU-aware read hotspot scheduling. ### `label [store ]` From 66f495eae7ac283759706ee9cf2511d3ca6bf386 Mon Sep 17 00:00:00 2001 From: Grace Cai Date: Thu, 2 Jul 2026 15:09:10 +0800 Subject: [PATCH 09/15] Update pd-control.md --- pd-control.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pd-control.md b/pd-control.md index aa6d315d2e9fb..4e907c32961ff 100644 --- a/pd-control.md +++ b/pd-control.md @@ -1116,7 +1116,7 @@ Usage: > > If a cluster component is earlier than v5.2, the configuration of the `query` dimension does not take effect. If some components are upgraded to v5.2 or later, the `byte` and `key` dimensions still by default have the priority for hot Region scheduling. After all components of the cluster are upgraded to v5.2 or later, such a configuration still takes effect for compatibility. > - > Starting from v8.5.7, TiKV reports read CPU usage for hot Region scheduling. For clusters that support read CPU reporting, the default `read-priorities` value is `cpu,byte`. For clusters that do not support read CPU reporting, PD automatically falls back to `query,byte`, or to `byte,key` if the cluster does not support the `query` dimension either. You can view the real-time configuration using the `pd-ctl` command. Usually, you do not need to modify these configurations. + > Starting from v8.5.7 and v9.0.0, TiKV reports read CPU usage for hot Region scheduling. For clusters that support read CPU reporting, the default `read-priorities` value is `cpu,byte`. For clusters that do not support read CPU reporting, PD automatically falls back to `query,byte`, or to `byte,key` if the cluster does not support the `query` dimension either. You can view the real-time configuration using the `pd-ctl` command. Usually, you do not need to modify these configurations. ```bash scheduler config balance-hot-region-scheduler set read-priorities cpu,byte From cc0882dbd27ca1984ba3222c33c94c774f352330 Mon Sep 17 00:00:00 2001 From: Grace Cai Date: Mon, 6 Jul 2026 15:15:34 +0800 Subject: [PATCH 10/15] Update troubleshoot-hot-spot-issues.md --- troubleshoot-hot-spot-issues.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/troubleshoot-hot-spot-issues.md b/troubleshoot-hot-spot-issues.md index a2a432c62cb3f..91b91dc7154f5 100644 --- a/troubleshoot-hot-spot-issues.md +++ b/troubleshoot-hot-spot-issues.md @@ -185,6 +185,8 @@ For more details, see [Coprocessor Cache](/coprocessor-cache.md). In a read hotspot scenario, the hotspot TiKV node cannot process read requests in time, resulting in the read requests queuing. However, not all TiKV resources are exhausted at this time. To reduce latency, TiDB v7.1.0 introduces the load-based replica read feature, which allows TiDB to read data from other TiKV nodes without queuing on the hotspot TiKV node. You can control the queue length of read requests using the [`tidb_load_based_replica_read_threshold`](/system-variables.md#tidb_load_based_replica_read_threshold-new-in-v700) system variable. When the estimated queue time of the leader node exceeds this threshold, TiDB prioritizes reading data from follower nodes. This feature can improve read throughput by 70% to 200% in a read hotspot scenario compared to not scattering read hotspots. +### CPU-aware hot Region scheduling for read hotspots + Starting from v8.5.7 and v9.0.0, PD supports CPU-aware hot Region scheduling for read hotspots. TiKV reports per-Region read CPU usage in store heartbeats, and PD can use CPU usage as a scheduling dimension. This mechanism helps PD identify read hotspots whose QPS or byte throughput appears balanced but whose TiKV CPU usage remains uneven, such as workloads with queries that have different CPU costs or clusters where TiKV nodes handle different workload patterns. - For clusters that support read CPU reporting, the default `read-priorities` value of `balance-hot-region-scheduler` is `cpu,byte`. From 13c8de345a1286502cede6f1edd71861b35f1032 Mon Sep 17 00:00:00 2001 From: Grace Cai Date: Mon, 6 Jul 2026 17:18:54 +0800 Subject: [PATCH 11/15] Update pd-control.md --- pd-control.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pd-control.md b/pd-control.md index 4e907c32961ff..d96a3507736c3 100644 --- a/pd-control.md +++ b/pd-control.md @@ -1076,7 +1076,7 @@ Usage: scheduler config balance-hot-region-scheduler set min-hot-query-rate 10 ``` -- `min-hot-cpu-rate` specifies the minimum CPU usage of read requests to count. The value is measured as a percentage of one CPU core, and the default value is usually `10`, which means 10% of one CPU core. +- `min-hot-cpu-rate` specifies the minimum CPU usage required for a read request to be included in hotspot statistics. The value is measured as a percentage of one CPU core, and the default value is usually `10`, which means 10% of one CPU core. ```bash scheduler config balance-hot-region-scheduler set min-hot-cpu-rate 10 From 8189ea625d46c9b8990fbae5a51b5133a91b8b90 Mon Sep 17 00:00:00 2001 From: Grace Cai Date: Mon, 6 Jul 2026 17:22:32 +0800 Subject: [PATCH 12/15] Update pd-control.md --- pd-control.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pd-control.md b/pd-control.md index d96a3507736c3..0ad71ab39201b 100644 --- a/pd-control.md +++ b/pd-control.md @@ -1076,7 +1076,7 @@ Usage: scheduler config balance-hot-region-scheduler set min-hot-query-rate 10 ``` -- `min-hot-cpu-rate` specifies the minimum CPU usage required for a read request to be included in hotspot statistics. The value is measured as a percentage of one CPU core, and the default value is usually `10`, which means 10% of one CPU core. +- `min-hot-cpu-rate` specifies the minimum CPU usage required for a read request to be included in hotspot statistics. The value is measured as a percentage of one CPU core, and the default value is `10`, which means 10% of one CPU core. ```bash scheduler config balance-hot-region-scheduler set min-hot-cpu-rate 10 From 15af37c0a629d76ac7982bc9275d90b72797f949 Mon Sep 17 00:00:00 2001 From: Grace Cai Date: Mon, 6 Jul 2026 17:29:45 +0800 Subject: [PATCH 13/15] minor wording updates --- pd-control.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pd-control.md b/pd-control.md index 0ad71ab39201b..af4abe1c75543 100644 --- a/pd-control.md +++ b/pd-control.md @@ -1114,7 +1114,7 @@ Usage: > **Note:** > - > If a cluster component is earlier than v5.2, the configuration of the `query` dimension does not take effect. If some components are upgraded to v5.2 or later, the `byte` and `key` dimensions still by default have the priority for hot Region scheduling. After all components of the cluster are upgraded to v5.2 or later, such a configuration still takes effect for compatibility. + > If any component in the cluster is earlier than v5.2, the configuration of the `query` dimension does not take effect. After some components are upgraded to v5.2 or later, the scheduler still prioritizes hotspot balancing based on the `byte` and `key` dimensions by default. After all components in the cluster are upgraded to v5.2 or later, such a configuration still takes effect for compatibility. > > Starting from v8.5.7 and v9.0.0, TiKV reports read CPU usage for hot Region scheduling. For clusters that support read CPU reporting, the default `read-priorities` value is `cpu,byte`. For clusters that do not support read CPU reporting, PD automatically falls back to `query,byte`, or to `byte,key` if the cluster does not support the `query` dimension either. You can view the real-time configuration using the `pd-ctl` command. Usually, you do not need to modify these configurations. From 66b8b121f29c3a5d9877f1fa1b8df3457624efbf Mon Sep 17 00:00:00 2001 From: Grace Cai Date: Mon, 6 Jul 2026 17:31:15 +0800 Subject: [PATCH 14/15] Update troubleshoot-hot-spot-issues.md --- troubleshoot-hot-spot-issues.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/troubleshoot-hot-spot-issues.md b/troubleshoot-hot-spot-issues.md index 91b91dc7154f5..a71ad0f36f3ae 100644 --- a/troubleshoot-hot-spot-issues.md +++ b/troubleshoot-hot-spot-issues.md @@ -187,7 +187,7 @@ In a read hotspot scenario, the hotspot TiKV node cannot process read requests i ### CPU-aware hot Region scheduling for read hotspots -Starting from v8.5.7 and v9.0.0, PD supports CPU-aware hot Region scheduling for read hotspots. TiKV reports per-Region read CPU usage in store heartbeats, and PD can use CPU usage as a scheduling dimension. This mechanism helps PD identify read hotspots whose QPS or byte throughput appears balanced but whose TiKV CPU usage remains uneven, such as workloads with queries that have different CPU costs or clusters where TiKV nodes handle different workload patterns. +Starting from v8.5.7 and v9.0.0, PD supports CPU-aware hot Region scheduling for read hotspots. TiKV reports hot Region read CPU usage in store heartbeats, and PD can use CPU usage as a scheduling dimension. This mechanism helps PD identify read hotspots whose QPS or byte throughput appears balanced but whose TiKV CPU usage remains uneven, such as workloads with queries that have different CPU costs or clusters where TiKV nodes handle different workload patterns. - For clusters that support read CPU reporting, the default `read-priorities` value of `balance-hot-region-scheduler` is `cpu,byte`. - For clusters that do not support read CPU reporting, PD automatically falls back to `query,byte`, or to `byte,key` if the cluster does not support the `query` dimension either. From 56171ae79f07aa48aecd56b925d056420ab93857 Mon Sep 17 00:00:00 2001 From: qiancai Date: Mon, 6 Jul 2026 17:50:02 +0800 Subject: [PATCH 15/15] docs: remove v9.0.0 mention for v8.5.7 --- grafana-pd-dashboard.md | 2 +- pd-control.md | 4 ++-- troubleshoot-hot-spot-issues.md | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/grafana-pd-dashboard.md b/grafana-pd-dashboard.md index a33da055390e6..4dc20d2b99739 100644 --- a/grafana-pd-dashboard.md +++ b/grafana-pd-dashboard.md @@ -88,7 +88,7 @@ The following is the description of PD Dashboard metrics items: - Total read bytes on hot peer Regions: The total read bytes of peers that have become read hotspots on each TiKV instance - Store read rate bytes: The total read bytes of each TiKV instance - Store read rate keys: The total read keys of each TiKV instance -- Store read cpu: The read CPU usage of each TiKV instance. PD uses this metric for CPU-aware read hotspot scheduling starting from v8.5.7 and v9.0.0. +- Store read cpu: The read CPU usage of each TiKV instance. PD uses this metric for CPU-aware read hotspot scheduling starting from v8.5.7. - Hot cache read entry number: The number of peers that are in the read hotspot statistics module on each TiKV instance ![PD Dashboard - Hot read metrics](/media/pd-dashboard-hotread-v4.png) diff --git a/pd-control.md b/pd-control.md index af4abe1c75543..f6c839c5fdd79 100644 --- a/pd-control.md +++ b/pd-control.md @@ -547,7 +547,7 @@ Usage: } ``` -Starting from v8.5.7 and v9.0.0, the output of the `hot read` and `hot history` commands includes the `flow_cpu` field, and the output of the `hot store` command includes the `cpu-read-rate` field. These fields show the read CPU usage for CPU-aware read hotspot scheduling. +Starting from v8.5.7, the output of the `hot read` and `hot history` commands includes the `flow_cpu` field, and the output of the `hot store` command includes the `cpu-read-rate` field. These fields show the read CPU usage for CPU-aware read hotspot scheduling. ### `label [store ]` @@ -1116,7 +1116,7 @@ Usage: > > If any component in the cluster is earlier than v5.2, the configuration of the `query` dimension does not take effect. After some components are upgraded to v5.2 or later, the scheduler still prioritizes hotspot balancing based on the `byte` and `key` dimensions by default. After all components in the cluster are upgraded to v5.2 or later, such a configuration still takes effect for compatibility. > - > Starting from v8.5.7 and v9.0.0, TiKV reports read CPU usage for hot Region scheduling. For clusters that support read CPU reporting, the default `read-priorities` value is `cpu,byte`. For clusters that do not support read CPU reporting, PD automatically falls back to `query,byte`, or to `byte,key` if the cluster does not support the `query` dimension either. You can view the real-time configuration using the `pd-ctl` command. Usually, you do not need to modify these configurations. + > Starting from v8.5.7, TiKV reports read CPU usage for hot Region scheduling. For clusters that support read CPU reporting, the default `read-priorities` value is `cpu,byte`. For clusters that do not support read CPU reporting, PD automatically falls back to `query,byte`, or to `byte,key` if the cluster does not support the `query` dimension either. You can view the real-time configuration using the `pd-ctl` command. Usually, you do not need to modify these configurations. ```bash scheduler config balance-hot-region-scheduler set read-priorities cpu,byte diff --git a/troubleshoot-hot-spot-issues.md b/troubleshoot-hot-spot-issues.md index a71ad0f36f3ae..5cb9a5ca9ceb6 100644 --- a/troubleshoot-hot-spot-issues.md +++ b/troubleshoot-hot-spot-issues.md @@ -187,7 +187,7 @@ In a read hotspot scenario, the hotspot TiKV node cannot process read requests i ### CPU-aware hot Region scheduling for read hotspots -Starting from v8.5.7 and v9.0.0, PD supports CPU-aware hot Region scheduling for read hotspots. TiKV reports hot Region read CPU usage in store heartbeats, and PD can use CPU usage as a scheduling dimension. This mechanism helps PD identify read hotspots whose QPS or byte throughput appears balanced but whose TiKV CPU usage remains uneven, such as workloads with queries that have different CPU costs or clusters where TiKV nodes handle different workload patterns. +Starting from v8.5.7, PD supports CPU-aware hot Region scheduling for read hotspots. TiKV reports hot Region read CPU usage in store heartbeats, and PD can use CPU usage as a scheduling dimension. This mechanism helps PD identify read hotspots whose QPS or byte throughput appears balanced but whose TiKV CPU usage remains uneven, such as workloads with queries that have different CPU costs or clusters where TiKV nodes handle different workload patterns. - For clusters that support read CPU reporting, the default `read-priorities` value of `balance-hot-region-scheduler` is `cpu,byte`. - For clusters that do not support read CPU reporting, PD automatically falls back to `query,byte`, or to `byte,key` if the cluster does not support the `query` dimension either.