1
+ .. SPDX-License-Identifier: GPL-2.0
2
+
3
+ ==============================
1
4
Using RCU's CPU Stall Detector
5
+ ==============================
2
6
3
7
This document first discusses what sorts of issues RCU's CPU stall
4
8
detector can locate, and then discusses kernel parameters and Kconfig
@@ -7,39 +11,40 @@ this document explains the stall detector's "splat" format.
7
11
8
12
9
13
What Causes RCU CPU Stall Warnings?
14
+ ===================================
10
15
11
16
So your kernel printed an RCU CPU stall warning. The next question is
12
17
"What caused it?" The following problems can result in RCU CPU stall
13
18
warnings:
14
19
15
- o A CPU looping in an RCU read-side critical section.
20
+ - A CPU looping in an RCU read-side critical section.
16
21
17
- o A CPU looping with interrupts disabled.
22
+ - A CPU looping with interrupts disabled.
18
23
19
- o A CPU looping with preemption disabled.
24
+ - A CPU looping with preemption disabled.
20
25
21
- o A CPU looping with bottom halves disabled.
26
+ - A CPU looping with bottom halves disabled.
22
27
23
- o For !CONFIG_PREEMPT kernels, a CPU looping anywhere in the kernel
28
+ - For !CONFIG_PREEMPT kernels, a CPU looping anywhere in the kernel
24
29
without invoking schedule(). If the looping in the kernel is
25
30
really expected and desirable behavior, you might need to add
26
31
some calls to cond_resched().
27
32
28
- o Booting Linux using a console connection that is too slow to
33
+ - Booting Linux using a console connection that is too slow to
29
34
keep up with the boot-time console-message rate. For example,
30
35
a 115Kbaud serial console can be -way- too slow to keep up
31
36
with boot-time message rates, and will frequently result in
32
37
RCU CPU stall warning messages. Especially if you have added
33
38
debug printk()s.
34
39
35
- o Anything that prevents RCU's grace-period kthreads from running.
40
+ - Anything that prevents RCU's grace-period kthreads from running.
36
41
This can result in the "All QSes seen" console-log message.
37
42
This message will include information on when the kthread last
38
43
ran and how often it should be expected to run. It can also
39
- result in the " rcu_.*kthread starved for" console-log message,
44
+ result in the `` rcu_.*kthread starved for `` console-log message,
40
45
which will include additional debugging information.
41
46
42
- o A CPU-bound real-time task in a CONFIG_PREEMPT kernel, which might
47
+ - A CPU-bound real-time task in a CONFIG_PREEMPT kernel, which might
43
48
happen to preempt a low-priority task in the middle of an RCU
44
49
read-side critical section. This is especially damaging if
45
50
that low-priority task is not permitted to run on any other CPU,
@@ -48,7 +53,7 @@ o A CPU-bound real-time task in a CONFIG_PREEMPT kernel, which might
48
53
While the system is in the process of running itself out of
49
54
memory, you might see stall-warning messages.
50
55
51
- o A CPU-bound real-time task in a CONFIG_PREEMPT_RT kernel that
56
+ - A CPU-bound real-time task in a CONFIG_PREEMPT_RT kernel that
52
57
is running at a higher priority than the RCU softirq threads.
53
58
This will prevent RCU callbacks from ever being invoked,
54
59
and in a CONFIG_PREEMPT_RCU kernel will further prevent
@@ -63,28 +68,28 @@ o A CPU-bound real-time task in a CONFIG_PREEMPT_RT kernel that
63
68
can increase your system's context-switch rate and thus degrade
64
69
performance.
65
70
66
- o A periodic interrupt whose handler takes longer than the time
71
+ - A periodic interrupt whose handler takes longer than the time
67
72
interval between successive pairs of interrupts. This can
68
73
prevent RCU's kthreads and softirq handlers from running.
69
74
Note that certain high-overhead debugging options, for example
70
75
the function_graph tracer, can result in interrupt handler taking
71
76
considerably longer than normal, which can in turn result in
72
77
RCU CPU stall warnings.
73
78
74
- o Testing a workload on a fast system, tuning the stall-warning
79
+ - Testing a workload on a fast system, tuning the stall-warning
75
80
timeout down to just barely avoid RCU CPU stall warnings, and then
76
81
running the same workload with the same stall-warning timeout on a
77
82
slow system. Note that thermal throttling and on-demand governors
78
83
can cause a single system to be sometimes fast and sometimes slow!
79
84
80
- o A hardware or software issue shuts off the scheduler-clock
85
+ - A hardware or software issue shuts off the scheduler-clock
81
86
interrupt on a CPU that is not in dyntick-idle mode. This
82
87
problem really has happened, and seems to be most likely to
83
88
result in RCU CPU stall warnings for CONFIG_NO_HZ_COMMON=n kernels.
84
89
85
- o A bug in the RCU implementation.
90
+ - A bug in the RCU implementation.
86
91
87
- o A hardware failure. This is quite unlikely, but has occurred
92
+ - A hardware failure. This is quite unlikely, but has occurred
88
93
at least once in real life. A CPU failed in a running system,
89
94
becoming unresponsive, but not causing an immediate crash.
90
95
This resulted in a series of RCU CPU stall warnings, eventually
@@ -109,6 +114,7 @@ see include/trace/events/rcu.h.
109
114
110
115
111
116
Fine-Tuning the RCU CPU Stall Detector
117
+ ======================================
112
118
113
119
The rcuupdate.rcu_cpu_stall_suppress module parameter disables RCU's
114
120
CPU stall detector, which detects conditions that unduly delay RCU grace
@@ -118,6 +124,7 @@ The stall detector's idea of what constitutes "unduly delayed" is
118
124
controlled by a set of kernel configuration variables and cpp macros:
119
125
120
126
CONFIG_RCU_CPU_STALL_TIMEOUT
127
+ ----------------------------
121
128
122
129
This kernel configuration parameter defines the period of time
123
130
that RCU will wait from the beginning of a grace period until it
@@ -137,6 +144,7 @@ CONFIG_RCU_CPU_STALL_TIMEOUT
137
144
/sys/module/rcupdate/parameters/rcu_cpu_stall_suppress.
138
145
139
146
RCU_STALL_DELAY_DELTA
147
+ ---------------------
140
148
141
149
Although the lockdep facility is extremely useful, it does add
142
150
some overhead. Therefore, under CONFIG_PROVE_RCU, the
@@ -145,6 +153,7 @@ RCU_STALL_DELAY_DELTA
145
153
macro, not a kernel configuration parameter.)
146
154
147
155
RCU_STALL_RAT_DELAY
156
+ -------------------
148
157
149
158
The CPU stall detector tries to make the offending CPU print its
150
159
own warnings, as this often gives better-quality stack traces.
@@ -155,6 +164,7 @@ RCU_STALL_RAT_DELAY
155
164
parameter.)
156
165
157
166
rcupdate.rcu_task_stall_timeout
167
+ -------------------------------
158
168
159
169
This boot/sysfs parameter controls the RCU-tasks stall warning
160
170
interval. A value of zero or less suppresses RCU-tasks stall
@@ -168,9 +178,10 @@ rcupdate.rcu_task_stall_timeout
168
178
169
179
170
180
Interpreting RCU's CPU Stall-Detector "Splats"
181
+ ==============================================
171
182
172
183
For non-RCU-tasks flavors of RCU, when a CPU detects that it is stalling,
173
- it will print a message similar to the following:
184
+ it will print a message similar to the following::
174
185
175
186
INFO: rcu_sched detected stalls on CPUs/tasks:
176
187
2-...: (3 GPs behind) idle=06c/0/0 softirq=1453/1455 fqs=0
@@ -223,7 +234,7 @@ an estimate of the total number of RCU callbacks queued across all CPUs
223
234
(625 in this case).
224
235
225
236
In kernels with CONFIG_RCU_FAST_NO_HZ, more information is printed
226
- for each CPU:
237
+ for each CPU::
227
238
228
239
0: (64628 ticks this GP) idle=dd5/3fffffffffffffff/0 softirq=82/543 last_accelerate: a345/d342 dyntick_enabled: 1
229
240
@@ -235,7 +246,7 @@ processing is enabled.
235
246
236
247
If the grace period ends just as the stall warning starts printing,
237
248
there will be a spurious stall-warning message, which will include
238
- the following:
249
+ the following::
239
250
240
251
INFO: Stall ended before state dump start
241
252
@@ -248,7 +259,7 @@ which is overkill for this sort of problem.
248
259
249
260
If all CPUs and tasks have passed through quiescent states, but the
250
261
grace period has nevertheless failed to end, the stall-warning splat
251
- will include something like the following:
262
+ will include something like the following::
252
263
253
264
All QSes seen, last rcu_preempt kthread activity 23807 (4297905177-4297881370), jiffies_till_next_fqs=3, root ->qsmask 0x0
254
265
@@ -261,7 +272,7 @@ which is way less than 23807. Finally, the root rcu_node structure's
261
272
262
273
If the relevant grace-period kthread has been unable to run prior to
263
274
the stall warning, as was the case in the "All QSes seen" line above,
264
- the following additional line is printed:
275
+ the following additional line is printed::
265
276
266
277
kthread starved for 23807 jiffies! g7075 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x1 ->cpu=5
267
278
@@ -276,6 +287,7 @@ kthread last ran on CPU 5.
276
287
277
288
278
289
Multiple Warnings From One Stall
290
+ ================================
279
291
280
292
If a stall lasts long enough, multiple stall-warning messages will be
281
293
printed for it. The second and subsequent messages are printed at
@@ -285,9 +297,10 @@ of the stall and the first message.
285
297
286
298
287
299
Stall Warnings for Expedited Grace Periods
300
+ ==========================================
288
301
289
302
If an expedited grace period detects a stall, it will place a message
290
- like the following in dmesg:
303
+ like the following in dmesg::
291
304
292
305
INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 7-... } 21119 jiffies s: 73 root: 0x2/.
293
306
0 commit comments