Skip to content

Commit af9c191

Browse files
torvaldstorvalds
committed
Merge tag 'trace-ring-buffer-v6.12' of git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace
Pull ring-buffer updates from Steven Rostedt: - tracing/ring-buffer: persistent buffer across reboots This allows for the tracing instance ring buffer to stay persistent across reboots. The way this is done is by adding to the kernel command line: trace_instance=boot_map@0x285400000:12M This will reserve 12 megabytes at the address 0x285400000, and then map the tracing instance "boot_map" ring buffer to that memory. This will appear as a normal instance in the tracefs system: /sys/kernel/tracing/instances/boot_map A user could enable tracing in that instance, and on reboot or kernel crash, if the memory is not wiped by the firmware, it will recreate the trace in that instance. For example, if one was debugging a shutdown of a kernel reboot: # cd /sys/kernel/tracing # echo function > instances/boot_map/current_tracer # reboot [..] # cd /sys/kernel/tracing # tail instances/boot_map/trace swapper/0-1 [000] d..1. 164.549800: restore_boot_irq_mode <-native_machine_shutdown swapper/0-1 [000] d..1. 164.549801: native_restore_boot_irq_mode <-native_machine_shutdown swapper/0-1 [000] d..1. 164.549802: disconnect_bsp_APIC <-native_machine_shutdown swapper/0-1 [000] d..1. 164.549811: hpet_disable <-native_machine_shutdown swapper/0-1 [000] d..1. 164.549812: iommu_shutdown_noop <-native_machine_restart swapper/0-1 [000] d..1. 164.549813: native_machine_emergency_restart <-__do_sys_reboot swapper/0-1 [000] d..1. 164.549813: tboot_shutdown <-native_machine_emergency_restart swapper/0-1 [000] d..1. 164.549820: acpi_reboot <-native_machine_emergency_restart swapper/0-1 [000] d..1. 164.549821: acpi_reset <-acpi_reboot swapper/0-1 [000] d..1. 164.549822: acpi_os_write_port <-acpi_reboot On reboot, the buffer is examined to make sure it is valid. The validation check even steps through every event to make sure the meta data of the event is correct. If any test fails, it will simply reset the buffer, and the buffer will be empty on boot. - Allow the tracing persistent boot buffer to use the "reserve_mem" option Instead of having the admin find a physical address to store the persistent buffer, which can be very tedious if they have to administrate several different machines, allow them to use the "reserve_mem" option that will find a location for them. It is not as reliable because of KASLR, as the loading of the kernel in different locations can cause the memory allocated to be inconsistent. Booting with "nokaslr" can make reserve_mem more reliable. - Have function graph tracer handle offsets from a previous boot. The ring buffer output from a previous boot may have different addresses due to kaslr. Have the function graph tracer handle these by using the delta from the previous boot to the new boot address space. - Only reset the saved meta offset when the buffer is started or reset In the persistent memory meta data, it holds the previous address space information, so that it can calculate the delta to have function tracing work. But this gets updated after being read to hold the new address space. But if the buffer isn't used for that boot, on reboot, the delta is now calculated from the previous boot and not the boot that holds the data in the ring buffer. This causes the functions not to be shown. Do not save the address space information of the current kernel until it is being recorded. - Add a magic variable to test the valid meta data Add a magic variable in the meta data that can also be used for validation. The validator of the previous buffer doesn't need this magic data, but it can be used if the meta data is changed by a new kernel, which may have the same format that passes the validator but is used differently. This magic number can also be used as a "versioning" of the meta data. - Align user space mapped ring buffer sub buffers to improve TLB entries Linus mentioned that the mapped ring buffer sub buffers were misaligned between the meta page and the sub-buffers, so that if the sub-buffers were bigger than PAGE_SIZE, it wouldn't allow the TLB to use bigger entries. - Add new kernel command line "traceoff" to disable tracing on boot for instances If tracing is enabled for a boot instance, there needs a way to be able to disable it on boot so that new events do not get entered into the ring buffer and be mixed with events from a previous boot, as that can be confusing. - Allow trace_printk() to go to other instances Currently, trace_printk() can only go to the top level instance. When debugging with a persistent buffer, it is really useful to be able to add trace_printk() to go to that buffer, so that you have access to them after a crash. - Do not use "bin_printk()" for traces to a boot instance The bin_printk() saves only a pointer to the printk format in the ring buffer, as the reader of the buffer can still have access to it. But this is not the case if the buffer is from a previous boot. If the trace_printk() is going to a "persistent" buffer, it will use the slower version that writes the printk format into the buffer. - Add command line option to allow trace_printk() to go to an instance Allow the kernel command line to define which instance the trace_printk() goes to, instead of forcing the admin to set it for every boot via the tracefs options. - Start a document that explains how to use tracefs to debug the kernel - Add some more kernel selftests to test user mapped ring buffer * tag 'trace-ring-buffer-v6.12' of git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace: (28 commits) selftests/ring-buffer: Handle meta-page bigger than the system selftests/ring-buffer: Verify the entire meta-page padding tracing/Documentation: Start a document on how to debug with tracing tracing: Add option to set an instance to be the trace_printk destination tracing: Have trace_printk not use binary prints if boot buffer tracing: Allow trace_printk() to go to other instance buffers tracing: Add "traceoff" flag to boot time tracing instances ring-buffer: Align meta-page to sub-buffers for improved TLB usage ring-buffer: Add magic and struct size to boot up meta data ring-buffer: Don't reset persistent ring-buffer meta saved addresses tracing/fgraph: Have fgraph handle previous boot function addresses tracing: Allow boot instances to use reserve_mem boot memory tracing: Fix ifdef of snapshots to not prevent last_boot_info file ring-buffer: Use vma_pages() helper function tracing: Fix NULL vs IS_ERR() check in enable_instances() tracing: Add last boot delta offset for stack traces tracing: Update function tracing output for previous boot buffer tracing: Handle old buffer mappings for event strings and functions tracing/ring-buffer: Add last_boot_info file to boot instance ring-buffer: Save text and data locations in mapped meta data ...
2 parents dd609b8 + 75d7ff9 commit af9c191

File tree

10 files changed

+1483
-152
lines changed

10 files changed

+1483
-152
lines changed

‎Documentation/admin-guide/kernel-parameters.txt

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6808,6 +6808,51 @@
68086808
the same thing would happen if it was left off). The irq_handler_entry
68096809
event, and all events under the "initcall" system.
68106810

6811+
Flags can be added to the instance to modify its behavior when it is
6812+
created. The flags are separated by '^'.
6813+
6814+
The available flags are:
6815+
6816+
traceoff - Have the tracing instance tracing disabled after it is created.
6817+
traceprintk - Have trace_printk() write into this trace instance
6818+
(note, "printk" and "trace_printk" can also be used)
6819+
6820+
trace_instance=foo^traceoff^traceprintk,sched,irq
6821+
6822+
The flags must come before the defined events.
6823+
6824+
If memory has been reserved (see memmap for x86), the instance
6825+
can use that memory:
6826+
6827+
memmap=12M$0x284500000 trace_instance=boot_map@0x284500000:12M
6828+
6829+
The above will create a "boot_map" instance that uses the physical
6830+
memory at 0x284500000 that is 12Megs. The per CPU buffers of that
6831+
instance will be split up accordingly.
6832+
6833+
Alternatively, the memory can be reserved by the reserve_mem option:
6834+
6835+
reserve_mem=12M:4096:trace trace_instance=boot_map@trace
6836+
6837+
This will reserve 12 megabytes at boot up with a 4096 byte alignment
6838+
and place the ring buffer in this memory. Note that due to KASLR, the
6839+
memory may not be the same location each time, which will not preserve
6840+
the buffer content.
6841+
6842+
Also note that the layout of the ring buffer data may change between
6843+
kernel versions where the validator will fail and reset the ring buffer
6844+
if the layout is not the same as the previous kernel.
6845+
6846+
If the ring buffer is used for persistent bootups and has events enabled,
6847+
it is recommend to disable tracing so that events from a previous boot do not
6848+
mix with events of the current boot (unless you are debugging a random crash
6849+
at boot up).
6850+
6851+
reserve_mem=12M:4096:trace trace_instance=boot_map^traceoff^traceprintk@trace,sched,irq
6852+
6853+
See also Documentation/trace/debugging.rst
6854+
6855+
68116856
trace_options=[option-list]
68126857
[FTRACE] Enable or disable tracer options at boot.
68136858
The option-list is a comma delimited list of options

‎Documentation/trace/debugging.rst

Lines changed: 159 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,159 @@
1+
==============================
2+
Using the tracer for debugging
3+
==============================
4+
5+
Copyright 2024 Google LLC.
6+
7+
:Author: Steven Rostedt <[email protected]>
8+
:License: The GNU Free Documentation License, Version 1.2
9+
(dual licensed under the GPL v2)
10+
11+
- Written for: 6.12
12+
13+
Introduction
14+
------------
15+
The tracing infrastructure can be very useful for debugging the Linux
16+
kernel. This document is a place to add various methods of using the tracer
17+
for debugging.
18+
19+
First, make sure that the tracefs file system is mounted::
20+
21+
$ sudo mount -t tracefs tracefs /sys/kernel/tracing
22+
23+
24+
Using trace_printk()
25+
--------------------
26+
27+
trace_printk() is a very lightweight utility that can be used in any context
28+
inside the kernel, with the exception of "noinstr" sections. It can be used
29+
in normal, softirq, interrupt and even NMI context. The trace data is
30+
written to the tracing ring buffer in a lockless way. To make it even
31+
lighter weight, when possible, it will only record the pointer to the format
32+
string, and save the raw arguments into the buffer. The format and the
33+
arguments will be post processed when the ring buffer is read. This way the
34+
trace_printk() format conversions are not done during the hot path, where
35+
the trace is being recorded.
36+
37+
trace_printk() is meant only for debugging, and should never be added into
38+
a subsystem of the kernel. If you need debugging traces, add trace events
39+
instead. If a trace_printk() is found in the kernel, the following will
40+
appear in the dmesg::
41+
42+
**********************************************************
43+
** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE **
44+
** **
45+
** trace_printk() being used. Allocating extra memory. **
46+
** **
47+
** This means that this is a DEBUG kernel and it is **
48+
** unsafe for production use. **
49+
** **
50+
** If you see this message and you are not debugging **
51+
** the kernel, report this immediately to your vendor! **
52+
** **
53+
** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE **
54+
**********************************************************
55+
56+
Debugging kernel crashes
57+
------------------------
58+
There is various methods of acquiring the state of the system when a kernel
59+
crash occurs. This could be from the oops message in printk, or one could
60+
use kexec/kdump. But these just show what happened at the time of the crash.
61+
It can be very useful in knowing what happened up to the point of the crash.
62+
The tracing ring buffer, by default, is a circular buffer than will
63+
overwrite older events with newer ones. When a crash happens, the content of
64+
the ring buffer will be all the events that lead up to the crash.
65+
66+
There are several kernel command line parameters that can be used to help in
67+
this. The first is "ftrace_dump_on_oops". This will dump the tracing ring
68+
buffer when a oops occurs to the console. This can be useful if the console
69+
is being logged somewhere. If a serial console is used, it may be prudent to
70+
make sure the ring buffer is relatively small, otherwise the dumping of the
71+
ring buffer may take several minutes to hours to finish. Here's an example
72+
of the kernel command line::
73+
74+
ftrace_dump_on_oops trace_buf_size=50K
75+
76+
Note, the tracing buffer is made up of per CPU buffers where each of these
77+
buffers is broken up into sub-buffers that are by default PAGE_SIZE. The
78+
above trace_buf_size option above sets each of the per CPU buffers to 50K,
79+
so, on a machine with 8 CPUs, that's actually 400K total.
80+
81+
Persistent buffers across boots
82+
-------------------------------
83+
If the system memory allows it, the tracing ring buffer can be specified at
84+
a specific location in memory. If the location is the same across boots and
85+
the memory is not modified, the tracing buffer can be retrieved from the
86+
following boot. There's two ways to reserve memory for the use of the ring
87+
buffer.
88+
89+
The more reliable way (on x86) is to reserve memory with the "memmap" kernel
90+
command line option and then use that memory for the trace_instance. This
91+
requires a bit of knowledge of the physical memory layout of the system. The
92+
advantage of using this method, is that the memory for the ring buffer will
93+
always be the same::
94+
95+
memmap==12M$0x284500000 trace_instance=boot_map@0x284500000:12M
96+
97+
The memmap above reserves 12 megabytes of memory at the physical memory
98+
location 0x284500000. Then the trace_instance option will create a trace
99+
instance "boot_map" at that same location with the same amount of memory
100+
reserved. As the ring buffer is broke up into per CPU buffers, the 12
101+
megabytes will be broken up evenly between those CPUs. If you have 8 CPUs,
102+
each per CPU ring buffer will be 1.5 megabytes in size. Note, that also
103+
includes meta data, so the amount of memory actually used by the ring buffer
104+
will be slightly smaller.
105+
106+
Another more generic but less robust way to allocate a ring buffer mapping
107+
at boot is with the "reserve_mem" option::
108+
109+
reserve_mem=12M:4096:trace trace_instance=boot_map@trace
110+
111+
The reserve_mem option above will find 12 megabytes that are available at
112+
boot up, and align it by 4096 bytes. It will label this memory as "trace"
113+
that can be used by later command line options.
114+
115+
The trace_instance option creates a "boot_map" instance and will use the
116+
memory reserved by reserve_mem that was labeled as "trace". This method is
117+
more generic but may not be as reliable. Due to KASLR, the memory reserved
118+
by reserve_mem may not be located at the same location. If this happens,
119+
then the ring buffer will not be from the previous boot and will be reset.
120+
121+
Sometimes, by using a larger alignment, it can keep KASLR from moving things
122+
around in such a way that it will move the location of the reserve_mem. By
123+
using a larger alignment, you may find better that the buffer is more
124+
consistent to where it is placed::
125+
126+
reserve_mem=12M:0x2000000:trace trace_instance=boot_map@trace
127+
128+
On boot up, the memory reserved for the ring buffer is validated. It will go
129+
through a series of tests to make sure that the ring buffer contains valid
130+
data. If it is, it will then set it up to be available to read from the
131+
instance. If it fails any of the tests, it will clear the entire ring buffer
132+
and initialize it as new.
133+
134+
The layout of this mapped memory may not be consistent from kernel to
135+
kernel, so only the same kernel is guaranteed to work if the mapping is
136+
preserved. Switching to a different kernel version may find a different
137+
layout and mark the buffer as invalid.
138+
139+
Using trace_printk() in the boot instance
140+
-----------------------------------------
141+
By default, the content of trace_printk() goes into the top level tracing
142+
instance. But this instance is never preserved across boots. To have the
143+
trace_printk() content, and some other internal tracing go to the preserved
144+
buffer (like dump stacks), either set the instance to be the trace_printk()
145+
destination from the kernel command line, or set it after boot up via the
146+
trace_printk_dest option.
147+
148+
After boot up::
149+
150+
echo 1 > /sys/kernel/tracing/instances/boot_map/options/trace_printk_dest
151+
152+
From the kernel command line::
153+
154+
reserve_mem=12M:4096:trace trace_instance=boot_map^traceprintk^traceoff@trace
155+
156+
If setting it from the kernel command line, it is recommended to also
157+
disable tracing with the "traceoff" flag, and enable tracing after boot up.
158+
Otherwise the trace from the most recent boot will be mixed with the trace
159+
from the previous boot, and may make it confusing to read.

‎Documentation/trace/ftrace.rst

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1186,6 +1186,18 @@ Here are the available options:
11861186
trace_printk
11871187
Can disable trace_printk() from writing into the buffer.
11881188

1189+
trace_printk_dest
1190+
Set to have trace_printk() and similar internal tracing functions
1191+
write into this instance. Note, only one trace instance can have
1192+
this set. By setting this flag, it clears the trace_printk_dest flag
1193+
of the instance that had it set previously. By default, the top
1194+
level trace has this set, and will get it set again if another
1195+
instance has it set then clears it.
1196+
1197+
This flag cannot be cleared by the top level instance, as it is the
1198+
default instance. The only way the top level instance has this flag
1199+
cleared, is by it being set in another instance.
1200+
11891201
annotate
11901202
It is sometimes confusing when the CPU buffers are full
11911203
and one CPU buffer had a lot of events recently, thus

‎include/linux/ring_buffer.h

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -89,6 +89,14 @@ void ring_buffer_discard_commit(struct trace_buffer *buffer,
8989
struct trace_buffer *
9090
__ring_buffer_alloc(unsigned long size, unsigned flags, struct lock_class_key *key);
9191

92+
struct trace_buffer *__ring_buffer_alloc_range(unsigned long size, unsigned flags,
93+
int order, unsigned long start,
94+
unsigned long range_size,
95+
struct lock_class_key *key);
96+
97+
bool ring_buffer_last_boot_delta(struct trace_buffer *buffer, long *text,
98+
long *data);
99+
92100
/*
93101
* Because the ring buffer is generic, if other users of the ring buffer get
94102
* traced by ftrace, it can produce lockdep warnings. We need to keep each
@@ -100,6 +108,18 @@ __ring_buffer_alloc(unsigned long size, unsigned flags, struct lock_class_key *k
100108
__ring_buffer_alloc((size), (flags), &__key); \
101109
})
102110

111+
/*
112+
* Because the ring buffer is generic, if other users of the ring buffer get
113+
* traced by ftrace, it can produce lockdep warnings. We need to keep each
114+
* ring buffer's lock class separate.
115+
*/
116+
#define ring_buffer_alloc_range(size, flags, order, start, range_size) \
117+
({ \
118+
static struct lock_class_key __key; \
119+
__ring_buffer_alloc_range((size), (flags), (order), (start), \
120+
(range_size), &__key); \
121+
})
122+
103123
typedef bool (*ring_buffer_cond_fn)(void *data);
104124
int ring_buffer_wait(struct trace_buffer *buffer, int cpu, int full,
105125
ring_buffer_cond_fn cond, void *data);

0 commit comments

Comments
 (0)