Skip to content

Commit 9b2964b

Browse files
simensrostadjorgenmk
authored andcommitted
modules: memfault: Add support for uploading modem traces to Memfault
Add support for uploading modem traces to Memfault upon a coredump. This is useful to get the full picture of what happened leading up to a crash. The feature is optional and guarded by a Kconfig option. It is enabled by default in the overlay `overlay-modem_trace_to_memfault.conf` added to the memfault sample. Tests and docs has been updated accordingly. Signed-off-by: Simen S. Røstad <[email protected]>
1 parent 50bf382 commit 9b2964b

File tree

14 files changed

+349
-1
lines changed

14 files changed

+349
-1
lines changed

doc/nrf/libraries/debug/memfault_ncs.rst

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -72,6 +72,21 @@ This feature is useful when you want to post the coredump as soon as possible af
7272
Alternatively, you can manually trigger the coredump upload by calling the :c:func:`memfault_zephyr_port_post_data` function.
7373
You can use the :c:func:`memfault_coredump_has_valid_coredump` function to check whether a coredump is available.
7474

75+
Automatic sending of modem trace to Memfault
76+
=============================================
77+
78+
.. memfault_coredump_send_start
79+
80+
To post modem traces to Memfault together with coredumps, set the :kconfig:option:`CONFIG_MEMFAULT_NCS_POST_MODEM_TRACE_ON_COREDUMP` Kconfig option to ``y``.
81+
This option is only supported for nRF91 Series devices.
82+
83+
The modem trace will be posted as a CDR (Custom Data Record) to Memfault and depends on the :ref:`modem_trace_module` being configured to use a trace backend that persists traces across reboots.
84+
Depending on the Memfault account type you use, different rate limits are imposed on the number of CDRs that can be posted.
85+
During development you can enable Developer Mode for selected devices, allowing them to post more CDRs.
86+
For more information, see the `Memfault Server-Side Developer Mode`_.
87+
88+
.. memfault_coredump_send_end
89+
7590
Configuration options in Memfault SDK
7691
=====================================
7792

@@ -107,6 +122,7 @@ Configuration options in |NCS|
107122
The Kconfig options for Memfault that are defined in |NCS| provide some additional features compared to the options that are already implemented in Memfault SDK:
108123

109124
* :kconfig:option:`CONFIG_MEMFAULT_NCS_POST_COREDUMP_ON_NETWORK_CONNECTED`
125+
* :kconfig:option:`CONFIG_MEMFAULT_NCS_POST_MODEM_TRACE_ON_COREDUMP`
110126
* :kconfig:option:`CONFIG_MEMFAULT_NCS_PROJECT_KEY`
111127
* :kconfig:option:`CONFIG_MEMFAULT_NCS_PROVISION_CERTIFICATES`
112128
* :kconfig:option:`CONFIG_MEMFAULT_NCS_INTERNAL_FLASH_BACKED_COREDUMP`

doc/nrf/links.txt

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1355,6 +1355,7 @@
13551355
.. _`Memfault: Collecting Device Metrics`: https://docs.memfault.com/docs/embedded/metrics-api
13561356
.. _`Memfault: Error Tracking with Trace Events`: https://docs.memfault.com/docs/embedded/trace-events/
13571357
.. _`Memfault Demo CLI`: https://docs.memfault.com/docs/mcu/demo-cli/
1358+
.. _`Memfault Server-Side Developer Mode`: https://docs.memfault.com/docs/platform/rate-limiting#server-side-developer-mode
13581359

13591360
.. _`Memfault debugging`: https://docs.memfault.com/docs/platform/introduction#debugging
13601361
.. _`Memfault OTA`: https://docs.memfault.com/docs/platform/introduction#ota

doc/nrf/releases_and_maturity/releases/release-notes-changelog.rst

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1107,6 +1107,10 @@ Memfault integration
11071107
To enable this feature, set the :kconfig:option:`CONFIG_MEMFAULT_NCS_POST_COREDUMP_ON_NETWORK_CONNECTED` Kconfig option to ``y``.
11081108
Only supported for nRF91 Series devices.
11091109

1110+
* Added a new feature to automatically capture and upload modem traces to Memfault with coredumps upon a crash.
1111+
To enable this feature, set the :kconfig:option:`CONFIG_MEMFAULT_NCS_POST_MODEM_TRACE_ON_COREDUMP` Kconfig option to ``y``.
1112+
Only supported for nRF91 Series devices.
1113+
11101114
AVSystem integration
11111115
--------------------
11121116

lib/nrf_modem_lib/shell/trace.c

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -146,7 +146,10 @@ static void modem_trace_dump_uart(const struct shell *sh, size_t argc, char **ar
146146
return;
147147
}
148148

149+
#ifdef CONFIG_NRF_MODEM_LIB_SHELL_TRACE_UART
149150
modem_trace_uart_send(sh, size);
151+
#endif /* CONFIG_NRF_MODEM_LIB_SHELL_TRACE_UART */
152+
150153
} else {
151154
shell_error(sh, "Missing chosen node for nordic,modem-trace-uart. "
152155
"Please configure which uart to use.");

modules/memfault-firmware-sdk/CMakeLists.txt

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,10 @@ zephyr_library_sources_ifdef(
3535
CONFIG_MEMFAULT_NCS_POST_COREDUMP_ON_NETWORK_CONNECTED
3636
memfault_lte_coredump.c)
3737

38+
zephyr_library_sources_ifdef(
39+
CONFIG_MEMFAULT_NCS_POST_MODEM_TRACE_ON_COREDUMP
40+
memfault_lte_coredump_modem_trace.c)
41+
3842
zephyr_library_sources_ifdef(
3943
CONFIG_MEMFAULT_NCS_ETB_CAPTURE
4044
memfault_etb_trace_capture.c)

modules/memfault-firmware-sdk/Kconfig

Lines changed: 14 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -191,7 +191,7 @@ config MEMFAULT_NCS_IMPLEMENT_METRICS_COLLECTION
191191
Implement the Memfault 'memfault_metrics_heartbeat_collect_data()' function
192192
for the selected metrics. Disable this to override the implementation.
193193

194-
config MEMFAULT_NCS_POST_COREDUMP_ON_NETWORK_CONNECTED
194+
menuconfig MEMFAULT_NCS_POST_COREDUMP_ON_NETWORK_CONNECTED
195195
bool "Post coredump on network connected"
196196
depends on PDN
197197
depends on LTE_LINK_CONTROL
@@ -203,6 +203,17 @@ config MEMFAULT_NCS_POST_COREDUMP_ON_NETWORK_CONNECTED
203203
Post coredump to Memfault when the device is connected to LTE network.
204204
This option is only supported for nRF91 targets.
205205

206+
if MEMFAULT_NCS_POST_COREDUMP_ON_NETWORK_CONNECTED
207+
208+
config MEMFAULT_NCS_POST_MODEM_TRACE_ON_COREDUMP
209+
bool "Post modem trace on coredump"
210+
select MEMFAULT_CDR_ENABLE
211+
depends on NRF_MODEM_LIB_TRACE
212+
depends on NRF_MODEM_LIB_TRACE_BACKEND_FLASH || NRF_MODEM_LIB_TRACE_BACKEND_RAM
213+
help
214+
Capture modem traces continuously to flash or RAM (no init RAM), and post the traces
215+
together with the coredump to Memfault in the event of a crash.
216+
206217
config MEMFAULT_NCS_POST_COREDUMP_AFTER_INITIAL_DELAY
207218
bool "Post coredump after initial delay"
208219
default y
@@ -228,6 +239,8 @@ config MEMFAULT_NCS_POST_COREDUMP_THREAD_STACK_SIZE
228239
int "Post coredump thread size"
229240
default 512
230241

242+
endif # MEMFAULT_NCS_POST_COREDUMP_ON_NETWORK_CONNECTED
243+
231244
config MEMFAULT_NCS_ETB_CAPTURE
232245
bool "Enable ETB trace capture"
233246
depends on ETB_TRACE
Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
/*
2+
* Copyright (c) 2025 Nordic Semiconductor ASA
3+
*
4+
* SPDX-License-Identifier: LicenseRef-Nordic-5-Clause
5+
*/
6+
7+
8+
/**
9+
* @brief Memfault LTE coredump modem trace initialization. Shall only be called once.
10+
* Ths function sets up a CDR (Custom Data Recording) source for modem traces.
11+
*
12+
* @retval -EALREADY if the module has already been initialized.
13+
* @retval -EIO if the modem trace CDR source registration failed.
14+
* @retval 0 if the modem trace backend was successfully initialized.
15+
*/
16+
int memfault_lte_coredump_modem_trace_init(void);
17+
18+
/**
19+
* @brief Prepare modem trace data for upload
20+
*
21+
* @retval -ENOTSUP if the current modem trace backend is not supported.
22+
* @retval -ENODATA if there are no modem traces to send.
23+
* @retval 0 if the modem trace data is ready for upload.
24+
*
25+
* @returns Negative error if an error occurred preparing the modem trace.
26+
*/
27+
int memfault_lte_coredump_modem_trace_prepare_for_upload(void);

modules/memfault-firmware-sdk/memfault_lte_coredump.c

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,8 @@
1818
#include <modem/pdn.h>
1919
#include <nrf_modem_at.h>
2020

21+
#include "memfault_lte_coredump_modem_trace.h"
22+
2123
LOG_MODULE_DECLARE(memfault_ncs, CONFIG_MEMFAULT_NCS_LOG_LEVEL);
2224

2325
NRF_MODEM_LIB_ON_INIT(memfault_ncs_lte_coredump_init_hook, on_modem_lib_init, NULL);
@@ -372,6 +374,21 @@ static void state_coredump_send_attempt_entry(void *o)
372374

373375
memfault_metrics_heartbeat_debug_trigger();
374376

377+
if (IS_ENABLED(CONFIG_MEMFAULT_NCS_POST_MODEM_TRACE_ON_COREDUMP)) {
378+
err = memfault_lte_coredump_modem_trace_init();
379+
if (err == -EIO) {
380+
LOG_ERR("memfault_lte_coredump_modem_trace_init, error: %d", err);
381+
__ASSERT_NO_MSG(false);
382+
return;
383+
}
384+
385+
err = memfault_lte_coredump_modem_trace_prepare_for_upload();
386+
if (err == -ENOTSUP) {
387+
__ASSERT_NO_MSG(false);
388+
return;
389+
}
390+
}
391+
375392
err = memfault_zephyr_port_post_data();
376393
if (err) {
377394
LOG_DBG("Failed to post data to Memfault");
Lines changed: 184 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,184 @@
1+
/*
2+
* Copyright (c) 2025 Nordic Semiconductor ASA
3+
*
4+
* SPDX-License-Identifier: LicenseRef-Nordic-5-Clause
5+
*/
6+
7+
#include <zephyr/kernel.h>
8+
#include <zephyr/logging/log.h>
9+
#include <zephyr/task_wdt/task_wdt.h>
10+
#include <zephyr/zbus/zbus.h>
11+
#include "memfault/components.h"
12+
#include <memfault/ports/zephyr/http.h>
13+
#include <memfault/metrics/metrics.h>
14+
#include <memfault/core/data_packetizer.h>
15+
#include <memfault/core/trace_event.h>
16+
#include "memfault/panics/coredump.h"
17+
#include <modem/nrf_modem_lib_trace.h>
18+
#include <modem/nrf_modem_lib.h>
19+
20+
LOG_MODULE_DECLARE(memfault_ncs, CONFIG_MEMFAULT_NCS_LOG_LEVEL);
21+
22+
/* Forward declarations */
23+
static bool has_cdr_cb(sMemfaultCdrMetadata *metadata);
24+
static void mark_cdr_read_cb(void);
25+
static bool read_data_cb(uint32_t offset, void *buf, size_t buf_len);
26+
27+
static const char * const mimetypes[] = {
28+
MEMFAULT_CDR_BINARY
29+
};
30+
31+
static sMemfaultCdrMetadata trace_recording_metadata = {
32+
.start_time.type = kMemfaultCurrentTimeType_Unknown,
33+
.mimetypes = (char const **)mimetypes,
34+
.num_mimetypes = 1,
35+
.data_size_bytes = 0,
36+
.collection_reason = "modem traces",
37+
};
38+
39+
static sMemfaultCdrSourceImpl recording = {
40+
.has_cdr_cb = has_cdr_cb,
41+
.read_data_cb = read_data_cb,
42+
.mark_cdr_read_cb = mark_cdr_read_cb,
43+
};
44+
45+
static int modem_trace_enable(void)
46+
{
47+
int err;
48+
49+
err = nrf_modem_lib_trace_level_set(NRF_MODEM_LIB_TRACE_LEVEL_LTE_AND_IP);
50+
if (err) {
51+
LOG_ERR("nrf_modem_lib_trace_level_set, error: %d", err);
52+
return err;
53+
}
54+
55+
return 0;
56+
}
57+
58+
/* Enable modem traces as soon as the modem is initialized if there is no valid coredump.
59+
* Modem traces are disabled by default via CONFIG_NRF_MODEM_LIB_TRACE_LEVEL_OFF.
60+
*/
61+
NRF_MODEM_LIB_ON_INIT(memfault_init_hook, on_modem_lib_init, NULL)
62+
63+
static void on_modem_lib_init(int ret, void *ctx)
64+
{
65+
int err;
66+
67+
if (memfault_coredump_has_valid_coredump(NULL)) {
68+
return;
69+
}
70+
71+
LOG_DBG("No coredump valid, clearing modem trace buffer");
72+
73+
err = nrf_modem_lib_trace_clear();
74+
if (err) {
75+
LOG_ERR("Failed to clear modem trace data: %d", err);
76+
return;
77+
}
78+
79+
err = modem_trace_enable();
80+
if (err) {
81+
LOG_ERR("Failed to enable modem traces: %d", err);
82+
return;
83+
}
84+
}
85+
86+
static bool has_cdr_cb(sMemfaultCdrMetadata *metadata)
87+
{
88+
if (trace_recording_metadata.data_size_bytes == 0) {
89+
return false;
90+
}
91+
92+
*metadata = trace_recording_metadata;
93+
return true;
94+
}
95+
96+
static void mark_cdr_read_cb(void)
97+
{
98+
int err;
99+
100+
LOG_DBG("Modem trace data has been processed / uploaded");
101+
102+
/* It's not guaranteed that the trace backend clears the stored traces after readout,
103+
* so we clear them before enabling traces just in case.
104+
*/
105+
err = nrf_modem_lib_trace_clear();
106+
if (err) {
107+
LOG_ERR("Failed to clear modem trace data: %d", err);
108+
109+
/* Continue */
110+
}
111+
112+
err = modem_trace_enable();
113+
if (err) {
114+
LOG_ERR("Failed to enable modem traces after upload: %d", err);
115+
116+
/* Continue */
117+
}
118+
119+
/* Set CDR data length to 0 once the modem trace has been processed (uploaded) */
120+
trace_recording_metadata.data_size_bytes = 0;
121+
}
122+
123+
static bool read_data_cb(uint32_t offset, void *buf, size_t buf_len)
124+
{
125+
ARG_UNUSED(offset);
126+
127+
int err = nrf_modem_lib_trace_read(buf, buf_len);
128+
129+
if (err == -ENODATA) {
130+
LOG_WRN("No more modem trace data to read");
131+
return false;
132+
} else if (err < 0) {
133+
LOG_ERR("Failed to read modem trace data: %d", err);
134+
return false;
135+
}
136+
137+
return true;
138+
}
139+
140+
int memfault_lte_coredump_modem_trace_init(void)
141+
{
142+
static bool initialized;
143+
144+
if (initialized) {
145+
LOG_ERR("Already initialized");
146+
return -EALREADY;
147+
}
148+
149+
if (!memfault_cdr_register_source(&recording)) {
150+
LOG_ERR("Failed to register modem trace CDR source, storage full");
151+
return -EIO;
152+
}
153+
154+
initialized = true;
155+
156+
LOG_DBG("Modem trace CDR source initialized");
157+
158+
return 0;
159+
}
160+
161+
int memfault_lte_coredump_modem_trace_prepare_for_upload(void)
162+
{
163+
size_t trace_size = 0;
164+
165+
trace_size = nrf_modem_lib_trace_data_size();
166+
167+
if (trace_size == -ENOTSUP) {
168+
LOG_ERR("The current modem trace backend is not supported");
169+
return -ENOTSUP;
170+
} else if (trace_size < 0) {
171+
LOG_ERR("Failed to get modem trace size: %d", trace_size);
172+
return trace_size;
173+
} else if (trace_size == 0) {
174+
LOG_DBG("No modem traces to send");
175+
return -ENODATA;
176+
}
177+
178+
LOG_DBG("Preparing modem trace data upload of: %d bytes", trace_size);
179+
180+
trace_recording_metadata.duration_ms = 0;
181+
trace_recording_metadata.data_size_bytes = trace_size;
182+
183+
return 0;
184+
}

samples/debug/memfault/README.rst

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -147,6 +147,16 @@ If :kconfig:option:`CONFIG_MEMFAULT_NCS_INTERNAL_FLASH_BACKED_COREDUMP` is enabl
147147
:start-after: modem_lib_sending_traces_UART_start
148148
:end-before: modem_lib_sending_traces_UART_end
149149

150+
Automatic capture and upload modem traces with coredumps
151+
--------------------------------------------------------
152+
153+
You can configure the sample to upload modem traces to Memfault when an application coredump is triggered.
154+
To enable this feature, include the :file:`overlay-modem_trace_to_memfault.conf` overlay in your project.
155+
156+
.. include:: /libraries/debug/memfault_ncs.rst
157+
:start-after: memfault_coredump_send_start
158+
:end-before: memfault_coredump_send_end
159+
150160
Building and running
151161
********************
152162

0 commit comments

Comments
 (0)