From 637d6c2722a81ad8af1f1eecf39df35bddef3487 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Sat, 15 Nov 2025 19:44:13 +0000 Subject: [PATCH 1/4] Initial plan From 268520402bcf3dd1b9d50ee036a39c0490a12f61 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Sat, 15 Nov 2025 19:52:01 +0000 Subject: [PATCH 2/4] Add stack high watermark tracking to thread analyzer Co-authored-by: kartben <128251+kartben@users.noreply.github.com> --- doc/services/debugging/thread-analyzer.rst | 55 ++++++++++- include/zephyr/debug/thread_analyzer.h | 91 +++++++++++++++++-- include/zephyr/kernel/thread.h | 10 ++ kernel/thread.c | 7 ++ subsys/debug/thread_analyzer/Kconfig | 27 ++++++ .../debug/thread_analyzer/thread_analyzer.c | 51 +++++++++++ 6 files changed, 233 insertions(+), 8 deletions(-) diff --git a/doc/services/debugging/thread-analyzer.rst b/doc/services/debugging/thread-analyzer.rst index d6ff834d07c0f..fa951883f9187 100644 --- a/doc/services/debugging/thread-analyzer.rst +++ b/doc/services/debugging/thread-analyzer.rst @@ -10,6 +10,50 @@ runtime statistics. The analysis is performed on demand when the application calls :c:func:`thread_analyzer_run` or :c:func:`thread_analyzer_print`. +Stack painting and high watermark tracking +******************************************* + +Stack painting +============== + +Stack painting is a debugging technique where thread stacks are initialized +with a known pattern (0xaa) at thread creation time. This is enabled via the +:kconfig:option:`CONFIG_INIT_STACKS` option, which is automatically selected +when :kconfig:option:`CONFIG_THREAD_ANALYZER` is enabled. + +The thread analyzer uses this known pattern to detect stack usage by scanning +from the bottom of the stack upward until it encounters memory that has been +modified. This allows accurate measurement of: + +* Current stack usage (bytes of stack that have been written to) +* Unused stack space (bytes still containing the 0xaa pattern) +* Stack utilization percentage + +High watermark tracking +======================== + +When :kconfig:option:`CONFIG_THREAD_ANALYZER_STACK_HIGH_WATERMARK` is enabled, +the thread analyzer tracks the maximum stack usage observed for each thread +since its creation. This feature provides: + +* Peak stack usage detection across the thread's lifetime +* Worst-case stack usage patterns +* Historical view of stack utilization +* Better insight for stack size optimization + +The high watermark is stored in the thread's stack_info structure and is +updated automatically each time the thread analyzer runs. This allows +developers to identify threads that may need larger stacks or threads +with stacks that can be safely reduced. + +Benefits of high watermark tracking: + +* **Optimization**: Identify threads with oversized stacks that can be reduced +* **Safety**: Ensure threads have adequate stack margins for worst-case scenarios +* **Debugging**: Detect intermittent stack-intensive operations that might be + missed with single-point measurements +* **Profiling**: Understand stack growth patterns over time + For example, to build the synchronization sample with Thread Analyser enabled, do the following: @@ -68,7 +112,16 @@ Configuration Configure this module using the following options. :kconfig:option:`CONFIG_THREAD_ANALYZER` - Enable the module. + Enable the module. This automatically selects :kconfig:option:`CONFIG_INIT_STACKS` + to enable stack painting for accurate stack usage measurement. +:kconfig:option:`CONFIG_INIT_STACKS` + Initialize stack areas with a known value (0xaa) for stack painting. This is + automatically enabled by :kconfig:option:`CONFIG_THREAD_ANALYZER`. +:kconfig:option:`CONFIG_THREAD_ANALYZER_STACK_HIGH_WATERMARK` + Enable tracking of stack high watermark (peak usage) for each thread. When + enabled, the thread analyzer maintains a record of the maximum stack usage + observed for each thread, allowing detection of worst-case stack requirements. + Default: enabled. :kconfig:option:`CONFIG_THREAD_ANALYZER_USE_PRINTK` Use printk for thread statistics. :kconfig:option:`CONFIG_THREAD_ANALYZER_USE_LOG` diff --git a/include/zephyr/debug/thread_analyzer.h b/include/zephyr/debug/thread_analyzer.h index c89607ab6a282..0440247d67b63 100644 --- a/include/zephyr/debug/thread_analyzer.h +++ b/include/zephyr/debug/thread_analyzer.h @@ -16,23 +16,100 @@ extern "C" { /** @defgroup thread_analyzer Thread analyzer * @ingroup debug - * @brief Module for analyzing threads + * @brief Module for analyzing thread stack usage and performance + * + * The thread analyzer provides comprehensive runtime analysis of thread + * behavior, including stack usage, CPU utilization, and performance metrics. + * + * ## Stack Painting + * + * Stack painting is a technique where thread stacks are initialized with a + * known pattern (0xaa) when CONFIG_INIT_STACKS is enabled. This allows the + * thread analyzer to detect stack usage by scanning from the bottom of the + * stack until it finds memory that has been modified. This mechanism provides + * accurate measurement of: + * - Current stack usage (how much stack is currently in use) + * - Unused stack space (pristine stack area with 0xaa pattern) + * + * ## High Watermark Tracking + * + * When CONFIG_THREAD_ANALYZER_STACK_HIGH_WATERMARK is enabled, the analyzer + * tracks the maximum stack usage observed for each thread since creation. + * This feature helps: + * - Identify worst-case stack usage patterns + * - Optimize stack size configurations + * - Detect intermittent stack-intensive operations + * - Ensure adequate stack margins for safety + * + * The high watermark is stored in the thread's stack_info structure and + * persists across multiple analyzer runs, providing a historical view of + * peak stack utilization. + * + * ## Usage + * + * To use the thread analyzer: + * 1. Enable CONFIG_THREAD_ANALYZER in your project configuration + * 2. Enable CONFIG_INIT_STACKS for stack painting support + * 3. Enable CONFIG_THREAD_ANALYZER_STACK_HIGH_WATERMARK for peak usage tracking + * 4. Call thread_analyzer_print() or thread_analyzer_run() to analyze threads + * + * Alternatively, enable CONFIG_THREAD_ANALYZER_AUTO to run analysis + * automatically at periodic intervals. * - * This module implements functions and the configuration that simplifies - * thread analysis. * @{ */ +/** + * @brief Thread analyzer information structure + * + * Contains detailed information about a thread's stack usage and + * performance characteristics. This structure is populated by the + * thread analyzer and passed to callback functions. + */ struct thread_analyzer_info { - /** The name of the thread or stringified address of the thread handle - * if name is not set. + /** Thread name or stringified thread handle address. + * + * If the thread has a name set via k_thread_name_set(), this field + * contains that name. Otherwise, it contains the hexadecimal string + * representation of the thread structure pointer. */ const char *name; - /** The total size of the stack*/ + + /** Total stack size in bytes. + * + * Represents the total size of the stack buffer allocated for this + * thread, including any reserved areas and adjustments. + */ size_t stack_size; - /** Stack size in used */ + + /** Current stack usage in bytes. + * + * The amount of stack space currently used by the thread. This is + * calculated by scanning the stack for the initialized pattern (0xaa) + * when CONFIG_INIT_STACKS is enabled. The value represents the number + * of bytes that have been modified since thread creation. + */ size_t stack_used; +#ifdef CONFIG_THREAD_ANALYZER_STACK_HIGH_WATERMARK + /** High watermark - maximum stack usage in bytes. + * + * Tracks the peak stack utilization observed since thread creation. + * This field is only available when CONFIG_THREAD_ANALYZER_STACK_HIGH_WATERMARK + * is enabled. The high watermark is updated each time the thread analyzer + * runs if the current usage exceeds the previously recorded maximum. + * + * Use cases: + * - Identifying worst-case stack requirements + * - Optimizing stack size allocations + * - Detecting stack growth patterns over time + * - Ensuring adequate safety margins + * + * Note: Requires CONFIG_INIT_STACKS to be enabled for accurate measurement. + */ + size_t stack_high_watermark; +#endif + #ifdef CONFIG_THREAD_RUNTIME_STATS unsigned int utilization; #ifdef CONFIG_SCHED_THREAD_USAGE diff --git a/include/zephyr/kernel/thread.h b/include/zephyr/kernel/thread.h index db7713d6d407d..c51a6691de322 100644 --- a/include/zephyr/kernel/thread.h +++ b/include/zephyr/kernel/thread.h @@ -171,6 +171,16 @@ struct _thread_stack_info { size_t sz; } mapped; #endif /* CONFIG_THREAD_STACK_MEM_MAPPED */ + +#if defined(CONFIG_THREAD_ANALYZER_STACK_HIGH_WATERMARK) + /* High watermark - maximum stack usage observed since thread creation. + * This field tracks the peak stack utilization over time. When + * CONFIG_INIT_STACKS is enabled, the stack is initialized with a known + * pattern (0xaa), enabling accurate high watermark detection through + * the thread analyzer. + */ + size_t high_watermark; +#endif /* CONFIG_THREAD_ANALYZER_STACK_HIGH_WATERMARK */ }; typedef struct _thread_stack_info _thread_stack_info_t; diff --git a/kernel/thread.c b/kernel/thread.c index 6d603564df927..b806e6371927b 100644 --- a/kernel/thread.c +++ b/kernel/thread.c @@ -495,6 +495,13 @@ static char *setup_thread_stack(struct k_thread *new_thread, new_thread->stack_info.start = (uintptr_t)stack_buf_start; new_thread->stack_info.size = stack_buf_size; new_thread->stack_info.delta = delta; + +#ifdef CONFIG_THREAD_ANALYZER_STACK_HIGH_WATERMARK + /* Initialize high watermark to 0. The thread analyzer will update this + * value as it tracks stack usage over time. + */ + new_thread->stack_info.high_watermark = 0; +#endif /* CONFIG_THREAD_ANALYZER_STACK_HIGH_WATERMARK */ #endif /* CONFIG_THREAD_STACK_INFO */ stack_ptr -= delta; diff --git a/subsys/debug/thread_analyzer/Kconfig b/subsys/debug/thread_analyzer/Kconfig index a3937ade25edd..c30120527c70a 100644 --- a/subsys/debug/thread_analyzer/Kconfig +++ b/subsys/debug/thread_analyzer/Kconfig @@ -16,6 +16,12 @@ menuconfig THREAD_ANALYZER stack size configuration to find stack overflow or to find stacks which may be optimized. + The thread analyzer automatically enables INIT_STACKS, which implements + stack painting - a technique where stacks are initialized with a known + pattern (0xaa). This allows the analyzer to detect stack usage by + scanning for memory that has been modified, providing accurate + measurements of current and peak stack utilization. + if THREAD_ANALYZER module = THREAD_ANALYZER module-str = thread analyzer @@ -49,6 +55,27 @@ config THREAD_ANALYZER_PRIV_STACK_USAGE help Print privileged stack usage for user threads. +config THREAD_ANALYZER_STACK_HIGH_WATERMARK + bool "Track stack high watermark" + default y + help + Enable tracking of stack high watermark (peak usage) for each thread. + When enabled, the thread analyzer will track and display the maximum + stack usage observed since thread creation or last reset. This feature + requires CONFIG_INIT_STACKS to function properly, as it relies on the + stack painting mechanism (initializing stack memory with 0xaa pattern) + to detect stack usage. + + High watermark tracking is useful for: + - Understanding worst-case stack usage patterns + - Optimizing stack size configurations + - Identifying threads that may need larger stacks + - Debugging intermittent stack-related issues + + Note: The high watermark is stored in the thread structure and is + maintained across multiple thread analyzer runs, allowing you to + observe peak usage over the lifetime of the thread. + config THREAD_ANALYZER_RUN_UNLOCKED bool "Run analysis with interrupts unlocked" default y diff --git a/subsys/debug/thread_analyzer/thread_analyzer.c b/subsys/debug/thread_analyzer/thread_analyzer.c index 8f136034f8fe1..4ff234457f90d 100644 --- a/subsys/debug/thread_analyzer/thread_analyzer.c +++ b/subsys/debug/thread_analyzer/thread_analyzer.c @@ -6,6 +6,29 @@ /** @file * @brief Thread analyzer implementation + * + * The thread analyzer provides runtime analysis of thread stack usage and + * performance metrics. It enables developers to optimize stack sizes and + * understand thread behavior. + * + * Key features: + * - Stack usage tracking via stack painting (CONFIG_INIT_STACKS) + * - High watermark tracking for peak stack usage detection + * - CPU utilization statistics per thread + * - ISR stack usage analysis + * - Privilege stack usage tracking (for user mode threads) + * + * Stack Painting: + * When CONFIG_INIT_STACKS is enabled, thread stacks are initialized with + * a known pattern (0xaa). The thread analyzer scans the stack from the + * bottom to detect how much of the stack has been modified (used), allowing + * accurate measurement of stack usage. + * + * High Watermark Tracking: + * When CONFIG_THREAD_ANALYZER_STACK_HIGH_WATERMARK is enabled, the thread + * analyzer maintains a record of the maximum stack usage observed for each + * thread since its creation. This helps identify worst-case stack usage + * patterns that may not be visible during typical operation. */ #include @@ -56,6 +79,16 @@ static void thread_print_cb(struct thread_analyzer_info *info) info->stack_size, pcnt, info->utilization); +#ifdef CONFIG_THREAD_ANALYZER_STACK_HIGH_WATERMARK + /* Display high watermark (peak stack usage) */ + size_t hwm_pcnt = (info->stack_high_watermark * 100U) / info->stack_size; + + THREAD_ANALYZER_PRINT( + THREAD_ANALYZER_FMT( + " %-20s: High watermark: %zu / %zu (%zu %%)"), + " ", info->stack_high_watermark, info->stack_size, hwm_pcnt); +#endif + #ifdef CONFIG_THREAD_ANALYZER_PRIV_STACK_USAGE if (info->priv_stack_size > 0) { pcnt = (info->priv_stack_used * 100U) / info->priv_stack_size; @@ -89,6 +122,16 @@ static void thread_print_cb(struct thread_analyzer_info *info) THREAD_ANALYZER_VSTR(info->name), info->stack_size - info->stack_used, info->stack_used, info->stack_size, pcnt); + +#ifdef CONFIG_THREAD_ANALYZER_STACK_HIGH_WATERMARK + /* Display high watermark (peak stack usage) */ + size_t hwm_pcnt = (info->stack_high_watermark * 100U) / info->stack_size; + + THREAD_ANALYZER_PRINT( + THREAD_ANALYZER_FMT( + " %-20s: High watermark: %zu / %zu (%zu %%)"), + " ", info->stack_high_watermark, info->stack_size, hwm_pcnt); +#endif #endif } @@ -134,6 +177,14 @@ static void thread_analyze_cb(const struct k_thread *cthread, void *user_data) info.stack_size = size; info.stack_used = size - unused; +#ifdef CONFIG_THREAD_ANALYZER_STACK_HIGH_WATERMARK + /* Update high watermark if current usage exceeds it */ + if (info.stack_used > thread->stack_info.high_watermark) { + thread->stack_info.high_watermark = info.stack_used; + } + info.stack_high_watermark = thread->stack_info.high_watermark; +#endif + #ifdef CONFIG_THREAD_ANALYZER_PRIV_STACK_USAGE ret = arch_thread_priv_stack_space_get(cthread, &size, &unused); if (ret == 0) { From 5b3d26d0640d9e0cfc68ed1075eb1ffafbbdf856 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Sat, 15 Nov 2025 19:54:37 +0000 Subject: [PATCH 3/4] Add tests for stack high watermark feature Co-authored-by: kartben <128251+kartben@users.noreply.github.com> --- tests/subsys/debug/thread_analyzer/prj.conf | 1 + tests/subsys/debug/thread_analyzer/testcase.yaml | 11 +++++++++++ 2 files changed, 12 insertions(+) diff --git a/tests/subsys/debug/thread_analyzer/prj.conf b/tests/subsys/debug/thread_analyzer/prj.conf index c3dd979fc061a..09b8b96de2d6d 100644 --- a/tests/subsys/debug/thread_analyzer/prj.conf +++ b/tests/subsys/debug/thread_analyzer/prj.conf @@ -1,5 +1,6 @@ CONFIG_THREAD_ANALYZER=y CONFIG_THREAD_ANALYZER_AUTO=y +CONFIG_THREAD_ANALYZER_STACK_HIGH_WATERMARK=y # log immediate mode requires a bit more stack to print CONFIG_THREAD_ANALYZER_AUTO_STACK_SIZE=2048 diff --git a/tests/subsys/debug/thread_analyzer/testcase.yaml b/tests/subsys/debug/thread_analyzer/testcase.yaml index 9ae49081889cd..32018206f7f0f 100644 --- a/tests/subsys/debug/thread_analyzer/testcase.yaml +++ b/tests/subsys/debug/thread_analyzer/testcase.yaml @@ -80,3 +80,14 @@ tests: regex: - "(.*)0x([0-9a-fA-F]+)([ ]+) : STACK: unused [0-9]+ usage [0-9]+ / [0-9]+ (.*)" - "(.*)ISR0([ ]+) : STACK: unused [0-9]+ usage [0-9]+ / [0-9]+ (.*)" + debug.thread_analyzer.high_watermark: + extra_configs: + - CONFIG_THREAD_ANALYZER_USE_PRINTK=y + - CONFIG_THREAD_ANALYZER_STACK_HIGH_WATERMARK=y + harness: console + harness_config: + type: multi_line + regex: + - "(.*)0x([0-9a-fA-F]+)([ ]+) : STACK: unused [0-9]+ usage [0-9]+ / [0-9]+ (.*)" + - "(.*)High watermark: [0-9]+ / [0-9]+ (.*)" + - "(.*)ISR0([ ]+) : STACK: unused [0-9]+ usage [0-9]+ / [0-9]+ (.*)" From 3071efc59b74c0ff670e6b75533ed679ec9e0158 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Sat, 15 Nov 2025 19:58:28 +0000 Subject: [PATCH 4/4] Add example output for high watermark feature in documentation Co-authored-by: kartben <128251+kartben@users.noreply.github.com> --- doc/services/debugging/thread-analyzer.rst | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/doc/services/debugging/thread-analyzer.rst b/doc/services/debugging/thread-analyzer.rst index fa951883f9187..25871effaf993 100644 --- a/doc/services/debugging/thread-analyzer.rst +++ b/doc/services/debugging/thread-analyzer.rst @@ -106,6 +106,26 @@ information from Thread Analyzer:: thread_a: Hello World from cpu 0 on qemu_x86! thread_b: Hello World from cpu 0 on qemu_x86! +Example output with high watermark tracking +============================================ + +When :kconfig:option:`CONFIG_THREAD_ANALYZER_STACK_HIGH_WATERMARK` is enabled +(default), the thread analyzer will also display the high watermark (peak stack +usage) for each thread:: + + Thread analyze: + thread_b : STACK: unused 648 usage 376 / 1024 (36 %); CPU: 7 % + : High watermark: 412 / 1024 (40 %) + thread_analyzer : STACK: unused 8 usage 504 / 512 (98 %); CPU: 0 % + : High watermark: 508 / 512 (99 %) + thread_a : STACK: unused 648 usage 376 / 1024 (36 %); CPU: 9 % + : High watermark: 428 / 1024 (41 %) + idle : STACK: unused 204 usage 116 / 320 (36 %); CPU: 82 % + : High watermark: 128 / 320 (40 %) + +In this example, you can see that while thread_b's current usage is 376 bytes, +it has peaked at 412 bytes at some point since creation. This information helps +identify whether threads need more stack space or if they can be safely reduced. Configuration *************