Add stack high watermark tracking to thread analyzer

doc/services/debugging/thread-analyzer.rst

@@ -10,6 +10,50 @@ runtime statistics.
 The analysis is performed on demand when the application calls
 :c:func:`thread_analyzer_run` or :c:func:`thread_analyzer_print`.
 
+Stack painting and high watermark tracking
+*******************************************
+
+Stack painting
+==============
+
+Stack painting is a debugging technique where thread stacks are initialized
+with a known pattern (0xaa) at thread creation time. This is enabled via the
+:kconfig:option:`CONFIG_INIT_STACKS` option, which is automatically selected
+when :kconfig:option:`CONFIG_THREAD_ANALYZER` is enabled.
+
+The thread analyzer uses this known pattern to detect stack usage by scanning
+from the bottom of the stack upward until it encounters memory that has been
+modified. This allows accurate measurement of:
+
+* Current stack usage (bytes of stack that have been written to)
+* Unused stack space (bytes still containing the 0xaa pattern)
+* Stack utilization percentage
+
+High watermark tracking
+========================
+
+When :kconfig:option:`CONFIG_THREAD_ANALYZER_STACK_HIGH_WATERMARK` is enabled,
+the thread analyzer tracks the maximum stack usage observed for each thread
+since its creation. This feature provides:
+
+* Peak stack usage detection across the thread's lifetime
+* Worst-case stack usage patterns
+* Historical view of stack utilization
+* Better insight for stack size optimization
+
+The high watermark is stored in the thread's stack_info structure and is
+updated automatically each time the thread analyzer runs. This allows
+developers to identify threads that may need larger stacks or threads
+with stacks that can be safely reduced.
+
+Benefits of high watermark tracking:
+
+* **Optimization**: Identify threads with oversized stacks that can be reduced
+* **Safety**: Ensure threads have adequate stack margins for worst-case scenarios
+* **Debugging**: Detect intermittent stack-intensive operations that might be
+  missed with single-point measurements
+* **Profiling**: Understand stack growth patterns over time
+
 For example, to build the synchronization sample with Thread Analyser enabled,
 do the following:
 
@@ -62,13 +106,42 @@ information from Thread Analyzer::
 	thread_a: Hello World from cpu 0 on qemu_x86!
 	thread_b: Hello World from cpu 0 on qemu_x86!
 
+Example output with high watermark tracking
+============================================
+
+When :kconfig:option:`CONFIG_THREAD_ANALYZER_STACK_HIGH_WATERMARK` is enabled
+(default), the thread analyzer will also display the high watermark (peak stack
+usage) for each thread::
+
+	Thread analyze:
+	 thread_b            : STACK: unused 648 usage 376 / 1024 (36 %); CPU: 7 %
+	                     : High watermark: 412 / 1024 (40 %)
+	 thread_analyzer     : STACK: unused 8 usage 504 / 512 (98 %); CPU: 0 %
+	                     : High watermark: 508 / 512 (99 %)
+	 thread_a            : STACK: unused 648 usage 376 / 1024 (36 %); CPU: 9 %
+	                     : High watermark: 428 / 1024 (41 %)
+	 idle                : STACK: unused 204 usage 116 / 320 (36 %); CPU: 82 %
+	                     : High watermark: 128 / 320 (40 %)
+
+In this example, you can see that while thread_b's current usage is 376 bytes,
+it has peaked at 412 bytes at some point since creation. This information helps
+identify whether threads need more stack space or if they can be safely reduced.
 
 Configuration
 *************
 Configure this module using the following options.
 
 :kconfig:option:`CONFIG_THREAD_ANALYZER`
-   Enable the module.
+   Enable the module. This automatically selects :kconfig:option:`CONFIG_INIT_STACKS`
+   to enable stack painting for accurate stack usage measurement.
+:kconfig:option:`CONFIG_INIT_STACKS`
+   Initialize stack areas with a known value (0xaa) for stack painting. This is
+   automatically enabled by :kconfig:option:`CONFIG_THREAD_ANALYZER`.
+:kconfig:option:`CONFIG_THREAD_ANALYZER_STACK_HIGH_WATERMARK`
+   Enable tracking of stack high watermark (peak usage) for each thread. When
+   enabled, the thread analyzer maintains a record of the maximum stack usage
+   observed for each thread, allowing detection of worst-case stack requirements.
+   Default: enabled.
 :kconfig:option:`CONFIG_THREAD_ANALYZER_USE_PRINTK`
    Use printk for thread statistics.
 :kconfig:option:`CONFIG_THREAD_ANALYZER_USE_LOG`

include/zephyr/debug/thread_analyzer.h

@@ -16,23 +16,100 @@ extern "C" {
 
 /** @defgroup thread_analyzer Thread analyzer
  *  @ingroup debug
- *  @brief Module for analyzing threads
+ *  @brief Module for analyzing thread stack usage and performance
+ *
+ *  The thread analyzer provides comprehensive runtime analysis of thread
+ *  behavior, including stack usage, CPU utilization, and performance metrics.
+ *
+ *  ## Stack Painting
+ *
+ *  Stack painting is a technique where thread stacks are initialized with a
+ *  known pattern (0xaa) when CONFIG_INIT_STACKS is enabled. This allows the
+ *  thread analyzer to detect stack usage by scanning from the bottom of the
+ *  stack until it finds memory that has been modified. This mechanism provides
+ *  accurate measurement of:
+ *  - Current stack usage (how much stack is currently in use)
+ *  - Unused stack space (pristine stack area with 0xaa pattern)
+ *
+ *  ## High Watermark Tracking
+ *
+ *  When CONFIG_THREAD_ANALYZER_STACK_HIGH_WATERMARK is enabled, the analyzer
+ *  tracks the maximum stack usage observed for each thread since creation.
+ *  This feature helps:
+ *  - Identify worst-case stack usage patterns
+ *  - Optimize stack size configurations
+ *  - Detect intermittent stack-intensive operations
+ *  - Ensure adequate stack margins for safety
+ *
+ *  The high watermark is stored in the thread's stack_info structure and
+ *  persists across multiple analyzer runs, providing a historical view of
+ *  peak stack utilization.
+ *
+ *  ## Usage
+ *
+ *  To use the thread analyzer:
+ *  1. Enable CONFIG_THREAD_ANALYZER in your project configuration
+ *  2. Enable CONFIG_INIT_STACKS for stack painting support
+ *  3. Enable CONFIG_THREAD_ANALYZER_STACK_HIGH_WATERMARK for peak usage tracking
+ *  4. Call thread_analyzer_print() or thread_analyzer_run() to analyze threads
+ *
+ *  Alternatively, enable CONFIG_THREAD_ANALYZER_AUTO to run analysis
+ *  automatically at periodic intervals.
  *
- *  This module implements functions and the configuration that simplifies
- *  thread analysis.
  *  @{
  */
 
+/**
+ * @brief Thread analyzer information structure
+ *
+ * Contains detailed information about a thread's stack usage and
+ * performance characteristics. This structure is populated by the
+ * thread analyzer and passed to callback functions.
+ */
 struct thread_analyzer_info {
-	/** The name of the thread or stringified address of the thread handle
-	 * if name is not set.
+	/** Thread name or stringified thread handle address.
+	 *
+	 * If the thread has a name set via k_thread_name_set(), this field
+	 * contains that name. Otherwise, it contains the hexadecimal string
+	 * representation of the thread structure pointer.
 	 */
 	const char *name;
-	/** The total size of the stack*/
+
+	/** Total stack size in bytes.
+	 *
+	 * Represents the total size of the stack buffer allocated for this
+	 * thread, including any reserved areas and adjustments.
+	 */
 	size_t stack_size;
-	/** Stack size in used */
+
+	/** Current stack usage in bytes.
+	 *
+	 * The amount of stack space currently used by the thread. This is
+	 * calculated by scanning the stack for the initialized pattern (0xaa)
+	 * when CONFIG_INIT_STACKS is enabled. The value represents the number
+	 * of bytes that have been modified since thread creation.
+	 */
 	size_t stack_used;
 
+#ifdef CONFIG_THREAD_ANALYZER_STACK_HIGH_WATERMARK
+	/** High watermark - maximum stack usage in bytes.
+	 *
+	 * Tracks the peak stack utilization observed since thread creation.
+	 * This field is only available when CONFIG_THREAD_ANALYZER_STACK_HIGH_WATERMARK
+	 * is enabled. The high watermark is updated each time the thread analyzer
+	 * runs if the current usage exceeds the previously recorded maximum.
+	 *
+	 * Use cases:
+	 * - Identifying worst-case stack requirements
+	 * - Optimizing stack size allocations
+	 * - Detecting stack growth patterns over time
+	 * - Ensuring adequate safety margins
+	 *
+	 * Note: Requires CONFIG_INIT_STACKS to be enabled for accurate measurement.
+	 */
+	size_t stack_high_watermark;
+#endif
+
 #ifdef CONFIG_THREAD_RUNTIME_STATS
 	unsigned int utilization;
 #ifdef CONFIG_SCHED_THREAD_USAGE

include/zephyr/kernel/thread.h

@@ -171,6 +171,16 @@ struct _thread_stack_info {
 		size_t sz;
 	} mapped;
 #endif /* CONFIG_THREAD_STACK_MEM_MAPPED */
+
+#if defined(CONFIG_THREAD_ANALYZER_STACK_HIGH_WATERMARK)
+	/* High watermark - maximum stack usage observed since thread creation.
+	 * This field tracks the peak stack utilization over time. When
+	 * CONFIG_INIT_STACKS is enabled, the stack is initialized with a known
+	 * pattern (0xaa), enabling accurate high watermark detection through
+	 * the thread analyzer.
+	 */
+	size_t high_watermark;
+#endif /* CONFIG_THREAD_ANALYZER_STACK_HIGH_WATERMARK */
 };
 
 typedef struct _thread_stack_info _thread_stack_info_t;

kernel/thread.c

@@ -495,6 +495,13 @@ static char *setup_thread_stack(struct k_thread *new_thread,
 	new_thread->stack_info.start = (uintptr_t)stack_buf_start;
 	new_thread->stack_info.size = stack_buf_size;
 	new_thread->stack_info.delta = delta;
+
+#ifdef CONFIG_THREAD_ANALYZER_STACK_HIGH_WATERMARK
+	/* Initialize high watermark to 0. The thread analyzer will update this
+	 * value as it tracks stack usage over time.
+	 */
+	new_thread->stack_info.high_watermark = 0;
+#endif /* CONFIG_THREAD_ANALYZER_STACK_HIGH_WATERMARK */
 #endif /* CONFIG_THREAD_STACK_INFO */
 	stack_ptr -= delta;
 

subsys/debug/thread_analyzer/Kconfig

@@ -16,6 +16,12 @@ menuconfig THREAD_ANALYZER
 	  stack size configuration to find stack overflow or to find stacks
 	  which may be optimized.
 
+	  The thread analyzer automatically enables INIT_STACKS, which implements
+	  stack painting - a technique where stacks are initialized with a known
+	  pattern (0xaa). This allows the analyzer to detect stack usage by
+	  scanning for memory that has been modified, providing accurate
+	  measurements of current and peak stack utilization.
+
 if THREAD_ANALYZER
 module = THREAD_ANALYZER
 module-str = thread analyzer
@@ -49,6 +55,27 @@ config THREAD_ANALYZER_PRIV_STACK_USAGE
 	help
 	  Print privileged stack usage for user threads.
 
+config THREAD_ANALYZER_STACK_HIGH_WATERMARK
+	bool "Track stack high watermark"
+	default y
+	help
+	  Enable tracking of stack high watermark (peak usage) for each thread.
+	  When enabled, the thread analyzer will track and display the maximum
+	  stack usage observed since thread creation or last reset. This feature
+	  requires CONFIG_INIT_STACKS to function properly, as it relies on the
+	  stack painting mechanism (initializing stack memory with 0xaa pattern)
+	  to detect stack usage.
+
+	  High watermark tracking is useful for:
+	  - Understanding worst-case stack usage patterns
+	  - Optimizing stack size configurations
+	  - Identifying threads that may need larger stacks
+	  - Debugging intermittent stack-related issues
+
+	  Note: The high watermark is stored in the thread structure and is
+	  maintained across multiple thread analyzer runs, allowing you to
+	  observe peak usage over the lifetime of the thread.
+
 config THREAD_ANALYZER_RUN_UNLOCKED
 	bool "Run analysis with interrupts unlocked"
 	default y

subsys/debug/thread_analyzer/thread_analyzer.c

@@ -6,6 +6,29 @@
 
 /** @file
  *  @brief Thread analyzer implementation
+ *
+ * The thread analyzer provides runtime analysis of thread stack usage and
+ * performance metrics. It enables developers to optimize stack sizes and
+ * understand thread behavior.
+ *
+ * Key features:
+ * - Stack usage tracking via stack painting (CONFIG_INIT_STACKS)
+ * - High watermark tracking for peak stack usage detection
+ * - CPU utilization statistics per thread
+ * - ISR stack usage analysis
+ * - Privilege stack usage tracking (for user mode threads)
+ *
+ * Stack Painting:
+ * When CONFIG_INIT_STACKS is enabled, thread stacks are initialized with
+ * a known pattern (0xaa). The thread analyzer scans the stack from the
+ * bottom to detect how much of the stack has been modified (used), allowing
+ * accurate measurement of stack usage.
+ *
+ * High Watermark Tracking:
+ * When CONFIG_THREAD_ANALYZER_STACK_HIGH_WATERMARK is enabled, the thread
+ * analyzer maintains a record of the maximum stack usage observed for each
+ * thread since its creation. This helps identify worst-case stack usage
+ * patterns that may not be visible during typical operation.
  */
 
 #include <zephyr/kernel.h>
@@ -56,6 +79,16 @@ static void thread_print_cb(struct thread_analyzer_info *info)
 		info->stack_size, pcnt,
 		info->utilization);
 
+#ifdef CONFIG_THREAD_ANALYZER_STACK_HIGH_WATERMARK
+	/* Display high watermark (peak stack usage) */
+	size_t hwm_pcnt = (info->stack_high_watermark * 100U) / info->stack_size;
+
+	THREAD_ANALYZER_PRINT(
+		THREAD_ANALYZER_FMT(
+			" %-20s: High watermark: %zu / %zu (%zu %%)"),
+		" ", info->stack_high_watermark, info->stack_size, hwm_pcnt);
+#endif
+
 #ifdef CONFIG_THREAD_ANALYZER_PRIV_STACK_USAGE
 	if (info->priv_stack_size > 0) {
 		pcnt = (info->priv_stack_used * 100U) / info->priv_stack_size;
@@ -89,6 +122,16 @@ static void thread_print_cb(struct thread_analyzer_info *info)
 		THREAD_ANALYZER_VSTR(info->name),
 		info->stack_size - info->stack_used, info->stack_used,
 		info->stack_size, pcnt);
+
+#ifdef CONFIG_THREAD_ANALYZER_STACK_HIGH_WATERMARK
+	/* Display high watermark (peak stack usage) */
+	size_t hwm_pcnt = (info->stack_high_watermark * 100U) / info->stack_size;
+
+	THREAD_ANALYZER_PRINT(
+		THREAD_ANALYZER_FMT(
+			" %-20s: High watermark: %zu / %zu (%zu %%)"),
+		" ", info->stack_high_watermark, info->stack_size, hwm_pcnt);
+#endif
 #endif
 }
 
@@ -134,6 +177,14 @@ static void thread_analyze_cb(const struct k_thread *cthread, void *user_data)
 	info.stack_size = size;
 	info.stack_used = size - unused;
 
+#ifdef CONFIG_THREAD_ANALYZER_STACK_HIGH_WATERMARK
+	/* Update high watermark if current usage exceeds it */
+	if (info.stack_used > thread->stack_info.high_watermark) {
+		thread->stack_info.high_watermark = info.stack_used;
+	}
+	info.stack_high_watermark = thread->stack_info.high_watermark;
+#endif
+
 #ifdef CONFIG_THREAD_ANALYZER_PRIV_STACK_USAGE
 	ret = arch_thread_priv_stack_space_get(cthread, &size, &unused);
 	if (ret == 0) {

tests/subsys/debug/thread_analyzer/prj.conf

@@ -1,5 +1,6 @@
 CONFIG_THREAD_ANALYZER=y
 CONFIG_THREAD_ANALYZER_AUTO=y
+CONFIG_THREAD_ANALYZER_STACK_HIGH_WATERMARK=y
 
 # log immediate mode requires a bit more stack to print
 CONFIG_THREAD_ANALYZER_AUTO_STACK_SIZE=2048

tests/subsys/debug/thread_analyzer/testcase.yaml

@@ -80,3 +80,14 @@ tests:
       regex:
         - "(.*)0x([0-9a-fA-F]+)([ ]+) : STACK: unused [0-9]+ usage [0-9]+ / [0-9]+ (.*)"
         - "(.*)ISR0([ ]+) : STACK: unused [0-9]+ usage [0-9]+ / [0-9]+ (.*)"
+  debug.thread_analyzer.high_watermark:
+    extra_configs:
+      - CONFIG_THREAD_ANALYZER_USE_PRINTK=y
+      - CONFIG_THREAD_ANALYZER_STACK_HIGH_WATERMARK=y
+    harness: console
+    harness_config:
+      type: multi_line
+      regex:
+        - "(.*)0x([0-9a-fA-F]+)([ ]+) : STACK: unused [0-9]+ usage [0-9]+ / [0-9]+ (.*)"
+        - "(.*)High watermark: [0-9]+ / [0-9]+ (.*)"
+        - "(.*)ISR0([ ]+) : STACK: unused [0-9]+ usage [0-9]+ / [0-9]+ (.*)"