traceconfig: generate an annotated Markdown document with source refs #95808

pillo79 · 2025-09-10T16:20:47Z

This pull request adds a new traceconfig build target to generate Markdown documentation for Kconfig symbols that details the reason a value was chosen for each symbol in the configuration. The list of symbols is split in three sorted groups:

Visible symbols: parameters that have a value and are user-configurable
Invisible symbols: parameters that have a value which are hidden by default (often target of selects or implicit defaults).
Unset symbols: parameters that do not have a value assigned

Symbols that have a value include a readable and clickable (if the viewer supports it) link to the location of the statement that resulted in that value being applied, be it an explicit assignment or a default. For the symbols that are select-ed or imply-ed by multiple sources, the set of matching requirements are printed.

Documentation for the new feature is available and a test suite has been added to validate Kconfig source reference functionality.

Sample generated documentation

The new target generates a zephyr/kconfig-trace.md file in the project's build folder. This is an example of a full generated documentation for the sample/drivers/spi_flash project running on the arduino_opta//m7 board.

The objective is for the generated document to be easily readable by humans with a simple editor. The entries are rendered like this:

As a raw source

| Type | Name                             |      Value |  Source  | Location                                                                                                                                  |
|-----:|:---------------------------------|-----------:|:--------:|:------------------------------------------------------------------------------------------------------------------------------------------|
| bool | `CONFIG_SERIAL`                  |          y | user set | [boards/arduino/opta/arduino\_opta\_stm32h747xx\_m7\_defconfig:27](<../../boards/arduino/opta/arduino_opta_stm32h747xx_m7_defconfig#L27>) |
| bool | `CONFIG_SIZE_OPTIMIZATIONS`      |          y | default  | [Kconfig.zephyr:485](<../../Kconfig.zephyr#L485>)                                                                                         |
| bool | `CONFIG_SOC_EARLY_INIT_HOOK`     |          y | selected | `SOC_SERIES_STM32H7X && SOC_FAMILY_STM32`                                                                                                 |
| bool | `CONFIG_SOC_FLASH_STM32`         |          y | default  | [drivers/flash/Kconfig.stm32:22](<../../drivers/flash/Kconfig.stm32#L22>)                                                                 |
|  hex | `CONFIG_SRAM_BASE_ADDRESS`       | 0x24000000 | default  | [arch/Kconfig:228](<../../arch/Kconfig#L228>)                                                                                             |
| bool | `CONFIG_SRAM_REGION_PERMISSIONS` |          y | selected | `ARM_MPU && CPU_HAS_MPU`                                                                                                                  |
|  int | `CONFIG_SRAM_SIZE`               |        512 | default  | [arch/Kconfig:220](<../../arch/Kconfig#L220>)                                                                                             |
| bool | `CONFIG_STACK_ALIGN_DOUBLE_WORD` |          y | default  | [arch/arm/core/Kconfig:152](<../../arch/arm/core/Kconfig#L152>)                                                                           |
| bool | `CONFIG_STDOUT_CONSOLE`          |          y | user set | [samples/drivers/spi\_flash/prj.conf:1](<../../samples/drivers/spi_flash/prj.conf#L1>)                                                    |

As a Github table

Note: links here have been edited to refer to the upstream used in the PR generation. They are usually relative to where the file was generated.

Type	Name	Value	Source	Location
bool	`CONFIG_SERIAL`	y	user set	boards/arduino/opta/arduino_opta_stm32h747xx_m7_defconfig:27
bool	`CONFIG_SIZE_OPTIMIZATIONS`	y	default	Kconfig.zephyr:485
bool	`CONFIG_SOC_EARLY_INIT_HOOK`	y	selected	`SOC_SERIES_STM32H7X && SOC_FAMILY_STM32`
bool	`CONFIG_SOC_FLASH_STM32`	y	default	drivers/flash/Kconfig.stm32:22
hex	`CONFIG_SRAM_BASE_ADDRESS`	0x24000000	default	arch/Kconfig:228
bool	`CONFIG_SRAM_REGION_PERMISSIONS`	y	selected	`ARM_MPU && CPU_HAS_MPU`
int	`CONFIG_SRAM_SIZE`	512	default	arch/Kconfig:220
bool	`CONFIG_STACK_ALIGN_DOUBLE_WORD`	y	default	arch/arm/core/Kconfig:152
bool	`CONFIG_STDOUT_CONSOLE`	y	user set	samples/drivers/spi_flash/prj.conf:1

Rendered in VSCode:

Review notes

Kconfiglib already used locs (a tuple of filename and line number) in a few specific places to store line information.
A refactor was first applied to expand the usage to all statements modifying the value, with the same loc element shared with multiple entities.
The source reference concept is then added to Symbols to collapse this information in a common form, which is exported for later usage by external tools.
A processor that reads this information and writes the user-visible Markdown document is then added, along with a test suite for the functionality.

Copilot

Pull Request Overview

This PR adds a new feature to generate Markdown documentation for Kconfig configuration parameters that details how each symbol received its final value. The documentation includes source references linking to the exact locations in Kconfig files or configuration files where values were set.

Extends kconfiglib to track source references for all symbol value assignments
Adds a new traceconfig build target that generates kconfig-trace.md documentation
Implements comprehensive test coverage for the source reference functionality

Reviewed Changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
scripts/kconfig/kconfiglib.py	Core implementation of source reference tracking in kconfiglib
scripts/kconfig/kconfig.py	Adds writing of source reference data in JSON and pickle formats
scripts/kconfig/traceconfig.py	New script to generate Markdown documentation from source reference data
cmake/modules/kconfig.cmake	Adds the `traceconfig` build target
tests/kconfig/srcrefs/*	Test suite validating source reference functionality
doc/build/kconfig/tracing.rst	Documentation for the new tracing feature

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

tests/kconfig/srcrefs/validate_srcrefs.py

scripts/kconfig/traceconfig.py

mathieuchopstm

Seems fairly reasonable.

scripts/kconfig/kconfiglib.py

mathieuchopstm · 2025-09-17T15:40:48Z

scripts/kconfig/traceconfig.py

+SRCREF_TO_STR = {
+    kconfiglib.SRCREF_UNSET: "unset",
+    kconfiglib.SRCREF_DEFAULT: "default",
+    kconfiglib.SRCREF_USER: "user set",


maybe set by user?

tests/kconfig/srcrefs/validate_srcrefs.py

doc/build/kconfig/tracing.rst

scripts/kconfig/traceconfig.py

tejlmand

Looks good.

But please note that changes to Kconfiglib should go upstream first: https://github.com/zephyrproject-rtos/Kconfiglib and then brought down to Zephyr with a reference.

Note also that kconfiglib is used by other projects.

It even states that Kconfiglib is python2/3 compliant (perhaps we should remove python2 as it's no longer maintained, but guess that's another discussion 😉 )

scripts/kconfig/kconfiglib.py

pillo79 · 2025-09-19T17:03:01Z

v2:

Renamed the concept of "srcrefs" to "origin" (fingers and eyes are a lot happier!)
Reworked kconfiglib changes into Symbol: track value origin Kconfiglib#18 (first 5 commits). Now, only the origin information is computed in the common kconfiglib, bundling of all other fields and the export code is in scripts/kconfig.py (in Zephyr) - this cut looks cleaner to me than it was before.
No change to final output.
All comments but auto-run addressed.

@erwango Tried to look into adding optional auto-run via .west/config, but I would have to import west stuff into "kconfig target" code. Not sure if it is OK to do so? If not, should I rework traceconfig to be a west command? Any advice on other ways to optionally run the doc generation at every instance? TIA

erwango · 2025-09-22T09:01:57Z

@erwango Tried to look into adding optional auto-run via .west/config, but I would have to import west stuff into "kconfig target" code. Not sure if it is OK to do so? If not, should I rework traceconfig to be a west command? Any advice on other ways to optionally run the doc generation at every instance? TIA

Tbh, I won't help much here. @carlescufi maybe ?

mathieuchopstm · 2025-09-22T09:22:52Z

tests/kconfig/tracing/validate_tracing.py

+# fmt: off
+
+# Indices for the fields in the trace entries
+# Keep these in sync with the generation code in scripts/kconfig/kconfig.py!


Nit: I'd also place a warning in kconfig.py, since it's more likely that changes there break this test than the other way around.

Replace 'filename' and 'linenr' usages with a single 'loc' attribute, which is a (filename, linenr) tuple. This simplifies code dealing with locations, and makes it explicitly constant. The original attributes are now provided via properties for compatibility. Signed-off-by: Luca Burelli <[email protected]>

Replace filename and linenr parameters to _warn() with a single loc element, which is a constant (filename, linenr) tuple that can be more efficiently passed and stored. Signed-off-by: Luca Burelli <[email protected]>

The 'user_loc' stores the location (filename and line number) where a symbol or choice was last set via a direct user selection. Signed-off-by: Luca Burelli <[email protected]>

Store the location (filename and line number) where a 'select', 'imply', 'range' or 'default' was added to a Symbol or Choice. Signed-off-by: Luca Burelli <[email protected]>

Track information about what caused a value to be set in the 'origin' property of Symbol. 'imply' and 'select' dependencies do not have a location associated with them, so in those cases the location is a string representation of the dependency expression. For defaults and user values, the location in the Kconfig file is stored. Signed-off-by: Luca Burelli <[email protected]>

Collect and save trace data for all symbols in the merged configuration. This includes information about where each symbol was defined or assigned, which can be useful for debugging complex Kconfig setups. The trace data includes the following information for each symbol: - Name - Visibility - Type - Value - Kind and location of value origin (as defined in kconfiglib) The trace data is saved for later use in two formats: a binary pickle file and a human-readable JSON file. Signed-off-by: Luca Burelli <[email protected]>

The new 'traceconfig' target generates a Markdown file listing all configuration symbols, their values, and where those values originated (user assignment, default, selection, implication, or unset). Signed-off-by: Luca Burelli <[email protected]>

Add a test case that verifies Kconfig value origin traces are correctly generated in both JSON and pickle formats. The test includes a sample Kconfig configuration with dependencies, selections, and choices to cover various scenarios. A validation script checks the generated source origin data against expected values. In addition, the 'traceconfig' target is called to test the generation of the Markdown file with the Kconfig trace report. Signed-off-by: Luca Burelli <[email protected]>

sonarqubecloud · 2025-10-14T15:24:15Z

Please retry analysis of this Pull-Request directly on SonarQube Cloud

pillo79 · 2025-10-15T16:07:39Z

@tejlmand and others, please re-review, would be useful to have in 4.3...

pillo79 force-pushed the pr-kconf-trace branch 3 times, most recently from 20e4518 to 67704a7 Compare September 17, 2025 11:09

pillo79 requested a review from Copilot September 17, 2025 12:13

Copilot AI reviewed Sep 17, 2025

View reviewed changes

tests/kconfig/srcrefs/validate_srcrefs.py Show resolved Hide resolved

scripts/kconfig/traceconfig.py Outdated Show resolved Hide resolved

pillo79 changed the title ~~kconfiglib: generate an annotated config.md document with source refs~~ kconfiglib: generate an annotated Markdown document with source refs Sep 17, 2025

pillo79 marked this pull request as ready for review September 17, 2025 12:22

zephyrbot added area: Kconfig area: Build System labels Sep 17, 2025

zephyrbot requested review from 57300, jeremybettis, nashif, nordicjm and tejlmand September 17, 2025 12:23

zephyrbot assigned tejlmand Sep 17, 2025

pillo79 requested review from fabiobaltieri, kartben, mathieuchopstm and pdgendt September 17, 2025 12:25

mathieuchopstm reviewed Sep 17, 2025

View reviewed changes

tejlmand reviewed Sep 18, 2025

View reviewed changes

scripts/kconfig/kconfiglib.py Show resolved Hide resolved

pillo79 force-pushed the pr-kconf-trace branch 2 times, most recently from f50932d to 985e967 Compare September 19, 2025 16:45

mathieuchopstm reviewed Sep 22, 2025

View reviewed changes

pillo79 changed the title ~~kconfiglib: generate an annotated Markdown document with source refs~~ traceconfig: generate an annotated Markdown document with source refs Sep 28, 2025

pillo79 force-pushed the pr-kconf-trace branch from 985e967 to 579f819 Compare October 14, 2025 14:56

zephyrbot added the area: Tests Issues related to a particular existing or missing test label Oct 14, 2025

pillo79 added 7 commits October 14, 2025 17:13

kconfiglib: refactor Kconfig._warn() to use 'loc'

680d2af

Replace filename and linenr parameters to _warn() with a single loc element, which is a constant (filename, linenr) tuple that can be more efficiently passed and stored. Signed-off-by: Luca Burelli <[email protected]>

kconfiglib: add 'user_loc' to Symbol and Choice classes

a49ed46

The 'user_loc' stores the location (filename and line number) where a symbol or choice was last set via a direct user selection. Signed-off-by: Luca Burelli <[email protected]>

kconfiglib: add 'loc' to reverse dependencies

25cca15

Store the location (filename and line number) where a 'select', 'imply', 'range' or 'default' was added to a Symbol or Choice. Signed-off-by: Luca Burelli <[email protected]>

pillo79 force-pushed the pr-kconf-trace branch from 579f819 to eea6715 Compare October 14, 2025 15:13

mathieuchopstm approved these changes Oct 17, 2025

View reviewed changes

mathieuchopstm added this to the v4.3.0 milestone Oct 17, 2025

erwango approved these changes Oct 17, 2025

View reviewed changes

traceconfig: generate an annotated Markdown document with source refs #95808

Are you sure you want to change the base?

traceconfig: generate an annotated Markdown document with source refs #95808

Conversation

pillo79 commented Sep 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Sample generated documentation

As a raw source

As a Github table

Rendered in VSCode:

Review notes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

mathieuchopstm left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mathieuchopstm Sep 17, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tejlmand left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

pillo79 commented Sep 19, 2025

Uh oh!

erwango commented Sep 22, 2025

Uh oh!

mathieuchopstm Sep 22, 2025

Choose a reason for hiding this comment

Uh oh!

sonarqubecloud bot commented Oct 14, 2025

Uh oh!

pillo79 commented Oct 15, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

pillo79 commented Sep 10, 2025 •

edited

Loading