Snippets

[mbolivar-nordic]

Is your enhancement proposal related to a problem?

Zephyr developers frequently need to provide interrelated build system settings to accomplish their goals. There is currently no extensible, composable, future-proof way to do this in the build system that lets you store these settings in the file system, give them a name, and apply them easily by name when building an application.

For example, required devicetree overlays and Kconfig fragments to configure and enable some feature are often interrelated configuration when building applications. These overrides are often specific to the target hardware, even when they perform the same "logical" change.

There is currently no way other than shields to pair these related configuration overrides together in a way that is searchable across modules, customizable on a per-board basis, and able to be passed to e.g. west build with a single name. However, shields are not an appropriate feature for overriding the default configuration on an existing board without involvement of daughter boards. This feature gap creates usability problems.

CMake preset files are another option for application configuration. However, these are not composable: there must be a preset file in the application source directory, and it must explicitly include anything else it wants. This doesn't meet the goals for snippets: it should be easy to ask the build system to apply zero, one, or N snippets by name, at west build time, without modifying the application source directory.

Example use case: CDC-ACM console

As a user, I want to override the default UART console configuration on my board with a CDC-ACM console.

As a part of this, I need to both:

• provide a devicetree overlay overriding the default zephyr,console chosen node
• ensure related Kconfig settings, such as setting CONFIG_USB_DEVICE_STACK=y

I want to keep these related settings in a single place within my west workspace, so that I can build multiple applications with the default console overridden. I want this "single place" to support storing the same logical configuration overrides for different boards.

I can currently do this "manually" with command lines like this:

$ west build -b board-1 app-1 -- \
    -DDTC_OVERLAY_FILE=/path/to/cdc-acm-console/board-1.overlay \
    -DOVERLAY_CONFIG=/path/to/cdc-acm-console/board-1.conf
$ west build -b board-2 app-1 -- \
    -DDTC_OVERLAY_FILE=/path/to/cdc-acm-console/board-2.overlay \
    -DOVERLAY_CONFIG=/path/to/cdc-acm-console/board-2.conf

$ west build -b board-1 app-2 -- \
    -DDTC_OVERLAY_FILE=/path/to/cdc-acm-console/board-1.overlay \
    -DOVERLAY_CONFIG=/path/to/cdc-acm-console/board-1.conf
$ west build -b board-2 app-2 -- \
    -DDTC_OVERLAY_FILE=/path/to/cdc-acm-console/board-2.overlay \
    -DOVERLAY_CONFIG=/path/to/cdc-acm-console/board-2.conf

This is very annoying to me as a user. (It is also difficult to document for other personas like "Zephyr maintainer" or "downstream Zephyr distributor".)

A proposal to handle this using SHIELD was raised in #40645 but rejected at dev-review for the reason described above: this is not a "shield" in the sense that term is usually defined in this context.

Describe the solution you'd like

A way to name and store interrelated build system settings within my workspace that is:

• extensible: the build system can search for and discover them, including in third party modules unknown to the upstream zephyr project
• composable: I should be able to apply multiple snippets, one snippet, or none at all
• specializable: I want to be able to specialize the settings to my target hardware as needed
• future-proof and backwards compatible: Zephyr should be able to extend this feature without breaking users, including out of tree users
• DRY: ("don't repeat yourself") I should be able to apply pattern matching or other rules to my settings so that they can match various different applicable target hardware, etc

We have previously called these entities "snippets", see e.g. #40669. So that's the name I'm going with here.

I am intentionally going to keep the implementation details out of this description for now and keep it high level.

Example solution usage: CDC-ACM

As a user, I would like to be able to do the following instead of the above, and have the same effects:

$ west build -b board-1 --snippet cdc-acm-console app-1
$ west build -b board-2 --snippet cdc-acm-console app-1

$ west build -b board-1 --snippet cdc-acm-console app-2
$ west build -b board-2 --snippet cdc-acm-console app-2

Extensible via modules

As a user, I would like to be able to define "snippets" in my own modules, and have them added to the search path used by the build system via existing module mechanisms, and have those "snippets" picked up by default by the build system when I use west build.

I additionally want to be able to extend existing "snippets" defined elsewhere. For example, if the zephyr repository defines a standard cdc-acm-console snippet, I want to be able to extend it with support for my board by adding files to a module I control, without any changes to the zephyr repository itself.

Multiple snippets at once

As a user, I want to be able to combine multiple "snippets" in a single application build. For example, if I want to both use a CDC-ACM console and override the default settings partition for my board to one on an external flash IC, I should be able to do this:

$ west build -b board --snippet cdc-acm-console --snippet my-settings-partition app

West not required

As a user, I want to be able to use the "snippets" feature even if I am not using west. I should be able to do this by defining a CMake variable that declares the snippets I want to use. For example:

cmake ... -DBOARD=board -DSNIPPET=cdc-acm-console;my-settings-partition
ninja ...

Future work: per-SoC configuration

As a user, I often want to specify "snippets" on a per-SoC basis, not a per-board basis. This occurs whenever the needed overrides are always the same for any two boards with the same SoC. The CDC-ACM console is a good example here too: it is common for microcontroller SoCs to only have a single USB device peripheral, so the "snippet" for any two boards with the same such SoC is going to be the same.

This is difficult to do within the current build system, but should be possible to address as follow-on work if the Zephyr project adopts build system changes to make SoCs more "first class citizens". This is intentionally left out of scope of the initial work to avoid muddying the waters in this issue.

Future work: code as well as configuration

It should be possible to enhance the feature so that a snippet can contain source code files that should be added to the build system, as well as build system settings.

Future work: twister integration

Suggested by @JordanYates in #53640 (review).

Future work: .cmake file support

Suggested by @JordanYates in #53640 (comment)

Future work: board-revision-specific snippets

These would use the following syntax:

name: foo
boards:
  bar:
    revision: 1.2.3
      DTC_OVERLAY_FILE: bar-1.2.3.overlay

Discussed with @nordicjm in #53640 (comment)

Describe alternatives you've considered

• cmake: introduce snippets for common configuration #40669
• shields: add generic shield to use CDC ACM UART as backend #40645

Additional context

The original discussion was started by @JordanYates at Zephyr Developers Summit 2021. The community at the time agreed the idea was solid, but we haven't been able to agree on specifics.

Change history:

• 2023-03-22: renamed "future directions" to "future work", added future work for board-revision-specific snippet settings
• 2023-03-21: added twister integration and .cmake file support future directions
• 2023-03-09: renamed back to "snippets", reworked some specifics, added "code as well as configuration"
• 2022-10-31: initial proposal

[jfischer-no]

I assume "mixins" is just a project name, temporary until something better is found.

[aborisovich]

Reason for implementing "Snippets" seems perfectly clear - end user should have a possibility to gather a set of needed configurations that alter a board definition under single name. However, I would like some concerns.

The naming

Personally do not agree with the term you are trying to use. I will not elaborate how important is the matter of giving proper names to abstractions we create, as every developer understands this. So, the name "snippets" does not indicate in any way what it actually is:

• name "snippet" does not indicate that it is a set (a group of configurations given common name)
• name "snippet" does not indicate that it is related to pre-build configuration
• name "snippet" misleadingly indicates that is it something lightweight and temporary (one of the things that comes to my mind is cut out part of text or code to be reused/pasted somewhere else), while it is actually something very static and of key importance

I would suggest some alternative names:

1. a "config_group" - this name perfectly clear indicates what it is - a set of configurations that were given a name. However, it may lead to confusion whether a "group" is a "collection" or maybe something we "assign to".
2. a "config_set" - this name also indicates clearly what it is, but may be confused with the meaning of "set" part (somebody can think of it as setting something).
3. a "config_bundle" - looks intuitive. "Bundle" word seems not to have too much alternative meanings and the verb has quite similar meaning to a noun. I would go for this one personally.

Feature of inclusion & exclusion

CMakePresets you mentioned suffer from the combinatory explosion issue. SOF project I contribute to also suffers from lack of this (snippets) feature. Let's think about a concrete example:
"User would like to build a firmware for MTL board, to be run on FPGA, with profiling features enabled and disabled logging".
Some context:

• MTL is a board (intel_adsp_ace15_mtpm.conf) that contains set of configurations to build production fw for this hardware (for silicon).
• We use overlay fpga_overlay.conf file that changes fws clock speed to be run on FPGA
• We use an overlay profiling_overlay.conf file to perform performance tests and include some special profiling code in fw and disable logging to have more accurate performance results.

So in this scenarion we have 2 overlays added to the build.
Lets say, that we use the "Snippets" you are proposing and give it a name "mtl_fpga_profiling.snippet" and this file contains information that we want to use fpga_overlay.conf and profiling_overlay.conf files for the build.
After some later time, I'm developing a new driver version that is supposed to replace old driver version. I would like a build fw with my new driver enabled, old one disabled and run performance tests on FPGA. Let's say I've created a new overlay driverA_overlay.conf (that contains CONFIG_DRIVER_A=n, CONFIG_NEW_DRIVER_A=y).
Next, I would create a "Snippet" to build my set using a single name, lets call it "mtl_fpga_new_driver_a_profiling.snippet".
And here we came to my:

• feature request: snippet files may include (or exclude too!) other snipper files (resolved by the order of occurrence) to avoid combinatory explosion. So in my example I would like my "mtl_fpga_new_driver_a_profiling.snippet" to include "mtl_fpga_profiling.snippet" and use additional driverA_overlay.conf overlay.

[nordicjm]

feature request: snippet files may include (or exclude too!)

I don't personally see the merit in excluding files, if you want a different set of files, create a new snippet. Likewise in a project with build files, if you want to exclude files, create a new project or create a separate configuration directory and set APP_CONFIGURATION_DIR to point to those files instead of hacking around cmake files trying to alter zephyr variables.

[aborisovich]

feature request: snippet files may include (or exclude too!)

I don't personally see the merit in excluding files, if you want a different set of files, create a new snippet. Likewise in a project with build files, if you want to exclude files, create a new project or create a separate configuration directory and set APP_CONFIGURATION_DIR to point to those files instead of hacking around cmake files trying to alter zephyr variables.

Hmmm yes, maybe excluding files does not have that much sense indeed. Inclusion would suffice.

[mbolivar-nordic]

@aborisovich thanks for your input.

I see your point that people might initially think snippets are more trivial than they really are, but I don't think this is going to be a real problem in practice once we have several nontrivial built-in examples. I personally find the name "config_bundle" too verbose and too similar to other names. I wanted a name that clearly stood out and would not be easy to mistake for some other build system feature.

That said, we haven't made any zephyr release with this feature as it is currently named, so if this is a serious issue for you, please escalate to architecture WG or dev-review to see if others agree with you.

Regarding your feature request, it seems like a reasonable extension. I guess the main things to get right would be to detect or avoid circular includes, deciding on semantics if snippets A and B both include C, etc. Perhaps you can propose an extension to the YAML syntax and associated semantics that would resolve these types of issues.