Introduce a new CMake conf variable for app-overridable configurations

[kail]

Problem Description

Background
Currently there are two ways to pass in the primary application config into the build:

1. providing all configuration files via the CONF_FILE variable, and
2. leveraging the mechanism in configuration_files.cmake:38 to find the appropriate app-board-variant configurations in their default locations.

Problem
The problem with the second approach is that there is no way to provide common configurations that can be overriden by the application. This is because the EXTRA_CONF_FILE variable -- which is intended to allow additional project configurations -- is included after the CONF_FILE_AS_LIST variable in kconfig.cmake:296, and the only configurations that come before it are board-specific.

Why this is a problem
This is an issue for projects that are organized such that there is a common platform with multiple configuration files in between the board and app layer and a high cardinality of applications. Since there is no way to override common configurations by the applications, all application configurations must be specified via CONF_FILE for every application, resulting in a more fragile build setup and a lot of boilerplate code across all applications' build setup.

Proposed Change (Summary)

The proposed change is to introduce a new PRE_CONF_FILE build variable (open to other name suggestions) that is equivalent to the existing EXTRA_CONF_FILE in processing and usage. This variable will be included in the merge_config_files list before CONF_FILE_AS_LIST, resulting in merging behavior that allows the configs in CONF_FILE to override their behavior.

Proposed Change (Detailed)

1. Introduce a new build variable PRE_CONF_FILE that is the equivalent of EXTRA_CONF_FILE in how it's handled/processed
2. Add a pre-user-files entry in build-schema.yml under the kconfig element
3. In configuration_files.cmake call zephyr_get for PRE_CONF_FILE
4. In kconfig.cmake:81 check for the existence of PRE_CONF_FILE and set up PRE_CONF_FILE_AS_LIST, if the variable is set
5. Add PRE_CONF_FILE_AS_LIST ahead of CONF_FILE_AS_LIST in merge_config_files within kconfig.cmake
6. Modify the west tool to enable passing in this argument

Overall, this change is very low-risk and is unlikely to break any existing behavior, unless the exact variable name is used by a project. This also has a low maintenance cost for maintainers of the project, since it is scoped to three files, and has limited interaction with the overall build process.

Example

Before
Let's say your "platform" has a platform.conf file with CONFIG_PLATFORM_INIT_PRIO=10. If you have a project with a board.conf, platform.conf, prj.conf, the Kconfig merge step will appear as follows (assuming EXTRA_CONF_FILE=/tmp/zephyr/platform/platform.conf):

Parsing /tmp/zephyr/Kconfig
Loaded configuration '/tmp/zephyr/boards/arm/board/board_defconfig'
Merged configuration '/tmp/zephyr/boards/arm/board/board.conf'
Merged configuration '/tmp/zephyr/apps/myapp/prj.conf'
Merged configuration '/tmp/zephyr/apps/myapp/boards/board.conf'
Merged configuration '/tmp/zephyr/platform/platform.conf'

The list above is in reverse order of precedence, meaning there is no way to override platform behavior with the prj.conf or the app/boards/board.conf file. So if your application needs to set CONFIG_PLATFORM_INIT_PRIO=15, based on some product-specific need, there is no way to do this.

After
With this change, users can set set(PRE_CONF_FILE platform/platform.conf) and the Kconfig processing will look as follows, therefore allowing the application/project to override the platform:

Parsing /tmp/zephyr/Kconfig
Loaded configuration '/tmp/zephyr/boards/arm/board/board_defconfig'
Merged configuration '/tmp/zephyr/boards/arm/board/board.conf'
Merged configuration '/tmp/zephyr/platform/platform.conf'
Merged configuration '/tmp/zephyr/apps/myapp/prj.conf'
Merged configuration '/tmp/zephyr/apps/myapp/boards/board.conf'

Dependencies

No response

Concerns and Unresolved Questions

1. Naming of the new variable. Names considered: PRE_PRJ_CONF_FILES, PRE_CONF_FILES, EARLY_EXTRA_CONF_FILES, EXTRA_PRE_CONF_FILES, PRE_EXTRA_CONF_FILES
2. Are there other patterns that projects use to address this kind of issue that we have not considered

Alternatives Considered

1. Using board_extension_conf_files for this purpose. This is not really the purpose of this variable, and not all configurations are board-specific.
2. Putting all configurations into CONF_FILE. This resulted in a less clean CMake file and required manually setting up all app-board configurations, even though this is already handled by Zephyr (i.e. duplicates functionality)
3. Copying the zephyr_file logic in configuration_files.cmake into our internal CMake file. This is a bit cleaner than (2), however it still duplicates functionality, and it's likely that others would benefit from this change, so chose to make it an RFC
4. Shields: this doesn't work cleanly, because it's the inverse of what is desired (i.e. shields override board and project config behaviour) and isn't what shields are meant for (as discussed in shields: add generic shield to use CDC ACM UART as backend #40645)
5. Snippets: snippets have a similar issue as shields, in that they provide overrides to project configurations, not the other way around

[github-actions]

Hi @kail! We appreciate you submitting your first issue for our open-source project. 🌟

Even though I'm a bot, I can assure you that the whole community is genuinely grateful for your time and effort. 🤖💙

[nordicjm]

Introduce a new build variable PRE_CONF_FILE that is the equivalent of EXTRA_CONF_FILE in how it's handled/processed

Please, no, this is not wanted behaviour

[kail]

@nordicjm Can you elaborate and provide a concise explanation as to why you don’t believe this is a good fit?

[felixturgeonmeta]

@nordicjm Reading the RFC, the behavior is optional for users. If a user wants to specify pre-configs, then they can. If they do not, then there is no behavior change.