Switch to managing the Linux kernel configuration by fragments

[fhunleth]

This discussion was started on Slack.

From @trarbr:

So I just watched this talk (https://www.youtube.com/watch?v=J8zxekwYMDY) on using fragments.
I guess the process for layering our own config on top of the linux.defconfig would be something like:

1. Run make linux-menuconfig
2. Search for the modules we need to enable (and their dependencies)
3. Write down ourselves which modules we enabled in a fragment file, e.g. "linux-mysystem-fragment.config"
4. Specify BR2_LINUX_KERNEL_CONFIG_FRAGMENT_FILES="${NERVES_DEFCONFIG_DIR}/linux-mysystem-fragment.config" in nerves_defconfig
   Note that the fragment file should only include the changes we explicitly made - any defaults that would automatically be applied to the defconfig on save don't have to be put in the fragment file, as it will happen automatically when Buildroot merges the defconfig with the fragment.
   I don't think this would require any changes to how Nerves does things, it's just on us to keep our own fragments updated. I think this would work for us.

As mentioned later, this breaks make linux-update-defconfig. On the other side, it also makes it possible to have comments and be very clear where Linux options come from - are they Nerves related, VintageNet support, etc?

[jjcarstens]

So this essentially becomes more of an organizational change?
The process to introduce new kernel pieces would be make a PR here and add a fragment config (like vintage-net-WiFi-fragment.config) or fork the system to make your own and update it, correct?

[fhunleth]

I think we'll have to play around with it to see what granularity is right. The change being proposed is only to delete linux-5.10.defconfig from this repository and replace it with a hierarchy that looks something like this:

nerves_defconfig
...
linux/
      base.config
      nerves.config
      vintage_net.config
      lte.config

Then update the nerves_defconfig to merge all of the config fragments above.

If you as a user want to change the Linux configuration, you still have to fork the system like before. Instead of running make linux-menuconfig and make linux-update-defconfig to enable your option, the new way will be to run make linux-menuconfig, write down what options you want, create a new .config file in your fork and then put the config strings in there.

This is a much more manual way of handling Linux configurations. The hope, though, is that it makes it a whole lot easier to figure out why options are enabled and maintain systems long term.

[trarbr]

Just to add some context, I want to note that the discussion was spurred by me complaining about the process we use at my company for maintaining our custom Nerves systems. This could easily be a ME problem - maybe we just don't upgrade often enough, or maybe we should follow another process. So here's what we do:

• We maintain a set of custom systems for RPi3, RPi4 and x86_64, all of which were originally forked from an official Nerves system.
• In our systems we have customised the fwup config files, nerves_defconfig and the linux-*.defconfig file.
• We keep our systems up to date by periodically pulling the latest tag from upstream and merging it into our main branch.
• Whenever we do this there's always some confusion around changes to the linux-*.defconfig file:
  • Why were these modules enabled/disabled?
  • Do they conflict with some of our changes?
  • Do we want this to be enabled/disabled?

A lot of this comes down to how the linux-*.defconfig file contains both explicit changes and implicit changes, and no clean corollary between them. Was CONFIG_FOO enabled because it is a dependency of CONFIG_BAR? Or for some other feature? Or was it automatically selected when CONFIG_BAZ was enabled?

In the end, in order to be feel confident about the merge, we usually perform manual testing on as many different hardware configurations as we have laying around. This can be pretty time consuming.

---

@fhunleth mentioned kernel config fragments, and I think they look interesting. I think in our case, my company would benefit from putting all our own changes in a fragment, and leaving the linux-.defconfig file alone. Whether the linux-.defconfig file should also be split into fragments, I am not sure. How disruptive would such a change be to other users? Will this make it easier for you to maintain the official systems?

The main advantage I see of fragments is that you only have to specify the changes you need - all the other modules which are automatically selected/unselected is not specified. So the intent becomes much clearer. Also, because the tooling doesn't manage the fragment files, it's easy to group a set of modules inside a fragment file and say "These all have to be enabled/disabled in order to run XYZ".

This also leads to the main drawbacks:

• You have to edit the files yourself 😄 There's no nice make linux-save-fragment command.
• There is no easy way to generate a defconfig file with the complete set of modules you enable/disable (for inspection purposes).
• When the kernel version is bumped, the default modules which are selected may change. If your system depends on a default select, this could break your system without warning.

[trarbr]

I finally took some time to experiment with kernel fragments. I'm working from the perspective of a downstream system maintainer, who just needs to add a few things to the official Nerves systems.

In this commit you can see how I added a fragment file. When running make linux-menuconfig after this, I saw this in the output (before the menuconfig-window popped open):

Using /home/troels/nerves_system_rpi0/.nerves/artifacts/nerves_system_rpi0-portable-1.20.0/build/linux-1.20220331/.config as base
Merging /home/troels/nerves_system_rpi0/linux_config/ax88179.config
#
# merged configuration written to /home/troels/nerves_system_rpi0/.nerves/artifacts/nerves_system_rpi0-portable-1.20.0/build/linux-1.20220331/.config (needs make)
#

When browsing around in the menuconfig window, I can see that the fragment has been applied (it has CONFIG_USB=y, which would be CONFIG_USB=n without the fragment).

I then built some firmware based on this system and booted an RPi 0W. This is the resulting config (taken from /proc/config.gz): https://gist.github.com/trarbr/42690130150d246e6754f6fa2fee6ccc

By skimming through the config file, it seems it has been applied on top of the linux-5.15.defconfig file, as it has CONFIG_WIREGUARD=m. I was afraid it might be based on a default defconfig file, so this is a huge win. It means that downstream system maintainers can use kernel fragments, without the upstream Nerves systems first being converted to using kernel fragments.

---

I have continued to play around with fragments to learn more of how they work. For example, what if we want to use several fragments?

• We can use Bash brace expansion: BR2_LINUX_KERNEL_CONFIG_FRAGMENT_FILES="${NERVES_DEFCONFIG_DIR}/linux_config/{one.config,two.config}"
• Or we can use globs: BR2_LINUX_KERNEL_CONFIG_FRAGMENT_FILES="${NERVES_DEFCONFIG_DIR}/linux_config/*.config"

Fragments are applied in the order they are listed. If using globs, then they are applied alphabetically.

Changes in later fragments can override changes in earlier fragments. In this case, a notice will be printed when the fragments are merged. It looks something like this:

Using /home/troels/nerves_system_rpi0/.nerves/artifacts/nerves_system_rpi0-portable-1.20.0/build/linux-1.20220331/.config as base
Merging /home/troels/nerves_system_rpi0/linux_config/ax88179.config
Merging /home/troels/nerves_system_rpi0/linux_config/bad.config
Value of CONFIG_USB is redefined by fragment /home/troels/nerves_system_rpi0/linux_config/bad.config:
Previous value: CONFIG_USB=y
New value: CONFIG_USB=n

#
# merged configuration written to /home/troels/nerves_system_rpi0/.nerves/artifacts/nerves_system_rpi0-portable-1.20.0/build/linux-1.20220331/.config (needs make)
#

I would like to find a way to merge the fragments into the base config without running make linux-menuconfig, but so far I have not succeeded. make linux-configure will do it, but it will also do all the linux configure stuff.