Offer to maintain 6.12-metal kernel under bottlerocket-os org

[mikn]

After the community meeting and the info that the decision has been made to drop metal driver support from the 6.12 LTS kernel we had a conversation internally about the removal of metal drivers from the 6.12 kernel.
An alternative we would like to discuss with the maintainers is that we, as users of the metal compatible config of the current kernel, could offer to within the https://github.com/bottlerocket-os/bottlerocket-kernel-kit maintain the 6.12 kernel with metal drivers as a separate RPM package.

It would simplify our footprint if we could just carry a separate config set in the upstream and maintain it together with you as a separate RPM package rather than building a kernel from scratch/forking the kernel-kit and publishing it under our own org.
It would also reduce the work (potentially?) for you in getting tooling and testing in to support installing an upstream kernel.

We could commit to keeping the config up to date with the hardware we run on (a mixture of Dell and Supermicro for now), but also respond to any user requests for driver inclusions.

Is this something you could consider? We could use CODEOWNERS to allow us/me to only merge against the metal config so that we have no authority over the mainline variants still.

[larvacea]

The Bottlerocket team is exploring alternatives for providing a 6.12 kernel with a much wider range of drivers. We don't have final details yet, but we do want to address your needs. If you can share either a list of drivers, or a general-purpose Linux distribution that includes what you need, that would help us.

[bcressey]

It would simplify our footprint if we could just carry a separate config set in the upstream and maintain it together with you as a separate RPM package rather than building a kernel from scratch/forking the kernel-kit and publishing it under our own org.
It would also reduce the work (potentially?) for you in getting tooling and testing in to support installing an upstream kernel.

We're working under these constraints:

1. We need to publish security fixes for known vulnerabilities in any software we ship. This applies to the artifacts as built; if we disable a particular feature, and the vulnerability is not present in the built artifact, then there's no need to publish a fix.
2. For the kernel, we depend on the Amazon Linux kernel team to triage and patch vulnerabilities. However, we can't expect them to patch vulnerabilities in features that they don't enable, such as additional drivers.

So that leaves us in a bind: if we enable additional modules, then they come into scope for patching, and we can't guarantee that patches will be applied.

Some paths forward we've considered:

1. Provide reference kernel configs (somewhere) that we can collectively maintain, which could be used with a corresponding vendor's kernel, but that's disconnected from any shipped binary. This avoids the patching problem but requires downstreams to assemble their own kernel configs and packages, potentially fragmenting the ecosystem.
2. Select additional upstream vendors for kernels, add the Bottlerocket-specific configuration settings, and otherwise pass-through their module choices. This solves the patching problem provided that we turn around new builds quickly, assuming that the upstream vendor publishes fixes for their kernel.

If none of the other distro vendor kernels are sufficiently lean, then one path might be to maintain a kernel config, pair it with the upstream kernel.org LTS release, and ship that as an alternative kernel package.

That may be what you were suggesting, but just to spell it out for clarity: relative to the current set of kernels, the critical difference is that Amazon Linux wouldn't be in the loop at all. All the patching for that kernel would happen upstream, and all the testing would happen downstream.

[mikn]

I have spent some significant time and research on this and the main reason I wanted to keep the Amazon Linux upstream is because I know the AL team maintain a significant patch stack (gregkh/linux@linux-6.12.y...amazonlinux:linux:amazon-6.12.y/mainline) which is quite relevant to us also. If I understand it correctly, the AL team follows the weekly releases of the kernel.org releases relatively well (much more so than most more established upstreams with larger patch stacks) and the kernel.org team has no OOB patch mechanism other than emailing out to the open vendors mailing list whenever a CVE fix is available.

The problem for the metal kernel would be the same, in other words, if we were using the kernel.org kernel or the AL kernel upstream, which is that someone would need to add themselves to the vendor list and make sure to cherrypick out critical CVE patches that the AL team would ignore because of irrelevance outside of this regular weekly release cycle. The burden would if anything be lighter with the AL upstream because at least some of (or most/all?) the patches would be cherrypicked for us as they are within the purview of the AL team. Maybe you could figure out if and how they filter those internally (?)

We do not have a full-time person who could be readily available to cherrypick security patches, and for our use-case, a worst case 1 week response time is fine, and will be fine for a long time.
I see a future where allowing live patching similar to AL would be interesting, and that being part of the Bottlerocket core would be ideal.

If relying on the weekly kernel release also for security patches on the kernel is not good enough in general to fulfil your obligations, is there a way you could label, mark or document the metal kernel in particular as an "unsupported" artefact? or a "community supported" artefact perhaps?

[larvacea]

Thanks for the response. It's helpful, in particular, to know what you value in the current Amazon Linux upstream.

[bcressey]

If relying on the weekly kernel release also for security patches on the kernel is not good enough in general to fulfil your obligations, is there a way you could label, mark or document the metal kernel in particular as an "unsupported" artefact? or a "community supported" artefact perhaps?

In areas other than security, documenting the limitations of technical support would be fine. I don't think it provides any air cover here: the distributed artifacts will either be free of known vulnerabilities or they won't be.

On my list of fun things to explore is RPM's new declarative builds mechanism. It could clean up a lot of boilerplate - most of the specs for cross-compiling autotools-based libraries are very similar.

In the best possible version of this, a custom kernel build would only need a few lines for a package spec, something like:

Name: %{_cross_os}custom-kernel-6.12

# This would come from the kernel kit.
BuildRequires: %{_cross_os}kernel-6.12-source-rpm

# This would be maintained locally.
Source1: custom-kernel-config

# This would handle %prep, %build, and %install.
BuildSystem: cross-kernel

%files -f cross-kernel.list

Ideally, Bottlerocket's own kernel builds would be converted to the same build system so we tripped over any problems first. Conceptually those builds are just Amazon Linux kernel RPMs with a custom kernel config and some patches, so it should work for them if it's going to work downstream.

If something like that existed, would it meet your needs? You'd only need to maintain the kernel config and after that the custom kernel would be rebuilt whenever the kit dependencies were updated.

[rajivr]

Even though I am not actively working on metal use-case, it would be really nice to see a bottlerocket-community org.

With SystemReady IR, we will soon see many reasonably priced IoT boards that might have proper UEFI/SecureBoot support. bottlerocket-community org will be a great place where we can share our experiments working with such boards.

[bcressey]

If there could be a way for you to create a bottlerocket-community org or similar for us to clearly separate, but still share infra for a built artefact (a metal kernel kit), I think that would largely help also. It would create a clear space for community members to open PRs and ask for support specifically around the metal kernel also.

Let me try to clarify the request so I can make sure we're on the same page 😀

1. Create bottlerocket-community org (or similar).
2. Create bottlerocket-metal-kit (or similar) within that org.
3. Adopt an explicit governance model for the above (e.g. MVG).
4. Optionally, provide PR testing via GitHub actions (unsure about costs for this).
5. Optionally, provide ECR hosting for published kit artifacts (unsure about budget for this).

Does that sound about right?

[mikn]

Yes, something like that :)
Of course ECR hosting would make this a lot easier to use, but I guess it may end up in the same problem space as distributing it directly in the kernel kit?

[bcressey]

I don't have anything substantial to announce, but I've started the discussion with internal stakeholders and made some progress.

[bcressey]

We now have the bottlerocket-community org, so we're one step closer!

The last blocker is figuring out the right content for the .github project there, and then I should be ready to create bottlerocket-metal-kit.