Make k0s a true "self-extracting ZIP archive"

[twz123]

Is your feature request related to a problem? Please describe.

K0s is a single binary that includes all of its runtime dependencies. This stems from the fact that k0s was historically designed to have controller nodes without a container runtime, among other things. K0s calls these "embedded binaries". Embedding is currently achieved by compressing and concatenating the binaries, then compiling the resulting byte offsets into k0s and appending the binary blob to the executable.

This has severe implications for the k0s build system. Before k0s can be built, the build system requires all the embedded binaries to be in place so it can compress them, record the offsets, and incorporate them into the k0s build process. This creates tight coupling between the k0s build and these dependencies. Binaries cannot be built in parallel with k0s itself, nor can they be brought in after the fact. K0s even exists in two variants: one with embedded binaries and one without. There's a Go build tag that distinguishes between the two at compile time. Over time, the k0s build system has grown complex, and a significant portion of that complexity stems from this coupling.

Moreover, this method of embedding binaries is self-invented by k0s. There are no standard tools being leveraged, nor is there any obvious prior art being followed. Those unfamiliar with the process usually have a hard time understanding how it works and why it is designed the way it is.

Describe the solution you would like

Indeed, there is one industry standard that has been in use for decades: The ZIP file format. It's ideal for self-extracting executables because it stores its index at the end of the file, which is exactly where it needs to be when appended to an arbitrary executable.

Migrating to ZIP-based embedding would bring several improvements:

• Byte offsets no longer need to be computed at compile time; the ZIP index provides them.
• The ZIP payload can be assembled independently of the k0s build and even added or modified after the k0s binary has been produced.
• There is no longer any need for a dedicated Go build tag; there would be a single k0s build regardless of whether additional binaries are appended.
• Standard tooling can be used to create ZIP archives.
• Standard tooling can also be used to inspect or extract embedded binaries directly from the k0s binary. (Although I would consider the naming convention for the files stored in the ZIP archive an implementation detail for the time being.)
• The implementation would rely on a well-known, well-documented format that is easier to understand and maintain long-term.

Describe alternatives you've considered

¯\_(ツ)_/¯

Additional context

Using a standard way (ZIP) to piggy-back arbitrary content onto the k0s binary without recompilation opens interesting opportunities for user customization. For example, in the future, k0s could automatically apply manifests via the stack applier whenever they are found in the appended ZIP file. Similarly, k0s could install OCI images from the ZIP file into containerd, as it currently does with OCI image bundle files in the <data-dir>/manifests folder.

From a security perspective, the ZIP payload generated by the CI is considered part of the k0s release artifact. K0s would still be shipped in the same way. Official builds are signed and hashed as a whole and k0s treats the appended ZIP payload as trusted content. If users choose to customize k0s by modifying or appending ZIP content, the resulting executable should be treated as a derivative artifact: upstream signatures will no longer apply, and it becomes the responsibility of the customizer to generate and distribute their own hashes or signatures if integrity verification is required. We may also choose to publish a signature file for the "plain" k0s executable without an appended ZIP payload as an additional release artifact. This would allow users to verify the integrity of the core k0s binary against an upstream signature, even if the ZIP payload has been modified.

[jnummelin]

I think we could also use this to embed all the needed k0s own system manifests/templates in the same zip instead of using the bin-data stuff

[twz123]

I think we could also use this to embed all the needed k0s own system manifests/templates in the same zip instead of using the bin-data stuff

Initially I was thinking the same, but the story is a bit different for the manifests: They are actually required in a k0s executable that's being built without embedded binaries. Also, the way we embed them has changed. They are pulled in via go:embed, and most of the drivers mentioned in the issue description don't apply to them: They don't need to be built, they're static assets. Well-known standard Golang tooling is used to embed them. They don't add much complication to the build system. In fact, trying to add them to the embedded payload could make things more difficult, I suppose.