Replace `pixz` with `pigz` for faster RAUC bundle creation?

[yfyf]

Currently RAUC bundles are compressed using pixz. This takes very long, on my machine:

real    12m9.685s
user    36m54.183s
sys     0m18.055s

The speed-up with pigz is 10x:

real    1m17.513s
user    3m30.048s
sys     0m6.234s

This eliminates the indexing support that pixz provides, but is it used anywhere?

To ensure that nothing breaks, it's best to do this after #200 is merged - this way we can verify that older releases properly handle the differently-compressed RAUC bundles.

We can also be even more adventurous and replace xz (i.e. LZMA) with zstd for broader improvements in space efficiency as well.

[yfyf]

Trying out different compression options, on my laptop (while using it, so times are obviously only ballpark):

Compression	Time	RAUC bundle size
pixz (current)	5m3s	1323M
pixz -1	1m56s	1447M
pixz -2	2m33s	1417M
xz -T0 -0	2m36s	1502M
xz -T0 -1	2m2s	1441M
xz -T0 -1 --block-size=32MiB	2m21s	1432M
xs -T0 -3	3m54s	1393M
pigz	35s	1.7G
pigz -b 4096 -9	1m29s	1.6G
pigz -b 4096 -11	too long...	n/a
pzstd -9	45s	1.5G
pzstd -19	6m28s	1.4G

It seems pzstd -9 provides the best improvement of speed over compression loss (~10x faster compression, but ~15% larger bundle size), however....

Midway into this, I realized we cannot use non-xz (LZMA) compressed archives in any near future, because our build of RAUC does not have zstd or gzip in it's PATH, only xz. We could only prep for such a switch in the long-term by adding, say, zstd to PATH.

Even though modern xz is multi-threaded and theoretically could provide better compression than pixz, it seems that pixz is a bit faster with the types of data we are compressing, without significant loss in compression ratio (I am guessing since a lot of the stuff in the bundle is already pre-compressed, like initrd and kernel images).

My suggestion would be to keep pixz, but reduce the compression level to 1, which would speed up compression more than twice at the cost of a ~10% increase in bundle size. @knuton yay or nay?

[knuton]

Thanks, very useful!

My suggestion would be to keep pixz, but reduce the compression level to 1, which would speed up compression more than twice at the cost of a ~10% increase in bundle size. @knuton yay or nay?

Let's walk through the practical implications/tradeoffs:

• 10 % bigger upload and storage size: Non-issue
• 10 % bigger download size: Non-issue
• 10 % bigger size on target RAM (< 2025.3.3): Increased risk of OOM during update, this risk diminishes as most machines saturate with >= 2025.3.3
• 10 % bigger size on target disk (>= 2025.3.3): ???
• Probably less load on CPU during decompression on target system?

In wich situations do we want to bring down build time? Ideally in all, but e.g. we release to master channel rarely, maybe the tradeoffs are different?

Also pixz -2 may be worth considering, halving the time at < 10 % size increase.

Some ideas:

• Stick to current scheme and wait for saturation with non-tmpfs raucb downloaders
• Go to pixz -2 as first step, wait for saturation with non-tmpfs raucb downloaders, then possibly go to pixz -1
• Choose compression level according to build context

[yfyf]

In wich situations do we want to bring down build time?

Two primary motivations to speed up the compression:

1. RAUC bundle creation is a significant chunk of e2e tests and release-validation validation (even the "build only"), so this would bring down the CI times
2. Fast RAUC bundle builds enable faster turnover for various experiments (e.g. scp $(nix-build -A components.unsignedRaucBundle) <machine>:./rauc.bundle && ssh <machine> 'rauc install rauc.bundle && systemctl reboot' can probably run in <5 mins. Overall, just makes various things smoother.

Stick to current scheme and wait for saturation with non-tmpfs raucb downloaders

I would probably go with this approach, wait for one more release cycle, then switch to pixz -1, there is no rush with this.

10 % bigger size on target disk (>= 2025.3.3): ???

I think this is OK. We did the math before doing the change and left wiggle room for bundle growth.

Choose compression level according to build context

Yeah, I was thinking we could make QUICK_BUILD=1 be pixz -0, but this only helps with point 2 above, not point 1.

I guess you are also thinking about things like if channel == master then pixz -9, but I would be cautious about any divergences between environments, even if it's just size / compression speed.