Should images be stored in Git LFS?

[tpimh]

Images are binary files, and they make up a significant part of the repository size. Each rootfs is 20Mb and each cpio is ~3Mb. Getting more architectures to build means also having more corresponding images. Given that we have 1Gb free LFS storage, maybe we can make use of it.

[nickdesaulniers]

So IIRC, @nathanchance created the images. Looking in driver.sh, seems like there's a mix of cpio and ext4 image use. If the cpio's are significantly smaller, can we drop the ext4 images? Do we need to append root= when using an initrd? (I think I was being a stick in the mud about keeping the ext4 images, but if I had a good reason to be, I no longer recall it)

To your point about binaries in git; these images are essentially never changing, or changing infrequently enough that they'd take up significant space, so it's not problematic yet, IMO. But I don't feel strongly about it; mostly I don't know anything about LFS storage. Can you point me to more info on how that would work?

[tpimh]

I don't know for sure about platform support, but pretty sure all should be able to use cpio images as initrd, so ext4 images can then be dropped.

In short, when using LFS selected files are not stored in repository, but instead are uploaded to a file server and a link to the file is stored in repository instead. Better explanation can be found in official docs.

[tpimh]

@nathanchance recently published updated images: 02fa71e The repo is getting bigger with every update.
Nathan, do you think we can drop ext4 images and only use cpio rootfs instead? Can we use Git LFS for cpio images in the future?

[nathanchance]

I suppose I am fine with dropping the ext4 images (although I am fairly certain we have caught bugs that way because we deal with an actual file system rather than just the ramdisk code).

The cpio image sizes are rather benign, do we actually need to use LFS? I'm just not exactly a huge fan of adding another binary to my workflow if I don't need to but if you all feel it's really beneficial, then sure.

3.0M    images/arm/rootfs.cpio
8.7M    images/ppc64le/rootfs.cpio
3.3M    images/arm64/rootfs.cpio
5.0M    images/ppc32/rootfs.cpio
3.5M    images/x86_64/rootfs.cpio

[tpimh]

Given that these images are not updated frequently and are pretty slim, it should be fine to proceed without LFS. How about gzipping them?

[nickdesaulniers]

Sure, why don't you see what makes them smallest; gzip, bzip, lz4 etc?

[tpimh]

Here is compression test of current rootfs.cpio for every arch:

	arm	arm64	mipsel	ppc32	ppc64	ppc64le	x86_64
no comp	3061248	3405824	3805696	5239296	9157632	9049600	3565568
bzip2 -9	1355957	1464769	1470082	1319283	2636797	2740250	1491278
gzip -9	1427459	1518362	1580781	1457225	2872146	2958789	1533389
lz4 -9	1676880	1794616	1856926	1695866	3325128	3393928	1788177
lzma -9 -e	1063855	1142821	1148776	1037374	2005783	2143598	1251130
lzop -9	1549072	1658336	1727513	1575432	3112088	3198560	1661583
xz -9 -e	1064064	1143040	1148996	1037572	2006136	2143972	1251372

LZMA is showing the best compression.

[nickdesaulniers]

The tradeoff in more efficient compression may be extended time to decompress, but if the tests aren't timing out, than I'm ok with crushing the images to be as small as possible.

[nathanchance]

I should write a hyperfine benchmark that does both compression and decompression times to see where we get the best all around results; I think that finding the right balance of decompression speed with space savings should be the goal because compression is only going to happen when we add new architectures or update the images (so not often) but decompression will happen all the time.

[nathanchance]

Okay, I got around to running some benchmarks :)

Results

Click for results

arm (compression)

Command	Mean [ms]	Min [ms]	Max [ms]	Relative speed	Relative size
bzip2 -9	267.1 ± 3.2	261.9	271.8	2.58 ± 0.09	44%
gzip -9	428.4 ± 3.9	423.8	433.8	4.14 ± 0.14	46%
lz4 -9	103.6 ± 3.5	97.9	108.4	1.00	54%
lzma -9	1087.0 ± 6.0	1079.1	1104.1	10.49 ± 0.36	34%
lzop -9	611.4 ± 3.0	603.4	616.0	5.90 ± 0.20	50%
xz -9	1091.7 ± 6.3	1081.9	1106.9	10.54 ± 0.36	34%
zstd -19	673.0 ± 4.5	666.1	682.2	6.50 ± 0.22	40%

arm (decompression)

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
bzip2 -d	155.0 ± 3.2	150.1	159.1	15.18 ± 2.82
gzip -d	36.2 ± 2.9	30.9	40.6	3.55 ± 0.72
lz4 -d	10.2 ± 1.9	5.5	12.9	1.00
lzma -d	86.0 ± 2.8	81.0	89.6	8.43 ± 1.58
lzop -d	19.9 ± 3.0	14.2	23.8	1.95 ± 0.47
xz -d	88.5 ± 3.2	82.9	92.7	8.67 ± 1.63
zstd -d	17.5 ± 2.9	11.8	22.2	1.71 ± 0.42

arm64 (compression)

Command	Mean [ms]	Min [ms]	Max [ms]	Relative speed	Relative size
bzip2 -9	279.4 ± 3.2	273.5	284.5	2.57 ± 0.07	43%
gzip -9	551.7 ± 4.3	545.7	560.9	5.07 ± 0.14	44%
lz4 -9	108.7 ± 2.8	104.0	113.5	1.00	52%
lzma -9	1203.5 ± 4.0	1197.4	1216.2	11.07 ± 0.29	33%
lzop -9	818.5 ± 3.0	813.5	823.8	7.53 ± 0.20	48%
xz -9	1206.3 ± 3.7	1199.6	1213.1	11.09 ± 0.29	33%
zstd -19	730.1 ± 5.9	722.1	753.7	6.71 ± 0.18	39%

arm64 (decompression)

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
bzip2 -d	161.2 ± 3.5	157.6	167.0	16.51 ± 4.04
gzip -d	37.3 ± 3.3	33.0	41.4	3.82 ± 0.99
lz4 -d	9.8 ± 2.4	5.9	13.7	1.00
lzma -d	92.5 ± 3.1	86.6	95.6	9.47 ± 2.33
lzop -d	21.0 ± 3.2	14.8	24.9	2.15 ± 0.62
xz -d	95.5 ± 2.8	90.9	99.1	9.78 ± 2.40
zstd -d	18.9 ± 3.1	13.3	23.0	1.93 ± 0.57

mips (compression)

Command	Mean [ms]	Min [ms]	Max [ms]	Relative speed	Relative size
bzip2 -9	311.9 ± 3.4	307.3	316.4	2.26 ± 0.06	37%
gzip -9	712.1 ± 3.8	707.4	716.9	5.17 ± 0.13	41%
lz4 -9	137.8 ± 3.3	131.9	141.4	1.00	48%
lzma -9	1405.6 ± 5.2	1397.8	1421.2	10.20 ± 0.25	29%
lzop -9	878.2 ± 2.9	871.4	882.8	6.37 ± 0.15	45%
xz -9	1409.4 ± 4.5	1397.9	1420.5	10.23 ± 0.25	29%
zstd -19	889.9 ± 3.7	881.6	896.3	6.46 ± 0.16	35%

mips (decompression)

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
bzip2 -d	174.8 ± 2.9	168.6	178.5	14.70 ± 3.29
gzip -d	40.0 ± 3.2	35.5	45.0	3.36 ± 0.80
lz4 -d	11.9 ± 2.7	5.9	15.1	1.00
lzma -d	91.6 ± 3.9	85.6	96.3	7.70 ± 1.75
lzop -d	22.6 ± 2.8	17.0	27.2	1.90 ± 0.49
xz -d	97.3 ± 3.1	91.5	102.2	8.18 ± 1.84
zstd -d	20.1 ± 3.0	13.9	25.0	1.69 ± 0.45

mipsel (compression)

Command	Mean [ms]	Min [ms]	Max [ms]	Relative speed	Relative size
bzip2 -9	327.4 ± 3.4	323.0	332.8	2.41 ± 0.06	38%
gzip -9	508.9 ± 3.5	504.9	514.1	3.74 ± 0.08	41%
lz4 -9	136.0 ± 2.9	131.0	139.9	1.00	48%
lzma -9	1432.6 ± 4.7	1422.8	1439.9	10.54 ± 0.23	30%
lzop -9	809.0 ± 3.5	802.9	813.9	5.95 ± 0.13	45%
xz -9	1435.2 ± 4.5	1428.0	1444.3	10.55 ± 0.23	30%
zstd -19	886.6 ± 3.7	880.5	892.1	6.52 ± 0.14	35%

mipsel (decompression)

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
bzip2 -d	179.8 ± 2.9	174.4	184.0	15.02 ± 2.81
gzip -d	41.0 ± 3.5	35.9	45.0	3.43 ± 0.70
lz4 -d	12.0 ± 2.2	6.6	16.5	1.00
lzma -d	90.7 ± 3.1	84.7	94.9	7.57 ± 1.43
lzop -d	22.1 ± 3.1	16.4	26.8	1.85 ± 0.43
xz -d	95.7 ± 2.1	90.2	98.8	8.00 ± 1.50
zstd -d	19.6 ± 3.0	13.8	23.4	1.64 ± 0.40

ppc32 (compression)

Command	Mean [ms]	Min [ms]	Max [ms]	Relative speed	Relative size
bzip2 -9	450.5 ± 3.7	444.3	455.8	2.37 ± 0.04	25%
gzip -9	565.8 ± 3.7	561.8	570.8	2.98 ± 0.05	27%
lz4 -9	189.7 ± 3.2	183.4	193.3	1.00	32%
lzma -9	1118.2 ± 4.9	1109.9	1132.9	5.89 ± 0.10	19%
lzop -9	705.0 ± 3.7	697.7	710.7	3.72 ± 0.07	30%
xz -9	1122.8 ± 3.8	1117.2	1132.3	5.92 ± 0.10	19%
zstd -19	688.2 ± 3.3	684.9	694.5	3.63 ± 0.06	23%

ppc32 (decompression)

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
bzip2 -d	155.4 ± 3.4	151.0	160.8	14.01 ± 3.77
gzip -d	44.4 ± 3.6	40.2	50.0	4.00 ± 1.12
lz4 -d	11.1 ± 3.0	6.5	16.1	1.00
lzma -d	86.9 ± 3.2	82.4	91.5	7.83 ± 2.12
lzop -d	24.3 ± 2.2	19.7	28.2	2.19 ± 0.62
xz -d	93.1 ± 3.2	87.3	97.0	8.39 ± 2.27
zstd -d	20.4 ± 2.6	14.0	24.1	1.84 ± 0.55

ppc64 (compression)

Command	Mean [ms]	Min [ms]	Max [ms]	Relative speed	Relative size
bzip2 -9	618.8 ± 3.3	611.7	624.4	2.46 ± 0.03	28%
gzip -9	2176.2 ± 3.2	2171.0	2180.7	8.66 ± 0.10	31%
lz4 -9	251.4 ± 2.9	246.5	256.0	1.00	36%
lzma -9	2615.5 ± 7.8	2603.6	2631.8	10.40 ± 0.12	21%
lzop -9	2276.9 ± 4.2	2269.7	2283.7	9.06 ± 0.10	33%
xz -9	2623.6 ± 4.7	2614.4	2634.7	10.44 ± 0.12	21%
zstd -19	1725.3 ± 6.5	1715.4	1749.4	6.86 ± 0.08	25%

ppc64 (decompression)

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
bzip2 -d	304.1 ± 3.0	299.8	308.4	20.86 ± 4.05
gzip -d	77.1 ± 2.4	72.5	80.1	5.29 ± 1.04
lz4 -d	14.6 ± 2.8	9.6	19.6	1.00
lzma -d	161.2 ± 2.7	157.0	165.5	11.06 ± 2.15
lzop -d	36.3 ± 3.0	32.2	41.2	2.49 ± 0.53
xz -d	168.0 ± 3.1	163.8	173.1	11.52 ± 2.25
zstd -d	30.1 ± 3.0	26.3	35.1	2.07 ± 0.45

ppc64le (compression)

Command	Mean [ms]	Min [ms]	Max [ms]	Relative speed	Relative size
bzip2 -9	652.2 ± 4.2	646.1	659.8	2.58 ± 0.04	30%
gzip -9	1118.3 ± 3.7	1114.0	1123.6	4.42 ± 0.06	32%
lz4 -9	253.1 ± 3.3	248.5	257.8	1.00	37%
lzma -9	2602.8 ± 11.0	2584.7	2631.2	10.28 ± 0.14	23%
lzop -9	1722.7 ± 4.7	1717.5	1731.9	6.81 ± 0.09	35%
xz -9	2608.8 ± 7.1	2599.0	2623.7	10.31 ± 0.14	23%
zstd -19	1574.9 ± 12.7	1563.8	1619.3	6.22 ± 0.10	27%

ppc64le (decompression)

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
bzip2 -d	319.3 ± 4.0	313.2	331.5	21.05 ± 3.52
gzip -d	77.9 ± 2.2	72.0	80.7	5.14 ± 0.87
lz4 -d	15.2 ± 2.5	9.6	18.8	1.00
lzma -d	166.5 ± 2.4	161.0	170.9	10.98 ± 1.84
lzop -d	36.7 ± 3.4	32.0	42.3	2.42 ± 0.46
xz -d	175.4 ± 2.3	169.9	178.7	11.56 ± 1.93
zstd -d	30.0 ± 3.2	26.6	36.1	1.97 ± 0.39

x86_64 (compression)

Command	Mean [ms]	Min [ms]	Max [ms]	Relative speed	Relative size
bzip2 -9	268.9 ± 3.8	259.8	272.9	2.13 ± 0.06	41%
gzip -9	652.2 ± 3.4	647.4	656.9	5.17 ± 0.13	43%
lz4 -9	126.1 ± 3.2	120.5	129.5	1.00	50%
lzma -9	1196.0 ± 5.8	1189.3	1208.2	9.48 ± 0.24	35%
lzop -9	942.8 ± 3.9	937.5	950.6	7.48 ± 0.19	46%
xz -9	1201.0 ± 5.9	1193.6	1209.9	9.52 ± 0.25	35%
zstd -19	732.3 ± 2.8	727.5	737.5	5.81 ± 0.15	38%

x86_64 (decompression)

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
bzip2 -d	165.4 ± 3.0	158.1	169.4	15.70 ± 3.72
gzip -d	38.7 ± 3.3	33.3	43.0	3.67 ± 0.92
lz4 -d	10.5 ± 2.5	5.5	14.1	1.00
lzma -d	98.4 ± 3.6	93.5	104.0	9.34 ± 2.23
lzop -d	20.3 ± 3.1	15.1	25.0	1.93 ± 0.54
xz -d	100.7 ± 3.2	95.5	105.6	9.56 ± 2.28
zstd -d	19.8 ± 2.2	13.9	23.3	1.88 ± 0.49

TL;DR: I believe we should use zstd because it has the second fastest decompression speed with the second most compression. The compression speed is the middle of the pack but better than lzma, which was worst in terms of decompression.

Now... the reason I did this now is that I am writing a set of regression/unit tests (basically what this repo does on a bit more of a grand scale) that will live in the tc-build repo but it needs these images for its boot tests. As a result, I propose that we split the images and buildroot folders off into their own repo, where we can compress them properly from the get go, then we just shallow clone that repo into this one in driver.sh initially, and I can use it for those tests in that repo. Thoughts? @tpimh @nickdesaulniers

Script

#!/usr/bin/env bash

BASE=$(cd "$(dirname "$(readlink -f "${BASH_SOURCE[0]}")")" && pwd)
CPIO_IMG=rootfs.cpio
NINE_K_IMG=( -9 -k "${CPIO_IMG}" )
BZ2_COMP=( bzip2 "${NINE_K_IMG[@]}" )
GZ_COMP=( gzip "${NINE_K_IMG[@]}" )
LZ4_COMP=( lz4 -9 "${CPIO_IMG}" "${CPIO_IMG}".lz4 )
LZMA_COMP=( lzma "${NINE_K_IMG[@]}" )
LZOP_COMP=( lzop -9 "${CPIO_IMG}" )
XZ_COMP=( xz "${NINE_K_IMG[@]}" )
ZSTD_COMP=( zstd -19 "${CPIO_IMG}" )

# Create results folder
RESULTS=${BASE}/results
mkdir -p "${RESULTS}"
# Benchmark compression
for FOLDER in "${BASE}"/images/*; do
    COMP_RESULTS=${RESULTS}/${FOLDER##*/}-comp-results.md
    cd "${FOLDER}" && \
    hyperfine --export-markdown ${COMP_RESULTS} \
              --prepare "rm -vrf ${CPIO_IMG}.*" \
              --warmup 1 \
              --runs 25 \
              "${BZ2_COMP[*]}" \
              "${GZ_COMP[*]}" \
              "${LZ4_COMP[*]}" \
              "${LZMA_COMP[*]}" \
              "${LZOP_COMP[*]}" \
              "${XZ_COMP[*]}" \
              "${ZSTD_COMP[*]}" && \
    rm -vrf "${CPIO_IMG}".zst && \
    "${BZ2_COMP[@]}" && \
    "${GZ_COMP[@]}" && \
    "${LZ4_COMP[@]}" && \
    "${LZMA_COMP[@]}" && \
    "${LZOP_COMP[@]}" && \
    "${XZ_COMP[@]}" && \
    "${ZSTD_COMP[@]}" && \
    echo >> "${COMP_RESULTS}"
    LINE=3
    for COMP_FILE in "${CPIO_IMG}".*; do
        PER_DIFF=$(echo "scale=2;$(stat --format=%s "${COMP_FILE}")/$(stat --format=%s "${COMP_FILE%.*}")*100" | bc | sed 's/\..*$/\%/') && \
        sed -i "${LINE}s/$/ ${PER_DIFF} |/" "${COMP_RESULTS}" && \
        LINE=$(( LINE + 1 ))
    done
    hyperfine --export-markdown ${RESULTS}/${FOLDER##*/}-decomp-results.md \
              --prepare "rm -vrf ${CPIO_IMG}" \
              --warmup 1 \
              --runs 25 \
              "bzip2 -d -k ${CPIO_IMG}.bz2" \
              "gzip -d -k ${CPIO_IMG}.gz" \
              "lz4 -d ${CPIO_IMG}.lz4" \
              "lzma -d -k ${CPIO_IMG}.lzma" \
              "lzop -d ${CPIO_IMG}.lzo" \
              "xz -d -k ${CPIO_IMG}.xz" \
              "zstd -d ${CPIO_IMG}.zst"
done

# Format files
for COMP_RESULT in "${RESULTS}"/*-comp-results*; do
    sed -i -e 's/Relative |/Relative speed | Relative size |/' -e '2s/$/---:|/' -e 's/9 .*r.*`/9`/' -e 's/9 .* -k.*`/9`/' "${COMP_RESULT}"
done
for DECOMP_RESULT in "${RESULTS}"/*-decomp-results*; do
    sed -i 's/ -d.*` / -d` /' "${DECOMP_RESULT}"
done

[tpimh]

I don't think we really should care about the compression speed as it's done so rarely the expenses can be neglected.

Ideally we can move buildroot and images directories to a separate repository (possibly, even preserving their git history), then set the images up to be built on travis if any of the configs is changed like we do with docker images.