Systematically benchmark compression algorithm, compression factor, block size

[probonopd]

Execute a systematic and reproducible benchmark to find the optimal combination of

• Compression algorithm
• Compression factor
• Block size

in terms of

• File size
• Applicartion startup time
• zsync efficiency (delta update size)

for

• zlib
• lzma
• zstandard
• dwarfs
• libdeflate

References:

• Regression? higher startup time with zstd compression appimagetool#64
• Consider using DwarFS #36

[probonopd]

This test matrix includes 5 compression algorithms (zlib, lzma, zstandard, dwarfs, and libdeflate), 3 compression factors (low, mid, high), and 3 block sizes (256KB, 512KB, and 1MB), resulting in a total of 45 test cases.

Compression Algorithm	Compression Factor	Block Size	File Size	Application Startup Time	zsync Efficiency (Delta Update Size)
squashfs with zlib	6	256KB
squashfs with zlib	6	512KB
squashfs with zlib	6	1MB
squashfs with zlib	9	256KB
squashfs with zlib	9	512KB
squashfs with zlib	9	1MB
squashfs with lzma	0 (fast)	256KB
squashfs with lzma	0 (fast)	512KB
squashfs with lzma	0 (fast)	1MB
squashfs with lzma	6 (normal)	256KB
squashfs with lzma	6 (normal)	512KB
squashfs with lzma	6 (normal)	1MB
squashfs with lzma	9 (ultra)	256KB
squashfs with lzma	9 (ultra)	512KB
squashfs with lzma	9 (ultra)	1MB
squashfs with zstandard	3	256KB
squashfs with zstandard	3	512KB
squashfs with zstandard	3	1MB
squashfs with zstandard	10	256KB
squashfs with zstandard	10	512KB
squashfs with zstandard	10	1MB
squashfs with zstandard	19	256KB
squashfs with zstandard	19	512KB
squashfs with zstandard	19	1MB
dwarfs	128	256KB
dwarfs	128	512KB
dwarfs	128	1MB
dwarfs	256	256KB
dwarfs	256	512KB
dwarfs	256	1MB
libdeflate	6	256KB
libdeflate	6	512KB
libdeflate	6	1MB
libdeflate	9	256KB
libdeflate	9	512KB
libdeflate	9	1MB

[probonopd]

Volunteers?

[Samueru-sama]

Something that needs to be considered as well is to take into account how long it normally takes for the same application not as an appimage to start.

For example we might see that on a very big application a certain algo is 30% faster, but that application even when not being an appimage due to its size takes several seconds to start anyway, and that 30% ends up being a very small percentage of the overall delay for the app.

Same way for very small applications, the speed difference might not matter much, because they are very small and take no time regardless.

Where problems can happen is with the mid size applications that you normally expect to start fast, those are web-browsers in other words.

Right now zstd with the current default block size is actually very good, I will do some benchmarks comparing the size and startup times, however I can't measure zsync efficiency since that seems quite a bit more work.

I'm also interested to know how this affects very old hardware, IE, some pre sandy bridge cpu for example, my hardware is from 2016 and not the worst I would say lol.

[probonopd]

Application startup times also depend on the hardware. On systems with slow disk but fast CPU, a highly compressed image may lead to faster application launch times than uncompressed files (iirc, I have seen this myself with a large application in the past, likely with a spinning drive). It always depends where the performance bottleneck is on a particular system.

So for this to be really scientific, we'd have to execute the test matrix for typical defined machines.

But then, we are not exactly writing a dissertation here ;-)

[Drsheppard01]

I'm probably a bit late, but I think EROFS is an interesting option as well.

• wide kernel support (since 5.4, Ubuntu 20.04, Alpine since 3.11)
• fuse support,
• compression:
  • zstd,
  • lzma,
  • lz4

dwarfs is incredibly performant, but gpl3 makes it impossible to use with proprietary programs packaged in appimage. At the same time, AppImage packaging is used by large companies, so the introduction of dwarfs will cut off a significant part of the audience

[CarlGao4]

I got these results running on a VM with 12 GiB Memory and 4 cores:
results.csv

Compression Method	Compression Parameters	Block Size	Compression Time	FS Size	Startup Time (average of 5 runs)	docx to pdf Time (average of 5 runs)	zsync download size
SquashFS gzip	-Xcompression-level 1	16k	18110.2 ms	1119.14 MiB	1810.8 ms	3494.1 ms	168.85 MiB
SquashFS gzip	-Xcompression-level 1	64k	16387.5 ms	930.97 MiB	1781.4 ms	3579.7 ms	171.56 MiB
SquashFS gzip	-Xcompression-level 1	256k	15713.5 ms	892.99 MiB	1942.4 ms	3737.5 ms	207.79 MiB
SquashFS gzip	-Xcompression-level 1	512k	15696.9 ms	886.75 MiB	1942.3 ms	3743.1 ms	227.78 MiB
SquashFS gzip	-Xcompression-level 1	1M	15614.7 ms	883.83 MiB	2047.6 ms	3991.9 ms	262.65 MiB
SquashFS gzip	-Xcompression-level 6	16k	24765.2 ms	1017.00 MiB	1849.4 ms	3423.8 ms	159.67 MiB
SquashFS gzip	-Xcompression-level 6	64k	25149.3 ms	803.89 MiB	1795.3 ms	3436.5 ms	159.65 MiB
SquashFS gzip	-Xcompression-level 6	256k	26578.8 ms	745.39 MiB	1777.4 ms	3512.6 ms	190.23 MiB
SquashFS gzip	-Xcompression-level 6	512k	27269.3 ms	735.77 MiB	1840.6 ms	3662.1 ms	207.10 MiB
SquashFS gzip	-Xcompression-level 6	1M	26230.7 ms	731.28 MiB	1950.7 ms	3830.9 ms	236.83 MiB
SquashFS gzip	-Xcompression-level 9	16k	31317.2 ms	1013.75 MiB	1769.3 ms	3371.3 ms	159.16 MiB
SquashFS gzip	-Xcompression-level 9	64k	42813.5 ms	798.62 MiB	1733.3 ms	3434.9 ms	159.42 MiB
SquashFS gzip	-Xcompression-level 9	256k	48116.2 ms	739.41 MiB	1778.2 ms	3483.0 ms	188.72 MiB
SquashFS gzip	-Xcompression-level 9	512k	49885.5 ms	729.69 MiB	1873.3 ms	3685.6 ms	206.14 MiB
SquashFS gzip	-Xcompression-level 9	1M	48892.3 ms	725.17 MiB	1955.9 ms	3940.3 ms	235.60 MiB
SquashFS lzo	-Xcompression-level 1	16k	21924.3 ms	1267.65 MiB	1534.1 ms	3096.9 ms	169.46 MiB
SquashFS lzo	-Xcompression-level 1	64k	20306.3 ms	1012.04 MiB	1577.4 ms	3065.8 ms	159.91 MiB
SquashFS lzo	-Xcompression-level 1	256k	19828.8 ms	940.37 MiB	1592.8 ms	3145.6 ms	152.45 MiB
SquashFS lzo	-Xcompression-level 1	512k	20631.9 ms	928.72 MiB	1619.6 ms	3172.9 ms	146.83 MiB
SquashFS lzo	-Xcompression-level 1	1M	20098.3 ms	923.09 MiB	1755.0 ms	3458.9 ms	142.23 MiB
SquashFS lzo	-Xcompression-level 6	16k	27793.1 ms	1192.38 MiB	1510.0 ms	2987.9 ms	159.45 MiB
SquashFS lzo	-Xcompression-level 6	64k	29080.7 ms	915.13 MiB	1544.1 ms	3012.1 ms	146.54 MiB
SquashFS lzo	-Xcompression-level 6	256k	31329.9 ms	833.88 MiB	1567.2 ms	3074.6 ms	136.98 MiB
SquashFS lzo	-Xcompression-level 6	512k	32028.0 ms	820.70 MiB	1585.2 ms	3182.4 ms	130.70 MiB
SquashFS lzo	-Xcompression-level 6	1M	31352.5 ms	814.33 MiB	1691.9 ms	3361.3 ms	126.41 MiB
SquashFS lzo	-Xcompression-level 9	16k	50543.6 ms	1183.98 MiB	1570.2 ms	3069.0 ms	158.34 MiB
SquashFS lzo	-Xcompression-level 9	64k	67672.1 ms	902.67 MiB	1556.2 ms	3041.0 ms	144.56 MiB
SquashFS lzo	-Xcompression-level 9	256k	79792.8 ms	819.64 MiB	1534.2 ms	3039.5 ms	134.66 MiB
SquashFS lzo	-Xcompression-level 9	512k	83141.9 ms	806.20 MiB	1602.6 ms	3198.3 ms	128.41 MiB
SquashFS lzo	-Xcompression-level 9	1M	81887.2 ms	799.70 MiB	1651.4 ms	3217.6 ms	124.16 MiB
SquashFS lz4		16k	12959.3 ms	1472.75 MiB	1362.8 ms	2783.3 ms	211.53 MiB
SquashFS lz4		64k	11803.8 ms	1165.39 MiB	1385.0 ms	2751.1 ms	210.94 MiB
SquashFS lz4		256k	11077.0 ms	1092.33 MiB	1445.8 ms	2792.0 ms	224.16 MiB
SquashFS lz4		512k	10743.2 ms	1080.90 MiB	1423.5 ms	2791.9 ms	224.01 MiB
SquashFS lz4		1M	10781.8 ms	1075.60 MiB	1419.8 ms	2815.3 ms	220.75 MiB
SquashFS xz	-Xdict-size 100%	16k	157594.6 ms	962.06 MiB	2672.6 ms	4816.1 ms	167.56 MiB
SquashFS xz	-Xdict-size 100%	64k	142406.6 ms	708.87 MiB	2704.6 ms	5130.0 ms	173.12 MiB
SquashFS xz	-Xdict-size 100%	256k	149722.1 ms	604.63 MiB	2615.8 ms	5099.6 ms	187.62 MiB
SquashFS xz	-Xdict-size 100%	512k	155167.6 ms	573.89 MiB	2811.0 ms	5681.2 ms	195.02 MiB
SquashFS xz	-Xdict-size 100%	1M	175004.0 ms	549.29 MiB	2983.1 ms	5923.1 ms	210.65 MiB
SquashFS zstd	-Xcompression-level 1	16k	13508.4 ms	1079.95 MiB	1460.5 ms	2922.0 ms	167.31 MiB
SquashFS zstd	-Xcompression-level 1	64k	12735.8 ms	863.07 MiB	1484.1 ms	2968.1 ms	170.44 MiB
SquashFS zstd	-Xcompression-level 1	256k	11946.1 ms	771.79 MiB	1527.9 ms	2962.3 ms	182.90 MiB
SquashFS zstd	-Xcompression-level 1	512k	11370.3 ms	761.88 MiB	1463.8 ms	2896.1 ms	194.40 MiB
SquashFS zstd	-Xcompression-level 1	1M	10859.0 ms	751.70 MiB	1522.0 ms	3112.5 ms	213.06 MiB
SquashFS zstd	-Xcompression-level 6	16k	20443.0 ms	1009.18 MiB	1547.2 ms	3005.2 ms	150.32 MiB
SquashFS zstd	-Xcompression-level 6	64k	17160.3 ms	786.94 MiB	1529.5 ms	3104.2 ms	146.00 MiB
SquashFS zstd	-Xcompression-level 6	256k	16392.8 ms	695.48 MiB	1577.8 ms	3073.1 ms	165.67 MiB
SquashFS zstd	-Xcompression-level 6	512k	16497.4 ms	666.01 MiB	1559.6 ms	3123.8 ms	176.20 MiB
SquashFS zstd	-Xcompression-level 6	1M	15757.7 ms	642.93 MiB	1582.3 ms	3094.2 ms	192.88 MiB
SquashFS zstd	-Xcompression-level 11	16k	56160.3 ms	1005.07 MiB	1542.4 ms	3037.8 ms	149.72 MiB
SquashFS zstd	-Xcompression-level 11	64k	38756.4 ms	773.32 MiB	1496.7 ms	2955.2 ms	142.71 MiB
SquashFS zstd	-Xcompression-level 11	256k	47508.0 ms	674.18 MiB	1597.6 ms	2994.8 ms	161.33 MiB
SquashFS zstd	-Xcompression-level 11	512k	23702.5 ms	651.57 MiB	1595.3 ms	2999.3 ms	174.39 MiB
SquashFS zstd	-Xcompression-level 11	1M	24029.9 ms	626.26 MiB	1530.3 ms	3077.0 ms	189.20 MiB
SquashFS zstd	-Xcompression-level 15	16k	159752.1 ms	975.35 MiB	1596.3 ms	3044.4 ms	143.35 MiB
SquashFS zstd	-Xcompression-level 15	64k	92158.8 ms	744.79 MiB	1587.4 ms	3090.8 ms	142.75 MiB
SquashFS zstd	-Xcompression-level 15	256k	102846.7 ms	638.89 MiB	1586.0 ms	3145.9 ms	165.22 MiB
SquashFS zstd	-Xcompression-level 15	512k	59426.8 ms	646.09 MiB	1531.4 ms	3066.1 ms	169.68 MiB
SquashFS zstd	-Xcompression-level 15	1M	70819.5 ms	620.51 MiB	1582.0 ms	3139.0 ms	184.97 MiB
SquashFS zstd	-Xcompression-level 19	16k	306681.6 ms	971.67 MiB	1573.4 ms	2985.6 ms	140.86 MiB
SquashFS zstd	-Xcompression-level 19	64k	337880.4 ms	738.67 MiB	1489.3 ms	2988.1 ms	137.11 MiB
SquashFS zstd	-Xcompression-level 19	256k	293177.2 ms	633.54 MiB	1513.3 ms	3009.7 ms	160.65 MiB
SquashFS zstd	-Xcompression-level 19	512k	257798.8 ms	601.88 MiB	1537.7 ms	3064.6 ms	170.96 MiB
SquashFS zstd	-Xcompression-level 19	1M	250558.4 ms	575.98 MiB	1571.6 ms	3140.7 ms	189.89 MiB
SquashFS zstd	-Xcompression-level 22	16k	357046.5 ms	971.59 MiB	1523.3 ms	3011.1 ms	140.85 MiB
SquashFS zstd	-Xcompression-level 22	64k	451164.6 ms	738.42 MiB	1478.1 ms	2979.2 ms	137.81 MiB
SquashFS zstd	-Xcompression-level 22	256k	395652.6 ms	633.24 MiB	1544.2 ms	3036.7 ms	160.58 MiB
SquashFS zstd	-Xcompression-level 22	512k	353558.6 ms	601.49 MiB	1528.2 ms	3072.5 ms	171.46 MiB
SquashFS zstd	-Xcompression-level 22	1M	342044.9 ms	575.53 MiB	1566.8 ms	3127.2 ms	189.75 MiB

My script

import itertools
import os
import pathlib
import re
import shlex
import subprocess
import time

file_ver1 = "./LibreOffice-24.8.3.2.full.help-x86_64.AppImage"
file_ver2 = "./LibreOffice-24.8.4.2.full.help-x86_64.AppImage"

block_sizes = ["16k", "64k", "256k", "512k", "1M"]
comps_squashfs = {
    "gzip": {"-Xcompression-level": ["1", "6", "9"]},
    "lzo": {
        "-Xcompression-level": ["1", "6", "9"],
    },
    "lz4": {},
    "xz": {
        "-Xdict-size": ["100%"],
    },
    "zstd": {
        "-Xcompression-level": ["1", "6", "11", "15", "19", "22"],
    },
}

output = open("results.csv", "w")
output.write(
    "Compression Method,Compression Parameters,Block Size,Compression Time,"
    "FS Size,Startup Time (average of 5 runs),docx to pdf Time (average of 5 runs),zsync download size"
)
output.flush()

try:
    os.mkdir("zsyncout")
except FileExistsError:
    pass

if not os.path.exists("ver1.AppDir"):
    subprocess.Popen([file_ver1, "--appimage-extract"]).wait()
    os.rename("squashfs-root", "ver1.AppDir")
if not os.path.exists("ver2.AppDir"):
    subprocess.Popen([file_ver2, "--appimage-extract"]).wait()
    os.rename("squashfs-root", "ver2.AppDir")

with open("uruntime-appimage-x86_64", "rb") as f:
    runtime = f.read()

for comp in comps_squashfs:
    # product all possible combinations of compression parameters
    comp_params = list(
        itertools.product(*list(list([i, j] for j in comps_squashfs[comp][i]) for i in comps_squashfs[comp]))
    )
    for comp_param in comp_params:
        for block_size in block_sizes:
            output.write("\n")
            output.write(f"SquashFS {comp},{shlex.join(sum(comp_param, []))},{block_size},")
            output.flush()

            print(
                "Preparing AppImages for compression method %s with parameters %s and block size %s"
                % (comp, sum(comp_param, []), block_size)
            )
            # Prepare the AppImages
            command = ["mksquashfs", "ver1.AppDir", "file_ver1.squashfs", "-root-owned", "-noappend", "-comp", comp]
            command.extend(["-b", block_size])
            for i in comp_param:
                command.extend(i)
            # file_ver1 is only the base file, so do not measure efficiency
            p = subprocess.Popen(command)
            if p.wait() != 0:
                print("Error in compression")
                continue
            with open("file_ver1.squashfs", "rb") as f:
                squashfs = f.read()
            os.remove("file_ver1.squashfs")
            with open("file_ver1.AppImage", "wb") as f:
                f.write(runtime)
                f.write(squashfs)
            del squashfs
            try:
                os.chmod("file_ver1.AppImage", 0o755)
            except PermissionError:
                pass

            # Compression time
            command[1] = "ver2.AppDir"
            command[2] = "file_ver2.squashfs"
            start = time.time()
            p = subprocess.Popen(command)
            if p.wait() != 0:
                print("Error in compression")
                continue
            output.write(f"{(time.time() - start) * 1000:.1f}ms,")
            output.flush()
            with open("file_ver2.squashfs", "rb") as f:
                squashfs = f.read()
            os.remove("file_ver2.squashfs")
            with open("file_ver2.AppImage", "wb") as f:
                f.write(runtime)
                f.write(squashfs)
            output.write(f"{len(squashfs)},")
            output.flush()
            print("SquashFS size: %d" % len(squashfs))
            del squashfs
            try:
                os.chmod("file_ver2.AppImage", 0o755)
            except PermissionError:
                pass

            # Startup time
            print("Measuring startup time, 10 rounds, only last 5 rounds count")
            for _ in range(2):
                startup_times = []
                for i in range(5):
                    start = time.time()
                    p = subprocess.Popen(["./file_ver2.AppImage", "--terminate_after_init"])
                    p.wait()
                    used_time = time.time() - start
                    startup_times.append(used_time)
                    print("Startup time round %d: %f" % (i + 1, used_time))
            output.write(f"{sum(startup_times) * 1000 / 5:.1f} ms,")
            output.flush()

            # docx to pdf time
            print("Measuring docx to pdf time, 10 rounds, only last 5 rounds count")
            for _ in range(2):
                docx_to_pdf_times = []
                for i in range(5):
                    start = time.time()
                    p = subprocess.Popen(
                        [
                            "./file_ver2.AppImage",
                            "--convert-to",
                            "pdf",
                            pathlib.Path("test.docx").absolute(),
                            "--outdir",
                            pathlib.Path(".").absolute(),
                        ]
                    )
                    p.wait()
                    used_time = time.time() - start
                    docx_to_pdf_times.append(used_time)
                    print("docx to pdf time round %d: %f" % (i + 1, used_time))
            output.write(f"{sum(docx_to_pdf_times) * 1000 / 5:.1f} ms,")
            output.flush()

            # zsync download size
            print("Creating zsync file")
            p = subprocess.Popen(["zsyncmake", "file_ver2.AppImage"])
            if p.wait() != 0:
                print("Error in zsync")
                continue
            print("Calculating zsync download size")
            p = subprocess.Popen(
                ["zsync", "-i", "file_ver1.AppImage", "file_ver2.AppImage.zsync", "-o", "./zsyncout"],
                stderr=subprocess.PIPE,
            )
            zsync_output = p.stderr.read().decode("utf-8")
            percentage = re.search(r"(\d+\.\d+)%", zsync_output).group(1)
            p.wait()
            try:
                os.remove("zsyncout.part")
            except FileNotFoundError:
                pass
            output.write(f"{(1 - float(percentage) / 100) * os.path.getsize('file_ver2.AppImage'):.0f}")
            output.flush()

output.close()

[Samueru-sama]

This is amazing @CarlGao4 thank you.

I modified the script to not test as many options and also dropped lzo all together, tested with the Brave AppImage:

Tested on actual hardware: Xeon E5-2640 V4 + 16 GiB of mem.

Compression Method	Compression Parameters	Block Size	Compression Time	FS Size	Startup Time (average of 5 runs)	zsync download size
SquashFS gzip	-Xcompression-level 1	128k	696.1ms	185876480	3009.8 ms	143952033
SquashFS gzip	-Xcompression-level 1	512k	712.8ms	184889344	5273.6 ms	145323560
SquashFS gzip	-Xcompression-level 1	1M	899.6ms	184729600	6895.4 ms	146543394
SquashFS gzip	-Xcompression-level 6	128k	1627.3ms	172199936	2964.2 ms	133914685
SquashFS gzip	-Xcompression-level 6	512k	1701.2ms	170659840	4850.3 ms	135070067
SquashFS gzip	-Xcompression-level 6	1M	2268.4ms	170401792	6764.2 ms	136114317
SquashFS gzip	-Xcompression-level 9	128k	3142.5ms	171626496	2901.7 ms	133485752
SquashFS gzip	-Xcompression-level 9	512k	3480.5ms	170024960	4779.2 ms	134410068
SquashFS gzip	-Xcompression-level 9	1M	4406.3ms	169750528	6673.8 ms	135790727
SquashFS lz4		128k	262.4ms	231120896	1147.7 ms	170373122
SquashFS lz4		512k	233.2ms	229834752	1313.5 ms	166612261
SquashFS lz4		1M	1757.7ms	229613568	1495.2 ms	166220104
SquashFS xz	-Xdict-size 100%	128k	10155.6ms	148647936	7873.7 ms	112099874
SquashFS xz	-Xdict-size 100%	512k	11458.4ms	141467648	14832.4 ms	107812675
SquashFS xz	-Xdict-size 100%	1M	14598.5ms	138706944	20413.6 ms	106533330
SquashFS zstd	-Xcompression-level 1	128k	217.8ms	187547648	1512.8 ms	137036518
SquashFS zstd	-Xcompression-level 1	512k	221.8ms	186511360	2072.7 ms	137079302
SquashFS zstd	-Xcompression-level 1	1M	263.4ms	186064896	2559.2 ms	136955654
SquashFS zstd	-Xcompression-level 15	128k	5820.6ms	158380032	1777.1 ms	115151666
SquashFS zstd	-Xcompression-level 15	512k	3303.8ms	162115584	2154.9 ms	116065886
SquashFS zstd	-Xcompression-level 15	1M	4653.0ms	158838784	2715.6 ms	113483386
SquashFS zstd	-Xcompression-level 22	128k	17713.8ms	156803072	1808.3 ms	116179759
SquashFS zstd	-Xcompression-level 22	512k	13204.2ms	149749760	2600.0 ms	111798229
SquashFS zstd	-Xcompression-level 22	1M	15394.0ms	146882560	3256.2 ms	110212187

I don't know how to make the table look all good like you did here. 😅

EDIT: I had this hack running in the background to kill the brave window every time it opened:

while true; do
    window_class="$(xdotool getactivewindow getwindowclassname 2>/dev/null)"
    if echo "$window_class" | grep -q "Brave-browser"; then
        killall brave
    fi
    sleep 0.0001
done

[CarlGao4]

Oh, I opened the csv in Excel and just copy the table and paste it here, it will automatically form a table in markdown syntax
Besides, I changed some display format before exporting it here

[CarlGao4]

I'll try to test DwarFS later

[Samueru-sama]

I'll try to test DwarFS later

That's going to be a lot of block sizes btw, with dwarfs iirc it can be as high as 64MB. And the impact also varies greatly depending on the application that's tested.

With this AppImage of GIMP for example -S25(32MB) was good while with Cromite I had to lower it to -S20 (1MB) otherwise it takes too long for the browser to open.