COMMIT in podman build is much slower than in docker build

[carles-aiven]

Please have a look at this experiment (building a container with a 2GB file):

$ cat Dockerfile_bench 
# benchmark this with docker vs podman
FROM scratch
COPY ./output.dat .

$ dd if=/dev/random of=output.dat  bs=2G  count=1
0+1 records in
0+1 records out
2147479552 bytes (2.1 GB, 2.0 GiB) copied, 5.36423 s, 400 MB/s

$ time sudo docker build  -f Dockerfile_bench .
[+] Building 8.6s (5/5) FINISHED                                                                                                                            docker:default
 => [internal] load build definition from Dockerfile_bench                                                                                                            0.1s
 => => transferring dockerfile: 174B                                                                                                                                  0.0s
 => [internal] load .dockerignore                                                                                                                                     0.1s
 => => transferring context: 2B                                                                                                                                       0.0s
 => [internal] load build context                                                                                                                                     4.5s
 => => transferring context: 2.15GB                                                                                                                                   4.4s
 => [1/1] COPY ./output.dat .                                                                                                                                         0.6s
 => exporting to image                                                                                                                                                3.1s
 => => exporting layers                                                                                                                                               3.1s
 => => writing image sha256:52667b9ddbd1da14e740b55ae37f877ef31bb434c71e9a18c364135736af5ed4                                                                          0.0s

real    0m12.196s
user    0m0.025s
sys     0m0.012s

$ time podman build  -f Dockerfile_bench .
STEP 1/2: FROM scratch
STEP 2/2: COPY ./output.dat .
COMMIT
--> 216ae3fcb52f
216ae3fcb52f5bd6f64f7b9deb1e2e17e67fc7239699ef7255bef50aceef8326

real    0m25.554s
user    0m24.398s
sys     0m6.823s

$ podman --version
podman version 4.8.3

$ docker --version
Docker version 25.0.3, build 4debf41

The host is Fedora 38, the filesystem is btrfs, and podman is using native overlay.

You can try this experiment with a more realistic file, such as a big git repository and you should see the same result.

Podman commit is much slower than the equivalent in Docker. Is there any room for optimization? Could the Podman developers apply the same optimizations that buildkit is doing to buildah?

[vrothberg]

Can you run Podman as root as well for a more fair comparison?

A factor of 2 for that Dockerfile seems too off as it's a purely I/O bound operation. Buildkit is usually a bit more clever when it comes to running instructions in parallel, but that's not the case here.

[carles-aiven]

I get a very similar result. I've repeated the experiment in Debian, Ubuntu, OpenSuse, Fedora; in ext4, xfs, btrfs; the results always show that Docker is significantly faster.

I don't have a scientific benchmark to show, but I've written this script so you can reproduce it easily:

# Write Dockerfile_bench
echo -e "FROM scratch \nCOPY ./output.dat . \n" > Dockerfile_bench

# Generate random 2GB output.dat
dd if=/dev/urandom of=output.dat  bs=2G  count=1

# Benchmark rootful Podman
time sudo podman build  -f Dockerfile_bench .

# Benchmark Docker
time sudo docker build  -f Dockerfile_bench .

# Show versions
podman --version
sudo docker --version

[carles-aiven]

The difference is in the podman "COMMIT" vs docker "exporting layers"

[vrothberg]

@nalind @rhatdan @giuseppe WDYT?

Thank you, @carles-aiven !