Command startup is significantly slower than GNU coreutils

[ilyapopov]

Currently, uutils have a much slower command startup time compared to GNU coreutils.

This may not be strictly speaking a bug, but something to be aware of (and may be to track)

Context:

Ubuntu 25.10 has switched to uutils for some of the core utilities.

This caused a regression in run times of some third-party shell scripts (a custom build system) that call core utilities a lot. Each call makes very little work, but there many of them; it seems mostly ln is problematic (there are also a lot of cp and sed calls, but these are still GNU versions in Ubuntu). Yes, the scripts are probably not optimal and could be rewritten to make fewer calls, but it is what it is.

Benchmarking

touch src.txt
hyperfine -N --warmup 10 "ln -s -f src.txt dst.txt" "gnuln -s -f src.txt dst.tct"

Output:

Benchmark 1: ln -s -f src.txt dst.txt
  Time (mean ± σ):       2.0 ms ±   0.2 ms    [User: 0.5 ms, System: 1.5 ms]
  Range (min … max):     1.7 ms …   8.7 ms    1547 runs
 
  Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet system without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.
 
Benchmark 2: gnuln -s -f src.txt dst.tct
  Time (mean ± σ):     645.7 µs ±  60.9 µs    [User: 292.5 µs, System: 287.0 µs]
  Range (min … max):   520.2 µs … 883.3 µs    4263 runs
 
Summary
  gnuln -s -f src.txt dst.tct ran
    3.14 ± 0.44 times faster than ln -s -f src.txt dst.txt

System info:

• Ubuntu 25.10
• uutils installed from Ubuntu repository, version 0.2.2-0ubuntu2.1; ln --version -> ln (uutils coreutils) 0.2.2
• Ryzen 5700G CPU (8 cores 16 threads), 32 GB RAM, fast SSD

I also made a benchmark on an older system (Intel 6200U laptop), with even bigger difference (about 4×)

Discussion

At least as it is shipped in Ubuntu, uutils are actually one multicall 10 MB binary that does different things depending on with which name it is called. Maybe this is a factor here, as it may present more work for a loader.

To my understanding, uutils currently has a benchmark suite that tracks performance of various algorithms, to not command startup time

[oech3]

#7928 (comment)

Would you show time for individual bins too?

[ilyapopov]

So i did:

Cloned coreutils repo and compiled individual binaries as specified in the README:

git clone https://github.com/uutils/coreutils.git
cargo build --release --bins --workspace --exclude coreutils --exclude uu_runcon --exclude uu_chcon

Then I run the same comparison:

 $ hyperfine -N --warmup 10 "ln -s -f src.txt dst.txt" "gnuln -s -f src.txt dst.tct" "./target/release/ln -s -f src.txt dst.txt"

Output:

Benchmark 1: ln -s -f src.txt dst.txt
  Time (mean ± σ):       2.1 ms ±   0.1 ms    [User: 0.4 ms, System: 1.5 ms]
  Range (min … max):     1.7 ms …   2.5 ms    1423 runs
 
Benchmark 2: gnuln -s -f src.txt dst.tct
  Time (mean ± σ):     652.4 µs ±  62.4 µs    [User: 289.4 µs, System: 294.8 µs]
  Range (min … max):   527.8 µs … 1443.3 µs    4342 runs
 
  Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet system without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.
 
Benchmark 3: ./target/release/ln -s -f src.txt dst.txt
  Time (mean ± σ):       1.0 ms ±   0.1 ms    [User: 0.3 ms, System: 0.7 ms]
  Range (min … max):     0.9 ms …   1.3 ms    2543 runs
 
Summary
  gnuln -s -f src.txt dst.tct ran
    1.60 ± 0.21 times faster than ./target/release/ln -s -f src.txt dst.txt
    3.17 ± 0.36 times faster than ln -s -f src.txt dst.txt

So indeed, an individual binary is much closer to GNU than the multicall binary, (however it is still quite big, 1.6 MB)

[pixelb]

I tested the other way and showed that GNU coreutils built as a multi-call binary are about as slow as uutils multi-call binary. I.e. the overhead is due to dynamically loading all the shared libs etc.

$ time seq 10000 | xargs -n1 true
real	0m8.634s
user	0m3.178s
sys	0m5.616s

$ time seq 10000 | xargs -n1 uu_true
real	0m22.137s
user	0m6.542s
sys	0m15.561s

$ ./configure --enable-single-binary --quiet && make -n $(nproc)

$ time seq 10000 | xargs -n1 src/true

real	0m21.595s
user	0m7.437s
sys	0m14.151s

The individual uutils binaries are bigger by default, but I saw someone mention that the uutils size can abe reduced by building with panic = abort.

I.e. this slowness seems to be a distribution packaging issue rather than a fundamental uutils issue.

Note cheap process creation is a fundamental idea in UNIX so I'm very surprised that ubuntu hasn't tried to optimize this for the 100 most invoked utilities.

[collinfunk]

Note cheap process creation is a fundamental idea in UNIX so I'm very surprised that ubuntu hasn't tried to optimize this for the 100 most invoked utilities.

Running ./configure scripts with both implementations would probably show the real world impact of this.

[oech3]

Ubuntu already have panic=abort and this project have release-fast profile with it (blocked by make due to wrong parsing). with buid-std hack (RUSTC_BOOTSTRAP=1, -Zbuild-std=std,panic_abort -Zbuild-std-features=panic_immediate_abort) on Arch with stripped bins, I have

>time seq 10000 | xargs -n1 'single-gnu-true'
Executed in    7.42 secs    fish           external
> time seq 10000 | xargs -n1 single-uu-true
Executed in   11.78 secs    fish           external
> time seq 10000 | xargs -n1 multi-hardlink-uu-true
Executed in   17.47 secs    fish           external

Total size for individual bins with build-std is still 100 MB+. So it is difficult to distribute it yet.

[oech3]

Arch uses system libonig.so for uutls-coreutils, not sure for other distributions. true is not linked with it.

[andreacorbellini]

Relevant to this discussion: #9103

[collinfunk]

@andreacorbellini That is probably part of the reason. Using gettext, the locale files aren't opened if they aren't needed.

For example true does not need messages if it is run without options:

$ strace -e trace=openat -- /bin/true >/dev/null
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
+++ exited with 0 +++

However, it needs them if we want the output of --help:

$ strace -e trace=openat -- /bin/true --help >/dev/null
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/usr/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/usr/share/locale/locale.alias", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/usr/share/locale/en_US.UTF-8/LC_MESSAGES/coreutils.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/share/locale/en_US.utf8/LC_MESSAGES/coreutils.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/share/locale/en_US/LC_MESSAGES/coreutils.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/share/locale/en.UTF-8/LC_MESSAGES/coreutils.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/share/locale/en.utf8/LC_MESSAGES/coreutils.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/share/locale/en/LC_MESSAGES/coreutils.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
+++ exited with 0 +++

[sylvestre]

dup of #9103

[oech3]

Beware about shell's builtins. I post invalid results at here...