Skip to content

ls: Do not eagerly return FileType for each PathData, wait until necessary#8663

Closed
kimono-koans wants to merge 73 commits intouutils:mainfrom
kimono-koans:lazily_obtain_file_type
Closed

ls: Do not eagerly return FileType for each PathData, wait until necessary#8663
kimono-koans wants to merge 73 commits intouutils:mainfrom
kimono-koans:lazily_obtain_file_type

Conversation

@kimono-koans
Copy link
Contributor

See: #8350

Obtaining FileType from DirEntry is cheap, here, we prefer not to eagerly request Metadata when not yet needed, just to obtain a FileType eagerly.

@github-actions
Copy link

GNU testsuite comparison:

GNU test failed: tests/ls/follow-slink. tests/ls/follow-slink is passing on 'main'. Maybe you have to rebase?
GNU test failed: tests/ls/infloop. tests/ls/infloop is passing on 'main'. Maybe you have to rebase?
Skip an intermittent issue tests/timeout/timeout (fails in this run but passes in the 'main' branch)
Skipping an intermittent issue tests/misc/stdbuf (passes in this run but fails in the 'main' branch)
Note: The gnu test tests/ls/stat-failed is now being skipped but was previously passing.

@github-actions
Copy link

GNU testsuite comparison:

Skip an intermittent issue tests/timeout/timeout (fails in this run but passes in the 'main' branch)
Skipping an intermittent issue tests/misc/stdbuf (passes in this run but fails in the 'main' branch)
Skipping an intermittent issue tests/misc/tee (passes in this run but fails in the 'main' branch)
Note: The gnu test tests/ls/stat-failed is now being skipped but was previously passing.

@github-actions
Copy link

GNU testsuite comparison:

Skipping an intermittent issue tests/misc/stdbuf (passes in this run but fails in the 'main' branch)
Skipping an intermittent issue tests/misc/tee (passes in this run but fails in the 'main' branch)
Note: The gnu test tests/ls/stat-failed is now being skipped but was previously passing.

@github-actions
Copy link

GNU testsuite comparison:

Skipping an intermittent issue tests/misc/stdbuf (passes in this run but fails in the 'main' branch)
Note: The gnu test tests/ls/stat-failed is now being skipped but was previously passing.

@github-actions
Copy link

GNU testsuite comparison:

Skipping an intermittent issue tests/misc/stdbuf (passes in this run but fails in the 'main' branch)
Skipping an intermittent issue tests/misc/tee (passes in this run but fails in the 'main' branch)
Note: The gnu test tests/ls/stat-failed is now being skipped but was previously passing.

@github-actions
Copy link

GNU testsuite comparison:

Skip an intermittent issue tests/timeout/timeout (fails in this run but passes in the 'main' branch)
Skipping an intermittent issue tests/misc/stdbuf (passes in this run but fails in the 'main' branch)
Skipping an intermittent issue tests/misc/tee (passes in this run but fails in the 'main' branch)

@github-actions
Copy link

GNU testsuite comparison:

GNU test failed: tests/ls/stat-free-color. tests/ls/stat-free-color is passing on 'main'. Maybe you have to rebase?
Skip an intermittent issue tests/timeout/timeout (fails in this run but passes in the 'main' branch)
Skipping an intermittent issue tests/misc/stdbuf (passes in this run but fails in the 'main' branch)
Skipping an intermittent issue tests/misc/tee (passes in this run but fails in the 'main' branch)

@github-actions
Copy link

GNU testsuite comparison:

GNU test failed: tests/ls/stat-free-color. tests/ls/stat-free-color is passing on 'main'. Maybe you have to rebase?
Skip an intermittent issue tests/timeout/timeout (fails in this run but passes in the 'main' branch)
Skipping an intermittent issue tests/misc/tee (passes in this run but fails in the 'main' branch)

@github-actions
Copy link

GNU testsuite comparison:

GNU test failed: tests/ls/stat-free-color. tests/ls/stat-free-color is passing on 'main'. Maybe you have to rebase?
Skip an intermittent issue tests/timeout/timeout (fails in this run but passes in the 'main' branch)
Skipping an intermittent issue tests/misc/tee (passes in this run but fails in the 'main' branch)

@codspeed-hq
Copy link

codspeed-hq bot commented Sep 20, 2025

CodSpeed Performance Report

Merging #8663 will degrade performances by 6.56%

Comparing kimono-koans:lazily_obtain_file_type (13c2091) with main (32eef06)

Summary

⚡ 14 improvements
❌ 1 regression
✅ 78 untouched
⏩ 1 skipped1

⚠️ Please fix the performance issues or acknowledge them on CodSpeed.

Benchmarks breakdown

Benchmark BASE HEAD Change
ls_recursive_balanced_tree[(3, 4, 8)] 436 µs 415.9 µs +4.83%
ls_recursive_balanced_tree[(4, 3, 6)] 428.7 µs 410.4 µs +4.46%
ls_recursive_balanced_tree[(5, 2, 10)] 435.1 µs 424.5 µs +2.51%
ls_recursive_deep_tree[(100, 1)] 395.7 µs 385.8 µs +2.56%
ls_recursive_deep_tree[(20, 3)] 391.1 µs 418.6 µs -6.56%
ls_recursive_long_all_balanced_tree[(3, 4, 8)] 534.9 µs 501.6 µs +6.63%
ls_recursive_long_all_balanced_tree[(4, 3, 6)] 514.5 µs 492.2 µs +4.52%
ls_recursive_long_all_balanced_tree[(5, 2, 10)] 525 µs 503.2 µs +4.33%
ls_recursive_long_all_deep_tree[(50, 2)] 481.7 µs 457 µs +5.42%
ls_recursive_long_all_wide_tree[(1000, 200)] 6.8 ms 5.9 ms +15.34%
ls_recursive_long_all_wide_tree[(5000, 500)] 32.5 ms 27.3 ms +18.77%
ls_recursive_mixed_tree 515.8 µs 495.3 µs +4.15%
ls_recursive_wide_tree[(1000, 200)] 5.3 ms 4.2 ms +27.24%
ls_recursive_wide_tree[(10000, 1000)] 48.4 ms 38.7 ms +25.28%
ls_recursive_wide_tree[(5000, 500)] 23.9 ms 19.3 ms +23.91%

Footnotes

  1. 1 benchmark was skipped, so the baseline result was used instead. If it was deleted from the codebase, click here and archive it to remove it from the performance reports.

@github-actions
Copy link

GNU testsuite comparison:

GNU test failed: tests/ls/stat-free-color. tests/ls/stat-free-color is passing on 'main'. Maybe you have to rebase?

@github-actions
Copy link

GNU testsuite comparison:

GNU test failed: tests/ls/stat-free-color. tests/ls/stat-free-color is passing on 'main'. Maybe you have to rebase?
GNU test failed: tests/tail/overlay-headers. tests/tail/overlay-headers is passing on 'main'. Maybe you have to rebase?
Skip an intermittent issue tests/misc/tee (fails in this run but passes in the 'main' branch)

@github-actions
Copy link

GNU testsuite comparison:

GNU test failed: tests/ls/stat-free-color. tests/ls/stat-free-color is passing on 'main'. Maybe you have to rebase?
GNU test failed: tests/tail/overlay-headers. tests/tail/overlay-headers is passing on 'main'. Maybe you have to rebase?
Skip an intermittent issue tests/misc/tee (fails in this run but passes in the 'main' branch)
Skip an intermittent issue tests/timeout/timeout (fails in this run but passes in the 'main' branch)

@kimono-koans
Copy link
Contributor Author

GNU testsuite comparison:

GNU test failed: tests/ls/stat-free-color. tests/ls/stat-free-color is passing on 'main'. Maybe you have to rebase?
GNU test failed: tests/tail/overlay-headers. tests/tail/overlay-headers is passing on 'main'. Maybe you have to rebase?
Skip an intermittent issue tests/misc/tee (fails in this run but passes in the 'main' branch)
Skip an intermittent issue tests/timeout/timeout (fails in this run but passes in the 'main' branch)

Again, this failing GNU test is really about the GNU implementation, and not about proper ls behavior. The test states that its purpose in a comment:

# Show that --color need not use stat, as long as we have d_type support.

While this might be interesting to some, is it important in a world where statx/etc are faster than getdents64?

17.46    0.000915           9        98         7 statx
17.29    0.000906          26        34           getdents64 

Yes, we do use DirEntry metadata as much as possible, but right now we are faster and use less overall syscalls than GNU, but more statx/etc calls, but I simply don't think that matters. See:

This branch:

 strace -c ~/program/coreutils/target/release/ls -la --color .
...
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ------------------
 20.16    0.000810         810         1           execve
 12.29    0.000494          14        34           getdents64
  9.56    0.000384           6        56         7 statx
  8.64    0.000347           8        42           listxattr
  8.09    0.000325          25        13           mmap
  7.62    0.000306          11        27         2 openat
  7.47    0.000300          15        20        14 readlink
  3.63    0.000146           5        29           close
  3.61    0.000145          13        11           read
  3.19    0.000128           5        25           fstat
  2.96    0.000119          23         5           mprotect
  2.89    0.000116          16         7           rt_sigaction
  1.64    0.000066          33         2           munmap
  1.34    0.000054          54         1           write
  1.34    0.000054           5        10           brk
  1.14    0.000046           9         5           ioctl
  0.62    0.000025          12         2           pread64
  0.55    0.000022           7         3           sigaltstack
  0.55    0.000022          11         2           getrandom
  0.50    0.000020          20         1         1 access
  0.45    0.000018          18         1           poll
  0.45    0.000018           9         2           prlimit64
  0.30    0.000012          12         1           sched_getaffinity
  0.27    0.000011          11         1           set_tid_address
  0.22    0.000009           9         1           arch_prctl
  0.22    0.000009           9         1           rseq
  0.20    0.000008           8         1           set_robust_list
  0.10    0.000004           1         3           lseek
  0.00    0.000000           0         4           socket
  0.00    0.000000           0         4         4 connect
  0.00    0.000000           0         4           newfstatat
------ ----------- ----------- --------- --------- ------------------
100.00    0.004018          12       319        28 total

GNU:

strace -c ls -la --color .
...
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 15.29    0.000797         797         1           execve
 10.26    0.000535          17        31        17 openat
  9.93    0.000518          12        43        43 lgetxattr
  9.30    0.000485          24        20           mmap
  9.11    0.000475          11        43           write
  7.86    0.000410           9        43           statx
  7.25    0.000378           8        43           listxattr
  5.73    0.000299          23        13           read
  4.30    0.000224           6        36           rt_sigaction
  3.61    0.000188           9        20           close
  3.45    0.000180          10        17           fstat
  2.45    0.000128         128         1           munmap
  1.98    0.000103          25         4         4 connect
  1.75    0.000091          18         5           mprotect
  1.34    0.000070          17         4           socket
  1.27    0.000066          33         2           getdents64
  0.69    0.000036          12         3           newfstatat
  0.65    0.000034          17         2         2 statfs
  0.61    0.000032          16         2         2 access
  0.59    0.000031          10         3           ioctl
  0.58    0.000030          10         3           brk
  0.56    0.000029          14         2           pread64
  0.46    0.000024           8         3           lseek
  0.17    0.000009           9         1           futex
  0.17    0.000009           9         1           getrandom
  0.15    0.000008           8         1           prlimit64
  0.13    0.000007           7         1           arch_prctl
  0.12    0.000006           6         1           set_tid_address
  0.12    0.000006           6         1           set_robust_list
  0.12    0.000006           6         1           rseq
------ ----------- ----------- --------- --------- ----------------
100.00    0.005214          14       351        68 total

@github-actions
Copy link

GNU testsuite comparison:

GNU test failed: tests/ls/stat-free-color. tests/ls/stat-free-color is passing on 'main'. Maybe you have to rebase?

@github-actions
Copy link

GNU testsuite comparison:

GNU test failed: tests/ls/stat-free-color. tests/ls/stat-free-color is passing on 'main'. Maybe you have to rebase?
Skip an intermittent issue tests/misc/stdbuf (fails in this run but passes in the 'main' branch)
Skipping an intermittent issue tests/tail/overlay-headers (passes in this run but fails in the 'main' branch)

@github-actions
Copy link

GNU testsuite comparison:

GNU test failed: tests/ls/stat-free-color. tests/ls/stat-free-color is passing on 'main'. Maybe you have to rebase?
Skip an intermittent issue tests/misc/usage_vs_getopt (fails in this run but passes in the 'main' branch)
Skip an intermittent issue tests/tail/overlay-headers (fails in this run but passes in the 'main' branch)

@github-actions
Copy link

GNU testsuite comparison:

GNU test failed: tests/ls/stat-free-color. tests/ls/stat-free-color is passing on 'main'. Maybe you have to rebase?
Skip an intermittent issue tests/tail/overlay-headers (fails in this run but passes in the 'main' branch)
Skip an intermittent issue tests/timeout/timeout (fails in this run but passes in the 'main' branch)

@github-actions
Copy link

GNU testsuite comparison:

GNU test failed: tests/ls/stat-free-color. tests/ls/stat-free-color is passing on 'main'. Maybe you have to rebase?
Skip an intermittent issue tests/tail/overlay-headers (fails in this run but passes in the 'main' branch)
Skipping an intermittent issue tests/misc/tee (passes in this run but fails in the 'main' branch)

@github-actions
Copy link

GNU testsuite comparison:

GNU test failed: tests/ls/stat-free-color. tests/ls/stat-free-color is passing on 'main'. Maybe you have to rebase?
Skip an intermittent issue tests/tail/overlay-headers (fails in this run but passes in the 'main' branch)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants