sort: fix newline handling across large and/or multiple files

[jacob-greenfield]

When the sort utility is searching for newlines in a large buffer, skip past any previously-searched data. This fixes a quadratic-time overhead that would occur in case of a line that is far longer than the configured buffer size (and START_BUFFER_SIZE).

On my M1 MacBook Pro:

$ dd if=/dev/zero bs=1M count=4096 status=progress | tr '\0' 'A' | head -c 4294967295 > oneline_4G.txt && cargo build --release -p uu_sort && time ./target/release/sort oneline_4G.txt >out.txt
...
./target/release/sort oneline_4G.txt > out.txt  1.96s user 3.32s system 78% cpu 6.687 total

This PR also fixes a separate issue where the check last_file_target_size != leftover_len was used to determine whether a file is non-empty; however this could fail if the buffer was recently resized, since leftover_len accounts for the additional capacity but last_file_target_size does not. This can cause two files to be concatenated without a newline in between. To reproduce: run head -c 8000 /dev/zero | tr '\0' 'b' >b.txt; echo aaa >a.txt; cargo run sort b.txt a.txt. I added a new test test_sort::test_start_buffer to cover this.

Fixes #8583.

[jacob-greenfield]

I am a bit embarrassed to admit I did not notice #8652 before submitting this PR. I am not sure how this affects or overlaps with 8652 but I don't doubt that 8652 is also a valid approach to the problem. :)

Edit: If nothing else, this PR still fixes the last_file_target_size issue, and those commits could be cherry-picked.

[github-actions]

GNU testsuite comparison:

Skip an intermittent issue tests/tail/overlay-headers (fails in this run but passes in the 'main' branch)
Skipping an intermittent issue tests/timeout/timeout (passes in this run but fails in the 'main' branch)

[codspeed-hq]

CodSpeed Performance Report

Merging #8746 will not alter performance

_{Comparing jacob-greenfield:sort-fix-buffer-read (25e269c) with main (ffbcb4b)}

Summary

✅ 106 untouched
⏩ 73 skipped¹

Footnotes

1. 73 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

[github-actions]

GNU testsuite comparison:

GNU test failed: tests/wc/wc-files0-from. tests/wc/wc-files0-from is passing on 'main'. Maybe you have to rebase?
Skip an intermittent issue tests/tail/overlay-headers (fails in this run but passes in the 'main' branch)
Skip an intermittent issue tests/timeout/timeout (fails in this run but passes in the 'main' branch)

[github-actions]

GNU testsuite comparison:

Skip an intermittent issue tests/tail/overlay-headers (fails in this run but passes in the 'main' branch)

[jacob-greenfield]

Test fixtures are now generated programmatically, let me know if I should squash to remove the hardcoded fixtures from the git history?

[kehrazy]

I am a bit embarrassed to admit I did not notice #8652 before submitting this PR. I am not sure how this affects or overlaps with 8652 but I don't doubt that 8652 is also a valid approach to the problem. :)

Edit: If nothing else, this PR still fixes the last_file_target_size issue, and those commits could be cherry-picked.

if anything, this change is much leaner and easier to review (and passes CI, at the very least), as opposed to 8652 :)

[github-actions]

GNU testsuite comparison:

Skip an intermittent issue tests/misc/tee (fails in this run but passes in the 'main' branch)
Skip an intermittent issue tests/tail/overlay-headers (fails in this run but passes in the 'main' branch)

[github-actions]

GNU testsuite comparison:

Skipping an intermittent issue tests/rm/rm1 (passes in this run but fails in the 'main' branch)

[github-actions]

GNU testsuite comparison:

Skip an intermittent issue tests/tail/overlay-headers (fails in this run but passes in the 'main' branch)
Skipping an intermittent issue tests/misc/tee (passes in this run but fails in the 'main' branch)

[sylvestre]

well done (instead of timeout) :)

/usr/bin/sort oneline_4G.txt ran
2.39 ± 0.04 times faster than ./target/release/coreutils sort oneline_4G.txt

[github-actions]

GNU testsuite comparison:

Skip an intermittent issue tests/misc/tee (fails in this run but passes in the 'main' branch)
Skipping an intermittent issue tests/timeout/timeout (passes in this run but fails in the 'main' branch)

[github-actions]

GNU testsuite comparison:

Skip an intermittent issue tests/misc/tee (fails in this run but passes in the 'main' branch)