diff: use custom Myers-based engine to improve performance

Until now we have used the engine provided by the 'diff' crate to do
the actual diffing. It implements a Longest Common Subsequence algorithm
that explores all potential offsets between the two collections.

This produces high quality diffs, but has the downside of requiring a
huge amount of memory for the left x right lines matrix, which makes it
unable to process big files (~36MB):

  > ./target/release/diffutils diff test-data/huge-base test-data/huge-very-similar
  memory allocation of 2202701222500 bytes failed
  fish: Job 1, './target/release/diffutils diff…' terminated by signal SIGABRT (Abort)

The author has begun an implementation of the Myers algorithm, which
will be offered as an alternative to the full LCS one, but has not
made any progress on merging it for months, and has not been responsive.

It probably makes sense for us to have our own engine, in any case,
so that we can evolve it along with the tool and make any adjustments
or apply any heuristics we decide could be helpful for matching GNU
diff's behavior or performance.

The Myers algorithm is a more efficient implementation of LCS, as it
only uses a couple vectors with 2 * (m + n) positions, rather than the
m * n positions used by the full LCS matrix, where m and n are the
number of lines of each file.

With this new engine we outperform GNU diff significantly when comparing
those two big files that are largely equal, with changes at the top and
bottom while producing the exact same diff and using almost exactly the
same amount of memory at the peak:

  Benchmark 1: ./target/release/diffutils diff test-data/huge-base test-data/huge-very-similar
    Time (mean ± σ):     105.0 ms ±   2.5 ms    [User: 62.2 ms, System: 41.6 ms]
    Range (min … max):   101.7 ms … 111.3 ms    28 runs

    Warning: Ignoring non-zero exit code.

  Benchmark 2: diff test-data/huge-base test-data/huge-very-similar
    Time (mean ± σ):      1.119 s ±  0.003 s    [User: 1.068 s, System: 0.044 s]
    Range (min … max):    1.115 s …  1.126 s    10 runs

    Warning: Ignoring non-zero exit code.

  Summary
    ./target/release/diffutils diff test-data/huge-base test-data/huge-very-similar ran
     10.66 ± 0.26 times faster than diff test-data/huge-base test-data/huge-very-similar

It's not all flowers, however. Without heuristics we suffer on files
which are very different, especially if they are large, but even if
they are small. Diffing two ~36MB and completely different  files may
take tens of minutes - but it at least works. This is where our ability
to add custom heuristics is helpful, though - we can avoid some of the
most pathological cases. Those come on the next couple commits.

  Benchmark 1: ./target/release/diffutils diff test-data/LGPL2.1 test-data/GPL3
    Time (mean ± σ):       6.5 ms ±   0.3 ms    [User: 5.5 ms, System: 0.8 ms]
    Range (min … max):     6.1 ms …   8.0 ms    435 runs

    Warning: Ignoring non-zero exit code.

  Benchmark 2: diff test-data/LGPL2.1 test-data/GPL3
    Time (mean ± σ):       1.5 ms ±   0.1 ms    [User: 1.1 ms, System: 0.3 ms]
    Range (min … max):     1.4 ms …   4.1 ms    1968 runs

    Warning: Ignoring non-zero exit code.
    Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet system without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.

  Benchmark 3: ./target/release/diffutils.old diff test-data/LGPL2.1 test-data/GPL3
    Time (mean ± σ):       2.1 ms ±   0.2 ms    [User: 1.2 ms, System: 0.8 ms]
    Range (min … max):     1.8 ms …   2.9 ms    1435 runs

    Warning: Ignoring non-zero exit code.

  Summary
    diff test-data/LGPL2.1 test-data/GPL3 ran
      1.42 ± 0.17 times faster than ./target/release/diffutils.old diff test-data/LGPL2.1 test-data/GPL3
      4.35 ± 0.43 times faster than ./target/release/diffutils diff test-data/LGPL2.1 test-data/GPL3

It is worth pointing out as well that the reason GNU diff is outperformed
in that best case scenario is because it does a lot more work to enable
other optimizations we do not implement such as hashing each line and
separating out those that only appear on one of the files. That work
adds up on big files, but allows GNU diff to outperform by a similar
factor when the files are not just different by rearranging lines, but
by having completely different lines.

Author: kov

Cargo.lock

@@ -16,11 +16,12 @@ path = "src/main.rs"
 
 [dependencies]
 chrono = "0.4.38"
-diff = "0.1.13"
 itoa = "1.0.11"
 regex = "1.10.4"
 same-file = "1.0.6"
 unicode-width = "0.2.0"
+tracing = "0.1.40"
+tracing-subscriber = { version = "0.3", features = ["env-filter"] }
 
 [dev-dependencies]
 pretty_assertions = "1.4.0"
@@ -42,6 +43,11 @@ ci = ["github"]
 # The installers to generate for each app
 installers = []
 # Target platforms to build apps for (Rust target-triple syntax)
-targets = ["aarch64-apple-darwin", "x86_64-apple-darwin", "x86_64-unknown-linux-gnu", "x86_64-pc-windows-msvc"]
+targets = [
+    "aarch64-apple-darwin",
+    "x86_64-apple-darwin",
+    "x86_64-unknown-linux-gnu",
+    "x86_64-pc-windows-msvc",
+]
 # Publish jobs to run in CI
 pr-run-mode = "plan"

Cargo.toml

@@ -6,6 +6,7 @@
 use std::collections::VecDeque;
 use std::io::Write;
 
+use crate::engine::{self, Edit};
 use crate::params::Params;
 use crate::utils::do_write_line;
 use crate::utils::get_modification_time;
@@ -77,9 +78,9 @@ fn make_diff(
     // Rust only allows allocations to grow to isize::MAX, and this is bigger than that.
     let mut expected_lines_change_idx: usize = !0;
 
-    for result in diff::slice(&expected_lines, &actual_lines) {
+    for result in engine::diff(&expected_lines, &actual_lines) {
         match result {
-            diff::Result::Left(str) => {
+            Edit::Delete(str) => {
                 if lines_since_mismatch > context_size && lines_since_mismatch > 0 {
                     results.push(mismatch);
                     mismatch = Mismatch::new(
@@ -101,7 +102,7 @@ fn make_diff(
                 line_number_expected += 1;
                 lines_since_mismatch = 0;
             }
-            diff::Result::Right(str) => {
+            Edit::Insert(str) => {
                 if lines_since_mismatch > context_size && lines_since_mismatch > 0 {
                     results.push(mismatch);
                     mismatch = Mismatch::new(
@@ -132,7 +133,7 @@ fn make_diff(
                 line_number_actual += 1;
                 lines_since_mismatch = 0;
             }
-            diff::Result::Both(str, _) => {
+            Edit::Keep(str) => {
                 expected_lines_change_idx = !0;
                 // if one of them is missing a newline and the other isn't, then they don't actually match
                 if (line_number_actual > actual_lines_count)

src/context_diff.rs

@@ -5,6 +5,7 @@
 
 use std::io::Write;
 
+use crate::engine::{self, Edit};
 use crate::params::Params;
 use crate::utils::do_write_line;
 
@@ -71,21 +72,21 @@ fn make_diff(expected: &[u8], actual: &[u8], stop_early: bool) -> Result<Vec<Mis
         return Err(DiffError::MissingNL);
     }
 
-    for result in diff::slice(&expected_lines, &actual_lines) {
+    for result in engine::diff(&expected_lines, &actual_lines) {
         match result {
-            diff::Result::Left(str) => {
+            Edit::Delete(str) => {
                 if !mismatch.actual.is_empty() {
                     results.push(mismatch);
                     mismatch = Mismatch::new(line_number_expected, line_number_actual);
                 }
                 mismatch.expected.push(str.to_vec());
                 line_number_expected += 1;
             }
-            diff::Result::Right(str) => {
+            Edit::Insert(str) => {
                 mismatch.actual.push(str.to_vec());
                 line_number_actual += 1;
             }
-            diff::Result::Both(_str, _) => {
+            Edit::Keep(_str) => {
                 line_number_expected += 1;
                 line_number_actual += 1;
                 if !mismatch.actual.is_empty() || !mismatch.expected.is_empty() {