Dupli-Annihilator-G v2.9.10

Dupli-Annihilator-G v2.9.10

Executive Summary

v2.9.10 fixes the desktop startup regression that affected the unsigned Windows binaries published in v2.9.9.

The root cause was a missing default plugins.updater block in the versioned Tauri config. With tauri-plugin-updater 2.10, omitting that block causes the app to panic during startup even when no updater signing keys are configured.

---

Fix: Desktop Startup On Unsigned Builds

What changed

• the default desktop config now ships with plugins.updater present and disabled,
• the disabled updater config includes the required pubkey field,
• the release preparation flow now runs configure-updater.mjs before desktop build steps.

Why it matters

Unsigned local and manual-release desktop builds now start correctly instead of exiting immediately on launch.

---

Included Assets In This Manual Release

This GitHub Release is published manually and includes ready-to-download Windows assets built from tag v2.9.10:

• Dupli-Annihilator-G_2.9.10_x64-setup.exe
• dedupe_desktop_tauri.exe
• dedupe_cli.exe
• SHA256SUMS.txt

---

Distribution Note

This release still does not include signed updater artifacts because the GitHub Actions signing credentials are not configured in this repository environment.

Included:

• source code for tag v2.9.10,
• release notes,
• unsigned Windows desktop installer,
• unsigned Windows desktop executable,
• Windows CLI executable,
• SHA-256 checksums for the attached binaries.

Not included:

• signed desktop installers,
• CI-published updater metadata,
• notarized macOS artifacts,
• signed Linux updater artifacts.

---

Validation

• desktop executable no longer panics with updater config deserialization on startup,
• cargo test --workspace
• cargo build -p dedupe_cli --release
• cargo tauri build --ci --no-sign

---

Compatibility

• No breaking API changes.
• No dedupe output format changes.
• Existing engine semantics are preserved.

Dupli-Annihilator-G v2.9.9

Dupli-Annihilator-G v2.9.9

Executive Summary

v2.9.9 is the clean follow-up release that packages the desktop and CLI artifacts from a fully consistent tagged state.

It carries forward the dedupe engine improvements introduced in v2.9.7 and the desktop NSIS packaging fix prepared after that release, while also fixing the release workflow so the desktop Cargo.lock is regenerated and committed as part of version preparation.

---

Fix: Release Consistency For Desktop Builds

What changed

• the release preparation script now regenerates apps/desktop/src-tauri/Cargo.lock,
• the same script now stages apps/desktop/src-tauri/Cargo.lock into the release commit,
• desktop packaging remains aligned with the valid NSIS install mode currentUser.

Why it matters

This ensures the published desktop assets come from the exact tagged source state instead of a post-tag lockfile refresh.

---

Included Assets In This Manual Release

This GitHub Release is published manually and includes ready-to-download Windows assets built from tag v2.9.9:

• Dupli-Annihilator-G_2.9.9_x64-setup.exe
• dedupe_desktop_tauri.exe
• dedupe_cli.exe
• SHA256SUMS.txt

---

Distribution Note

This release still does not include signed updater artifacts because the GitHub Actions signing credentials are not configured in this repository environment.

Included:

• source code for tag v2.9.9,
• release notes,
• unsigned Windows desktop installer,
• unsigned Windows desktop executable,
• Windows CLI executable,
• SHA-256 checksums for the attached binaries.

Not included:

• signed desktop installers,
• CI-published updater metadata,
• notarized macOS artifacts,
• signed Linux updater artifacts.

---

Validation

• node scripts/release/prepare-release.mjs 2.9.9 --commit --tag --push
• cargo build -p dedupe_cli --release
• cargo tauri build --ci --no-sign

---

Compatibility

• No breaking API changes.
• No dedupe output format changes.
• Existing engine semantics are preserved.

Dupli-Annihilator-G v2.9.7

Dupli-Annihilator-G v2.9.7

Executive Summary

v2.9.7 improves the exact deduplication engine for real-world large-file workloads. AUTO mode is now host-aware, using available system memory plus a lightweight corpus sample to choose between RAM and DISK, and the DISK path reduces buckets in parallel for the main overlap-heavy workload.

This is a patch release: no breaking API changes, no output format changes, and no workflow changes for existing desktop or CLI users.

---

Improvement: Host-Aware AUTO Mode

AUTO no longer relies on a single static size rule.

What changed

• rich inputs are still resolved before mode selection,
• the engine now reads host memory telemetry (available, free, total),
• AUTO samples the resolved input corpus to estimate token density, uniqueness, duplicate pressure, and projected RAM set size,
• multi-file partial-overlap workloads can now resolve to DISK even when raw input size alone would have chosen RAM,
• benchmark JSON and run summaries now expose auto_* decision telemetry.

Why it matters

The baseline real-corpus target (Test1.csv + Test3.csv) previously resolved to RAM even though DISK was substantially faster on the benchmark machine. This release fixes that misclassification.

---

Improvement: Faster DISK Preserve Path

The bucketed DISK pipeline now reduces buckets concurrently.

What changed

• bucket partitioning remains serial,
• bucket reduction runs with bounded parallelism,
• reduced bucket outputs are still concatenated in bucket order,
• existing DISK + PreserveFirstSeen behavior is preserved: exact dedupe without a global first-seen guarantee.

Why it matters

The main 2-file overlap benchmark improved enough to hit the phase-1 target on the baseline machine.

---

Improvement: Lower Hot-Path Overhead

Several engine hot paths were tightened around tokenization and I/O.

What changed

• ASCII-only lines now use a byte-scanning fast path in tokenization,
• readers and writers use larger explicit buffers,
• RAM store insertion avoids the previous double-lookup pattern,
• AUTO sampling also feeds a better reserve hint for the RAM store.

Why it matters

Unique-heavy single-file workloads improved in RAM mode, while the benchmark harness now reports the decision telemetry needed for the next tuning round.

---

Validation

• Workspace tests pass locally with cargo test --workspace --locked.
• Real-corpus benchmark suites were rerun against testfiles/.
• Benchmark history and current baselines were updated in docs/benchmark-history.md and docs/benchmarks.md.

---

Compatibility

• No breaking desktop workflow changes.
• No output format changes.
• Existing deduplication semantics are preserved; this release improves mode selection, benchmark observability, and throughput on the primary large-file overlap scenario.

---

Distribution Note

This GitHub Release was published manually because the signed desktop release workflow is not configured yet in this repository environment.

Included now:

• source code for tag v2.9.7
• release notes

Not included yet:

• signed desktop installers
• updater metadata published by CI
• notarized or signature-backed public desktop artifacts

Dupli-Annihilator-G v2.9.6

Dupli-Annihilator-G v2.9.6

Executive Summary

v2.9.6 improves the rich-input preprocessing path for PDF and EPUB workloads. The engine now extracts rich files with bounded parallelism, keeps progress reporting coherent under concurrent extraction, and reduces PDF memory overhead by streaming page text directly into temporary files.

This is a patch release: no output format changes, no workflow changes, and no compatibility break for existing desktop or CLI usage.

---

Improvement: Parallel Rich-Input Extraction

The preprocessing stage no longer resolves rich files strictly one by one.

What changed

• PDF and EPUB extraction now runs with a bounded worker pool sized from available CPU parallelism,
• the engine still reconstructs the resolved input list in the original user-supplied order,
• failed rich inputs are still reported as warnings and skipped instead of aborting the whole run.

Why it matters

Large batches of rich inputs can now use more than one core during preprocessing, which reduces startup time before tokenization begins.

---

Improvement: Lower-Memory PDF Extraction

The PDF path was refactored to avoid building one giant in-memory string per document.

What changed

• PDFs are opened once and processed page by page,
• extracted text is written directly into the temporary text file through a buffered writer,
• cancellation checks now happen between PDF pages,
• the previous encrypted-PDF empty-password behavior is preserved.

Why it matters

This reduces peak memory usage for large PDFs and improves responsiveness when canceling a job during extraction.

---

Improvement: Concurrent Extraction Telemetry

Progress reporting was adjusted so concurrent extraction does not produce misleading counters.

What changed

• extraction stage progress now tracks completed rich files explicitly,
• the current rich input path remains visible while extraction is in progress,
• the job runner no longer assumes a serial "started index" model for rich-input progress.

Why it matters

Users still get trustworthy extraction-stage feedback even when multiple rich files are being processed at the same time.

---

Validation

• Workspace tests pass locally with cargo test --workspace.
• Desktop frontend metadata was refreshed with npm --prefix apps/desktop install --package-lock-only.
• Release-managed versions were aligned to 2.9.6.

---

Compatibility

• No breaking desktop workflow changes.
• No output format changes.
• Existing deduplication behavior is preserved; this release improves extraction throughput, cancellation responsiveness, and PDF memory usage for rich inputs.

Dupli-Annihilator-G v2.9.5

Dupli-Annihilator-G v2.9.5

Executive Summary

v2.9.5 is the final corrective release for the rich-input startup and extraction improvements prepared in the previous patch attempts. The shipped functionality is the same verified payload: visible extraction telemetry, faster EPUB preprocessing, and more accurate AUTO mode selection for rich inputs.

This patch exists to publish the already-validated changes under a correctly aligned release commit and tag.

---

Feature: Visible Extraction Telemetry

Jobs that begin with EPUB or PDF extraction now expose useful progress immediately.

What changed

• the telemetry footer shows extraction progress before tokenization starts,
• extraction reports the current rich-file index and total rich files,
• the current input path is shown while a file is being extracted,
• the frontend polls immediately after start_job so the first visible state arrives sooner.

Why it matters

Large rich-input jobs no longer appear frozen during the initial preprocessing stage.

---

Improvement: Faster EPUB Extraction

The EPUB text extraction path was optimized to reduce startup overhead.

What changed

• chapter text is streamed through a buffered writer,
• HTML/XML tag stripping is performed in a single pass,
• common entities such as &,  , <, >, ", and ' are decoded inline,
• cancellation checks happen between EPUB chapters/resources.

Why it matters

This reduces allocation and copying overhead when many EPUBs are loaded at once and improves responsiveness during cancellation.

---

Improvement: Better AUTO Mode Choice For Rich Inputs

AUTO mode now decides between RAM and DISK after rich inputs are resolved to extracted text.

Why it matters

Compressed EPUB containers are no longer treated as if their on-disk size were the real processing footprint, which makes mode selection more accurate for large rich-input jobs.

---

Compatibility

• No breaking desktop workflow changes.
• No output format changes.
• Existing deduplication behavior is preserved; this release improves observability, extraction performance, and AUTO-mode selection accuracy for rich inputs.

Dupli-Annihilator-G v2.9.2

Dupli-Annihilator-G v2.9.2

Executive Summary

v2.9.2 ships the new desktop running focus mode and includes the Tauri-side compatibility fix required to restore successful desktop bundle builds across the release matrix.

---

Feature: Running Focus Mode

The main deduplication workflow now enters a more deliberate runtime state:

• background workspace panels are blurred, dimmed, and blocked from interaction,
• the Output panel stays sharp and elevated,
• telemetry remains clear as a persistent operations dock,
• responsive layout keeps the focused controls visible on narrower windows.

This improves readability and reduces accidental interaction while a job is actively running.

---

Fix: Desktop Release Build Compatibility

The Tauri desktop binary contained stale n-gram command bindings that no longer existed in dedupe_backend.

What was fixed

• removed dead imports for NgramRequest and NgramResponse,
• removed the obsolete run_ngram_extract Tauri command,
• aligned the desktop invoke handler with the actual backend command surface,
• refreshed the Tauri Cargo.lock after validating the corrected desktop crate.

Why it matters

Without this fix, cross-platform desktop bundle jobs failed during Rust compilation before artifacts could be published.

---

Files Changed

File	Change
apps/desktop/src/main.tsx	Added the running focus-mode layout and behavior
apps/desktop/src/styles.css	Added blur/glass/focus styling and responsive handling
apps/desktop/src-tauri/src/main.rs	Removed stale n-gram Tauri bindings that broke desktop bundle builds
apps/desktop/src-tauri/Cargo.lock	Refreshed resolved desktop dependencies after the Tauri fix

---

Compatibility

• No backend API changes.
• No output format changes.
• Desktop release only; existing job/event contracts remain unchanged.

Dupli-Annihilator-G v2.9.0

Dupli-Annihilator-G Release Notes

Summary

• Professional desktop deduplication workflow for very large text datasets.
• Native installers for Windows and macOS.
• High-performance Rust engine with RAM and DISK execution strategies.

Highlights

• Multi-file deduplication with deterministic output controls.
• Mission Report with exportable run diagnostics.
• In-app update checks and release automation pipeline.
• Multilingual UI support.

Core Capabilities

• Input handling:
  • File picker and drag-and-drop.
  • Multiple input files merged into one output stream.
• Processing:
  • Modes: AUTO, RAM, DISK.
  • Ordering: preserve_first_seen, alphabetical, unordered_fast.
  • Optional normalization: trim, drop_empty.
  • Custom output separator (escaped or raw).
• Observability:
  • Live stage/progress/throughput/ETA telemetry.
  • Final Mission Report with reduction metrics, timeline, and warnings.
  • Open output/folder, copy report, export JSON, run again.
• Operations:
  • Release version/tag coherence checks.
  • Tag-from-main enforcement in CI.
  • Automated cross-platform release publishing.

Notes

• No breaking changes expected for standard desktop usage.
• Refer to README.md and docs/ for architecture and operational details.

Dupli-Annihilator-G v2.8.0

Dupli-Annihilator-G Release Notes

Summary

• Professional desktop deduplication workflow for very large text datasets.
• Native installers for Windows and macOS.
• High-performance Rust engine with RAM and DISK execution strategies.

Highlights

• Multi-file deduplication with deterministic output controls.
• Mission Report with exportable run diagnostics.
• In-app update checks and release automation pipeline.
• Multilingual UI support.

Core Capabilities

• Input handling:
  • File picker and drag-and-drop.
  • Multiple input files merged into one output stream.
• Processing:
  • Modes: AUTO, RAM, DISK.
  • Ordering: preserve_first_seen, alphabetical, unordered_fast.
  • Optional normalization: trim, drop_empty.
  • Custom output separator (escaped or raw).
• Observability:
  • Live stage/progress/throughput/ETA telemetry.
  • Final Mission Report with reduction metrics, timeline, and warnings.
  • Open output/folder, copy report, export JSON, run again.
• Operations:
  • Release version/tag coherence checks.
  • Tag-from-main enforcement in CI.
  • Automated cross-platform release publishing.

Notes

• No breaking changes expected for standard desktop usage.
• Refer to README.md and docs/ for architecture and operational details.

Dupli-Annihilator-G v2.7.0

Dupli-Annihilator-G Release Notes

Summary

• Professional desktop deduplication workflow for very large text datasets.
• Native installers for Windows and macOS.
• High-performance Rust engine with RAM and DISK execution strategies.

Highlights

• Multi-file deduplication with deterministic output controls.
• Mission Report with exportable run diagnostics.
• In-app update checks and release automation pipeline.
• Multilingual UI support.

Core Capabilities

• Input handling:
  • File picker and drag-and-drop.
  • Multiple input files merged into one output stream.
• Processing:
  • Modes: AUTO, RAM, DISK.
  • Ordering: preserve_first_seen, alphabetical, unordered_fast.
  • Optional normalization: trim, drop_empty.
  • Custom output separator (escaped or raw).
• Observability:
  • Live stage/progress/throughput/ETA telemetry.
  • Final Mission Report with reduction metrics, timeline, and warnings.
  • Open output/folder, copy report, export JSON, run again.
• Operations:
  • Release version/tag coherence checks.
  • Tag-from-main enforcement in CI.
  • Automated cross-platform release publishing.

Notes

• No breaking changes expected for standard desktop usage.
• Refer to README.md and docs/ for architecture and operational details.

Dupli-Annihilator-G v2.6.3

Dupli-Annihilator-G v2.6.3

Executive Summary

v2.6.3 fixes a display bug where the Per-File Breakdown table showed system temporary file names (e.g. .tmpXgIdbM) instead of the real input file names for EPUB and PDF inputs.

---

Bug Fix: Per-File Breakdown Shows Real File Names for EPUB/PDF

Problem

When an EPUB or PDF file was added as input, the engine extracted its text into a temporary file before processing. The FileStats struct then recorded the temp file path (e.g. .tmpqqLOq5, .tmpXgIdbM) rather than the original input path. This temp name was forwarded all the way to the UI and shown verbatim in the Per-File Breakdown table of the Mission Report.

Fix

A path_aliases map (HashMap<PathBuf, PathBuf>) is now built inside resolve_rich_inputs as each temp file is created. It maps every temp path back to the original input path. The run_ram function receives this map and uses it when constructing FileStats, so both the display path and the file-size metadata source reflect the real original file — not the transient temp file.

File changed: crates/core/src/engine.rs

+    let mut path_aliases: HashMap<PathBuf, PathBuf> = HashMap::new();
     ...
     // PDF branch
+    let temp_path = tmp.path().to_path_buf();
+    path_aliases.insert(temp_path.clone(), path.clone());
-    resolved_inputs.push(tmp.path().to_path_buf());
+    resolved_inputs.push(temp_path);
     // EPUB branch (same pattern)
     ...
+    let display_path = path_aliases.get(path).cloned().unwrap_or_else(|| path.clone());
+    let file_bytes = std::fs::metadata(&display_path).map(|m| m.len()).ok();
     per_file.push(FileStats {
-        path: path.clone(),
+        path: display_path,
         file_bytes,

---

Files Changed

File	Change
crates/core/src/engine.rs	Track temp→original path aliases in resolve_rich_inputs; use them in run_ram when building FileStats

---

Compatibility

• No breaking changes. Pure internal fix — no public API changes.
• Only RAM mode is affected (Disk mode does not produce per-file stats, unchanged).
• All formats unaffected by this fix (plain text, CSV, TSV, LOG) continue to work identically.