Dedup Initial Implementation#889

Draft

ikreymer wants to merge 24 commits intomainfrom

hash-based-dedup

Commits on Sep 14, 2025

dedup work:
ikreymer
committed

Commits on Sep 17, 2025

args: add separate --dedupIndexUrl to support separate redis for dedup
ikreymer
committed

Commits on Sep 20, 2025

Commits on Sep 23, 2025

dedup indexing: strip hash prefix from digest, as cdx does not have it
ikreymer
committed
use dedup redis for queue up wacz files that need to be updated
ikreymer
committed

Commits on Sep 25, 2025

dedup post requests and non-404s as well!
ikreymer
committed

Commits on Oct 1, 2025

Merge branch 'main' into hash-based-dedup
tw4l
authored

Commits on Oct 14, 2025

Commits on Oct 18, 2025

- track source index for each hash, so entry becomes '<source index> <date> <url>'
ikreymer
committed

Commits on Oct 24, 2025

Commits on Oct 25, 2025