Dedup Backend Initial Implementation #2868
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
tw4l
reviewed
Sep 30, 2025
ikreymer
force-pushed
the
dedup-initial
branch
from
fdbcaa1 to
d8c3a05
Compare
SuaYoo
reviewed
Oct 13, 2025
|
Also, updating |
SuaYoo
reviewed
Oct 13, 2025
- add CollIndex crd - add new operator - added as btrix-crds 0.2.0 operator collindex: - redis_dedupe_memory and redis_dedupe_storage configurable - dedupe_importer_channel can configure crawler channel for index imports operator crawl: - add 'waiting_for_dedupe_index' state to indicate crawl is awaiting dedup index - on crawl success, get list of required crawls deduped against and set 'requiredCrawls' and 'requiredByCrawls' collection: - supports creating, updating, deleting CollIndex via CrawlManager - tracks via hasDedupeIndex - mark CollIndex object for updates by updating collItemsUpdatedAt on items add or remove - CollIndex deleted on collection deleted workflow: - has dedupeCollId if dedupe is enabled - dedupeCollId included in autoAdd collections - update empty string dedupeCollId to remove dedupe collection from workflow
ikreymer
force-pushed
the
dedup-initial
branch
from
1bccfad to
d6e7f3e
Compare
Co-authored-by: Tessa Walsh <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #2867
Set to feature branch as base.
The backend implementation involves:
hasDedupIndexfielddedupCollIdmust also be a collection that the crawl is auto-added to.waiting_for_dedup_indexthat is entered if a crawl is starting, but index is not yet ready.For indexing, dependent on version of crawler (1.9.0 beta 0 or higher) that supports indexing mode.
Testing
This is ready for initial frontend work and testing:
dedupCollIdcan be set on the workflow to enable dedup for future crawls.hasDedupIndexto indicate if an index is enabled for it.kubectl get pods -n crawlers collindexshould work