Skip to content

Conversation

@georgeh0
Copy link
Member

@georgeh0 georgeh0 commented Nov 13, 2025

fixes #1057

@georgeh0 georgeh0 requested a review from Copilot November 13, 2025 23:07
Copilot finished reviewing on behalf of georgeh0 November 13, 2025 23:09
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR implements fine-grained data lineage tracking to detect logic changes more precisely. Instead of using a single plan-wide fingerprint for all import operations, it introduces per-source fingerprinting that only considers the export operations that actually depend on each source.

  • Introduces SourceLogicFingerprint to compute fingerprints based on which export operations use a specific source
  • Adds FieldDefFingerprint to track data lineage through field definitions, recording which source operations contribute to each field
  • Replaces the plan-level ExecutionPlanLogicFingerprint with two legacy fingerprint versions stored in a vector

Reviewed Changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 5 comments.

Show a summary per file
File Description
rust/cocoindex/src/execution/indexing_status.rs Adds SourceLogicFingerprint struct to compute per-source fingerprints based on dependent export operations
rust/cocoindex/src/execution/source_indexer.rs Updates to use SourceLogicFingerprint instead of plan-level fingerprint for tracking source versions
rust/cocoindex/src/execution/row_indexer.rs Changes parameter types to accept SourceLogicFingerprint in version comparison methods
rust/cocoindex/src/execution/evaluator.rs Adds source_logic_fp field to SourceRowEvaluationContext
rust/cocoindex/src/execution/dumper.rs Creates and threads SourceLogicFingerprint through evaluation logic
rust/cocoindex/src/builder/plan.rs Adds FieldDefFingerprint struct and updates ExecutionPlan to use legacy fingerprints vector
rust/cocoindex/src/builder/flow_builder.rs Updates flow builder to accept FieldDefFingerprint parameters (with incomplete implementation)
rust/cocoindex/src/builder/analyzer.rs Implements fingerprint propagation logic through field paths, collectors, and operation analysis
rust/cocoindex/src/service/flows.rs Updates service handlers to create and use SourceLogicFingerprint for evaluation contexts

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@georgeh0 georgeh0 merged commit 557f255 into main Nov 13, 2025
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[FEATURE] Skip reprocessing of entire rows based on finer-grain lineage information

2 participants