Delta lake #3
Draft
Delta lake #3
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
benjamin-awd
force-pushed
the
delta-lake
branch
2 times, most recently
from
779f822 to
494bf95
Compare
benjamin-awd
force-pushed
the
delta-lake
branch
from
494bf95 to
bfb8003
Compare
benjamin-awd
force-pushed
the
delta-lake
branch
from
dea9791 to
d2f12ea
Compare
benjamin-awd
force-pushed
the
delta-lake
branch
from
d2f12ea to
80949c7
Compare
benjamin-awd
force-pushed
the
delta-lake
branch
from
8bd3c63 to
a1ca534
Compare
benjamin-awd
force-pushed
the
delta-lake
branch
from
9ba0950 to
8d9da1b
Compare
benjamin-awd
force-pushed
the
delta-lake
branch
from
8d9da1b to
26f5eaf
Compare
benjamin-awd
force-pushed
the
delta-lake
branch
from
978c6d7 to
ab51706
Compare
benjamin-awd
force-pushed
the
delta-lake
branch
from
54a1e6f to
c705f91
Compare
benjamin-awd
force-pushed
the
delta-lake
branch
from
c582a48 to
2dc64b2
Compare
… parsing Replace fragile string-based error detection with proper pattern matching on DeltaTableError and ObjectStoreError variants. This provides: - Type-safe error classification that won't break with upstream changes - Cleaner separation of concerns with dedicated helper functions - More reliable detection of concurrent conflicts and retriable errors The ObjectStoreError variants (Precondition, AlreadyExists, PermissionDenied, etc.) are now matched directly instead of searching for substrings like "FAILED_PRECONDITION" or "412" in error messages. Co-Authored-By: Claude <[email protected]>
…rKind enum Introduce WriteErrorKind enum as a single source of truth for error classification. This consolidates the duplicate error classification logic that was spread across DeltaLakeService and DeltaLakeRetryLogic. Benefits: - Single place to update when error handling needs change - Consistent classification between internal retry logic and Tower retry - More readable code with semantic error kinds instead of boolean flags - DeltaLakeRetryLogic reduced to a single line using the shared enum The enum variants (ConcurrentConflict, SchemaMismatch, PermissionDenied, NotFound, Transient, Permanent) map directly to retry strategies. Co-Authored-By: Claude <[email protected]>
Collapse 6 error variants into 4 by merging similar non-retriable cases: - ConcurrentConflict: internal retry with table reload - SchemaMismatch: internal retry with schema reload - Transient: Tower-level retry (IO, network) - NonRetriable: no retry (permissions, not found, config errors) This simplifies the error classification while maintaining the same behavior - all the merged variants had identical retry semantics. Co-Authored-By: Claude <[email protected]>
benjamin-awd
force-pushed
the
delta-lake
branch
from
2810655 to
3fca356
Compare
Replace manual concurrent conflict retry loop with delta-rs built-in retry mechanism via CommitProperties::with_max_retries(). This delegates conflict resolution to the library which handles: - Optimistic concurrency control - Automatic retry on version conflicts - Internal state management The schema mismatch retry loop is preserved since it requires explicit table reload to pick up schema changes, which the library doesn't do. This removes ~50 lines of retry logic including the ExponentialBackoff import and manual backoff/sleep handling. Co-Authored-By: Claude <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Vector configuration
How did you test this PR?
Change Type
Is this a breaking change?
Does this PR include user facing changes?
no-changeloglabel to this PR.References
Notes
@vectordotdev/vectorto reach out to us regarding this PR.pre-pushhook, please see this template.make fmtmake check-clippy(if there are failures it's possible some of them can be fixed withmake clippy-fix)make testgit merge origin masterandgit push.Cargo.lock), pleaserun
make build-licensesto regenerate the license inventory and commit the changes (if any). More details here.