refactor(replica): move resumable reader internal#1081
Closed
refactor(replica): move resumable reader internal#1081
Conversation
…restores During restore of large databases (~55 GB), the LTX compactor opens all S3 streams upfront but processes pages in page-number order. Incremental LTX files with high-numbered pages sit idle for minutes while lower pages are processed from the snapshot. Storage providers (S3, Tigris) close these idle connections, causing "unexpected EOF" errors. Wrap each LTX file stream in a ResumableReader that detects connection drops (both non-EOF errors and premature EOF) and transparently reopens from the current byte offset using range requests. Partial reads are returned without error so callers like io.ReadFull continue seamlessly. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Collaborator
Author
|
Closing in favor of #1082 which includes this refactor along with integration tests. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Move the ResumableReader helper into the internal package and relocate its tests, keeping restore behavior unchanged.
Problem
PR #1080 introduced an exported ResumableReader in the main litestream package primarily for testing. Ben suggested moving it to internal to avoid cluttering the public API.
Solution
Relocate ResumableReader and its tests to
internal/and update restore to useinternal.NewResumableReader.Scope
In scope:
internalinternalNot in scope:
Test Plan
go test ./...go test -race ./...Related