-
Notifications
You must be signed in to change notification settings - Fork 4.1k
release-26.1: sql/inspect: add protected timestamp for "now" AOST case #160587
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Previously, INSPECT jobs without a historical AS OF SYSTEM TIME clause
would not create protected timestamp records, but still used an AOST
clause with the current timestamp. If span processing took a long time
(especially with BulkLowQoS admission control), garbage collection could
occur before the query completed, resulting in "batch timestamp must be
after replica GC threshold" errors.
This change adds per-span protected timestamp protection when INSPECT
uses "now" as the AOST. The implementation uses a coordinator-based
approach where:
1. When a processor starts processing a span and picks "now" as the
timestamp, it sends a new "span started" progress message containing
the span and timestamp via InspectProcessorProgress.
2. The coordinator's progress tracker receives this message and calls
TryToProtectBeforeGC for the relevant tables in that span. This
waits until 80% of the GC TTL has elapsed before creating a PTS,
avoiding unnecessary PTS creation for quick operations.
3. When span processing completes (existing behavior), the coordinator
cleans up the PTS for that span. Any remaining PTS records are
cleaned up when the tracker terminates (e.g., on job cancellation).
This coordinator-based design keeps PTS management centralized rather
than distributed across processors, simplifying cleanup and error
handling. PTS failures are logged but don't fail the job since the
protection is best-effort.
Resolves: cockroachdb#159866
Release note: None
In addition to checkpointing in the job, now we also log progress to text logs periodically in order to enhance observability. Release note: None
Previously, TryToProtectBeforeGC accepted a catalog.TableDescriptor parameter but only used it to call GetID() in two places. This was unnecessarily restrictive and forced callers to load a full table descriptor just to pass the ID. This change simplifies the function signature to accept a descpb.ID directly. The most significant improvement is in inspect/progress.go, where this eliminates an unnecessary DescsTxn call that was only used to load the descriptor for its ID. Release note: None Epic: None
Previously, the INSPECT job called TryToProtectBeforeGC per span with different timestamps. Since the job only stores one PTS record, each new span's call to Protect would update the existing record's timestamp via UpdateTimestamp, which removes protection for older spans. To address this, this patch changes the PTS strategy to track the minimum (oldest) timestamp across all active spans and protect only at that timestamp. Since PROTECT_AFTER mode protects all data at or after the specified timestamp, protecting at the minimum covers all active spans. When the oldest span completes, the PTS is updated to the new minimum timestamp, allowing GC of data between the old and new minimum. Release note: None
|
Thanks for opening a backport. Before merging, please confirm that the change does not break backwards compatibility and otherwise complies with the backport policy. Include a brief release justification in the PR description explaining why the backport is appropriate. All backports must be reviewed by the TL for the owning area. While the stricter LTS policy does not yet apply, please exercise judgment and consider gating non-critical changes behind a disabled-by-default feature flag when appropriate. |
spilchen
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@spilchen reviewed 13 files and all commit messages, and made 1 comment.
Reviewable status:complete! 1 of 0 LGTMs obtained (waiting on @kev-cao).
Backport 4/4 commits from #160138.
/cc @cockroachdb/release
Previously, INSPECT jobs without a historical AS OF SYSTEM TIME clause
would not create protected timestamp records, but still used an AOST
clause with the current timestamp. If span processing took a long time
(especially with BulkLowQoS admission control), garbage collection could
occur before the query completed, resulting in "batch timestamp must be
after replica GC threshold" errors.
This change adds per-span protected timestamp protection when INSPECT
uses "now" as the AOST. The implementation uses a coordinator-based
approach where:
When a processor starts processing a span and picks "now" as the
timestamp, it sends a new "span started" progress message containing
the span and timestamp via InspectProcessorProgress.
The coordinator's progress tracker receives this message and calls
TryToProtectBeforeGC for the relevant tables in that span. This
waits until 80% of the GC TTL has elapsed before creating a PTS,
avoiding unnecessary PTS creation for quick operations.
When span processing completes (existing behavior), the coordinator
cleans up the PTS for that span. Any remaining PTS records are
cleaned up when the tracker terminates (e.g., on job cancellation).
This coordinator-based design keeps PTS management centralized rather
than distributed across processors, simplifying cleanup and error
handling. PTS failures are logged but don't fail the job since the
protection is best-effort.
sql/inspect: use minimum timestamp for PTS protection
Previously, the INSPECT job called TryToProtectBeforeGC per span with
different timestamps. Since the job only stores one PTS record, each
new span's call to Protect would update the existing record's timestamp
via UpdateTimestamp, which removes protection for older spans.
To address this, this patch changes the PTS strategy to track the
minimum (oldest) timestamp across all active spans and protect only at
that timestamp. Since PROTECT_AFTER mode protects all data at or after
the specified timestamp, protecting at the minimum covers all active
spans. When the oldest span completes, the PTS is updated to the new
minimum timestamp, allowing GC of data between the old and new minimum.
Resolves: #159866
Epic: None
Release note: None
Release justification: fix for feature becoming GA in 26.1