[Introspection] Support for partial results for internal queries#4485
Open
AhmedSoliman wants to merge 1 commit intomainfrom
Open
[Introspection] Support for partial results for internal queries#4485AhmedSoliman wants to merge 1 commit intomainfrom
AhmedSoliman wants to merge 1 commit intomainfrom
Conversation
AhmedSoliman
force-pushed
the
pr4485
branch
2 times, most recently
from
bf8ba6a to
22dab09
Compare
When distributed fan-out queries (e.g. `select * from loglet_workers`) encounter unreachable nodes, return results from the available nodes instead of failing the entire query. Per-node errors are surfaced as structured warnings alongside the partial result set. Implementation: - ErrorCatchingStream wraps each per-node stream in NodeFanOutExecutionPlan, catching errors into a shared NodeWarnings (Arc<parking_lot::Mutex<Vec>>) and terminating the individual stream gracefully. - QueryContext::execute() returns a QueryResult that bundles the record batch stream with collected warning handles, walking the physical plan tree to extract them from any NodeFanOutExecutionPlan nodes. - QueryWarningStream in the gRPC handler buffers the last response and attaches accumulated warnings to it before yielding, piggybacking on existing data rather than sending a trailing empty message. - Proto: added QueryWarning message and repeated warnings field to QueryResponse. - CLI: restatectl collects warnings and displays them on stderr after results.
Contributor
Author
|
Adding @tillrohrmann as reviewer to help driving this to completion. I'd like to get this merged. It's a low-impact PR since it only impacts internal datafusion tables (via restatectl). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When distributed fan-out queries (e.g.
select * from loglet_workers) encounterunreachable nodes, return results from the available nodes instead of failing the
entire query. Per-node errors are surfaced as structured warnings alongside the
partial result set.
Implementation:
catching errors into a shared NodeWarnings (Arc<parking_lot::Mutex>)
and terminating the individual stream gracefully.
stream with collected warning handles, walking the physical plan tree to
extract them from any NodeFanOutExecutionPlan nodes.
accumulated warnings to it before yielding, piggybacking on existing data
rather than sending a trailing empty message.