fix(stream): sync ops with visibility after noop elimination in HashDataDispatcher#24968
Open
jw-itq wants to merge 1 commit intorisingwavelabs:mainfrom
Open
fix(stream): sync ops with visibility after noop elimination in HashDataDispatcher#24968jw-itq wants to merge 1 commit intorisingwavelabs:mainfrom
jw-itq wants to merge 1 commit intorisingwavelabs:mainfrom
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.
What's changed and what's your intention?
Fix a bug where HashDataDispatcher uses pre-elimination ops with post-elimination visibility, causing downstream panic:
expect a U- before U+.Root Cause:
In HashDataDispatcher::dispatch_data,
new_opsis computed before output_mapping.apply(), which internally calls eliminate_adjacent_noop_update(). This function may:However,
combined_visis computed after the elimination, incorporating the new visibility. The resulting chunk uses stalenew_ops(without normalize) but updatedcombined_vis(with normalize), creating a mismatch — orphanUpdateInsertops become visible without a precedingUpdateDelete, causing downstream iterators to panic.Fix:
After output_mapping.apply(), merge
new_ops(which contains dist-key-changed rewrites) with chunk.ops() (which contains post-elimination normalized ops). This ensures ops and visibility are always consistent.How to Reproduce:
When the same stream key has multiple consecutive updates within a single chunk and the dispatcher's output columns are fewer than input columns (triggering noop elimination), the middle U-/U+ pair gets eliminated, leaving the outer pair's ops un-normalized.
Example:
U- row_A (visible) U+ row_B (eliminated by noop) U- row_B (eliminated by noop) U+ row_C (visible, but ops still says UpdateInsert → panic)
Related:
combined_visbut did not sync opsChecklist
Documentation
Release note
Fix a critical bug introduced in v2.8.0 where compute nodes crash with
expect a U- before U+panic when processing streams with consecutive updates on the same key through HashDataDispatcher with column pruning. This is caused by ops/visibility mismatch after noop update elimination in the dispatcher.