test: Demonstrate EnforceDistribution and EnforceSorting behavior on the latest DF branch.#60
Closed
test: Demonstrate EnforceDistribution and EnforceSorting behavior on the latest DF branch.#60
Conversation
… output to our test case
wiedld
commented
Feb 12, 2025
Comment on lines
+3486
to
+3493
| /// Same starting plan as [`repartitions_for_aggregate_after_sorted_union`], but adds a projection | ||
| /// as well between the union and aggregate. This change the outcome: | ||
| /// | ||
| /// * we still get repartitioning | ||
| /// * we still get another sort added | ||
| /// * we no longer get 2 additional sorts pushed down below the union, after the first run | ||
| #[test] | ||
| fn repartitions_for_aggregate_after_sorted_union_projection() -> Result<()> { |
Author
There was a problem hiding this comment.
This no longer inserts the coalesce. Even after 2 runs of enforce distribution.
wiedld
commented
Feb 12, 2025
| } | ||
|
|
||
| #[tokio::test] | ||
| async fn test_preserve_needed_coalesce() -> Result<()> { |
Author
There was a problem hiding this comment.
But this bug still occurs. EnforceSorting can make an valid plan, become invalid.
wiedld
commented
Feb 13, 2025
Comment on lines
+3419
to
+3424
| // Demonstrate: plan changes are not idempotent. | ||
| // | ||
| // After the second run, the optimizer: | ||
| // * replaces the SPM with a sort, and removes the sorts pushed down below the union. | ||
| // * swaps the repartition vs SPM. | ||
| let expected_after_second_run = &[ |
wiedld
commented
Feb 13, 2025
Comment on lines
+3530
to
+3534
| // Demonstrate: plan changes are not idempotent. | ||
| // | ||
| // After the second run, the optimizer: | ||
| // * replaces the SPM with a sort. | ||
| let expected_after_second_run = &[ |
Author
There was a problem hiding this comment.
I believe this is proof that it's not idempotent? 🤔
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow up to #58, which demonstrates:
In this PR, we tried to recreate on the latest apache main branch. It was found that:
Followups:
Upstream/apache fix is going to focus on the EnforceSorting bug.
Separately, going to make some of the test setup cleanups from the EnforceDistribution learnings.