[flink] Support specifying what partitions to scan in Flink#5090
Merged
JingsongLi merged 2 commits intoapache:masterfrom Feb 17, 2025
tsreaper:scan-partitions
Merged
[flink] Support specifying what partitions to scan in Flink#5090JingsongLi merged 2 commits intoapache:masterfrom tsreaper:scan-partitions
JingsongLi merged 2 commits intoapache:masterfrom
tsreaper:scan-partitions
Conversation
wwj6591812
reviewed
Feb 15, 2025
| private static final String MAX_PT = "max_pt()"; | ||
| private static final String MAX_TWO_PT = "max_two_pt()"; | ||
|
|
||
| protected final FileStoreTable table; |
Contributor
There was a problem hiding this comment.
Why use FileStoreTable instead of Table?
Contributor
Author
There was a problem hiding this comment.
We need specific methods like schema() in its subclass.
JingsongLi
reviewed
Feb 17, 2025
|
|
||
| @Override | ||
| public boolean checkRefresh() { | ||
| if (partitions.isEmpty()) { |
Contributor
There was a problem hiding this comment.
Just return false;, StaticPartitionLoader should never refresh partitions.
You can add partitions in open.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Lookup joins in streaming SQL is the same as normal joins in batch SQL. However, when specifying what partitions to scan in lookup joins, currently user can specify
max_pt()through SQL hints to read the latest partition, without specifying a fixed partition. Such SQL hint is not supported in batch joins.Paimon is a streaming-batch unified lake format. To also support streaming-batch unification in SQL, this PR introduces a new option
scan.partitions, which accepts bothmax_pt()(in lookup joins) and fixed partitions (in all joins). Users only need to change the value of this option to specify different partitions for streaming and batch jobs, and they don't need to change SQL itself.Tests
Unit tests and IT cases.
API and Format
No format changes.
Documentation
Document is also added.