[core] Add versioned split serializer#8482
Open
JingsongLi wants to merge 2 commits into
Open
Conversation
JingsongLi
force-pushed
the
codex/split-serializer-protocol
branch
from
541fdce to
d26fd7c
Compare
leaves12138
approved these changes
Jul 6, 2026
leaves12138
left a comment
leaves12138
left a comment
Contributor
There was a problem hiding this comment.
Looks good to me. The versioned split serializer protocol now has stable type ids and golden-file coverage, and the targeted paimon-core tests pass locally.
| serialize(new DataOutputViewStreamWrapper(out)); | ||
| } | ||
|
|
||
| private void readObject(ObjectInputStream in) throws IOException, ClassNotFoundException { |
Contributor
There was a problem hiding this comment.
This breaks Java serialization compatibility for QueryAuthSplit. Old Flink checkpoints use the previous default object stream, but readObject now expects the new SplitSerializer format. I reproduced restore as EOFException.
Contributor
Author
There was a problem hiding this comment.
This is expected. Firstly, cross version compatibility is not guaranteed, and secondly, QueryAuthSplit is generally not saved in Flink state.
However, I should indeed increase the serial Version UID to make the exception more apparent.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Introduce a versioned binary serializer for non-system table
Splitimplementations so split metadata can be exchanged by Java and Rust runtimes through a stable protocol.Changes
SplitSerializerwith aSPLIT_V1frame, serializer version, and stable type ids forDataSplit,IncrementalSplit,IndexedSplit,ChainSplit,QueryAuthSplit, and fallback split variants.QueryAuthSplitandFallbackSplitImplto use custom serialization paths instead of Java default object serialization.Testing
mvn -pl paimon-core -Pfast-build -DgenerateSplitGoldenFiles=true -Dtest=SplitSerializerTest,QueryAuthSplitTest -Dmaven.main.skip=true testmvn -pl paimon-core -Pfast-build -Dtest=SplitSerializerTest,QueryAuthSplitTest -Dmaven.main.skip=true testmvn -pl paimon-core spotless:checkgit diff --checkjavaccompile for the changed main classesNotes
FormatDataSplitand object table splits are intentionally excluded from this protocol.