[core] Supports only execute specified actions during the writing process.#4884
[core] Supports only execute specified actions during the writing process.#4884LinMingQiang wants to merge 3 commits intoapache:masterfrom
Conversation
LinMingQiang
force-pushed
the
master-feat-addconf-toskip-action
branch
2 times, most recently
from
9d3fcef to
ed49eb8
Compare
|
Thanks for @LinMingQiang prepare this pr. |
xuzifu666
left a comment
xuzifu666
left a comment
There was a problem hiding this comment.
Good job and left some comments.
LinMingQiang
force-pushed
the
master-feat-addconf-toskip-action
branch
from
7250389 to
b3867f8
Compare
|
+1 |
LinMingQiang
force-pushed
the
master-feat-addconf-toskip-action
branch
from
b3867f8 to
99dcebd
Compare
LinMingQiang
force-pushed
the
master-feat-addconf-toskip-action
branch
5 times, most recently
from
2d26f9d to
fd50ccd
Compare
LinMingQiang
changed the title
LinMingQiang
changed the title
|
@JingsongLi PTAL, thanks! |
| TAG_AUTOMATIC_CREATION("tag-automatic-creation"), | ||
|
|
||
| // Actions during writing. | ||
| MINOR_COMPACT("minor-compact"), |
There was a problem hiding this comment.
It is strange here, in streaming mode, it is hard to say just doing minor compact.
There was a problem hiding this comment.
We can only keep full-compact and minor-compact action can be removed. minor-compact keep the original default logic.
| || doWriteActions.contains(WriteAction.PARTITION_EXPIRE); | ||
| } | ||
|
|
||
| public boolean doSnapshotExpireAction(Set<WriteAction> doWriteActions) { |
There was a problem hiding this comment.
All these methods can be in WriteAction, they are just static methods.
LinMingQiang
force-pushed
the
master-feat-addconf-toskip-action
branch
from
fd50ccd to
3702c60
Compare
| TAG_AUTOMATIC_CREATION("tag-automatic-creation"), | ||
|
|
||
| // Actions during writing. | ||
| FULL_COMPACT("full-compact"); |
There was a problem hiding this comment.
I cannot get why you want to add this. Full compact should be separate option.
There was a problem hiding this comment.
Should I add parameter write-action.compact.enable separately to control whether to execute compaction?
In my original conception, compact is one of the write-actions types like the others.
There was a problem hiding this comment.
Is it more reasonable to directly use "compact"?
From the implementation code, this parameter only controls the compact behavior, while whether to perform a full compact or a minor compact is controlled by other parameters.
|
Do you intend to control the behavior of expire in compact action? Based on our experience, the expire behavior is quite heavy and may cause the checkpoint to timeout during compact commit. We expect to use separate and independent compact and expire job to handle these things. |
6 participants
Purpose
There are several operations during data writing such as
partition-expire,snapshot-expire,create-tag...Currently, these actions are either executed together or not executed at all in the framework.
The purpose of this PR is to supports skipping specified actions during writing.
For example : set
'write.skip-actions'='partition-expire', the partition expiration action will be skipped during data writing, butsnapshot-expire/auto create tag/full compactwill still execute normally.Linked issue: close #xxx
Tests
API and Format
Documentation