[Feature][seatunnel core][seatunnel-flink-start] Filnk engine supports task restore from checkpoint upon startup #10232
Conversation
…s task restore from checkpoint upon startup(apache#10220)
|
@yzeng1618 Please help review this PR, thx |
| + "kubernetes-session, yarn-application, kubernetes-application]") | ||
| private MasterType masterType; | ||
|
|
||
| /** restore checkpoint path */ |
There was a problem hiding this comment.
The comment is inconsistent with the actual description of the savepoint parameter.
| @Parameter( | ||
| names = {"-s", "--fromSavepoint"}, | ||
| description = | ||
| "Path to a savepoint to restore the job from (for example, flink run -s hdfs:///flink/checkpoints/3c298a925d9a2a7837bbf5a8e4966b4f/chk-7902).") |
There was a problem hiding this comment.
The example describes a checkpoint, which is also inconsistent
|
Could we separately add descriptions for checkpoint and savepoint, and update the corresponding documentation appropriately? |
Okay, let me modify everything to checkpoint description. I copied it directly from Flink's description |
…rrock, an exception prompt 'Unknown HostException' appears
…rrock, an exception prompt 'Unknown HostException' appears
docs/zh/other-engine/flink.md
Outdated
| ### 如何从检查点恢复任务 | ||
|
|
||
| 当你需要从检查点恢复任务时，只需启动任务时指定相关的检查点目录即可。 | ||
|
|
||
| ``` | ||
| bin/start-seatunnel-flink-13-connector-v2.sh --target local --fromCheckpoint hdfs:///flink/checkpoints/3c298a925d9a2a7837bbf5a8e4966b4f/chk-7902 --deploy-mode run --config ./config/example.conf | ||
| ``` | ||
|
|
There was a problem hiding this comment.
In Flink, the -s option works for both savepoints and （externalized/retained）checkpoints. You can add a description in the documentation to indicate that recovery from both savepoints and checkpoints is supported.
|
You also need to supplement the relevant parameters in docs/zh/command/usage.mdx and docs/en/command/usage.mdx. |
|
It's best to add a unit test to lock the output of buildCommands(), verifying whether -s is included when parameters are passed or not, to prevent regression during future refactoring. |
docs/en/other-engine/flink.md
Outdated
| } | ||
| ``` | ||
|
|
||
| ### How to recover tasks from checkpoint |
There was a problem hiding this comment.
How to recover tasks from savepoint / checkpoint
| @Parameter( | ||
| names = {"-s", "--fromCheckpoint"}, | ||
| description = | ||
| "Path to a checkpoint to restore the job from (for example, flink run -s hdfs:///flink/checkpoints/3c298a925d9a2a7837bbf5a8e4966b4f/chk-7902).") |
There was a problem hiding this comment.
Path to a checkpoint/savepoint to restore the job from
@parameter( names = {"-s", "--fromSavepoint", "--fromCheckpoint"}, description = "Path to a savepoint (or an externalized checkpoint) to restore the job from (equivalent to flink run -s ).")
…s task restore from checkpoint upon startup
|
LGTM if CI passes |
3 participants
Purpose of this pull request
Filnk engine supports task restore from checkpoint upon startup, and issue is #10220
Does this PR introduce any user-facing change?
How was this patch tested?
Check list
New License Guide
incompatible-changes.mdto describe the incompatibility caused by this PR.