add docs

MOBIN-F · MOBIN-F · commit ad5e18dc9238 · 2025-04-15T10:14:03.000+08:00
diff --git a/docs/content.zh/docs/connectors/pipeline-connectors/doris.md b/docs/content.zh/docs/connectors/pipeline-connectors/doris.md
@@ -46,7 +46,8 @@ sink:
 
 pipeline:
    parallelism: 1
-
+   flink:
+     execution.checkpointing.interval: 2min
 ```
 
 ## 连接器配置项
diff --git a/docs/content.zh/docs/connectors/pipeline-connectors/elasticsearch.md b/docs/content.zh/docs/connectors/pipeline-connectors/elasticsearch.md
@@ -58,6 +58,8 @@ route:
 pipeline:
   name: MySQL to Elasticsearch Pipeline
   parallelism: 2
+  flink:
+    execution.checkpointing.interval: 2min
 ```
 
 Pipeline Connector Options
diff --git a/docs/content.zh/docs/connectors/pipeline-connectors/kafka.md b/docs/content.zh/docs/connectors/pipeline-connectors/kafka.md
@@ -57,6 +57,8 @@ sink:
 pipeline:
   name: MySQL to Kafka Pipeline
   parallelism: 2
+  flink:
+    execution.checkpointing.interval: 2min
 ```
 
 Pipeline 连接器配置项
diff --git a/docs/content.zh/docs/connectors/pipeline-connectors/maxcompute.md b/docs/content.zh/docs/connectors/pipeline-connectors/maxcompute.md
@@ -63,6 +63,8 @@ sink:
 pipeline:
   name: MySQL to MaxCompute Pipeline
   parallelism: 2
+  flink:
+    execution.checkpointing.interval: 2min
 ```
 
 ## 连接器配置项
diff --git a/docs/content.zh/docs/connectors/pipeline-connectors/mysql.md b/docs/content.zh/docs/connectors/pipeline-connectors/mysql.md
@@ -75,6 +75,8 @@ sink:
 pipeline:
    name: MySQL to Doris Pipeline
    parallelism: 4
+   flink:
+     execution.checkpointing.interval: 2min
 ```
 
 ## 连接器配置项
diff --git a/docs/content.zh/docs/connectors/pipeline-connectors/oceanbase.md b/docs/content.zh/docs/connectors/pipeline-connectors/oceanbase.md
@@ -57,6 +57,8 @@ sink:
 pipeline:
   name: MySQL to OceanBase Pipeline
   parallelism: 1
+  flink:
+    execution.checkpointing.interval: 2min
 ```
 
 ## 连接器配置项
diff --git a/docs/content.zh/docs/connectors/pipeline-connectors/paimon.md b/docs/content.zh/docs/connectors/pipeline-connectors/paimon.md
@@ -58,6 +58,8 @@ sink:
 pipeline:
   name: MySQL to Paimon Pipeline
   parallelism: 2
+  flink:
+    execution.checkpointing.interval: 2min
 ```
 
 Pipeline 连接器配置项
diff --git a/docs/content.zh/docs/connectors/pipeline-connectors/starrocks.md b/docs/content.zh/docs/connectors/pipeline-connectors/starrocks.md
@@ -59,6 +59,8 @@ sink:
 pipeline:
    name: MySQL to StarRocks Pipeline
    parallelism: 2
+   flink:
+     execution.checkpointing.interval: 2min
 ```
 
 ## 连接器配置项
diff --git a/docs/content.zh/docs/core-concept/data-pipeline.md b/docs/content.zh/docs/core-concept/data-pipeline.md
@@ -60,6 +60,8 @@ under the License.
    pipeline:
      name: Sync MySQL Database to Doris
      parallelism: 2
+   flink:
+     execution.checkpointing.interval: 2min
 ```
 
 ## 包含可选部分
@@ -106,13 +108,16 @@ under the License.
          classpath: com.example.functions.AddOneFunctionClass
        - name: format
          classpath: com.example.functions.FormatFunctionClass
+   flink:
+     execution.checkpointing.interval: 2min
 ```
 
 # Pipeline 配置
 下面 是 Data Pipeline 的一些可选配置：
 
-| 参数              | 含义                                    | optional/required |
-|-----------------|---------------------------------------|-------------------|
-| name            | 这个 pipeline 的名称，会用在 Flink 集群中作为作业的名称。 | optional          |
-| parallelism     | pipeline的全局并发度，默认值是1。                 | optional          |
-| local-time-zone | 作业级别的本地时区。                            | optional          |
+| 参数              | 含义                                                                                                                                                            | optional/required |
+|-----------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------|
+| name            | 这个 pipeline 的名称，会用在 Flink 集群中作为作业的名称。                                                                                                                         | optional          |
+| parallelism     | pipeline的全局并发度，默认值是1。                                                                                                                                         | optional          |
+| local-time-zone | 作业级别的本地时区。                                                                                                                                                    | optional          |
+| flink           | 用于配置[Flink相关参数](https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/)。 <br/>Flink参数优先级：config.yaml < job command-line < pipeline.yaml | optional          |
diff --git a/docs/content.zh/docs/deployment/yarn.md b/docs/content.zh/docs/deployment/yarn.md
@@ -77,17 +77,6 @@ echo "stop" | ./bin/yarn-session.sh -id application_XXXXX_XXX
 
 启动 Yarn 会话之后，即可通过命令输出最后一行打印的 URL 或者 YARN ResourceManager Web UI 访问 Flink Web UI。
 
-然后，需要向 flink-conf.yaml 添加一些配置：
-
-```yaml
-rest.bind-port: {{REST_PORT}}
-rest.address: {{NODE_IP}}
-execution.target: yarn-session
-yarn.application.id: {{YARN_APPLICATION_ID}}
-```
-
-{{REST_PORT}} 和 {{NODE_IP}} 需要替换为 JobManager Web 接口的实际值，{{YARN_APPLICATION_ID}} 则需替换为 Flink 实际的 Yarn 应用 ID。
-
 ### 配置 Flink CDC
 从[发布页面](https://github.com/apache/flink-cdc/releases)下载 Flink CDC 的 tar 文件，然后提取该归档文件：
 
@@ -126,8 +115,16 @@ sink:
 pipeline:
  name: Sync MySQL Database to Doris
  parallelism: 2
-
+ flink:
+   rest.bind-port: {{REST_PORT}}
+   rest.address: {{NODE_IP}}
+   execution.target: yarn-session
+   yarn.application.id: {{YARN_APPLICATION_ID}}
+   execution.checkpointing.interval: 2min
+   #如果需要从savepoint恢复，则配置以下参数
+   #execution.savepoint.path: hdfs:///flink/savepoint-1537
 ```
+{{REST_PORT}} 和 {{NODE_IP}} 需要替换为 JobManager Web 接口的实际值，{{YARN_APPLICATION_ID}} 则需替换为 Flink 实际的 Yarn 应用 ID。
 
 你可以按需修改配置文件。
 最后，通过 Cli 将作业提交至 Flink Yarn Session 集群。
@@ -147,19 +144,28 @@ Job Description: Sync MySQL Database to Doris
 
 你可以通过 Flink Web UI 找到一个名为 `Sync MySQL Database to Doris` 的作业。
 
-# Yarn Application 模式
+## Yarn Application 模式
 Yarn Application 模式是 Yarn 集群上运行 Flink 作业的推荐模式。对资源的管理和分配更加灵活，可以更好地利用集群资源。
 
-通过Cli将作业提交至 Flink Yarn Application 集群。
-```bash
-cd /path/flink-cdc-*
-./bin/flink-cdc.sh -t yarn-application -Dexecution.checkpointing.interval=2min mysql-to-doris.yaml
+修改mysql-to-doris.yaml作业的运行方式为Yarn Application模式：
+```yaml
+...
+pipeline:
+ name: Sync MySQL Database to Doris
+ parallelism: 2
+ flink:
+   execution.target: yarn-application
+   execution.checkpointing.interval: 2min
+   #如果需要从savepoint恢复，则配置以下参数
+   #execution.savepoint.path: hdfs:///flink/savepoint-1537
 ```
-或者从savepoint恢复Flink-CDC作业：
+
+通过Cli将作业提交至 Flink Yarn Application 集群。
 ```bash
 cd /path/flink-cdc-*
-./bin/flink-cdc.sh -t yarn-application -s hdfs:///flink/savepoint-1537 -Dexecution.checkpointing.interval=2min mysql-to-doris.yaml
+./bin/flink-cdc.sh mysql-to-doris.yaml
 ```
+
 提交成功将返回如下信息：
 ```bash
 Pipeline has been submitted to cluster.
diff --git a/docs/content/docs/connectors/pipeline-connectors/doris.md b/docs/content/docs/connectors/pipeline-connectors/doris.md
@@ -46,7 +46,8 @@ sink:
 
 pipeline:
    parallelism: 1
-
+   flink:
+     execution.checkpointing.interval: 2min
 ```
 
 ## Connector Options
diff --git a/docs/content/docs/connectors/pipeline-connectors/elasticsearch.md b/docs/content/docs/connectors/pipeline-connectors/elasticsearch.md
@@ -58,6 +58,8 @@ route:
 pipeline:
   name: MySQL to Elasticsearch Pipeline
   parallelism: 2
+  flink:
+    execution.checkpointing.interval: 2min
 ```
 
 Pipeline Connector Options
diff --git a/docs/content/docs/connectors/pipeline-connectors/kafka.md b/docs/content/docs/connectors/pipeline-connectors/kafka.md
@@ -55,6 +55,8 @@ sink:
 pipeline:
   name: MySQL to Kafka Pipeline
   parallelism: 2
+  flink:
+    execution.checkpointing.interval: 2min
 ```
 
 Pipeline Connector Options
diff --git a/docs/content/docs/connectors/pipeline-connectors/maxcompute.md b/docs/content/docs/connectors/pipeline-connectors/maxcompute.md
@@ -63,6 +63,8 @@ sink:
 pipeline:
   name: MySQL to MaxCompute Pipeline
   parallelism: 2
+  flink:
+    execution.checkpointing.interval: 2min
 ```
 
 ## Connector Options
diff --git a/docs/content/docs/connectors/pipeline-connectors/mysql.md b/docs/content/docs/connectors/pipeline-connectors/mysql.md
@@ -76,6 +76,8 @@ sink:
 pipeline:
    name: MySQL to Doris Pipeline
    parallelism: 4
+   flink:
+     execution.checkpointing.interval: 2min
 ```
 
 ## Connector Options
diff --git a/docs/content/docs/connectors/pipeline-connectors/oceanbase.md b/docs/content/docs/connectors/pipeline-connectors/oceanbase.md
@@ -57,6 +57,8 @@ sink:
 pipeline:
   name: MySQL to OceanBase Pipeline
   parallelism: 1
+  flink:
+    execution.checkpointing.interval: 2min
 ```
 
 ## Connector Options
diff --git a/docs/content/docs/connectors/pipeline-connectors/paimon.md b/docs/content/docs/connectors/pipeline-connectors/paimon.md
@@ -58,6 +58,8 @@ sink:
 pipeline:
   name: MySQL to Paimon Pipeline
   parallelism: 2
+  flink:
+    execution.checkpointing.interval: 2min
 ```
 
 Pipeline Connector Options
diff --git a/docs/content/docs/connectors/pipeline-connectors/starrocks.md b/docs/content/docs/connectors/pipeline-connectors/starrocks.md
@@ -59,6 +59,8 @@ sink:
 pipeline:
    name: MySQL to StarRocks Pipeline
    parallelism: 2
+   flink:
+     execution.checkpointing.interval: 2min
 ```
 
 ## Connector Options
diff --git a/docs/content/docs/core-concept/data-pipeline.md b/docs/content/docs/core-concept/data-pipeline.md
@@ -129,8 +129,9 @@ We could use following yaml file to define a complicated Data Pipeline describin
 # Pipeline Configurations
 The following config options of Data Pipeline level are supported:
 
-| parameter       | meaning                                                                                 | optional/required |
-|-----------------|-----------------------------------------------------------------------------------------|-------------------|
-| name            | The name of the pipeline, which will be submitted to the Flink cluster as the job name. | optional          |
-| parallelism     | The global parallelism of the pipeline. Defaults to 1.                                  | optional          |
-| local-time-zone | The local time zone defines current session time zone id.                               | optional          |
+| parameter       | meaning                                                                                                                                                                                                   | optional/required |
+|-----------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------|
+| name            | The name of the pipeline, which will be submitted to the Flink cluster as the job name.                                                                                                                   | optional          |
+| parallelism     | The global parallelism of the pipeline. Defaults to 1.                                                                                                                                                    | optional          |
+| local-time-zone | The local time zone defines current session time zone id.                                                                                                                                                 | optional          |
+| flink           | Used to configure [Flink related parameters](https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/). <br/>Flink parameter priority: config.yaml < job command-line < pipeline.yaml | optional          |
diff --git a/docs/content/docs/deployment/yarn.md b/docs/content/docs/deployment/yarn.md
@@ -79,17 +79,6 @@ echo "stop" | ./bin/yarn-session.sh -id application_XXXXX_XXX
 
 After starting YARN session, you can now access the Flink Web UI through the URL printed in the last lines of the command output, or through the YARN ResourceManager web UI.
 
-Then, you need to add some configs to your flink-conf.yaml:
-
-```yaml
-rest.bind-port: {{REST_PORT}}
-rest.address: {{NODE_IP}}
-execution.target: yarn-session
-yarn.application.id: {{YARN_APPLICATION_ID}}
-```
-
-{{REST_PORT}} and {{NODE_IP}} should be replaced by the actual values of your JobManager Web Interface, and {{YARN_APPLICATION_ID}} should be replaced by the actual YARN application ID of Flink.
-
 ### Set up Flink CDC
 Download the tar file of Flink CDC from [release page](https://github.com/apache/flink-cdc/releases), then extract the archive:
 
@@ -128,8 +117,16 @@ sink:
 pipeline:
  name: Sync MySQL Database to Doris
  parallelism: 2
-
+ flink:
+   rest.bind-port: {{REST_PORT}}
+   rest.address: {{NODE_IP}}
+   execution.target: yarn-session
+   yarn.application.id: {{YARN_APPLICATION_ID}}
+   execution.checkpointing.interval: 2min
+   #If you need to restore from a savepoint, configure the following parameters:
+   #execution.savepoint.path: hdfs:///flink/savepoint-1537
 ```
+{{REST_PORT}} and {{NODE_IP}} should be replaced by the actual values of your JobManager Web Interface, and {{YARN_APPLICATION_ID}} should be replaced by the actual YARN application ID of Flink.
 
 You need to modify the configuration file according to your needs.
 Finally, submit job to Flink Yarn Session cluster using Cli.
@@ -149,19 +146,28 @@ Job Description: Sync MySQL Database to Doris
 
 You can find a job named `Sync MySQL Database to Doris` running through Flink Web UI.
 
-# Yarn Application Mode
+## Yarn Application Mode
 The Yarn Application mode is the recommended approach for running Flink jobs on a Yarn cluster. It offers more flexible resource management and allocation, enabling better utilization of cluster resources.
 
+Modify the running mode of the `mysql-to-doris.yaml` job to Yarn Application mode:
+```yaml
+...
+pipeline:
+ name: Sync MySQL Database to Doris
+ parallelism: 2
+ flink:
+   execution.target: yarn-application
+   execution.checkpointing.interval: 2min
+   #If you need to restore from a savepoint, configure the following parameters:
+   #execution.savepoint.path: hdfs:///flink/savepoint-1537
+```
+
 To submit a job to a Flink Yarn Application cluster using the CLI:
 ```bash
 cd /path/flink-cdc-*
-./bin/flink-cdc.sh -t yarn-application -Dexecution.checkpointing.interval=2min mysql-to-doris.yaml
+./bin/flink-cdc.sh mysql-to-doris.yaml
 ````
-Or resuming Flink-CDC job from Savepoint:
-```bash
-cd /path/flink-cdc-*
-./bin/flink-cdc.sh -t yarn-application -s hdfs:///flink/savepoint-1537 -Dexecution.checkpointing.interval=2min mysql-to-doris.yaml
-```
+
 After successful submission, the return information is as follows:
 ```bash
 Pipeline has been submitted to cluster.
