-
Notifications
You must be signed in to change notification settings - Fork 124
add dumpling-lightning-job #2732
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
ti-chi-bot
merged 76 commits into
pingcap:master
from
yiduoyunQ:yiduoyunQ-dumpling-lightning
Apr 24, 2025
Merged
Changes from all commits
Commits
Show all changes
76 commits
Select commit
Hold shift + click to select a range
3d02897
dumpling-lightning-job
yiduoyunQ 5498425
dumpling-lightning-job
yiduoyunQ f3bd026
dumpling-lightning-job
yiduoyunQ 3c507cd
Update zh/backup-to-blob-using-job.md
yiduoyunQ f456260
Update zh/backup-to-blob-using-job.md
yiduoyunQ d5fed6d
Update zh/backup-to-s3-using-job.md
yiduoyunQ 761eaae
Update zh/backup-to-s3-using-job.md
yiduoyunQ d4e9539
Update zh/restore-from-blob-using-job.md
yiduoyunQ bb31eb2
Update zh/restore-from-blob-using-job.md
yiduoyunQ b11dba8
Update zh/restore-from-gcs-using-job.md
yiduoyunQ 01c8428
Update zh/restore-from-s3-using-job.md
yiduoyunQ 6005953
Update zh/backup-to-blob-using-job.md
yiduoyunQ 530bfe2
Update zh/backup-to-blob-using-job.md
yiduoyunQ 382c330
Update zh/backup-to-blob-using-job.md
yiduoyunQ 9c29328
Update zh/backup-to-blob-using-job.md
yiduoyunQ 495bcb3
Update zh/backup-to-blob-using-job.md
yiduoyunQ 76c7579
Update zh/backup-to-s3-using-job.md
yiduoyunQ 4233e18
Update zh/backup-to-s3-using-job.md
yiduoyunQ d21e183
Update zh/backup-to-s3-using-job.md
yiduoyunQ ff59359
Update zh/backup-to-s3-using-job.md
yiduoyunQ 3a21b0d
Update zh/backup-to-s3-using-job.md
yiduoyunQ f626353
Update zh/restore-from-blob-using-job.md
yiduoyunQ 06465c7
Update zh/restore-from-blob-using-job.md
yiduoyunQ 5d183c2
Update zh/restore-from-blob-using-job.md
yiduoyunQ a05864c
Update zh/restore-from-blob-using-job.md
yiduoyunQ a587b1a
Update zh/restore-from-blob-using-job.md
yiduoyunQ f96ee79
Update zh/backup-to-gcs-using-job.md
yiduoyunQ ca0de1a
Update zh/backup-to-gcs-using-job.md
yiduoyunQ afd3e60
Update zh/backup-to-gcs-using-job.md
yiduoyunQ efa8952
Update zh/backup-to-gcs-using-job.md
yiduoyunQ 5175a0a
Update zh/backup-to-gcs-using-job.md
yiduoyunQ bce7121
Update zh/backup-to-s3-using-job.md
yiduoyunQ 22659d1
Update zh/restore-from-blob-using-job.md
yiduoyunQ c9f7d66
Update zh/backup-to-gcs-using-job.md
yiduoyunQ 94011c0
Update zh/restore-from-gcs-using-job.md
yiduoyunQ 0309ec7
Update zh/restore-from-gcs-using-job.md
yiduoyunQ b319cad
Update zh/restore-from-gcs-using-job.md
yiduoyunQ 996d4d3
Update zh/restore-from-gcs-using-job.md
yiduoyunQ 649d413
Update zh/restore-from-gcs-using-job.md
yiduoyunQ 8dd0d52
Update zh/restore-from-gcs-using-job.md
yiduoyunQ 07b4646
Update zh/restore-from-gcs-using-job.md
yiduoyunQ 5de789b
Update zh/restore-from-s3-using-job.md
yiduoyunQ 0629499
Update zh/restore-from-s3-using-job.md
yiduoyunQ e7450be
Update zh/restore-from-s3-using-job.md
yiduoyunQ b4dfd91
Update zh/restore-from-s3-using-job.md
yiduoyunQ 87f9f30
Update zh/restore-from-s3-using-job.md
yiduoyunQ 3a3fbb9
Update zh/restore-from-s3-using-job.md
yiduoyunQ 79b52ca
Update zh/restore-from-s3-using-job.md
yiduoyunQ 455c503
refine external storage names
hfxsd 81eec79
refine structures and wording
hfxsd fe1ec3d
Apply suggestions from code review
hfxsd d12cbfd
Update zh/backup-to-gcs.md
yiduoyunQ fca6219
Update zh/restore-from-s3.md
yiduoyunQ 7d14c44
Update zh/TOC.md
yiduoyunQ c695f3f
Update zh/backup-to-blob-using-job.md
yiduoyunQ 1dd9ac7
Update zh/backup-to-gcs-using-job.md
yiduoyunQ eba038f
Update zh/backup-to-s3-using-job.md
yiduoyunQ 6b10105
Update zh/restore-from-blob-using-job.md
yiduoyunQ 2676fa7
Update zh/restore-from-gcs-using-job.md
yiduoyunQ f4449b9
Update zh/restore-from-s3-using-job.md
yiduoyunQ 8a5019a
Update zh/restore-from-s3-using-job.md
yiduoyunQ 0193b18
Update zh/restore-from-s3-using-job.md
yiduoyunQ bb21f58
Update zh/restore-from-gcs-using-job.md
yiduoyunQ 5642d22
Update zh/backup-to-gcs-using-job.md
yiduoyunQ 0e3acd7
Update zh/restore-from-blob-using-job.md
yiduoyunQ 93a5a91
Update zh/restore-from-blob-using-job.md
yiduoyunQ ced8917
Update zh/restore-from-gcs-using-job.md
yiduoyunQ 14a0940
Update zh/restore-from-gcs-using-job.md
yiduoyunQ 5d69624
Update zh/restore-from-gcs-using-job.md
yiduoyunQ 1af9314
Update zh/backup-to-s3.md
yiduoyunQ efb2510
Update zh/backup-to-s3.md
yiduoyunQ 859f9a9
Update zh/restore-from-gcs.md
yiduoyunQ 1b0d998
Update zh/restore-from-gcs.md
yiduoyunQ 2204f4e
Update zh/TOC.md
yiduoyunQ 4762dda
Update zh/backup-to-blob-using-job.md
yiduoyunQ 0154b8a
Update zh/restore-from-s3.md
yiduoyunQ File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,115 @@ | ||
| --- | ||
| title: 使用 Dumpling 备份 TiDB 数据到 Azure Blob Storage | ||
| summary: 本文介绍如何使用 Dumpling 将 TiDB 集群数据备份到 Azure Blob Storage。 | ||
| --- | ||
|
|
||
| # 使用 Dumpling 备份 TiDB 数据到 Azure Blob Storage | ||
|
|
||
| 本文档介绍如何使用 [Dumpling](https://docs.pingcap.com/zh/tidb/stable/dumpling-overview/) 将部署在 Azure AKS 上的 TiDB 集群数据备份到 Azure Blob Storage。Dumpling 是一款数据导出工具,可将 TiDB 或 MySQL 中的数据导出为 SQL 或 CSV 格式,用于全量数据备份或导出。 | ||
|
|
||
| ## 准备 Dumpling 节点池 | ||
|
|
||
| 你可以在现有节点池中运行 Dumpling,也可以创建一个专用节点池。以下命令示例展示了如何创建一个新的节点池。使用前,请根据实际情况替换以下变量: | ||
|
|
||
| - `${clusterName}`:AKS 集群名称 | ||
| - `${resourceGroup}`:资源组名称 | ||
|
|
||
| ```shell | ||
| az aks nodepool add --name dumpling \ | ||
| --cluster-name ${clusterName} \ | ||
| --resource-group ${resourceGroup} \ | ||
| --zones 1 2 3 \ | ||
| --node-count 1 \ | ||
| --labels dedicated=dumpling | ||
| ``` | ||
|
|
||
| ## 部署 Dumpling Job | ||
|
|
||
| 本章节介绍如何配置、部署以及监控 Dumpling Job。 | ||
|
|
||
| ### 配置 Dumpling Job | ||
|
|
||
| Dumpling Job 的配置文件 (`dumpling_job.yaml`) 示例如下。使用前,请替换以下变量: | ||
|
|
||
| - `${name}`:Job 名称 | ||
| - `${namespace}`:Kubernetes 命名空间 | ||
| - `${version}`:Dumpling 镜像版本 | ||
| - Dumpling 的相关参数,请参考 [Dumpling 主要选项表](https://docs.pingcap.com/zh/tidb/stable/dumpling-overview/#dumpling-主要选项表)。 | ||
|
|
||
| ```yaml | ||
| # dumpling_job.yaml | ||
| --- | ||
| apiVersion: batch/v1 | ||
| kind: Job | ||
| metadata: | ||
| name: ${name} | ||
| namespace: ${namespace} | ||
| labels: | ||
| app.kubernetes.io/component: dumpling | ||
| spec: | ||
| template: | ||
| spec: | ||
| nodeSelector: | ||
| dedicated: dumpling | ||
| affinity: | ||
| podAntiAffinity: | ||
| requiredDuringSchedulingIgnoredDuringExecution: | ||
| - labelSelector: | ||
| matchExpressions: | ||
| - key: app.kubernetes.io/component | ||
| operator: In | ||
| values: | ||
| - dumpling | ||
| topologyKey: kubernetes.io/hostname | ||
| containers: | ||
| - name: ${name} | ||
| image: pingcap/dumpling:${version} | ||
| command: | ||
| - /bin/sh | ||
| - -c | ||
| - | | ||
| /dumpling \ | ||
| --host=basic-tidb \ | ||
| --port=4000 \ | ||
| --user=root \ | ||
| --password='' \ | ||
| --s3.region=us-west-2 \ | ||
| --threads=16 \ | ||
| --rows=20000 \ | ||
| --filesize=256MiB \ | ||
| --database=test \ | ||
| --filetype=csv \ | ||
| --output=azure://external/testfolder?account-name=${accountname}&account-key=${accountkey} | ||
| restartPolicy: Never | ||
| backoffLimit: 0 | ||
| ``` | ||
|
|
||
| ### 创建 Dumpling Job | ||
|
|
||
| 执行以下命令创建 Dumpling Job: | ||
|
|
||
| ```shell | ||
| export name=dumpling | ||
| export version=v8.5.1 | ||
| export namespace=tidb-cluster | ||
| export accountname=<your-account-name> | ||
| export accountkey=<your-account-key> | ||
|
|
||
| envsubst < dumpling_job.yaml | kubectl apply -f - | ||
| ``` | ||
|
|
||
| ### 查看 Dumpling Job 状态 | ||
|
|
||
| 运行以下命令查看 Dumpling Job 的 Pod 状态: | ||
|
|
||
| ```shell | ||
| kubectl -n ${namespace} get pod ${name} | ||
| ``` | ||
|
|
||
| ### 查看 Dumpling Job 日志 | ||
|
|
||
| 运行以下命令查看 Dumpling Job 的日志输出: | ||
|
|
||
| ```shell | ||
| kubectl -n ${namespace} logs pod ${name} | ||
| ``` |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,123 @@ | ||
| --- | ||
| title: 使用 Dumpling 备份 TiDB 数据到 Google Cloud Storage (GCS) | ||
| summary: 本文介绍如何使用 Dumpling 将 TiDB 集群数据备份到 Google Cloud Storage (GCS)。 | ||
| --- | ||
|
|
||
| # 使用 Dumpling 备份 TiDB 数据到 Google Cloud Storage (GCS) | ||
|
|
||
| 本文档介绍如何使用 [Dumpling](https://docs.pingcap.com/zh/tidb/stable/dumpling-overview/) 将部署在 Google GKE 上的 TiDB 集群数据备份到 [Google Cloud Storage (GCS)](https://cloud.google.com/storage/docs)。Dumpling 是一款数据导出工具,可将 TiDB 或 MySQL 中的数据导出为 SQL 或 CSV 格式,用于全量数据备份或导出。 | ||
|
|
||
| ## 准备 Dumpling 节点池 | ||
|
|
||
| 你可以在现有节点池中运行 Dumpling,也可以创建一个专用节点池。以下命令示例展示了如何创建一个新的节点池。请根据实际情况替换以下变量: | ||
|
|
||
| - `${clusterName}`:GKE 集群名称 | ||
|
|
||
| ```shell | ||
| gcloud container node-pools create dumpling \ | ||
| --cluster ${clusterName} \ | ||
| --machine-type n2-standard-4 \ | ||
| --num-nodes=1 \ | ||
| --node-labels=dedicated=dumpling | ||
| ``` | ||
|
|
||
| ## 部署 Dumpling Job | ||
|
|
||
| ### 创建凭证 ConfigMap | ||
|
|
||
| 将从 Google Cloud Console 下载的 `service account key` 文件保存为 `google-credentials.json`,然后使用以下命令创建 ConfigMap: | ||
|
|
||
| ```shell | ||
| kubectl -n ${namespace} create configmap google-credentials --from-file=google-credentials.json | ||
| ``` | ||
|
|
||
| ### 配置 Dumpling Job | ||
|
|
||
| Dumpling Job 的配置文件 (`dumpling_job.yaml`) 示例如下。使用前,请替换以下变量: | ||
|
|
||
| - `${name}`:Job 名称 | ||
| - `${namespace}`:Kubernetes 命名空间 | ||
| - `${version}`:Dumpling 镜像版本 | ||
| - Dumpling 的相关参数,请参考 [Dumpling 主要选项表](https://docs.pingcap.com/zh/tidb/stable/dumpling-overview/#dumpling-主要选项表)。 | ||
|
|
||
| ```yaml | ||
| # dumpling_job.yaml | ||
| --- | ||
| apiVersion: batch/v1 | ||
| kind: Job | ||
| metadata: | ||
| name: ${name} | ||
| namespace: ${namespace} | ||
| labels: | ||
| app.kubernetes.io/component: dumpling | ||
| spec: | ||
| template: | ||
| spec: | ||
| nodeSelector: | ||
| dedicated: dumpling | ||
| affinity: | ||
| podAntiAffinity: | ||
| requiredDuringSchedulingIgnoredDuringExecution: | ||
| - labelSelector: | ||
| matchExpressions: | ||
| - key: app.kubernetes.io/component | ||
| operator: In | ||
| values: | ||
| - dumpling | ||
| topologyKey: kubernetes.io/hostname | ||
| containers: | ||
| - name: ${name} | ||
| image: pingcap/dumpling:${version} | ||
| command: | ||
| - /bin/sh | ||
| - -c | ||
| - | | ||
| /dumpling \ | ||
| --host=basic-tidb \ | ||
| --port=4000 \ | ||
| --user=root \ | ||
| --password='' \ | ||
| --threads=16 \ | ||
| --rows=20000 \ | ||
| --filesize=256MiB \ | ||
| --database=test \ | ||
| --filetype=csv \ | ||
| --output=gcs://external/testfolder?credentials-file=/etc/config/google-credentials.json | ||
| volumeMounts: | ||
| - name: google-credentials | ||
| mountPath: /etc/config | ||
| volumes: | ||
| - name: google-credentials | ||
| configMap: | ||
| name: google-credentials | ||
| restartPolicy: Never | ||
| backoffLimit: 0 | ||
| ``` | ||
|
|
||
| ### 创建 Dumpling Job | ||
|
|
||
| 执行以下命令创建 Dumpling Job: | ||
|
|
||
| ```shell | ||
| export name=dumpling | ||
| export version=v8.5.1 | ||
| export namespace=tidb-cluster | ||
|
|
||
| envsubst < dumpling_job.yaml | kubectl apply -f - | ||
| ``` | ||
|
|
||
| ### 查看 Dumpling Job 状态 | ||
|
|
||
| 运行以下命令查看 Dumpling Job 的 Pod 状态: | ||
|
|
||
| ```shell | ||
| kubectl -n ${namespace} get pod ${name} | ||
| ``` | ||
|
|
||
| ### 查看 Dumpling Job 日志 | ||
|
|
||
| 运行以下命令查看 Dumpling Job 的日志输出: | ||
|
|
||
| ```shell | ||
| kubectl -n ${namespace} logs pod ${name} | ||
| ``` | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.