|
| 1 | +--- |
| 2 | +title: Deploy BR Federation on Kubernetes |
| 3 | +summary: Learn how to deploy BR Federation on Kubernetes. |
| 4 | +--- |
| 5 | + |
| 6 | +# Deploy BR Federation on Kubernetes |
| 7 | + |
| 8 | +This document describes how to deploy [BR Federation](br-federation-architecture.md#br-federation-architecture-and-processes) across multiple Kubernetes clusters. |
| 9 | + |
| 10 | +## Prerequisites |
| 11 | + |
| 12 | +Before deploy BR Federation on Kubernetes cluster, make sure you have met the following prerequisites: |
| 13 | + |
| 14 | +* Kubernetes version must be >= v1.24. |
| 15 | +* You must have multiple Kubernetes clusters. |
| 16 | +* You have deployed TiDB Operator for all the Kubernetes clusters that serve as data planes. |
| 17 | + |
| 18 | +## Step 1: Generate a kubeconfig file in data planes |
| 19 | + |
| 20 | +The BR Federation manages Kubernetes clusters of data planes by accessing their API servers. To authenticate and authorize itself in the API servers, BR Federation requires a kubeconfig file. The users or service accounts in the kubeconfig file need to have at least all the permissions of **backups.pingcap.com** and **restores.pingcap.com** CRD. |
| 21 | + |
| 22 | +You can get the kubeconfig file from the Kubernetes cluster administrator. However, if you have permission to access all the data planes, you can generate the kubeconfig file on your own. |
| 23 | + |
| 24 | +### Step 1.1: Create RBAC resources in data planes |
| 25 | + |
| 26 | +To enable the BR Federation to manipulate Backup and Restore CR, you need to create the following resources in every data plane. |
| 27 | + |
| 28 | +```yaml |
| 29 | +apiVersion: v1 |
| 30 | +kind: ServiceAccount |
| 31 | +metadata: |
| 32 | + name: br-federation-member |
| 33 | + namespace: tidb-admin |
| 34 | +--- |
| 35 | +apiVersion: rbac.authorization.k8s.io/v1 |
| 36 | +kind: ClusterRole |
| 37 | +metadata: |
| 38 | + name: br-federation-manager:br-federation-member |
| 39 | +rules: |
| 40 | +- apiGroups: |
| 41 | + - pingcap.com |
| 42 | + resources: |
| 43 | + - backups |
| 44 | + - restores |
| 45 | + verbs: |
| 46 | + - '*' |
| 47 | +--- |
| 48 | +apiVersion: rbac.authorization.k8s.io/v1 |
| 49 | +kind: ClusterRoleBinding |
| 50 | +metadata: |
| 51 | + name: br-federation-manager:br-federation-member |
| 52 | +roleRef: |
| 53 | + apiGroup: rbac.authorization.k8s.io |
| 54 | + kind: ClusterRole |
| 55 | + name: br-federation-manager:br-federation-member |
| 56 | +subjects: |
| 57 | +- kind: ServiceAccount |
| 58 | + name: br-federation-member |
| 59 | + namespace: tidb-admin |
| 60 | +``` |
| 61 | +
|
| 62 | +For Kubernetes >= v1.24, to let external applications access the Kubernetes API server, you need to manually create a service account secret as follows: |
| 63 | +
|
| 64 | +```yaml |
| 65 | +apiVersion: v1 |
| 66 | +kind: Secret |
| 67 | +type: kubernetes.io/service-account-token |
| 68 | +metadata: |
| 69 | + name: br-federation-member-secret |
| 70 | + namespace: tidb-admin |
| 71 | + annotations: |
| 72 | + kubernetes.io/service-account.name: "br-federation-member" |
| 73 | +``` |
| 74 | +
|
| 75 | +### Step 1.2: Generate kubeconfig files |
| 76 | +
|
| 77 | +Execute the following script for every data plane. |
| 78 | +
|
| 79 | +```shell |
| 80 | +# for Kubernetes < 1.24 |
| 81 | +export TOKEN_SECRET_NAME=$(kubectl -n tidb-admin get serviceaccount br-federation-member -o=jsonpath='{.secrets[0].name}') |
| 82 | +# for Kubernetes >= 1.24, the service account secret should be created manually as above, so you should use its name as value of TOKEN_SECRET_NAME |
| 83 | +# export TOKEN_SECRET_NAME=br-federation-member-secret |
| 84 | +export USER_TOKEN_VALUE=$(kubectl -n tidb-admin get secret/${TOKEN_SECRET_NAME} -o=go-template='{{.data.token}}' | base64 --decode) |
| 85 | +export CURRENT_CONTEXT=$(kubectl config current-context) |
| 86 | +export CURRENT_CLUSTER=$(kubectl config view --raw -o=go-template='{{range .contexts}}{{if eq .name "'''${CURRENT_CONTEXT}'''"}}{{ index .context "cluster" }}{{end}}{{end}}') |
| 87 | +export CLUSTER_CA=$(kubectl config view --raw -o=go-template='{{range .clusters}}{{if eq .name "'''${CURRENT_CLUSTER}'''"}}"{{with index .cluster "certificate-authority-data" }}{{.}}{{end}}"{{ end }}{{ end }}') |
| 88 | +export CLUSTER_SERVER=$(kubectl config view --raw -o=go-template='{{range .clusters}}{{if eq .name "'''${CURRENT_CLUSTER}'''"}}{{ .cluster.server }}{{end}}{{ end }}') |
| 89 | +# you should modify this value in different data plane |
| 90 | +export DATA_PLANE_SYMBOL="a" |
| 91 | + |
| 92 | +cat << EOF > {k8s-name}-kubeconfig |
| 93 | +apiVersion: v1 |
| 94 | +kind: Config |
| 95 | +current-context: ${DATA_PLANE_SYMBOL} |
| 96 | +contexts: |
| 97 | +- name: ${DATA_PLANE_SYMBOL} |
| 98 | + context: |
| 99 | + cluster: ${CURRENT_CLUSTER} |
| 100 | + user: br-federation-member-${DATA_PLANE_SYMBOL} |
| 101 | + namespace: kube-system |
| 102 | +clusters: |
| 103 | +- name: ${CURRENT_CLUSTER} |
| 104 | + cluster: |
| 105 | + certificate-authority-data: ${CLUSTER_CA} |
| 106 | + server: ${CLUSTER_SERVER} |
| 107 | +users: |
| 108 | +- name: br-federation-member-${DATA_PLANE_SYMBOL} |
| 109 | + user: |
| 110 | + token: ${USER_TOKEN_VALUE} |
| 111 | +EOF |
| 112 | +``` |
| 113 | + |
| 114 | +The environment variable `$DATA_PLANE_SYMBOL` represents the name of the data plane cluster. Make sure that you provide a brief and unique name. In the preceding script, you use this variable as the context name for kubeconfig. The context name will be used as `k8sClusterName` in both the `VolumeBackup` and `VolumeRestore` CR. |
| 115 | + |
| 116 | +### Step 1.3: Merge multiple kubeconfig files into one |
| 117 | + |
| 118 | +After following the previous steps to generate kubeconfig, you now have multiple kubeconfig files. You need to merge them into a single kubeconfig file. |
| 119 | + |
| 120 | +Assume that you have 3 kubeconfig files with file paths: `kubeconfig-path1`, `kubeconfig-path2`, `kubeconfig-path3`. To merge these files into one kubeconfig file with file path `data-planes-kubeconfig`, execute the following command: |
| 121 | + |
| 122 | +```shell |
| 123 | +KUBECONFIG=${kubeconfig-path1}:${kubeconfig-path2}:${kubeconfig-path3} kubectl config view --flatten > ${data-planes-kubeconfig} |
| 124 | +``` |
| 125 | + |
| 126 | +## Step 2: Deploy BR Federation in the control plane |
| 127 | + |
| 128 | +To deploy the BR Federation, you need to select one Kubernetes cluster as the control plane. The following steps **must be executed on the control plane**. |
| 129 | + |
| 130 | +### Step 2.1: Create CRD |
| 131 | + |
| 132 | +The BR Federation uses [Custom Resource Definition (CRD)](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/#customresourcedefinitions) to extend Kubernetes. Before using the BR Federation, you must create the CRD in your Kubernetes cluster. After using the BR Federation Manager, you only need to perform the operation once. |
| 133 | + |
| 134 | +```shell |
| 135 | +kubectl create -f https://raw.githubusercontent.com/pingcap/tidb-operator/{{{ .tidb_operator_version }}}/manifests/federation-crd.yaml |
| 136 | +``` |
| 137 | + |
| 138 | +### Step 2.2: Prepare the kubeconfig secret |
| 139 | + |
| 140 | +Now that you already have a kubeconfig file of data planes, you need to encode the kubeconfig file into a secret. Take the following steps: |
| 141 | + |
| 142 | +1. Encode the kubeconfig file: |
| 143 | + |
| 144 | + ```shell |
| 145 | + base64 -i ${kubeconfig-path} |
| 146 | + ``` |
| 147 | + |
| 148 | +2. Store the output from the previous step in a secret object. |
| 149 | + |
| 150 | + Note that the name of the secret and the data key of the kubeconfig field **must** match the following example: |
| 151 | + |
| 152 | + ```yaml |
| 153 | + apiVersion: v1 |
| 154 | + kind: Secret |
| 155 | + metadata: |
| 156 | + name: br-federation-kubeconfig |
| 157 | + type: Opaque |
| 158 | + data: |
| 159 | + kubeconfig: ${encoded-kubeconfig} |
| 160 | + ``` |
| 161 | + |
| 162 | +### Step 2.3: Install BR Federation |
| 163 | + |
| 164 | +This section describes how to install the BR Federation using [Helm 3](https://helm.sh/docs/intro/install/). |
| 165 | + |
| 166 | +- If you prefer to use the default configuration, follow the **Quick deployment** steps. |
| 167 | +- If you prefer to use a custom configuration, follow the **Custom deployment** steps. |
| 168 | + |
| 169 | +<SimpleTab> |
| 170 | +<div label="Quick deployment"> |
| 171 | + |
| 172 | +1. To create resources related to the BR Federation, create a namespace: |
| 173 | + |
| 174 | + ```shell |
| 175 | + kubectl create ns br-fed-admin |
| 176 | + ``` |
| 177 | + |
| 178 | +2. In the specified namespace, create a secret that contains all the encoded kubeconfig files: |
| 179 | + |
| 180 | + ```shell |
| 181 | + kubectl create -f ${secret-path} -n br-fed-admin |
| 182 | + ``` |
| 183 | + |
| 184 | +3. Add the PingCAP repository: |
| 185 | + |
| 186 | + ```shell |
| 187 | + helm repo add pingcap https://charts.pingcap.com/ |
| 188 | + ``` |
| 189 | + |
| 190 | +4. Install the BR Federation: |
| 191 | + |
| 192 | + ```shell |
| 193 | + helm install --namespace br-fed-admin br-federation pingcap/br-federation --version {{{ .tidb_operator_version }}} |
| 194 | + ``` |
| 195 | + |
| 196 | +</div> |
| 197 | +<div label="Custom deployment"> |
| 198 | + |
| 199 | +1. To create resources related to the BR Federation, create a namespace: |
| 200 | + |
| 201 | + ```shell |
| 202 | + kubectl create ns br-fed-admin |
| 203 | + ``` |
| 204 | + |
| 205 | +2. In the specified namespace, create a secret that contains all the encoded kubeconfig files: |
| 206 | + |
| 207 | + ```shell |
| 208 | + kubectl create -f ${secret-path} -n br-fed-admin |
| 209 | + ``` |
| 210 | + |
| 211 | +3. Add the PingCAP repository: |
| 212 | + |
| 213 | + ```shell |
| 214 | + helm repo add pingcap https://charts.pingcap.com/ |
| 215 | + ``` |
| 216 | + |
| 217 | +4. Get the `values.yaml` file of the desired `br-federation` chart for deployment. |
| 218 | + |
| 219 | + ```shell |
| 220 | + mkdir -p ${HOME}/br-federation && \ |
| 221 | + helm inspect values pingcap/br-federation --version={{{ .tidb_operator_version }}} > ${HOME}/br-federation/values.yaml |
| 222 | + ``` |
| 223 | + |
| 224 | +5. Configure the BR Federation by modifying fields such as `image`, `limits`, `requests`, and `replicas` according to your needs. |
| 225 | + |
| 226 | +6. Deploy the BR Federation. |
| 227 | + |
| 228 | + ```shell |
| 229 | + helm install --namespace br-fed-admin br-federation pingcap/br-federation --version {{{ .tidb_operator_version }}} -f ${HOME}/br-federation/values.yaml && \ |
| 230 | + kubectl get po -n br-fed-admin -l app.kubernetes.io/instance=br-federation |
| 231 | + ``` |
| 232 | + |
| 233 | +</div> |
| 234 | +</SimpleTab> |
| 235 | + |
| 236 | +## What's next |
| 237 | + |
| 238 | +After deploying BR Federation, you can now perform the following tasks: |
| 239 | + |
| 240 | +- [Back Up a TiDB Cluster across Multiple Kubernetes Using EBS Volume Snapshots](backup-by-ebs-snapshot-across-multiple-kubernetes.md) |
| 241 | +- [Restore a TiDB Cluster across Multiple Kubernetes from EBS Volume Snapshots](restore-from-ebs-snapshot-across-multiple-kubernetes.md) |
0 commit comments