Skip to content

Commit af02ec7

Browse files
andy-stark-redisandy-stark-redis
authored
Merge pull request #817 from redis/DOC-4367-k8s-docs
DOC-4367 RDI K8s docs
2 parents 07c6810 + f83207a commit af02ec7

File tree

9 files changed

+973
-414
lines changed

9 files changed

+973
-414
lines changed

‎content/integrate/redis-data-integration/architecture.md‎

Lines changed: 21 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -136,14 +136,29 @@ traffic. The diagram below shows this configuration:
136136

137137
{{< image filename="images/rdi/ingest/ingest-active-passive-vms.png" >}}
138138

139+
See [Install on VMs]({{< relref "/integrate/redis-data-integration/installation/install-vm" >}})
140+
for more information.
141+
139142
### RDI on Kubernetes
140143

141-
You can run RDI in a namespace on your own Kubernetes cluster with
142-
a separate deployment for each of the control plane
143-
components (REST API, operator, and metrics exporter). The operator
144-
creates and configures deployments for the collector and stream processor
145-
when you start a pipeline from the CLI tool. This means that you must
146-
provide the operator with a service account that has permissions to create and manipulate pods in the namespace.
144+
You can use the RDI [Helm chart](https://helm.sh/docs/topics/charts/) to install
145+
on [Kubernetes (K8s)](https://kubernetes.io/), including Red Hat
146+
[OpenShift](https://docs.openshift.com/). This creates:
147+
148+
- A K8s [namespace](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/) named `rdi`.
149+
- [Deployments](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/) for the
150+
[RDI operator]({{< relref "/integrate/redis-data-integration/architecture#how-rdi-is-deployed" >}}),
151+
[metrics exporter]({{< relref "/integrate/redis-data-integration/observability" >}}), and API server.
152+
- A [service account](https://kubernetes.io/docs/concepts/security/service-accounts/) along with a
153+
[role](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#restrictions-on-role-creation-or-update)
154+
and [role binding](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#rolebinding-and-clusterrolebinding) for the RDI operator.
155+
- A [Configmap](https://kubernetes.io/docs/concepts/configuration/configmap/)
156+
for the different components with RDI Redis database details.
157+
- [Secrets](https://kubernetes.io/docs/concepts/configuration/secret/)
158+
with the RDI Redis database credentials and TLS certificates.
159+
160+
See [Install on Kubernetes]({{< relref "/integrate/redis-data-integration/installation/install-k8s" >}})
161+
for more information.
147162

148163
### Secrets and security considerations
149164

‎content/integrate/redis-data-integration/data-pipelines/data-pipelines.md‎

Lines changed: 3 additions & 73 deletions
Original file line numberDiff line numberDiff line change
@@ -328,81 +328,11 @@ find the preparation guides for the databases that RDI supports in the
328328
[Prepare source databases]({{< relref "/integrate/redis-data-integration/data-pipelines/prepare-dbs" >}})
329329
section.
330330

331-
## Set secrets
332-
333-
Before you deploy your pipeline, you must set the authentication secrets for the
334-
source and target databases. Each secret has a corresponding property name that
335-
you can pass to the
336-
[`redis-di set-secret`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-set-secret" >}})
337-
command to set the property's value. You can then refer to these properties
338-
in `config.yaml` using the syntax "`${PROPERTY_NAME}`"
339-
(the sample [config.yaml file](#the-configyaml-file) shows these properties in use).
340-
For example, you would use the
341-
following command line to set the source database username to `myUserName`:
342-
343-
```bash
344-
redis-di set-secret SOURCE_DB_USERNAME myUserName
345-
```
346-
347-
The table below shows the property name for each secret. Note that the
348-
username and password are required for the source and target, but the other
349-
secrets are only relevant to TLS/mTLS connections.
350-
351-
| Property name | Description |
352-
| :-- | :-- |
353-
| `SOURCE_DB_USERNAME` | Username for the source database |
354-
| `SOURCE_DB_PASSWORD` | Password for the source database |
355-
| `SOURCE_DB_CACERT` | (For TLS only) Source database trust certificate |
356-
| `SOURCE_DB_KEY` | (For mTLS only) Source database private key |
357-
| `SOURCE_DB_CERT` | (For mTLS only) Source database public key |
358-
| `SOURCE_DB_KEY_PASSWORD` | (For mTLS only) Source database private key password |
359-
| `TARGET_DB_USERNAME` | Username for the target database |
360-
| `TARGET_DB_PASSWORD` | Password for the target database |
361-
| `TARGET_DB_CACERT` | (For TLS only) Target database trust certificate |
362-
| `TARGET_DB_KEY` | (For mTLS only) Target database private key |
363-
| `TARGET_DB_CERT` | (For mTLS only) Target database public key |
364-
| `TARGET_DB_KEY_PASSWORD` | (For mTLS only) Target database private key password |
365-
366331
## Deploy a pipeline
367332

368-
If you are hosting RDI on your own VMs, you can use the
369-
[`deploy`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-deploy" >}})
370-
command to deploy a configuration, including the jobs, once you have created them.
371-
372-
If your RDI CLI is deployed as a pod in a Kubernetes cluster, you should perform the following
373-
steps to deploy a pipeline:
374-
375-
- Create a [ConfigMap](https://kubernetes.io/docs/concepts/configuration/configmap/) from the
376-
YAML files in your `jobs` folder:
377-
378-
```bash
379-
kubectl create configmap redis-di-jobs --from-file=jobs/
380-
```
381-
382-
- Deploy your jobs:
383-
384-
```bash
385-
kubectl exec -it pod/redis-di-cli -- redis-di deploy
386-
```
387-
388-
{{< note >}}When you create or modify a ConfigMap, it will be available in the `redis-di-cli` pod
389-
after a short delay. Wait around 30 seconds before running the `redis-di deploy` command.{{< /note >}}
390-
391-
You have two options to update the ConfigMap:
392-
393-
- For smaller changes, you can edit the ConfigMap directly with this command:
394-
395-
```bash
396-
kubectl edit configmap redis-di-jobs
397-
```
398-
399-
- For bigger changes, such as adding another job file, edit the files in your local `jobs` folder and then run this command:
400-
401-
```bash
402-
kubectl create configmap redis-di-jobs --from-file=jobs/ --dry-run=client -o yaml | kubectl apply -f -
403-
```
404-
405-
{{< note >}} You must run `kubectl exec -it pod/redis-di-cli -- redis-di deploy` after updating the ConfigMap with either option.{{< /note >}}
333+
When your configuration is ready, you must deploy it to start using the pipeline. See
334+
[Deploy a pipeline]({{< relref "/integrate/redis-data-integration/data-pipelines/deploy" >}})
335+
to learn how to do this.
406336

407337
## Ingest pipeline lifecycle
408338

‎content/integrate/redis-data-integration/data-pipelines/deploy.md‎

Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,67 @@
1+
---
2+
Title: Deploy a pipeline
3+
aliases: /integrate/redis-data-integration/ingest/data-pipelines/data-type-handling/
4+
alwaysopen: false
5+
categories:
6+
- docs
7+
- integrate
8+
- rs
9+
- rdi
10+
description: Learn how to deploy an RDI pipeline
11+
group: di
12+
linkTitle: Deploy
13+
summary: Redis Data Integration keeps Redis in sync with the primary database in near
14+
real time.
15+
type: integration
16+
weight: 2
17+
---
18+
19+
The sections below explain how to deploy a pipeline after you have created the required
20+
[configuration]({{< relref "/integrate/redis-data-integration/data-pipelines/data-pipelines" >}}).
21+
22+
## Set secrets
23+
24+
Before you deploy your pipeline, you must set the authentication secrets for the
25+
source and target databases. Each secret has a corresponding property name that
26+
you can pass to the
27+
[`redis-di set-secret`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-set-secret" >}})
28+
command to set the property's value. You can then refer to these properties
29+
in `config.yaml` using the syntax "`${PROPERTY_NAME}`"
30+
(the sample [config.yaml file](#the-configyaml-file) shows these properties in use).
31+
For example, you would use the
32+
following command line to set the source database username to `myUserName`:
33+
34+
```bash
35+
redis-di set-secret SOURCE_DB_USERNAME myUserName
36+
```
37+
38+
The table below shows the property name for each secret. Note that the
39+
username and password are required for the source and target, but the other
40+
secrets are only relevant to TLS/mTLS connections.
41+
42+
| Property name | Description |
43+
| :-- | :-- |
44+
| `SOURCE_DB_USERNAME` | Username for the source database |
45+
| `SOURCE_DB_PASSWORD` | Password for the source database |
46+
| `SOURCE_DB_CACERT` | (For TLS only) Source database trust certificate |
47+
| `SOURCE_DB_KEY` | (For mTLS only) Source database private key |
48+
| `SOURCE_DB_CERT` | (For mTLS only) Source database public key |
49+
| `SOURCE_DB_KEY_PASSWORD` | (For mTLS only) Source database private key password |
50+
| `TARGET_DB_USERNAME` | Username for the target database |
51+
| `TARGET_DB_PASSWORD` | Password for the target database |
52+
| `TARGET_DB_CACERT` | (For TLS only) Target database trust certificate |
53+
| `TARGET_DB_KEY` | (For mTLS only) Target database private key |
54+
| `TARGET_DB_CERT` | (For mTLS only) Target database public key |
55+
| `TARGET_DB_KEY_PASSWORD` | (For mTLS only) Target database private key password |
56+
57+
## Deploy a pipeline
58+
59+
When you have created your configuration, including the [jobs]({{< relref "/integrate/redis-data-integration/data-pipelines/data-pipelines#job-files" >}}), use the
60+
[`redis-di deploy`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-deploy" >}})
61+
command to deploy them:
62+
63+
```bash
64+
redis-di deploy --dir <path to pipeline folder>
65+
```
66+
67+
You can also deploy a pipeline using [Redis Insight]({{< relref "/develop/connect/insight/rdi-connector" >}}).

‎content/integrate/redis-data-integration/data-pipelines/prepare-dbs/my-sql-mariadb.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,7 +37,11 @@ and then run the following commands:
3737
1. Grant the required permissions to the user:
3838

3939
```sql
40+
# MySQL <v8.0
4041
mysql> GRANT SELECT, RELOAD, SHOW DATABASES, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'user' IDENTIFIED BY 'password';
42+
43+
# MySQL v8.0 and above
44+
mysql> GRANT SELECT, RELOAD, SHOW DATABASES, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'user'@'localhost';
4145
```
4246

4347
1. Finalize the user's permissions:

0 commit comments

Comments
 (0)