Skip to content
27 changes: 21 additions & 6 deletions content/integrate/redis-data-integration/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -136,14 +136,29 @@ traffic. The diagram below shows this configuration:

{{< image filename="images/rdi/ingest/ingest-active-passive-vms.png" >}}

See [Install on VMs]({{< relref "/integrate/redis-data-integration/installation/install-vm" >}})
for more information.

### RDI on Kubernetes

You can run RDI in a namespace on your own Kubernetes cluster with
a separate deployment for each of the control plane
components (REST API, operator, and metrics exporter). The operator
creates and configures deployments for the collector and stream processor
when you start a pipeline from the CLI tool. This means that you must
provide the operator with a service account that has permissions to create and manipulate pods in the namespace.
You can use the RDI [Helm chart](https://helm.sh/docs/topics/charts/) to install
on [Kubernetes (K8s)](https://kubernetes.io/), including Red Hat
[OpenShift](https://docs.openshift.com/). This creates:

- A K8s [namespace](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/) named `rdi`.
- [Deployments](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/) for the
[RDI operator]({{< relref "/integrate/redis-data-integration/architecture#how-rdi-is-deployed" >}}),
[metrics exporter]({{< relref "/integrate/redis-data-integration/observability" >}}), and API server.
- A [service account](https://kubernetes.io/docs/concepts/security/service-accounts/) along with a
[role](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#restrictions-on-role-creation-or-update)
and [role binding](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#rolebinding-and-clusterrolebinding) for the RDI operator.
- A [Configmap](https://kubernetes.io/docs/concepts/configuration/configmap/)
for the different components with RDI Redis database details.
- [Secrets](https://kubernetes.io/docs/concepts/configuration/secret/)
with the RDI Redis database credentials and TLS certificates.

See [Install on Kubernetes]({{< relref "/integrate/redis-data-integration/installation/install-k8s" >}})
for more information.

### Secrets and security considerations

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -328,81 +328,11 @@ find the preparation guides for the databases that RDI supports in the
[Prepare source databases]({{< relref "/integrate/redis-data-integration/data-pipelines/prepare-dbs" >}})
section.

## Set secrets

Before you deploy your pipeline, you must set the authentication secrets for the
source and target databases. Each secret has a corresponding property name that
you can pass to the
[`redis-di set-secret`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-set-secret" >}})
command to set the property's value. You can then refer to these properties
in `config.yaml` using the syntax "`${PROPERTY_NAME}`"
(the sample [config.yaml file](#the-configyaml-file) shows these properties in use).
For example, you would use the
following command line to set the source database username to `myUserName`:

```bash
redis-di set-secret SOURCE_DB_USERNAME myUserName
```

The table below shows the property name for each secret. Note that the
username and password are required for the source and target, but the other
secrets are only relevant to TLS/mTLS connections.

| Property name | Description |
| :-- | :-- |
| `SOURCE_DB_USERNAME` | Username for the source database |
| `SOURCE_DB_PASSWORD` | Password for the source database |
| `SOURCE_DB_CACERT` | (For TLS only) Source database trust certificate |
| `SOURCE_DB_KEY` | (For mTLS only) Source database private key |
| `SOURCE_DB_CERT` | (For mTLS only) Source database public key |
| `SOURCE_DB_KEY_PASSWORD` | (For mTLS only) Source database private key password |
| `TARGET_DB_USERNAME` | Username for the target database |
| `TARGET_DB_PASSWORD` | Password for the target database |
| `TARGET_DB_CACERT` | (For TLS only) Target database trust certificate |
| `TARGET_DB_KEY` | (For mTLS only) Target database private key |
| `TARGET_DB_CERT` | (For mTLS only) Target database public key |
| `TARGET_DB_KEY_PASSWORD` | (For mTLS only) Target database private key password |

## Deploy a pipeline

If you are hosting RDI on your own VMs, you can use the
[`deploy`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-deploy" >}})
command to deploy a configuration, including the jobs, once you have created them.

If your RDI CLI is deployed as a pod in a Kubernetes cluster, you should perform the following
steps to deploy a pipeline:

- Create a [ConfigMap](https://kubernetes.io/docs/concepts/configuration/configmap/) from the
YAML files in your `jobs` folder:

```bash
kubectl create configmap redis-di-jobs --from-file=jobs/
```

- Deploy your jobs:

```bash
kubectl exec -it pod/redis-di-cli -- redis-di deploy
```

{{< note >}}When you create or modify a ConfigMap, it will be available in the `redis-di-cli` pod
after a short delay. Wait around 30 seconds before running the `redis-di deploy` command.{{< /note >}}

You have two options to update the ConfigMap:

- For smaller changes, you can edit the ConfigMap directly with this command:

```bash
kubectl edit configmap redis-di-jobs
```

- For bigger changes, such as adding another job file, edit the files in your local `jobs` folder and then run this command:

```bash
kubectl create configmap redis-di-jobs --from-file=jobs/ --dry-run=client -o yaml | kubectl apply -f -
```

{{< note >}} You must run `kubectl exec -it pod/redis-di-cli -- redis-di deploy` after updating the ConfigMap with either option.{{< /note >}}
When your configuration is ready, you must deploy it to start using the pipeline. See
[Deploy a pipeline]({{< relref "/integrate/redis-data-integration/data-pipelines/deploy" >}})
to learn how to do this.

## Ingest pipeline lifecycle

Expand Down
67 changes: 67 additions & 0 deletions content/integrate/redis-data-integration/data-pipelines/deploy.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
---
Title: Deploy a pipeline
aliases: /integrate/redis-data-integration/ingest/data-pipelines/data-type-handling/
alwaysopen: false
categories:
- docs
- integrate
- rs
- rdi
description: Learn how to deploy an RDI pipeline
group: di
linkTitle: Deploy
summary: Redis Data Integration keeps Redis in sync with the primary database in near
real time.
type: integration
weight: 2
---

The sections below explain how to deploy a pipeline after you have created the required
[configuration]({{< relref "/integrate/redis-data-integration/data-pipelines/data-pipelines" >}}).

## Set secrets

Before you deploy your pipeline, you must set the authentication secrets for the
source and target databases. Each secret has a corresponding property name that
you can pass to the
[`redis-di set-secret`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-set-secret" >}})
command to set the property's value. You can then refer to these properties
in `config.yaml` using the syntax "`${PROPERTY_NAME}`"
(the sample [config.yaml file](#the-configyaml-file) shows these properties in use).
For example, you would use the
following command line to set the source database username to `myUserName`:

```bash
redis-di set-secret SOURCE_DB_USERNAME myUserName
```

The table below shows the property name for each secret. Note that the
username and password are required for the source and target, but the other
secrets are only relevant to TLS/mTLS connections.

| Property name | Description |
| :-- | :-- |
| `SOURCE_DB_USERNAME` | Username for the source database |
| `SOURCE_DB_PASSWORD` | Password for the source database |
| `SOURCE_DB_CACERT` | (For TLS only) Source database trust certificate |
| `SOURCE_DB_KEY` | (For mTLS only) Source database private key |
| `SOURCE_DB_CERT` | (For mTLS only) Source database public key |
| `SOURCE_DB_KEY_PASSWORD` | (For mTLS only) Source database private key password |
| `TARGET_DB_USERNAME` | Username for the target database |
| `TARGET_DB_PASSWORD` | Password for the target database |
| `TARGET_DB_CACERT` | (For TLS only) Target database trust certificate |
| `TARGET_DB_KEY` | (For mTLS only) Target database private key |
| `TARGET_DB_CERT` | (For mTLS only) Target database public key |
| `TARGET_DB_KEY_PASSWORD` | (For mTLS only) Target database private key password |

## Deploy a pipeline

When you have created your configuration, including the [jobs]({{< relref "/integrate/redis-data-integration/data-pipelines/data-pipelines#job-files" >}}), use the
[`redis-di deploy`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-deploy" >}})
command to deploy them:

```bash
redis-di deploy --dir <path to pipeline folder>
```

You can also deploy a pipeline using [Redis Insight]({{< relref "/develop/connect/insight/rdi-connector" >}}).
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,11 @@ and then run the following commands:
1. Grant the required permissions to the user:

```sql
# MySQL <v8.0
mysql> GRANT SELECT, RELOAD, SHOW DATABASES, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'user' IDENTIFIED BY 'password';
# MySQL v8.0 and above
mysql> GRANT SELECT, RELOAD, SHOW DATABASES, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'user'@'localhost';
```

1. Finalize the user's permissions:
Expand Down
Loading
Loading