diff --git a/content/integrate/redis-data-integration/architecture.md b/content/integrate/redis-data-integration/architecture.md index f00c12a61d..2b558ee5d3 100644 --- a/content/integrate/redis-data-integration/architecture.md +++ b/content/integrate/redis-data-integration/architecture.md @@ -136,14 +136,29 @@ traffic. The diagram below shows this configuration: {{< image filename="images/rdi/ingest/ingest-active-passive-vms.png" >}} +See [Install on VMs]({{< relref "/integrate/redis-data-integration/installation/install-vm" >}}) +for more information. + ### RDI on Kubernetes -You can run RDI in a namespace on your own Kubernetes cluster with -a separate deployment for each of the control plane -components (REST API, operator, and metrics exporter). The operator -creates and configures deployments for the collector and stream processor -when you start a pipeline from the CLI tool. This means that you must -provide the operator with a service account that has permissions to create and manipulate pods in the namespace. +You can use the RDI [Helm chart](https://helm.sh/docs/topics/charts/) to install +on [Kubernetes (K8s)](https://kubernetes.io/), including Red Hat +[OpenShift](https://docs.openshift.com/). This creates: + +- A K8s [namespace](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/) named `rdi`. +- [Deployments](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/) for the + [RDI operator]({{< relref "/integrate/redis-data-integration/architecture#how-rdi-is-deployed" >}}), + [metrics exporter]({{< relref "/integrate/redis-data-integration/observability" >}}), and API server. +- A [service account](https://kubernetes.io/docs/concepts/security/service-accounts/) along with a + [role](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#restrictions-on-role-creation-or-update) + and [role binding](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#rolebinding-and-clusterrolebinding) for the RDI operator. +- A [Configmap](https://kubernetes.io/docs/concepts/configuration/configmap/) + for the different components with RDI Redis database details. +- [Secrets](https://kubernetes.io/docs/concepts/configuration/secret/) + with the RDI Redis database credentials and TLS certificates. + +See [Install on Kubernetes]({{< relref "/integrate/redis-data-integration/installation/install-k8s" >}}) +for more information. ### Secrets and security considerations diff --git a/content/integrate/redis-data-integration/data-pipelines/data-pipelines.md b/content/integrate/redis-data-integration/data-pipelines/data-pipelines.md index 71c3fab461..d43f98c39f 100644 --- a/content/integrate/redis-data-integration/data-pipelines/data-pipelines.md +++ b/content/integrate/redis-data-integration/data-pipelines/data-pipelines.md @@ -328,81 +328,11 @@ find the preparation guides for the databases that RDI supports in the [Prepare source databases]({{< relref "/integrate/redis-data-integration/data-pipelines/prepare-dbs" >}}) section. -## Set secrets - -Before you deploy your pipeline, you must set the authentication secrets for the -source and target databases. Each secret has a corresponding property name that -you can pass to the -[`redis-di set-secret`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-set-secret" >}}) -command to set the property's value. You can then refer to these properties -in `config.yaml` using the syntax "`${PROPERTY_NAME}`" -(the sample [config.yaml file](#the-configyaml-file) shows these properties in use). -For example, you would use the -following command line to set the source database username to `myUserName`: - -```bash -redis-di set-secret SOURCE_DB_USERNAME myUserName -``` - -The table below shows the property name for each secret. Note that the -username and password are required for the source and target, but the other -secrets are only relevant to TLS/mTLS connections. - -| Property name | Description | -| :-- | :-- | -| `SOURCE_DB_USERNAME` | Username for the source database | -| `SOURCE_DB_PASSWORD` | Password for the source database | -| `SOURCE_DB_CACERT` | (For TLS only) Source database trust certificate | -| `SOURCE_DB_KEY` | (For mTLS only) Source database private key | -| `SOURCE_DB_CERT` | (For mTLS only) Source database public key | -| `SOURCE_DB_KEY_PASSWORD` | (For mTLS only) Source database private key password | -| `TARGET_DB_USERNAME` | Username for the target database | -| `TARGET_DB_PASSWORD` | Password for the target database | -| `TARGET_DB_CACERT` | (For TLS only) Target database trust certificate | -| `TARGET_DB_KEY` | (For mTLS only) Target database private key | -| `TARGET_DB_CERT` | (For mTLS only) Target database public key | -| `TARGET_DB_KEY_PASSWORD` | (For mTLS only) Target database private key password | - ## Deploy a pipeline -If you are hosting RDI on your own VMs, you can use the -[`deploy`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-deploy" >}}) -command to deploy a configuration, including the jobs, once you have created them. - -If your RDI CLI is deployed as a pod in a Kubernetes cluster, you should perform the following -steps to deploy a pipeline: - -- Create a [ConfigMap](https://kubernetes.io/docs/concepts/configuration/configmap/) from the - YAML files in your `jobs` folder: - - ```bash - kubectl create configmap redis-di-jobs --from-file=jobs/ - ``` - -- Deploy your jobs: - - ```bash - kubectl exec -it pod/redis-di-cli -- redis-di deploy - ``` - -{{< note >}}When you create or modify a ConfigMap, it will be available in the `redis-di-cli` pod -after a short delay. Wait around 30 seconds before running the `redis-di deploy` command.{{< /note >}} - -You have two options to update the ConfigMap: - -- For smaller changes, you can edit the ConfigMap directly with this command: - - ```bash - kubectl edit configmap redis-di-jobs - ``` - -- For bigger changes, such as adding another job file, edit the files in your local `jobs` folder and then run this command: - - ```bash - kubectl create configmap redis-di-jobs --from-file=jobs/ --dry-run=client -o yaml | kubectl apply -f - - ``` - -{{< note >}} You must run `kubectl exec -it pod/redis-di-cli -- redis-di deploy` after updating the ConfigMap with either option.{{< /note >}} +When your configuration is ready, you must deploy it to start using the pipeline. See +[Deploy a pipeline]({{< relref "/integrate/redis-data-integration/data-pipelines/deploy" >}}) +to learn how to do this. ## Ingest pipeline lifecycle diff --git a/content/integrate/redis-data-integration/data-pipelines/deploy.md b/content/integrate/redis-data-integration/data-pipelines/deploy.md new file mode 100644 index 0000000000..438f15ad56 --- /dev/null +++ b/content/integrate/redis-data-integration/data-pipelines/deploy.md @@ -0,0 +1,67 @@ +--- +Title: Deploy a pipeline +aliases: /integrate/redis-data-integration/ingest/data-pipelines/data-type-handling/ +alwaysopen: false +categories: +- docs +- integrate +- rs +- rdi +description: Learn how to deploy an RDI pipeline +group: di +linkTitle: Deploy +summary: Redis Data Integration keeps Redis in sync with the primary database in near + real time. +type: integration +weight: 2 +--- + +The sections below explain how to deploy a pipeline after you have created the required +[configuration]({{< relref "/integrate/redis-data-integration/data-pipelines/data-pipelines" >}}). + +## Set secrets + +Before you deploy your pipeline, you must set the authentication secrets for the +source and target databases. Each secret has a corresponding property name that +you can pass to the +[`redis-di set-secret`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-set-secret" >}}) +command to set the property's value. You can then refer to these properties +in `config.yaml` using the syntax "`${PROPERTY_NAME}`" +(the sample [config.yaml file](#the-configyaml-file) shows these properties in use). +For example, you would use the +following command line to set the source database username to `myUserName`: + +```bash +redis-di set-secret SOURCE_DB_USERNAME myUserName +``` + +The table below shows the property name for each secret. Note that the +username and password are required for the source and target, but the other +secrets are only relevant to TLS/mTLS connections. + +| Property name | Description | +| :-- | :-- | +| `SOURCE_DB_USERNAME` | Username for the source database | +| `SOURCE_DB_PASSWORD` | Password for the source database | +| `SOURCE_DB_CACERT` | (For TLS only) Source database trust certificate | +| `SOURCE_DB_KEY` | (For mTLS only) Source database private key | +| `SOURCE_DB_CERT` | (For mTLS only) Source database public key | +| `SOURCE_DB_KEY_PASSWORD` | (For mTLS only) Source database private key password | +| `TARGET_DB_USERNAME` | Username for the target database | +| `TARGET_DB_PASSWORD` | Password for the target database | +| `TARGET_DB_CACERT` | (For TLS only) Target database trust certificate | +| `TARGET_DB_KEY` | (For mTLS only) Target database private key | +| `TARGET_DB_CERT` | (For mTLS only) Target database public key | +| `TARGET_DB_KEY_PASSWORD` | (For mTLS only) Target database private key password | + +## Deploy a pipeline + +When you have created your configuration, including the [jobs]({{< relref "/integrate/redis-data-integration/data-pipelines/data-pipelines#job-files" >}}), use the +[`redis-di deploy`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-deploy" >}}) +command to deploy them: + +```bash +redis-di deploy --dir +``` + +You can also deploy a pipeline using [Redis Insight]({{< relref "/develop/connect/insight/rdi-connector" >}}). diff --git a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/my-sql-mariadb.md b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/my-sql-mariadb.md index e4d38f16f1..2b7c097cc7 100644 --- a/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/my-sql-mariadb.md +++ b/content/integrate/redis-data-integration/data-pipelines/prepare-dbs/my-sql-mariadb.md @@ -37,7 +37,11 @@ and then run the following commands: 1. Grant the required permissions to the user: ```sql + # MySQL GRANT SELECT, RELOAD, SHOW DATABASES, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'user' IDENTIFIED BY 'password'; + + # MySQL v8.0 and above + mysql> GRANT SELECT, RELOAD, SHOW DATABASES, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'user'@'localhost'; ``` 1. Finalize the user's permissions: diff --git a/content/integrate/redis-data-integration/installation/_index.md b/content/integrate/redis-data-integration/installation/_index.md index 5655f16f6e..e5753809f0 100644 --- a/content/integrate/redis-data-integration/installation/_index.md +++ b/content/integrate/redis-data-integration/installation/_index.md @@ -17,337 +17,7 @@ type: integration weight: 2 --- -This guide explains how to install Redis Data Integration (RDI) and integrate it with -your source database. - -{{< note >}}We recommend you use RDI v1.2.8 version instead of v1.2.7. -RDI v1.2.7 had a CoreDNS issue. In some installations, the host DNS could not -resolve hostnames for the RDI components. This meant it couldn't connect to the RDI database -and resume the installation. This problem is fixed in v1.2.8. -{{< /note >}} - -## Install RDI on VMs - -You would normally install RDI on two VMs for high availability (HA) but you can also install -one just one VM if you don't need this. For example, you might not need HA during -development and testing. - -{{< note >}}You can't install RDI on a host where a Redis Enterprise cluster -is also installed, due to incompatible network rules. If you want to install RDI on a -host that you have previously used for Redis Enterprise then you must -use [`iptables`](https://www.netfilter.org/projects/iptables/index.html) to -"clean" the host before installation with the following command line: - -```bash - sudo iptables-save | awk '/^[*]/ { print $1 } - /^:[A-Z]+ [^-]/ { print $1 " ACCEPT" ; } - /COMMIT/ { print $0; }' | sudo iptables-restore -``` - -You may encounter problems if you use `iptables` v1.6.1 and earlier in -`nftables` mode. Use `iptables` versions later than v1.6.1 or enable the `iptables` -legacy mode with the following commands: - -```bash -update-alternatives --set iptables /usr/sbin/iptables-legacy -update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy -``` - -Also, `iptables` versions 1.8.0-1.8.4 have known issues that can prevent RDI -from working, especially on RHEL 8. Ideally, use `iptables` v1.8.8, which is -known to work correctly with RDI. -{{< /note >}} - -The supported OS versions for RDI are: - -- RHEL 8 & 9 -- Ubuntu 18.04 & 20.04 - -You must run the RDI installer as a privileged user because it installs -[containerd](https://containerd.io/) and registers services. However, you don't -need any special privileges to run RDI processes for normal operation. - -RDI has a few -requirements for cloud VMs that you must implement before running the -RDI installer, or else installation will fail. The following sections -give full pre-installation instructions for [RHEL](#firewall-rhel) and -[Ubuntu](#firewall-ubuntu). - -### RHEL {#firewall-rhel} - -We recommend you turn off -[`firewalld`](https://firewalld.org/documentation/) -before installation using the command: - -```bash -systemctl disable firewalld --now -``` - -However, if you do need to use `firewalld`, you must add the following rules: - -```bash -firewall-cmd --permanent --add-port=6443/tcp #apiserver -firewall-cmd --permanent --zone=trusted --add-source=10.42.0.0/16 #pods -firewall-cmd --permanent --zone=trusted --add-source=10.43.0.0/16 #services -firewall-cmd --reload -``` - -You should also add [port rules](https://firewalld.org/documentation/howto/open-a-port-or-service.html) -for all the [RDI services]({{< relref "/integrate/redis-data-integration/reference/ports" >}}) -you intend to use: - -```bash -firewall-cmd --permanent --add-port=8080/tcp # (Required) rdi-operator/rdi-api -firewall-cmd --permanent --add-port=9090/tcp # vm-dis-reloader -firewall-cmd --permanent --add-port=9092/tcp # prometheus-service -firewall-cmd --permanent --add-port=9121/tcp # rdi-metric-exporter -``` - -{{}}You may also need to add similar rules to open other ports if your setup requires them. -{{}} - -If you have `nm-cloud-setup.service` enabled, you must disable it and reboot the -node with the following commands: - -```bash -systemctl disable nm-cloud-setup.service nm-cloud-setup.timer -reboot -``` - -### Ubuntu {#firewall-ubuntu} - -We recommend you turn off -[Uncomplicated Firewall](https://wiki.ubuntu.com/UncomplicatedFirewall) (`ufw`) -before installation with the command: - -```bash -ufw disable -``` - -However, if you do need to use `ufw`, you must add the following rules: - -```bash -ufw allow 6443/tcp #apiserver -ufw allow from 10.42.0.0/16 to any #pods -ufw allow from 10.43.0.0/16 to any #services -``` - -You should also add [port rules](https://ubuntu.com/server/docs/firewalls) -for all the [RDI services]({{< relref "/integrate/redis-data-integration/reference/ports" >}}) -you intend to use: - -```bash -ufw allow 8080/tcp # (Required) rdi-operator/rdi-api -ufw allow 9090/tcp # vm-dis-reloader -ufw allow 9092/tcp # prometheus-service -ufw allow 9121/tcp # rdi-metric-exporter -``` - -{{}}You may also need to add similar rules to open other ports if your setup requires them. -{{}} - -## Hardware sizing - -RDI is mainly CPU and network bound. -Each of the RDI VMs should have: - -- CPU: A minimum of 4 CPU cores. You should consider adding - 2-6 extra cores on top of this if your dataset is big and you want to ingest the - baseline snapshot as fast as possible. -- RAM: 2GB -- Disk: 25GB of disk (this includes the OS footprint) -- 10GB or more network interface - -## Installation steps - -Follow the steps below for each of your VMs: - -1. Download the RDI installer from the [Redis download center](https://cloud.redis.io/#/rlec-downloads) - (under the *Modules, Tools & Integration* dropdown) - and extract it to your preferred installation folder. - -1. Go to the installation folder: - - ```bash - cd rdi_install/$RDI_VERSION - ``` - -1. Run the installer as a privileged user: - - ```bash - sudo ./install.sh - ``` -RDI uses a database on your Redis Enterprise cluster to store its state -information. *This requires Redis Enterprise v6.4 or greater*. - -The installer gives you instructions to help you create secrets and create your pipeline. -It will ask you for cluster admin credentials during installation. You should supply -these if you want the installer to create the RDI database for you. - -{{}}The installer does not create the RDI Redis database with -[TLS](https://en.wikipedia.org/wiki/Transport_Layer_Security)/ -[mTLS](https://en.wikipedia.org/wiki/Mutual_authentication#mTLS). -If you want to use TLS or other advanced options then you must create the Redis database -yourself using the Redis Enterprise console.{{}} - -If you don’t want the installation to create the RDI database for you: - -- Use the Redis console to create a database with 250MB RAM with 1 primary and 1 replica. -- If you are deploying RDI for a production environment then secure this database with a password - and TLS. -- Provide the installation with the required RDI database details. - -{{< note >}}If you specify `localhost` as the address of the RDI database server during -installation then the connection will fail if the actual IP address changes for the local -VM. For this reason, we recommend that you don't use `localhost` for the address. However, -if you do encounter this problem, you can fix it using the following commands on the VM -that is running RDI itself: - -```bash -sudo k3s kubectl delete nodes --all -sudo service k3s restart -``` -{{< /note >}} - -After the installation is finished, RDI is ready for use. - -## "Silent" installation - -You can use the -[installer script](#installation-steps) or the -[`redis-di install`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-install" >}}) -command with the `--file` option (or the shorter version `-f`) to supply answers -to the installer's questions automatically using properties from a -[TOML](https://toml.io/en/) file: - -```bash -./install.sh --file silent.toml -``` - -### Silent install example - -The following TOML file example shows the properties for a typical -silent install configuration: - -```toml -title = "RDI Silent Installer Config" - -scaffold = true -deploy_directory = "/opt/rdi/config" - -# If you are *not* using an existing RDI database and you want -# the installer to create one then remove the properties in this -# section, apart from : -# - `password` -# - `use_existing_rdi` - set this to `false` -# Also, uncomment the [rdi.cluster] section below. -[rdi.database] -host = "localhost" -port = 12001 -username = "username" -password = "password" -use_existing_rdi = true -ssl = true - -# Uncomment this section and remove properties from the -# [rdi.database] section as described above if you -# are *not* using an existing RDI database and you want -# the installer to create one. -# [rdi.cluster] -# host = "localhost" -# port = 9443 -# username = "username" -# password = "password" - - -# Uncomment the properties in this section only if the RDI -# database uses TLS/mTLS. -# [rdi.database.certificates] -# ca = "/home/ubuntu/rdi/certs/ca.crt" -# cert = "/home/ubuntu/rdi/certs/client.crt" -# key = "/home/ubuntu/rdi/certs/client.key" -# passphrase = "foobar" -``` - -The sections below describe the properties in more detail. - -### Silent install properties - -#### Root - -| Property | Description | -|-- |-- | -| `title` | Text to identify the file. RDI doesn't use use this, so you can use any text you like. | -| `high_availability` | Do you want to enable replication on the RDI database (true/false)? You should only use this if you ask the installer to create the RDI database for you. | -| `scaffold` | Do you want to enable [scaffolding]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-scaffold" >}}) during the install? (true/false) | -| `db_index` | Integer to specify the source database type for scaffolding. The options are 2 (MySQL/MariaDB), 3 (Oracle), 4 (PostgreSQL), and 5 (SQL Server). | -| `deploy_directory` | Path to the directory where you want to store the RDI configuration. | - -#### `rdi.database` - -Use the properties in this section if you want to use an existing RDI database. -See [`rdi.cluster`](#rdicluster) below if you want the installer to create a new -RDI database. However, you should still supply the `password` in this -section and set `use_existing_rdi` to `false` if the installer creates the -database. - -| Property | Description | -|-- |-- | -| `host` | Hostname for the Redis database to store RDI state. | -| `port` | Port for the RDI database. | -| `username` | Username for the RDI database. | -| `password` | Password for the RDI database. | -| `use_existing_rdi` | Do you want to use an existing RDI instance (true) or create a new one (false)? If you enable SSL (see the property below), this will be set to true, overriding the value you specify here. | -| `ssl` | Is SSL enabled for the RDI database (true/false)? If this is false then RDI will ignore the settings in the [`rdi.database.certificates`](#rdidatabasecertificates) section. | - -#### `rdi.cluster` - -Use the properties in this section if you are *not* using an existing RDI database -and you want the installer to create one. -See [`rdi.database`](#rdidatabase) above if you want to use an existing RDI database. - -| Property | Description | -|-- |-- | -| `host` | Hostname of the Redis cluster to use for RDI. | -| `port` | Port for the cluster. | -| `username` | Username for the cluster. | -| `password` | Password for the cluster. | - -#### `rdi.database.certificates` - -Use these properties only if the RDI database requires -[TLS](https://en.wikipedia.org/wiki/Transport_Layer_Security) or -[mTLS](https://en.wikipedia.org/wiki/Mutual_authentication#mTLS). -You must also set `ssl` to `true` in the -[`rdi.database`](#rdidatabase) section to enable these properties. - -| Property | Description | -|-- |-- | -| `ca` | Path to the CA certificate file. | -| `cert` | Path to the client certificate file. | -| `key` | Path to the key file. | -| `passphrase` | Password for the private key (string). | - -## Prepare your source database - -You must also configure your source database to use the Debezium connector for CDC. See the +The guides in this section explain the options you have for installing RDI. +Before you use RDI, you must also configure your source database to use the CDC connector. See the [Prepare source databases]({{< relref "/integrate/redis-data-integration/data-pipelines/prepare-dbs" >}}) -section to learn how to do this. - -## Uninstall RDI - -If you want to remove your RDI installation, go to the installation folder and run -the uninstall script as a privileged user: - -```bash -sudo ./uninstall.sh -``` - -The script will check you are sure before you proceed: - -``` -This will uninstall RDI and its dependencies, are you sure? [y, N] -``` - -If you type anything other than "y" here, the script will abort without making any changes -to RDI or your source database. +section to learn how to do this. \ No newline at end of file diff --git a/content/integrate/redis-data-integration/installation/install-k8s.md b/content/integrate/redis-data-integration/installation/install-k8s.md new file mode 100644 index 0000000000..00b3207ae0 --- /dev/null +++ b/content/integrate/redis-data-integration/installation/install-k8s.md @@ -0,0 +1,469 @@ +--- +Title: Install on Kubernetes +alwaysopen: false +categories: +- docs +- integrate +- rs +- rdi +description: Learn how to install RDI on Kubernetes +group: di +hideListLinks: false +linkTitle: Install on K8s +summary: Redis Data Integration keeps Redis in sync with the primary database in near + real time. +type: integration +weight: 2 +--- + +This guide explains how to use the RDI [Helm chart](https://helm.sh/docs/topics/charts/) +to install on [Kubernetes](https://kubernetes.io/) (K8s). You can also +[Install RDI on VMs]({{< relref "/integrate/redis-data-integration/installation/install-vm" >}}). + +The installation creates the following K8s objects: + +- A K8s [namespace](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/) named `rdi`. +- [Deployments](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/) for the + [RDI operator]({{< relref "/integrate/redis-data-integration/architecture#how-rdi-is-deployed" >}}), + [metrics exporter]({{< relref "/integrate/redis-data-integration/observability" >}}), and API server. +- A [service account](https://kubernetes.io/docs/concepts/security/service-accounts/) along with a + [role](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#restrictions-on-role-creation-or-update) + and [role binding](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#rolebinding-and-clusterrolebinding) for the RDI operator. +- A [Configmap](https://kubernetes.io/docs/concepts/configuration/configmap/) + for the different components with RDI Redis database details. +- [Secrets](https://kubernetes.io/docs/concepts/configuration/secret/) + with the RDI Redis database credentials and TLS certificates. + +You can use this installation on [OpenShift](https://docs.openshift.com/) and other K8s distributions +including cloud providers' K8s managed clusters. + +You can pull the RDI images from +[Docker Hub](https://hub.docker.com/) or from your own [private image registry](#using-a-private-image-registry). + +## Before you install + +Complete the following steps before running Helm: + +- [Create the RDI database](#create-the-rdi-database) on your Redis Enterprise cluster. +- Create a [user]({{< relref "/operate/rs/security/access-control/create-users" >}}) + for the RDI database if you prefer not to use the default password (see + [Access control]({{< relref "/operate/rs/security/access-control" >}}) for + more information). +- Download the RDI helm chart tar file from the [download center](https://cloud.redis.io/#rlec-downloads). +- If you want to use a private image registry, + [prepare it with the RDI images](#using-a-private-image-registry). + +### Create the RDI database + +RDI uses a database on your Redis Enterprise cluster to store its state +information. *This requires Redis Enterprise v6.4 or greater*. + +- Use the Redis console to create a database with 250MB RAM with one primary and one replica. +- If you are deploying RDI for a production environment then secure this database with a password + and [TLS](https://en.wikipedia.org/wiki/Transport_Layer_Security). + +You should then provide the details of this database in the [`values.yaml`](#the-valuesyaml-file) +file as described below. + +### Using a private image registry + +Add the RDI images from [Docker Hub](https://hub.docker.com/) to your local registry. +The example below shows how to specify the registry and image pull secret in the +[`values.yaml`](#the-valuesyaml-file) file for the Helm chart: + +```yaml +global: + imagePullSecrets: [] + # - name: "image-pull-secret" + + image: + registry: docker.io + repository: redis +``` + +To pull images from a local registry, you must provide the image pull secret and in some cases also set the permissions. Follow the links below to learn how to use a private registry with: + +- [Rancher](https://ranchermanager.docs.rancher.com/how-to-guides/new-user-guides/kubernetes-resources-setup/kubernetes-and-docker-registries#using-a-private-registry) +- [OpenShift](https://docs.openshift.com/container-platform/4.17/openshift_images/managing_images/using-image-pull-secrets.html) +- [Amazon Elastic Kubernetes Service (EKS)](https://docs.aws.amazon.com/AmazonECR/latest/userguide/ECR_on_EKS.html) +- [Google Kubernetes Engine (GKE)](https://cloud.google.com/artifact-registry/docs/pull-cached-dockerhub-images) +- [Azure Kubernetes Service (AKS)](https://learn.microsoft.com/en-us/azure/aks/cluster-container-registry-integration?tabs=azure-cli) + +## Install the RDI Helm chart + +1. Decompress the tar file: + + ```bash + tar -xvf rdi-k8s-.tar.gz + ``` + +1. Open the `values.yaml` file and set the appropriate values for your installation + (see [The `values.yaml` file](#the-valuesyaml-file) below for the full set of + available values). + +1. Start the installation: + + ```bash + helm install rdi-k8s//helm --create-namespace -n rdi + ``` + +### The `values.yaml` file + +The annotated [`values.yaml`](https://helm.sh/docs/topics/charts/#templates-and-values) +file below describes the values you can set for the RDI Helm installation: + +```yaml +# Default RDI values in YAML format. +# Variables to template configuration. + +global: + # Set this property when using a private image repository. + # Provide an array of image pull secrets. + # Example: + # imagePullSecrets: + # - name: pullSecret1 + # - name: pullSecret2 + imagePullSecrets: + - name: docker-config-jfrog + + # DO NOT modify this value. + vmMode: false + + # Indicates whether the deployment is intended for an OpenShift environment. + openShift: false + + image: + # Overrides the image tag for all RDI components. + tag: 0.0.0 + + # If using a private repository, update the default values accordingly. + # Docker registry. + registry: docker.io + + # Docker image repository. + repository: redis + + # Configuration for the RDI ConfigMap. + rdiSysConfig: + # Log level for all RDI components. Valid options: DEBUG, INFO, ERROR. + # If specific component log levels are not set, this value will be used. + RDI_LOG_LEVEL: INFO + + # Log level for the RDI API. Valid options: DEBUG, INFO, ERROR. + # If not set, RDI_LOG_LEVEL will be used. + # RDI_LOG_LEVEL_API: INFO + + # Log level for the RDI Operator. Valid options: DEBUG, INFO, ERROR. + # If not set, RDI_LOG_LEVEL will be used. + # RDI_LOG_LEVEL_OPERATOR: INFO + + # Log level for the RDI processor. Valid options: DEBUG, INFO, ERROR. + # If not set, RDI_LOG_LEVEL will be used. + # RDI_LOG_LEVEL_PROCESSOR: INFO + + # Specifies whether the RDI is configured to use TLS. + RDI_REDIS_SSL: false + + # RDI_IMAGE_REPO: redis + + # This value must be set to the same tag as global.image.tag. + # RDI_IMAGE_TAG: "" + + # If using a private repository, set this value to the same secret name as in global.imagePullSecrets. + # RDI_IMAGE_PULL_SECRET: [] + + # The service IP of the RDI database. + # RDI_REDIS_HOST: "" + + # The port for the RDI database. + # RDI_REDIS_PORT: "" + + # Enable authentication for the RDI API. + # RDI_API_AUTH_ENABLED: "1" + + # Specifies whether the API Collector should be deployed. + # RDI_API_COLLECTOR_ENABLED: "0" + + # Configuration for the RDI Secret. + rdiSysSecret: + # Username and password for RDI database. + # If using the default password, keep the username as an empty string. + # RDI_REDIS_USERNAME: "" + # RDI_REDIS_PASSWORD: "" + + # Uncomment this property when using a TLS connection from RDI to its Redis database. + # DO NOT modify this value. + # RDI_REDIS_CACERT: /etc/certificates/rdi_db/cacert + + # Uncomment these properties when using a TLS connection from RDI to its Redis database. + # DO NOT modify these values. + # RDI_REDIS_CERT: /etc/certificates/rdi_db/cert + # RDI_REDIS_KEY: /etc/certificates/rdi_db/key + + # The passphrase used to get the private key stored in the secret store when using mTLS. + # RDI_REDIS_KEY_PASSPHRASE: "" + + # The key used to encrypt the JWT token used by RDI API. + # JWT_SECRET_KEY: "" + + rdiDbSSLSecret: + # Set to `true` when using a TLS connection from RDI to its Redis database. + enabled: false + + # The content of the CA certificate PEM file. + # Uncomment and set this property when using a TLS connection from RDI to its Redis database. + # cacert: "" + + # The content of the certificate PEM file. + # Uncomment and set this property when using a TLS connection from RDI to its Redis database. + # cert: "" + + # The content of the private key PEM file. + # Uncomment and set this property when using a TLS connection from RDI to its Redis database. + # key: "" + + # Container default security context. + # ref: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#set-the-security-context-for-a-container + securityContext: + runAsNonRoot: true + runAsUser: 1000 + runAsGroup: 1000 + allowPrivilegeEscalation: false + +# Set `isOpenshift` to `true` if deploying on OpenShift. +reloader: + reloader: + isOpenshift: false + deployment: + containerSecurityContext: + allowPrivilegeEscalation: false + capabilities: + drop: + - ALL + securityContext: + runAsUser: null + +# Configuration of the RDI Operator. +operator: + image: + name: rdi-operator + + # Specify an imagePullPolicy. + # ref: https://kubernetes.io/docs/concepts/containers/images/#pre-pulled-images + pullPolicy: IfNotPresent + + # Extra optional options for liveness probe. + # ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/ + liveness: + failureThreshold: 6 + periodSeconds: 10 + + # Extra optional options for readiness probe. + # ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/ + readiness: + failureThreshold: 6 + periodSeconds: 30 + + # Extra optional options for startup probe. + # ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/ + startup: + failureThreshold: 60 + periodSeconds: 5 + +fluentd: + image: + name: rdi-fluentd + + # Specify an imagePullPolicy. + # ref: https://kubernetes.io/docs/concepts/containers/images/#pre-pulled-images + pullPolicy: IfNotPresent + + rdiLogsHostPath: "/opt/rdi/logs" + podLogsHostPath: "/var/log/pods" + logrotateMinutes: "5" + +rdiMetricsExporter: + image: + name: rdi-monitor + + # Specify an imagePullPolicy. + # ref: https://kubernetes.io/docs/concepts/containers/images/#pre-pulled-images + pullPolicy: IfNotPresent + + # The RDI metrics service is set to ClusterIP, allowing access only from within the cluster. + # ref: http://kubernetes.io/docs/user-guide/services/ + service: + protocol: TCP + port: 9121 + targetPort: 9121 + type: ClusterIP + + # Configure extra options for liveness probe. + # ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/#configure-probes + liveness: + failureThreshold: 6 + periodSeconds: 10 + + # Configure extra options for readiness probe. + # ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/#configure-probes + readiness: + failureThreshold: 6 + periodSeconds: 30 + + # Configure extra options for startupProbe. + # ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/#configure-probes + startup: + failureThreshold: 60 + periodSeconds: 5 + + # Configuration for the ServiceMonitor, which is used to scrape metrics from the RDI metrics service. + serviceMonitor: + # Set to `true` to activate the ServiceMonitor. + enabled: false + + # The endpoint from which Prometheus will scrape metrics. + path: /metrics + +# Configuration of the RDI API. +apiServer: + image: + name: rdi-api + + # Specify an imagePullPolicy. + # ref: https://kubernetes.io/docs/concepts/containers/images/#pre-pulled-images + pullPolicy: IfNotPresent + + # The RDI API service is set to ClusterIP, allowing access only from within the cluster. + # ref: http://kubernetes.io/docs/user-guide/services/ + service: + type: ClusterIP + name: rdi-api + port: 8080 + targetPort: 8081 + + # Configure extra options for liveness probe. + # ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/#configure-probes + liveness: + failureThreshold: 6 + periodSeconds: 10 + + # Configure extra options for readiness probe. + # ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/#configure-probes + readiness: + failureThreshold: 6 + periodSeconds: 30 + + # Configure extra options for startupProbe. + # ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/#configure-probes + startup: + failureThreshold: 60 + periodSeconds: 5 + +# Expose the RDI API service to be accessed from outside the cluster. +# ref: http://kubernetes.io/docs/user-guide/services/ +ingress: + # When `enabled` is set to `true`, RDI API Ingress will be created. + enabled: false + + # ref: https://kubernetes.io/blog/2020/04/02/improvements-to-the-ingress-api-in-kubernetes-1.18/#specifying-the-class-of-an-ingress + # className: "" + + # Ingress resources configure routes based on the requested host. + # The optional Host field defines the hosts for routing. If omitted, it matches all hosts. + # Example: + # hosts: + # - example.com + # - another-example.com + + # Annotations to be added to the IngressClass resource. + # Example: + # annotations: + # kubernetes.io/ingress.class: "nginx" + # nginx.ingress.kubernetes.io/rewrite-target: / + + tls: + # Specifies whether the Ingress should be configured to use TLS. + enabled: false + + # When `enabled` is set to `true`, set this property to the content of the crt file. + # crt: "" + + # When `enabled` is set to `true`, set this property to the content of the key file. + # key: "" + +# When `openShift` is set to `true`, Route will be created automatically. +# Route exposes RDI API outside the cluster. +route: + tls: + # Specifies whether the Route should be configured to use TLS. + enabled: false + + # When `enabled` is set to `true`, set this property to the content of the crt file. + # crt: "" + + # When `enabled` is set to `true`, set this property to the content of the key file. + # key: "" + +collectorSourceMetricsExporter: + # The collector-source metrics service is set to ClusterIP, allowing access only from within the cluster. + # ref: http://kubernetes.io/docs/user-guide/services/ + service: + type: ClusterIP + port: 9092 + targetPort: 19000 + + # Configuration for the ServiceMonitor, which is used to scrape metrics from the collector-source metrics service. + serviceMonitor: + # Set to `true` to activate the ServiceMonitor. + enabled: false + + # The endpoint from which Prometheus will scrape metrics. + path: /metrics +``` + +## Check the installation + +To verify the status of the K8s deployment, run the following command: + +```bash +helm list -n monitoring -n rdi +``` + +The output looks like the following. Check that `` is listed. + +``` +NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION + rdi 1 2024-10-10 16:53... +0300 IDT deployed rdi-1.0.0 +``` + + +Also, check that the following pods have `Running` status: + +```bash +sudo k3s kubectl get pod -n rdi + +NAME READY STATUS RESTARTS AGE +rdi-api- 1/1 Running 0 29m +rdi-metric-l 1/1 Running 0 29m +rdi-operator- 1/1 Running 0 29m +-reloader- 1/1 Running 0 29m +collector-api- 1/1 Running 0 29m +``` + +You can verify that the RDI API works by adding the server in +[Redis Insight]({{< relref "/develop/connect/insight/rdi-connector" >}}). + +## Prepare your source database + +You must also configure your source database to use the CDC connector. See the +[Prepare source databases]({{< relref "/integrate/redis-data-integration/data-pipelines/prepare-dbs" >}}) +section to learn how to do this. + +## Deploy a pipeline + +When the Helm installation is complete, and you have prepared the source database for CDC, +you are ready to start using RDI. See the guides to +[configuring]({{< relref "/integrate/redis-data-integration/data-pipelines/data-pipelines" >}}) and +[deploying]({{< relref "/integrate/redis-data-integration/data-pipelines/deploy" >}}) +RDI pipelines for more information. You can also configure and deploy a pipeline +using [Redis Insight]({{< relref "/develop/connect/insight/rdi-connector" >}}). diff --git a/content/integrate/redis-data-integration/installation/install-vm.md b/content/integrate/redis-data-integration/installation/install-vm.md new file mode 100644 index 0000000000..43a23e9185 --- /dev/null +++ b/content/integrate/redis-data-integration/installation/install-vm.md @@ -0,0 +1,360 @@ +--- +Title: Install on VMs +alwaysopen: false +categories: +- docs +- integrate +- rs +- rdi +description: Learn how to install RDI on one or more VMs +group: di +hideListLinks: false +linkTitle: Install on VMs +summary: Redis Data Integration keeps Redis in sync with the primary database in near + real time. +type: integration +weight: 1 +--- + +This guide explains how to install Redis Data Integration (RDI) on one or more VMs and integrate it with +your source database. You can also +[Install RDI on Kubernetes]({{< relref "/integrate/redis-data-integration/installation/install-k8s" >}}). + +{{< note >}}We recommend you use RDI v1.4.0. The previous version, RDI v1.2.8, +will not work on VMs where IPv6 is disabled. This problem is solved with version 1.4.0. +{{< /note >}} + +## Install RDI on VMs + +You would normally install RDI on two VMs for high availability (HA) but you can also install +one just one VM if you don't need this. For example, you might not need HA during +development and testing. + +{{< note >}}You can't install RDI on a host where a Redis Enterprise cluster +is also installed, due to incompatible network rules. If you want to install RDI on a +host that you have previously used for Redis Enterprise then you must +use [`iptables`](https://www.netfilter.org/projects/iptables/index.html) to +"clean" the host before installation with the following command line: + +```bash + sudo iptables-save | awk '/^[*]/ { print $1 } + /^:[A-Z]+ [^-]/ { print $1 " ACCEPT" ; } + /COMMIT/ { print $0; }' | sudo iptables-restore +``` + +You may encounter problems if you use `iptables` v1.6.1 and earlier in +`nftables` mode. Use `iptables` versions later than v1.6.1 or enable the `iptables` +legacy mode with the following commands: + +```bash +update-alternatives --set iptables /usr/sbin/iptables-legacy +update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy +``` + +Also, `iptables` versions 1.8.0-1.8.4 have known issues that can prevent RDI +from working, especially on RHEL 8. Ideally, use `iptables` v1.8.8, which is +known to work correctly with RDI. +{{< /note >}} + +The supported OS versions for RDI are: + +- RHEL 8 & 9 +- Ubuntu 18.04 & 20.04 + +You must run the RDI installer as a privileged user because it installs +[containerd](https://containerd.io/) and registers services. However, you don't +need any special privileges to run RDI processes for normal operation. + +RDI has a few +requirements for cloud VMs that you must implement before running the +RDI installer, or else installation will fail. The following sections +give full pre-installation instructions for [RHEL](#firewall-rhel) and +[Ubuntu](#firewall-ubuntu). + +### RHEL {#firewall-rhel} + +We recommend you turn off +[`firewalld`](https://firewalld.org/documentation/) +before installation using the command: + +```bash +systemctl disable firewalld --now +``` + +However, if you do need to use `firewalld`, you must add the following rules: + +```bash +firewall-cmd --permanent --add-port=6443/tcp #apiserver +firewall-cmd --permanent --zone=trusted --add-source=10.42.0.0/16 #pods +firewall-cmd --permanent --zone=trusted --add-source=10.43.0.0/16 #services +firewall-cmd --reload +``` + +You should also add [port rules](https://firewalld.org/documentation/howto/open-a-port-or-service.html) +for all the [RDI services]({{< relref "/integrate/redis-data-integration/reference/ports" >}}) +you intend to use: + +```bash +firewall-cmd --permanent --add-port=8080/tcp # (Required) rdi-operator/rdi-api +firewall-cmd --permanent --add-port=9090/tcp # vm-dis-reloader +firewall-cmd --permanent --add-port=9092/tcp # prometheus-service +firewall-cmd --permanent --add-port=9121/tcp # rdi-metric-exporter +``` + +{{}}You may also need to add similar rules to open other ports if your setup requires them. +{{}} + +If you have `nm-cloud-setup.service` enabled, you must disable it and reboot the +node with the following commands: + +```bash +systemctl disable nm-cloud-setup.service nm-cloud-setup.timer +reboot +``` + +### Ubuntu {#firewall-ubuntu} + +We recommend you turn off +[Uncomplicated Firewall](https://wiki.ubuntu.com/UncomplicatedFirewall) (`ufw`) +before installation with the command: + +```bash +ufw disable +``` + +However, if you do need to use `ufw`, you must add the following rules: + +```bash +ufw allow 6443/tcp #apiserver +ufw allow from 10.42.0.0/16 to any #pods +ufw allow from 10.43.0.0/16 to any #services +``` + +You should also add [port rules](https://ubuntu.com/server/docs/firewalls) +for all the [RDI services]({{< relref "/integrate/redis-data-integration/reference/ports" >}}) +you intend to use: + +```bash +ufw allow 8080/tcp # (Required) rdi-operator/rdi-api +ufw allow 9090/tcp # vm-dis-reloader +ufw allow 9092/tcp # prometheus-service +ufw allow 9121/tcp # rdi-metric-exporter +``` + +{{}}You may also need to add similar rules to open other ports if your setup requires them. +{{}} + +## Hardware sizing + +RDI is mainly CPU and network bound. +Each of the RDI VMs should have: + +- CPU: A minimum of 4 CPU cores. You should consider adding + 2-6 extra cores on top of this if your dataset is big and you want to ingest the + baseline snapshot as fast as possible. +- RAM: 2GB +- Disk: 25GB of disk (this includes the OS footprint) +- 10GB or more network interface + +## Installation steps + +Follow the steps below for each of your VMs: + +1. Download the RDI installer from the [Redis download center](https://cloud.redis.io/#/rlec-downloads) + (under the *Modules, Tools & Integration* dropdown) + and extract it to your preferred installation folder. + +1. Go to the installation folder: + + ```bash + cd rdi_install/$RDI_VERSION + ``` + +1. Run the installer as a privileged user: + + ```bash + sudo ./install.sh + ``` +RDI uses a database on your Redis Enterprise cluster to store its state +information. *This requires Redis Enterprise v6.4 or greater*. + +The installer gives you instructions to help you create secrets and create your pipeline. +It will ask you for cluster admin credentials during installation. You should supply +these if you want the installer to create the RDI database for you. + +{{}}The installer does not create the RDI Redis database with +[TLS](https://en.wikipedia.org/wiki/Transport_Layer_Security)/ +[mTLS](https://en.wikipedia.org/wiki/Mutual_authentication#mTLS). +If you want to use TLS or other advanced options then you must create the Redis database +yourself using the Redis Enterprise console.{{}} + +If you don’t want the installation to create the RDI database for you: + +- Use the Redis console to create a database with 250MB RAM with 1 primary and 1 replica. +- If you are deploying RDI for a production environment then secure this database with a password + and TLS. +- Provide the installation with the required RDI database details. + +{{< note >}}If you specify `localhost` as the address of the RDI database server during +installation then the connection will fail if the actual IP address changes for the local +VM. For this reason, we recommend that you don't use `localhost` for the address. However, +if you do encounter this problem, you can fix it using the following commands on the VM +that is running RDI itself: + +```bash +sudo k3s kubectl delete nodes --all +sudo service k3s restart +``` +{{< /note >}} + +After the installation is finished, RDI is ready for use. + +## "Silent" installation + +You can use the +[installer script](#installation-steps) or the +[`redis-di install`]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-install" >}}) +command with the `--file` option (or the shorter version `-f`) to supply answers +to the installer's questions automatically using properties from a +[TOML](https://toml.io/en/) file: + +```bash +./install.sh --file silent.toml +``` + +### Silent install example + +The following TOML file example shows the properties for a typical +silent install configuration: + +```toml +title = "RDI Silent Installer Config" + +scaffold = true +deploy_directory = "/opt/rdi/config" + +# If you are *not* using an existing RDI database and you want +# the installer to create one then remove the properties in this +# section, apart from : +# - `password` +# - `use_existing_rdi` - set this to `false` +# Also, uncomment the [rdi.cluster] section below. +[rdi.database] +host = "localhost" +port = 12001 +username = "username" +password = "password" +use_existing_rdi = true +ssl = true + +# Uncomment this section and remove properties from the +# [rdi.database] section as described above if you +# are *not* using an existing RDI database and you want +# the installer to create one. +# [rdi.cluster] +# host = "localhost" +# port = 9443 +# username = "username" +# password = "password" + + +# Uncomment the properties in this section only if the RDI +# database uses TLS/mTLS. +# [rdi.database.certificates] +# ca = "/home/ubuntu/rdi/certs/ca.crt" +# cert = "/home/ubuntu/rdi/certs/client.crt" +# key = "/home/ubuntu/rdi/certs/client.key" +# passphrase = "foobar" +``` + +The sections below describe the properties in more detail. + +### Silent install properties + +#### Root + +| Property | Description | +|-- |-- | +| `title` | Text to identify the file. RDI doesn't use use this, so you can use any text you like. | +| `high_availability` | Do you want to enable replication on the RDI database (true/false)? You should only use this if you ask the installer to create the RDI database for you. | +| `scaffold` | Do you want to enable [scaffolding]({{< relref "/integrate/redis-data-integration/reference/cli/redis-di-scaffold" >}}) during the install? (true/false) | +| `db_index` | Integer to specify the source database type for scaffolding. The options are 2 (MySQL/MariaDB), 3 (Oracle), 4 (PostgreSQL), and 5 (SQL Server). | +| `deploy_directory` | Path to the directory where you want to store the RDI configuration. | + +#### `rdi.database` + +Use the properties in this section if you want to use an existing RDI database. +See [`rdi.cluster`](#rdicluster) below if you want the installer to create a new +RDI database. However, you should still supply the `password` in this +section and set `use_existing_rdi` to `false` if the installer creates the +database. + +| Property | Description | +|-- |-- | +| `host` | Hostname for the Redis database to store RDI state. | +| `port` | Port for the RDI database. | +| `username` | Username for the RDI database. | +| `password` | Password for the RDI database. | +| `use_existing_rdi` | Do you want to use an existing RDI instance (true) or create a new one (false)? If you enable SSL (see the property below), this will be set to true, overriding the value you specify here. | +| `ssl` | Is SSL enabled for the RDI database (true/false)? If this is false then RDI will ignore the settings in the [`rdi.database.certificates`](#rdidatabasecertificates) section. | + +#### `rdi.cluster` + +Use the properties in this section if you are *not* using an existing RDI database +and you want the installer to create one. +See [`rdi.database`](#rdidatabase) above if you want to use an existing RDI database. + +| Property | Description | +|-- |-- | +| `host` | Hostname of the Redis cluster to use for RDI. | +| `port` | Port for the cluster. | +| `username` | Username for the cluster. | +| `password` | Password for the cluster. | + +#### `rdi.database.certificates` + +Use these properties only if the RDI database requires +[TLS](https://en.wikipedia.org/wiki/Transport_Layer_Security) or +[mTLS](https://en.wikipedia.org/wiki/Mutual_authentication#mTLS). +You must also set `ssl` to `true` in the +[`rdi.database`](#rdidatabase) section to enable these properties. + +| Property | Description | +|-- |-- | +| `ca` | Path to the CA certificate file. | +| `cert` | Path to the client certificate file. | +| `key` | Path to the key file. | +| `passphrase` | Password for the private key (string). | + +## Prepare your source database + +You must also configure your source database to use the CDC connector. See the +[Prepare source databases]({{< relref "/integrate/redis-data-integration/data-pipelines/prepare-dbs" >}}) +section to learn how to do this. + +## Deploy a pipeline + +When the installation is complete, and you have prepared the source database for CDC, +you are ready to start using RDI. See the guides to +[configuring]({{< relref "/integrate/redis-data-integration/data-pipelines/data-pipelines" >}}) and +[deploying]({{< relref "/integrate/redis-data-integration/data-pipelines/deploy" >}}) +RDI pipelines for more information. You can also configure and deploy a pipeline +using [Redis Insight]({{< relref "/develop/connect/insight/rdi-connector" >}}). + +## Uninstall RDI + +If you want to remove your RDI installation, go to the installation folder and run +the uninstall script as a privileged user: + +```bash +sudo ./uninstall.sh +``` + +The script will check you are sure before you proceed: + +``` +This will uninstall RDI and its dependencies, are you sure? [y, N] +``` + +If you type anything other than "y" here, the script will abort without making any changes +to RDI or your source database. diff --git a/content/integrate/redis-data-integration/quick-start-guide.md b/content/integrate/redis-data-integration/quick-start-guide.md index 4a8dc89400..df7d4b1108 100644 --- a/content/integrate/redis-data-integration/quick-start-guide.md +++ b/content/integrate/redis-data-integration/quick-start-guide.md @@ -44,7 +44,7 @@ and is already set up for the RDI collector to use. ### Install RDI Install RDI using the instructions in the -[installation guide]({{< relref "/integrate/redis-data-integration/installation" >}}) +[VM installation guide]({{< relref "/integrate/redis-data-integration/installation/install-vm" >}}). RDI will create the pipeline template for your chosen source database type at `/opt/rdi/config`. You will need this pathname later when you prepare the pipeline for deployment @@ -112,7 +112,8 @@ contexts. ### Deploy the pipeline -You can use Redis Insight to deploy the pipeline by adding a connection to the RDI API +You can use [Redis Insight]({{< relref "/develop/connect/insight/rdi-connector" >}}) +to deploy the pipeline by adding a connection to the RDI API endpoint (which has the same IP address as your RDI VM and uses port 8083) and then clicking the **Deploy** button. You can also deploy it with the following command: ```bash @@ -127,6 +128,9 @@ validates your pipeline and then deploys it if the configuration is correct. Once the pipeline is running, you can use Redis Insight to view the data flow using the pipeline metrics. You can also connect to your target database to see the keys that RDI has written there. +See [Deploy a pipeline]({{< relref "/integrate/redis-data-integration/data-pipelines/deploy" >}}) +for more information about deployment settings. + ### View RDI's response to data changes Once the pipeline has loaded a *snapshot* of all the existing data from the source, diff --git a/content/integrate/redis-data-integration/release-notes/rdi-1-4-0.md b/content/integrate/redis-data-integration/release-notes/rdi-1-4-0.md new file mode 100644 index 0000000000..87d7fbe3a4 --- /dev/null +++ b/content/integrate/redis-data-integration/release-notes/rdi-1-4-0.md @@ -0,0 +1,40 @@ +--- +Title: Redis Data Integration release notes 1.4.0 (October 2024) +alwaysopen: false +categories: +- docs +- operate +- rs +description: Installation on Kubernetes with a Helm chart. Improvements for installation on VMs. +linkTitle: 1.4.0 (October 2024) +toc: 'true' +weight: 997 +--- + +> This is a GA version of Redis Data Integration (RDI) that improves the installation of RDI. + +RDI’s mission is to help Redis customers sync Redis Enterprise with live data from their slow disk-based databases to: + +- Meet the required speed and scale of read queries and provide an excellent and predictable user experience. +- Save resources and time when building pipelines and coding data transformations. +- Reduce the total cost of ownership by saving money on expensive database read replicas. + +RDI keeps the Redis cache up to date with changes in the primary database, using a [_Change Data Capture (CDC)_](https://en.wikipedia.org/wiki/Change_data_capture) mechanism. +It also lets you _transform_ the data from relational tables into convenient and fast data structures that match your app's requirements. You specify the transformations using a configuration system, so no coding is required. + +## Headlines + +- Installation on [Kubernetes]({{< relref "/integrate/redis-data-integration/installation/install-k8s" >}}) using a [Helm chart](https://helm.sh/docs/). You can install on [OpenShift](https://docs.openshift.com/) or other flavours of K8s using Helm. + +- Improvements for installation on VMs: + - Installer checks if the OS firewall is enabled on Ubuntu and RHEL. + - Installer verifies DNS resolution from RDI components. + - Installer provides log lines from components that failed during RDI deployment if a problem occurs. + - Improved verification of RDI installation. + - Installer verifies if the RDI database is in use by another instance of RDI. + - Installer checks and warns if any [`iptables`](https://www.netfilter.org/projects/iptables/index.html) rules are set. + - Improved message when RDI tries to connect to its Redis database with invalid TLS keys. + +## Limitations + +RDI can write data to a Redis Active-Active database. However, it doesn't support writing data to two or more Active-Active replicas. Writing data from RDI to several Active-Active replicas could easily harm data integrity as RDI is not synchronous with the source database commits.