diff --git a/content/integrate/redis-data-integration/data-pipelines/transform-examples/redis-add-field-example.md b/content/integrate/redis-data-integration/data-pipelines/transform-examples/redis-add-field-example.md new file mode 100644 index 0000000000..5c7101bea1 --- /dev/null +++ b/content/integrate/redis-data-integration/data-pipelines/transform-examples/redis-add-field-example.md @@ -0,0 +1,176 @@ +--- +Title: Add new fields to a key +alwaysopen: false +categories: +- docs +- integrate +- rs +- rdi +description: null +group: di +linkTitle: Add new fields +summary: Redis Data Integration keeps Redis in sync with a primary database in near + real time. +type: integration +weight: 40 +--- + +By default, RDI adds fields to +[hash]({{< relref "/develop/data-types/hashes" >}}) or +[JSON]({{< relref "/develop/data-types/json" >}}) objects in the target +database that match the columns of the source table. +The examples below show how to add extra fields to the target data with the +[`add_field`]({{< relref "/integrate/redis-data-integration/reference/data-transformation/add_field" >}}) transformation. + +## Add a single field + +The first example adds a single field to the data. +The `source` section selects the `customer` table of the +[`chinook`](https://github.com/Redislabs-Solution-Architects/rdi-quickstart-postgres) +database (the optional `db` value here corresponds to the +`sources..connection.database` value defined in +[`config.yaml`]({{< relref "/integrate/redis-data-integration/data-pipelines/data-pipelines#the-configyaml-file" >}})). + +In the `transform` section, the `add_field` transformation adds an extra field called `localphone` +to the object, which is created by removing the country and area code from the `phone` +field with the +[JMESPath]({{< relref "/integrate/redis-data-integration/reference/jmespath-custom-functions" >}}) function `regex_replace()`. +You can also specify `sql` as the `language` if you prefer to create the new +field with an [SQL](https://en.wikipedia.org/wiki/SQL) expression. + +The `output` section specifies `hash` as the `data_type` to write to the target, which +overrides the default setting of `target_data_type` defined in `config.yaml`. Also, the +`output.with.key` section specifies a custom key format of the form `cust:` where +the `id` part is generated by the `uuid()` function. + +The full example is shown below: + +```yaml +source: + db: chinook + table: customer +transform: + - uses: add_field + with: + expression: regex_replace(phone, '\+[0-9]+ (\([0-9]+\) )?', '') + field: localphone + language: jmespath +output: + - uses: redis.write + with: + connection: target + data_type: hash + key: + expression: concat(['cust:', uuid()]) + language: jmespath +``` + +If you queried the generated target data from the default transformation +using [`redis-cli`]({{< relref "/develop/connect/cli" >}}), you would +see something like the following: + +``` + 1) "customerid" + 2) "27" + 3) "firstname" + 4) "Patrick" + 5) "lastname" + 6) "Gray" +. +. +17) "phone" +18) "+1 (520) 622-4200" +. +. +``` + +Using the job file above, the data also includes the new `localphone` field: + +``` + 1) "customerid" + 2) "27" + 3) "firstname" + 4) "Patrick" + 5) "lastname" + 6) "Gray" + . + . +23) "localphone" +24) "622-4200" +``` + +## Add multiple fields + +The `add_field` transformation can also add multiple fields at the same time +if you specify them under a `fields` subsection. The example below adds two +fields to the `track` objects. The first new field, `seconds`, is created using a SQL +expression to calculate the duration of the track in seconds from the +`milliseconds` field. +The second new field, `composerlist`, adds a JSON array using the `split()` function +to split the `composer` string field wherever it contains a comma. + +```yaml +source: + db: chinook + table: track +transform: + - uses: add_field + with: + fields: + - expression: floor(milliseconds / 1000) + field: seconds + language: sql + - expression: split(composer) + field: composerlist + language: jmespath +output: + - uses: redis.write + with: + connection: target + data_type: json + key: + expression: concat(['track:', trackid]) + language: jmespath +``` + +You can query the target database to see the new fields in +the JSON object: + +```bash +> JSON.GET track:1 $ + +"[{\"trackid\":1,\"name\":\"For Those About To Rock (We Salute You)\",\"albumid\":1,\"mediatypeid\":1,\"genreid\":1,\"composer\":\"Angus Young, Malcolm Young, Brian Johnson\",\"milliseconds\":343719,\"bytes\":11170334,\"unitprice\":\"0.99\",\"seconds\":343,\"composerlist\":[\"Angus Young\",\" Malcolm Young\",\" Brian Johnson\"]}]" +``` + +## Using `add_field` with `remove_field` + +You can use the `add_field` and +[`remove_field`]({{< relref "/integrate/redis-data-integration/data-pipelines/transform-examples/redis-remove-field-example" >}}) +transformations together to completely replace fields from the source. For example, +if you add a new `fullname` field, you might not need the separate `firstname` and +`lastname` fields. You can remove them with a job file like the following: + +```yaml +source: + db: chinook + table: customer +transform: + - uses: add_field + with: + expression: concat(firstname, ' ', lastname) + field: fullname + language: sql + - uses: remove_field + with: + fields: + - field: firstname + - field: lastname +output: + - uses: redis.write + with: + connection: target + data_type: hash + key: + expression: concat(['cust:', customerid]) + language: jmespath +``` diff --git a/content/integrate/redis-data-integration/data-pipelines/transform-examples/redis-remove-field-example.md b/content/integrate/redis-data-integration/data-pipelines/transform-examples/redis-remove-field-example.md new file mode 100644 index 0000000000..94438eba5e --- /dev/null +++ b/content/integrate/redis-data-integration/data-pipelines/transform-examples/redis-remove-field-example.md @@ -0,0 +1,166 @@ +--- +Title: Remove fields from a key +alwaysopen: false +categories: +- docs +- integrate +- rs +- rdi +description: null +group: di +linkTitle: Remove fields +summary: Redis Data Integration keeps Redis in sync with a primary database in near + real time. +type: integration +weight: 40 +--- + +By default, RDI adds fields to +[hash]({{< relref "/develop/data-types/hashes" >}}) or +[JSON]({{< relref "/develop/data-types/json" >}}) objects in the target +database for each of the columns of the source table. +The examples below show how to omit some of those fields from the target data with the +[`remove_field`]({{< relref "/integrate/redis-data-integration/reference/data-transformation/remove_field" >}}) transformation. + +## Remove a single field + +The first example removes a single field from the data. +The `source` section selects the `employee` table of the +[`chinook`](https://github.com/Redislabs-Solution-Architects/rdi-quickstart-postgres) +database (the optional `db` field here corresponds to the +`sources..connection.database` field defined in +[`config.yaml`]({{< relref "/integrate/redis-data-integration/data-pipelines/data-pipelines#the-configyaml-file" >}})). + +In the `transform` section, the `remove_field` transformation removes the +`hiredate` field. + +The `output` section specifies `hash` as the `data_type` to write to the target, which +overrides the default setting of `target_data_type` defined in `config.yaml`. Also, the +`output.with.key` section specifies a custom key format of the form `emp:`. +Note that any fields you remove in the `transform` section are not available for +the key calculation in the `output` section. + +The full example is shown below: + +```yaml +source: + db: chinook + table: employee +transform: + - uses: remove_field + with: + field: hiredate +output: + - uses: redis.write + with: + connection: target + data_type: hash + key: + expression: concat(['emp:', employeeid]) + language: jmespath +``` + +If you queried the generated target data from the default transformation +using [`redis-cli`]({{< relref "/develop/connect/cli" >}}), you would +see something like the following: + +```bash +> hgetall emp:8 + 1) "employeeid" + 2) "8" + 3) "lastname" + 4) "Callahan" + 5) "firstname" + 6) "Laura" + 7) "title" + 8) "IT Staff" + 9) "reportsto" +10) "6" +11) "birthdate" +12) "-62467200000000" +13) "hiredate" +14) "1078358400000000" +15) "address" +16) "923 7 ST NW" +. +. +``` + +Using the job file above, the data omits the `hiredate` field: + +```bash + > hgetall emp:8 + 1) "employeeid" + 2) "8" + 3) "lastname" + 4) "Callahan" + 5) "firstname" + 6) "Laura" + 7) "title" + 8) "IT Staff" + 9) "reportsto" +10) "6" +11) "birthdate" +12) "-62467200000000" +13) "address" +14) "923 7 ST NW" +. +. +``` + +## Remove multiple fields + +The `remove_field` transformation can also remove multiple fields at the same time +if you specify them under a `fields` subsection. The example below is similar +to the previous one but also removes the `birthdate` field: + +```yaml +source: + db: chinook + table: employee +transform: + - uses: remove_field + with: + fields: + - field: hiredate + - field: birthdate +output: + - uses: redis.write + with: + connection: target + data_type: hash + key: + expression: concat(['emp:', employeeid]) + language: jmespath +``` + +If you query the data, you can see that it also omits the +`birthdate` field: + +```bash +> hgetall emp:8 + 1) "employeeid" + 2) "8" + 3) "lastname" + 4) "Callahan" + 5) "firstname" + 6) "Laura" + 7) "title" + 8) "IT Staff" + 9) "reportsto" +10) "6" +11) "address" +12) "923 7 ST NW" +. +. +``` + +## Using `remove_field` with `add_field` + +The `remove_field` transformation is very useful in combination with +[`add_field`]({{< relref "/integrate/redis-data-integration/data-pipelines/transform-examples/redis-add-field-example" >}}). +For example, if you use `add_field` to concatenate a person's first +and last names, you may not need separate `firstname` and `lastname` +fields, so you can use `remove_field` to omit them. +See [Using `add_field` with `remove_field`]({{< relref "/integrate/redis-data-integration/data-pipelines/transform-examples/redis-add-field-example#using-add_field-with-remove_field" >}}) +for an example of how to do this.