DOC-4510 RDI add_field and remove_field job file examples

content/integrate/redis-data-integration/data-pipelines/transform-examples/redis-add-field-example.md

@@ -0,0 +1,176 @@
+---
+Title: Add new fields to a key
+alwaysopen: false
+categories:
+- docs
+- integrate
+- rs
+- rdi
+description: null
+group: di
+linkTitle: Add new fields
+summary: Redis Data Integration keeps Redis in sync with a primary database in near
+  real time.
+type: integration
+weight: 40
+---
+
+By default, RDI adds fields to
+[hash]({{< relref "/develop/data-types/hashes" >}}) or
+[JSON]({{< relref "/develop/data-types/json" >}}) objects in the target
+database that match the columns of the source table.
+The examples below show how to add extra fields to the target data with the
+[`add_field`]({{< relref "/integrate/redis-data-integration/reference/data-transformation/add_field" >}}) transformation.
+
+## Add a single field
+
+The first example adds a single field to the data.
+The `source` section selects the `customer` table of the
+[`chinook`](https://github.com/Redislabs-Solution-Architects/rdi-quickstart-postgres)
+database (the optional `db` value here corresponds to the
+`sources.<source-name>.connection.database` value defined in
+[`config.yaml`]({{< relref "/integrate/redis-data-integration/data-pipelines/data-pipelines#the-configyaml-file" >}})).
+
+In the `transform` section, the `add_field` transformation adds an extra field called `localphone`
+to the object, which is created by removing the country and area code from the `phone`
+field with the
+[JMESPath]({{< relref "/integrate/redis-data-integration/reference/jmespath-custom-functions" >}}) function `regex_replace()`.
+You can also specify `sql` as the `language` if you prefer to create the new
+field with an [SQL](https://en.wikipedia.org/wiki/SQL) expression.
+
+The `output` section specifies `hash` as the `data_type` to write to the target, which
+overrides the default setting of `target_data_type` defined in `config.yaml`. Also, the
+`output.with.key` section specifies a custom key format of the form `cust:<id>` where
+the `id` part is generated by the `uuid()` function.
+
+The full example is shown below:
+
+```yaml
+source:
+  db: chinook
+  table: customer
+transform:
+  - uses: add_field
+    with:
+      expression: regex_replace(phone, '\+[0-9]+ (\([0-9]+\) )?', '')
+      field: localphone
+      language: jmespath
+output:
+  - uses: redis.write
+    with:
+      connection: target
+      data_type: hash
+      key:
+        expression: concat(['cust:', uuid()])
+        language: jmespath
+```
+
+If you queried the generated target data from the default transformation
+using [`redis-cli`]({{< relref "/develop/connect/cli" >}}), you would
+see something like the following:
+
+```
+ 1) "customerid"
+ 2) "27"
+ 3) "firstname"
+ 4) "Patrick"
+ 5) "lastname"
+ 6) "Gray"
+.
+.
+17) "phone"
+18) "+1 (520) 622-4200"
+.
+.
+```
+
+Using the job file above, the data also includes the new `localphone` field:
+
+```
+ 1) "customerid"
+ 2) "27"
+ 3) "firstname"
+ 4) "Patrick"
+ 5) "lastname"
+ 6) "Gray"
+ .
+ .
+23) "localphone"
+24) "622-4200"
+```
+
+## Add multiple fields
+
+The `add_field` transformation can also add multiple fields at the same time
+if you specify them under a `fields` subsection. The example below adds two
+fields to the `track` objects. The first new field, `seconds`, is created using a SQL
+expression to calculate the duration of the track in seconds from the
+`milliseconds` field.
+The second new field, `composerlist`, adds a JSON array using the `split()` function
+to split the `composer` string field wherever it contains a comma.
+
+```yaml
+source:
+  db: chinook
+  table: track
+transform:
+  - uses: add_field
+    with:
+      fields:
+        - expression: floor(milliseconds / 1000)
+          field: seconds
+          language: sql
+        - expression: split(composer)
+          field: composerlist
+          language: jmespath
+output:
+  - uses: redis.write
+    with:
+      connection: target
+      data_type: json
+      key:
+        expression: concat(['track:', trackid])
+        language: jmespath
+```
+
+You can query the target database to see the new fields in
+the JSON object:
+
+```bash
+> JSON.GET track:1 $
+
+"[{\"trackid\":1,\"name\":\"For Those About To Rock (We Salute You)\",\"albumid\":1,\"mediatypeid\":1,\"genreid\":1,\"composer\":\"Angus Young, Malcolm Young, Brian Johnson\",\"milliseconds\":343719,\"bytes\":11170334,\"unitprice\":\"0.99\",\"seconds\":343,\"composerlist\":[\"Angus Young\",\" Malcolm Young\",\" Brian Johnson\"]}]"
+```
+
+## Using `add_field` with `remove_field`
+
+You can use the `add_field` and
+[`remove_field`]({{< relref "/integrate/redis-data-integration/data-pipelines/transform-examples/redis-remove-field-example" >}})
+transformations together to completely replace fields from the source. For example,
+if you add a new `fullname` field, you might not need the separate `firstname` and
+`lastname` fields. You can remove them with a job file like the following:
+
+```yaml
+source:
+  db: chinook
+  table: customer
+transform:
+  - uses: add_field
+    with:
+      expression: concat(firstname, ' ', lastname)
+      field: fullname
+      language: sql
+  - uses: remove_field
+    with:
+      fields:
+        - field: firstname
+        - field: lastname
+output:
+  - uses: redis.write
+    with:
+      connection: target
+      data_type: hash
+      key:
+        expression: concat(['cust:', customerid])
+        language: jmespath
+```

content/integrate/redis-data-integration/data-pipelines/transform-examples/redis-remove-field-example.md

@@ -0,0 +1,166 @@
+---
+Title: Remove fields from a key
+alwaysopen: false
+categories:
+- docs
+- integrate
+- rs
+- rdi
+description: null
+group: di
+linkTitle: Remove fields
+summary: Redis Data Integration keeps Redis in sync with a primary database in near
+  real time.
+type: integration
+weight: 40
+---
+
+By default, RDI adds fields to
+[hash]({{< relref "/develop/data-types/hashes" >}}) or
+[JSON]({{< relref "/develop/data-types/json" >}}) objects in the target
+database for each of the columns of the source table.
+The examples below show how to omit some of those fields from the target data with the
+[`remove_field`]({{< relref "/integrate/redis-data-integration/reference/data-transformation/remove_field" >}}) transformation.
+
+## Remove a single field
+
+The first example removes a single field from the data.
+The `source` section selects the `employee` table of the
+[`chinook`](https://github.com/Redislabs-Solution-Architects/rdi-quickstart-postgres)
+database (the optional `db` field here corresponds to the
+`sources.<source-name>.connection.database` field defined in
+[`config.yaml`]({{< relref "/integrate/redis-data-integration/data-pipelines/data-pipelines#the-configyaml-file" >}})).
+
+In the `transform` section, the `remove_field` transformation removes the
+`hiredate` field.
+
+The `output` section specifies `hash` as the `data_type` to write to the target, which
+overrides the default setting of `target_data_type` defined in `config.yaml`. Also, the
+`output.with.key` section specifies a custom key format of the form `emp:<employeeid>`.
+Note that any fields you remove in the `transform` section are not available for
+the key calculation in the `output` section.
+
+The full example is shown below:
+
+```yaml
+source:
+  db: chinook
+  table: employee
+transform:
+  - uses: remove_field
+    with:
+      field: hiredate
+output:
+  - uses: redis.write
+    with:
+      connection: target
+      data_type: hash
+      key:
+        expression: concat(['emp:', employeeid])
+        language: jmespath
+```
+
+If you queried the generated target data from the default transformation
+using [`redis-cli`]({{< relref "/develop/connect/cli" >}}), you would
+see something like the following:
+
+```bash
+> hgetall emp:8
+ 1) "employeeid"
+ 2) "8"
+ 3) "lastname"
+ 4) "Callahan"
+ 5) "firstname"
+ 6) "Laura"
+ 7) "title"
+ 8) "IT Staff"
+ 9) "reportsto"
+10) "6"
+11) "birthdate"
+12) "-62467200000000"
+13) "hiredate"
+14) "1078358400000000"
+15) "address"
+16) "923 7 ST NW"
+.
+.
+```
+
+Using the job file above, the data omits the `hiredate` field:
+
+```bash
+ > hgetall emp:8
+ 1) "employeeid"
+ 2) "8"
+ 3) "lastname"
+ 4) "Callahan"
+ 5) "firstname"
+ 6) "Laura"
+ 7) "title"
+ 8) "IT Staff"
+ 9) "reportsto"
+10) "6"
+11) "birthdate"
+12) "-62467200000000"
+13) "address"
+14) "923 7 ST NW"
+.
+.
+```
+
+## Remove multiple fields
+
+The `remove_field` transformation can also remove multiple fields at the same time
+if you specify them under a `fields` subsection. The example below is similar
+to the previous one but also removes the `birthdate` field: 
+
+```yaml
+source:
+  db: chinook
+  table: employee
+transform:
+  - uses: remove_field
+    with:
+      fields:
+        - field: hiredate
+        - field: birthdate
+output:
+  - uses: redis.write
+    with:
+      connection: target
+      data_type: hash
+      key:
+        expression: concat(['emp:', employeeid])
+        language: jmespath
+```
+
+If you query the data, you can see that it also omits the
+`birthdate` field:
+
+```bash
+> hgetall emp:8
+ 1) "employeeid"
+ 2) "8"
+ 3) "lastname"
+ 4) "Callahan"
+ 5) "firstname"
+ 6) "Laura"
+ 7) "title"
+ 8) "IT Staff"
+ 9) "reportsto"
+10) "6"
+11) "address"
+12) "923 7 ST NW"
+.
+.
+```
+
+## Using `remove_field` with `add_field`
+
+The `remove_field` transformation is very useful in combination with
+[`add_field`]({{< relref "/integrate/redis-data-integration/data-pipelines/transform-examples/redis-add-field-example" >}}).
+For example, if you use `add_field` to concatenate a person's first
+and last names, you may not need separate `firstname` and `lastname`
+fields, so you can use `remove_field` to omit them.
+See [Using `add_field` with `remove_field`]({{< relref "/integrate/redis-data-integration/data-pipelines/transform-examples/redis-add-field-example#using-add_field-with-remove_field" >}})
+for an example of how to do this.