Skip to content

Commit 6a62e3a

Browse files
andy-stark-redisandy-stark-redis
committed
DOC-4510 added remove_field examples and other updates
1 parent 46615e7 commit 6a62e3a

File tree

2 files changed

+195
-18
lines changed

2 files changed

+195
-18
lines changed

‎content/integrate/redis-data-integration/data-pipelines/transform-examples/redis-add-field-example.md‎

Lines changed: 29 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
---
2-
Title: Add a new field to a hash/JSON key
2+
Title: Add new fields to a key
33
alwaysopen: false
44
categories:
55
- docs
@@ -8,33 +8,38 @@ categories:
88
- rdi
99
description: null
1010
group: di
11-
linkTitle: Add a new hash/JSON field
11+
linkTitle: Add new fields
1212
summary: Redis Data Integration keeps Redis in sync with the primary database in near
1313
real time.
1414
type: integration
15-
weight: 30
15+
weight: 40
1616
---
1717

1818
By default, RDI adds fields to
1919
[hash]({{< relref "/develop/data-types/hashes" >}}) or
2020
[JSON]({{< relref "/develop/data-types/json" >}}) objects in the target
21-
database that closely correspond to the columns of the source table.
22-
This example shows how to add extra fields to the target data with the
21+
database that match the columns of the source table.
22+
The examples below show how to add extra fields to the target data with the
2323
[`add_field`]({{< relref "/integrate/redis-data-integration/reference/data-transformation/add_field" >}}) transformation.
2424

25+
## Add a single field
26+
27+
The first example adds a single field to the data.
2528
The `source` section selects the `customer` table of the
2629
[`chinook`](https://github.com/Redislabs-Solution-Architects/rdi-quickstart-postgres)
27-
database (the optional `db` field here corresponds to the
28-
`sources.<source-name>.connection.database` field defined in
30+
database (the optional `db` value here corresponds to the
31+
`sources.<source-name>.connection.database` value defined in
2932
[`config.yaml`]({{< relref "/integrate/redis-data-integration/data-pipelines/data-pipelines#the-configyaml-file" >}})).
3033

31-
In the `transform` section, the `add_field` transformation adds an extra field called `fullname`,
32-
which is created by concatenating the existing `firstname` and `lastname` fields using
34+
In the `transform` section, the `add_field` transformation adds an extra field called `fullname`
35+
to the object, which is created by concatenating the existing `firstname` and `lastname` fields using
3336
an [SQL expression](https://www.simplilearn.com/tutorials/sql-tutorial/concat-function-in-sql).
37+
You can also specify `jmespath` as the `language` if you prefer to create the new
38+
field with a [JMESPath](https://jmespath.org/) expression
3439

3540
The `output` section specifies `hash` as the `data_type` to write to the target, which
3641
overrides the default setting of `target_data_type` defined in `config.yaml`. Also, the
37-
`output.key` section specifies a custom key format of the form `cust:<customerid>`.
42+
`output.with.key` section specifies a custom key format of the form `cust:<customerid>`.
3843

3944
The full example is shown below:
4045

@@ -58,10 +63,11 @@ output:
5863
language: jmespath
5964
```
6065
61-
If you queried the generated target data from the default transformation, you would
66+
If you queried the generated target data from the default transformation
67+
using [`redis-cli`]({{< relref "/develop/connect/cli" >}}), you would
6268
see something like the following:
6369

64-
```bash
70+
```
6571
> hgetall cust:14
6672
1) "customerid"
6773
2) "14"
@@ -79,7 +85,7 @@ see something like the following:
7985

8086
Using the job file above, the data also includes the new `fullname` field:
8187

82-
```bash
88+
```
8389
1) "customerid"
8490
2) "14"
8591
3) "firstname"
@@ -92,6 +98,8 @@ Using the job file above, the data also includes the new `fullname` field:
9298
28) "Mark Philips"
9399
```
94100

101+
## Add multiple fields
102+
95103
The `add_field` transformation can also add multiple fields at the same time
96104
if you specify them under a `fields` subsection. The example below is similar
97105
to the previous one but also adds a `fulladdress` field and uses JSON as the
@@ -124,15 +132,18 @@ output:
124132
You can query the target database to see the new `fullname` field in
125133
the JSON object:
126134

127-
```bash
135+
```
128136
> JSON.GET cust:14 $.fulladdress
129137
"[\"8210 111 ST NW, Edmonton, Canada, T6G 2C7\"]"
130138
```
131139

132-
You can use the `add_field` and `remove_field` transformations together
133-
to completely replace fields from the source. For example, if you add
134-
a new `fullname` field, you might not need the separate `firstname` and `lastname`
135-
fields. You can remove them with a job file like the following:
140+
## Using `add_field` with `remove_field`
141+
142+
You can use the `add_field` and
143+
[`remove_field`]({{< relref "/integrate/redis-data-integration/data-pipelines/transform-examples/redis-remove-field-example" >}})
144+
transformations together to completely replace fields from the source. For example,
145+
if you add a new `fullname` field, you might not need the separate `firstname` and
146+
`lastname` fields. You can remove them with a job file like the following:
136147

137148
```yaml
138149
source:

‎content/integrate/redis-data-integration/data-pipelines/transform-examples/redis-remove-field-example.md‎

Lines changed: 166 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,166 @@
1+
---
2+
Title: Remove fields from a key
3+
alwaysopen: false
4+
categories:
5+
- docs
6+
- integrate
7+
- rs
8+
- rdi
9+
description: null
10+
group: di
11+
linkTitle: Remove fields
12+
summary: Redis Data Integration keeps Redis in sync with the primary database in near
13+
real time.
14+
type: integration
15+
weight: 40
16+
---
17+
18+
By default, RDI adds fields to
19+
[hash]({{< relref "/develop/data-types/hashes" >}}) or
20+
[JSON]({{< relref "/develop/data-types/json" >}}) objects in the target
21+
database for each of the columns of the source table.
22+
The examples below show how to omit some of those fields from the target data with the
23+
[`remove_field`]({{< relref "/integrate/redis-data-integration/reference/data-transformation/remove_field" >}}) transformation.
24+
25+
## Remove a single field
26+
27+
The first example removes a single field from the data.
28+
The `source` section selects the `employee` table of the
29+
[`chinook`](https://github.com/Redislabs-Solution-Architects/rdi-quickstart-postgres)
30+
database (the optional `db` field here corresponds to the
31+
`sources.<source-name>.connection.database` field defined in
32+
[`config.yaml`]({{< relref "/integrate/redis-data-integration/data-pipelines/data-pipelines#the-configyaml-file" >}})).
33+
34+
In the `transform` section, the `remove_field` transformation removes the
35+
`hiredate` field.
36+
37+
The `output` section specifies `hash` as the `data_type` to write to the target, which
38+
overrides the default setting of `target_data_type` defined in `config.yaml`. Also, the
39+
`output.with.key` section specifies a custom key format of the form `emp:<employeeid>`.
40+
Note that any fields you remove in the `transform` section are not available for
41+
the key calculation in the `output` section.
42+
43+
The full example is shown below:
44+
45+
```yaml
46+
source:
47+
db: chinook
48+
table: employee
49+
transform:
50+
- uses: remove_field
51+
with:
52+
field: hiredate
53+
output:
54+
- uses: redis.write
55+
with:
56+
connection: target
57+
data_type: hash
58+
key:
59+
expression: concat(['emp:', employeeid])
60+
language: jmespath
61+
```
62+
63+
If you queried the generated target data from the default transformation
64+
using [`redis-cli`]({{< relref "/develop/connect/cli" >}}), you would
65+
see something like the following:
66+
67+
```bash
68+
> hgetall emp:8
69+
1) "employeeid"
70+
2) "8"
71+
3) "lastname"
72+
4) "Callahan"
73+
5) "firstname"
74+
6) "Laura"
75+
7) "title"
76+
8) "IT Staff"
77+
9) "reportsto"
78+
10) "6"
79+
11) "birthdate"
80+
12) "-62467200000000"
81+
13) "hiredate"
82+
14) "1078358400000000"
83+
15) "address"
84+
16) "923 7 ST NW"
85+
.
86+
.
87+
```
88+
89+
Using the job file above, the data omits the `hiredate` field:
90+
91+
```bash
92+
> hgetall emp:8
93+
1) "employeeid"
94+
2) "8"
95+
3) "lastname"
96+
4) "Callahan"
97+
5) "firstname"
98+
6) "Laura"
99+
7) "title"
100+
8) "IT Staff"
101+
9) "reportsto"
102+
10) "6"
103+
11) "birthdate"
104+
12) "-62467200000000"
105+
13) "address"
106+
14) "923 7 ST NW"
107+
.
108+
.
109+
```
110+
111+
## Remove multiple fields
112+
113+
The `remove_field` transformation can also remove multiple fields at the same time
114+
if you specify them under a `fields` subsection. The example below is similar
115+
to the previous one but also removes the `birthdate` field:
116+
117+
```yaml
118+
source:
119+
db: chinook
120+
table: employee
121+
transform:
122+
- uses: remove_field
123+
with:
124+
fields:
125+
- field: hiredate
126+
- field: birthdate
127+
output:
128+
- uses: redis.write
129+
with:
130+
connection: target
131+
data_type: hash
132+
key:
133+
expression: concat(['emp:', employeeid])
134+
language: jmespath
135+
```
136+
137+
If you query the data, you can see that it also omits the
138+
`birthdate` field:
139+
140+
```bash
141+
> hgetall emp:8
142+
1) "employeeid"
143+
2) "8"
144+
3) "lastname"
145+
4) "Callahan"
146+
5) "firstname"
147+
6) "Laura"
148+
7) "title"
149+
8) "IT Staff"
150+
9) "reportsto"
151+
10) "6"
152+
11) "address"
153+
12) "923 7 ST NW"
154+
.
155+
.
156+
```
157+
158+
## Using `remove_field` with `add_field`
159+
160+
The `remove_field` transformation is very useful in combination with
161+
[`add_field`]({{< relref "/integrate/redis-data-integration/data-pipelines/transform-examples/redis-add-field-example" >}}).
162+
For example, if you use `add_field` to concatenate a person's first
163+
and last names, you may not need separate `firstname` and `lastname`
164+
fields, so you can use `remove_field` to omit them.
165+
See [Using `add_field` with `remove_field`]({{< relref "/integrate/redis-data-integration/data-pipelines/transform-examples/redis-add-field-example#using-add_field-with-remove_field" >}})
166+
for an example of how to do this.

0 commit comments

Comments
 (0)