From 76d4afd553423e5f6ea1b1afbbeed4ca15bab0e8 Mon Sep 17 00:00:00 2001 From: Ilian Iliev Date: Wed, 4 Jun 2025 13:45:43 +0300 Subject: [PATCH 1/4] Remapping the output --- .../transform-examples/map-example.md | 62 ++++++++----------- .../remapping-the-output.md | 38 ++++++++++++ 2 files changed, 63 insertions(+), 37 deletions(-) create mode 100644 content/integrate/redis-data-integration/data-pipelines/transform-examples/remapping-the-output.md diff --git a/content/integrate/redis-data-integration/data-pipelines/transform-examples/map-example.md b/content/integrate/redis-data-integration/data-pipelines/transform-examples/map-example.md index 296297828c..8f89554b01 100644 --- a/content/integrate/redis-data-integration/data-pipelines/transform-examples/map-example.md +++ b/content/integrate/redis-data-integration/data-pipelines/transform-examples/map-example.md @@ -19,48 +19,22 @@ By default, RDI adds fields to [hash]({{< relref "/develop/data-types/hashes" >}}) or [JSON]({{< relref "/develop/data-types/json" >}}) objects in the target database that closely match the columns of the source table. -The examples below show how you can create a completely new object structure -from existing fields using the +If you just want to limit the set fields in the output and/or rename some of them, you can use the +[`output mapping`]({{< relref "/integrate/redis-data-integration/data-pipelines/transform-examples/remapping-the-output" >}}) configuration option. + +For situations where you want to create a new object structure with multiple levels or use calculations for the field values, you can use the [`map`]({{< relref "/integrate/redis-data-integration/reference/data-transformation/map" >}}) -transformation. - -## Map to a new JSON structure - -The first -[job file]({{< relref "/integrate/redis-data-integration/data-pipelines/data-pipelines#job-files" >}}) -example creates a new [JSON]({{< relref "/develop/data-types/json" >}}) -object structure to write to the target. -The `source` section selects the `employee` table of the -[`chinook`](https://github.com/Redislabs-Solution-Architects/rdi-quickstart-postgres) -database (the optional `db` value here corresponds to the -`sources..connection.database` value defined in -[`config.yaml`]({{< relref "/integrate/redis-data-integration/data-pipelines/data-pipelines#the-configyaml-file" >}})). - -In the `transform` section, the `map` transformation uses a [JMESPath](https://jmespath.org/) -expression to specify the new JSON format. (Note that the vertical bar "|" in the `expression` -line indicates that the following indented lines should be interpreted as a single string.) -The expression resembles JSON notation but with data values supplied from -table fields and -[JMESPath functions]({{< relref "/integrate/redis-data-integration/reference/jmespath-custom-functions" >}}). - -Here, we rename the -`employeeid` field to `id` and create two nested objects for the `address` -and `contact` information. The `name` field is the concatenation of the existing -`firstname` and `lastname` fields, with `lastname` converted to uppercase. -In the `contact` subobject, the `email` address is obfuscated slightly, using the -`replace()` function to hide the '@' sign and dots. +transformation. Take a look at the following examples to see how to use the `map` transformation: -In the `output` section of the job file, we specify that we want to write -to a JSON object with a custom key. Note that in the `output` section, you must refer to -fields defined in the `map` transformation, so we use the new name `id` -for the key instead of `employeeid`. +## Creating multilevel JSON objects -The full example is shown below: +You can use the `map` transformation to create a new structure for the output data, which can include nested objects and calculated fields. The `map` transformation allows you to define a new structure using an expression language, such as SQL or JavaScript. ```yaml source: db: chinook table: employee + transform: - uses: map with: @@ -81,16 +55,30 @@ transform: } } language: jmespath + output: - uses: redis.write with: - connection: target data_type: json key: expression: concat(['emp:', id]) language: jmespath ``` + +The example above creates a new JSON object with the following structure: + - A top-level `id` field that is the same as the `employeeid` field in the source table. + - A `name` field that is a concatenation of the `firstname` and `lastname` fields, with the `lastname` converted to uppercase. + - An `address` subobject that contains the `address`, `city`, `state`, `postalcode`, and `country` fields. + - A `contact` subobject that contains the `phone` field and a modified version of the `email` field, where the '@' sign and dots are replaced with '_at_' and '_dot_' respectively. + +In the `output` section of the job file, we specify that we want to write +to a JSON object with a custom key. Note that in the `output` section, you must refer to +fields defined in the `map` transformation, so we use the new name `id` +for the key instead of `employeeid`. + + + If you query one of the new JSON objects, you see output like the following: ```bash @@ -118,7 +106,7 @@ Formatted in the usual JSON style, the output looks like the sample below: } ``` -## Map to a hash structure +## Creating hash structure This example creates a new [hash]({{< relref "/develop/data-types/hashes" >}}) object structure for items from the `track` table. Here, the `map` transformation uses @@ -167,4 +155,4 @@ like the following: 6) "3:35.0" 7) "storagesize" 8) "6.71MB" -``` \ No newline at end of file +``` diff --git a/content/integrate/redis-data-integration/data-pipelines/transform-examples/remapping-the-output.md b/content/integrate/redis-data-integration/data-pipelines/transform-examples/remapping-the-output.md new file mode 100644 index 0000000000..08ecaa8881 --- /dev/null +++ b/content/integrate/redis-data-integration/data-pipelines/transform-examples/remapping-the-output.md @@ -0,0 +1,38 @@ +--- +Title: Remapping the fields in the output +aliases: null +alwaysopen: false +categories: +- docs +- integrate +- rs +- rdi +description: null +group: di +linkTitle: Remapping the fields in the output +summary: Redis Data Integration keeps Redis in sync with the primary database in near + real time. +type: integration +weight: 40 +--- + +Sometimes, you may want to remap the fields in the output of a data pipeline. This can be done by defining a `mapping` section in the output configuration. + +```yaml +source: + table: Customer + +output: + - uses: redis.write + with: + data_type: hash + mapping: + - CustomerId: id + - FirstName: first_name + - LastName: last_name +``` + +The example above remaps the `CustomerId` field to `id`, `FirstName` to `first_name`, and `LastName` to `last_name` in the output. This allows you to customize the field names in the Redis data store according to your application's requirements. +You can also use `mapping` to include only the fields you need in the output and exclude the rest. + +Mapping only allows you to rename fields and limit the output to specific fields and define a single level structure. To create nested structures and/or perform operations on the field values you can use the [map transformation]({{< relref "/integrate/redis-data-integration/data-pipelines/transform-examples/map-example" >}}). From 379c91b014353dabdafdf84eaaa49148d5c8f270 Mon Sep 17 00:00:00 2001 From: ilianiliev-redis Date: Wed, 11 Jun 2025 14:52:25 +0300 Subject: [PATCH 2/4] Update content/integrate/redis-data-integration/data-pipelines/transform-examples/map-example.md Co-authored-by: andy-stark-redis <164213578+andy-stark-redis@users.noreply.github.com> --- .../data-pipelines/transform-examples/map-example.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/integrate/redis-data-integration/data-pipelines/transform-examples/map-example.md b/content/integrate/redis-data-integration/data-pipelines/transform-examples/map-example.md index 8f89554b01..210ea568a1 100644 --- a/content/integrate/redis-data-integration/data-pipelines/transform-examples/map-example.md +++ b/content/integrate/redis-data-integration/data-pipelines/transform-examples/map-example.md @@ -24,7 +24,7 @@ If you just want to limit the set fields in the output and/or rename some of the For situations where you want to create a new object structure with multiple levels or use calculations for the field values, you can use the [`map`]({{< relref "/integrate/redis-data-integration/reference/data-transformation/map" >}}) -transformation. Take a look at the following examples to see how to use the `map` transformation: +transformation, as described in the following sections. ## Creating multilevel JSON objects From 66a80ddba1a929267b14a5fcaadc909c176a4035 Mon Sep 17 00:00:00 2001 From: ilianiliev-redis Date: Wed, 11 Jun 2025 14:53:16 +0300 Subject: [PATCH 3/4] Update content/integrate/redis-data-integration/data-pipelines/transform-examples/map-example.md Co-authored-by: andy-stark-redis <164213578+andy-stark-redis@users.noreply.github.com> --- .../data-pipelines/transform-examples/map-example.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/integrate/redis-data-integration/data-pipelines/transform-examples/map-example.md b/content/integrate/redis-data-integration/data-pipelines/transform-examples/map-example.md index 210ea568a1..c66cdc6cc1 100644 --- a/content/integrate/redis-data-integration/data-pipelines/transform-examples/map-example.md +++ b/content/integrate/redis-data-integration/data-pipelines/transform-examples/map-example.md @@ -72,7 +72,7 @@ The example above creates a new JSON object with the following structure: - An `address` subobject that contains the `address`, `city`, `state`, `postalcode`, and `country` fields. - A `contact` subobject that contains the `phone` field and a modified version of the `email` field, where the '@' sign and dots are replaced with '_at_' and '_dot_' respectively. -In the `output` section of the job file, we specify that we want to write +The `output` section of the file configures the job to write to a JSON object with a custom key. Note that in the `output` section, you must refer to fields defined in the `map` transformation, so we use the new name `id` for the key instead of `employeeid`. From 3eaee5f488a92270fcbd5767c8d2bdbf926d3701 Mon Sep 17 00:00:00 2001 From: ilianiliev-redis Date: Wed, 11 Jun 2025 14:53:31 +0300 Subject: [PATCH 4/4] Update content/integrate/redis-data-integration/data-pipelines/transform-examples/remapping-the-output.md Co-authored-by: andy-stark-redis <164213578+andy-stark-redis@users.noreply.github.com> --- .../data-pipelines/transform-examples/remapping-the-output.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/integrate/redis-data-integration/data-pipelines/transform-examples/remapping-the-output.md b/content/integrate/redis-data-integration/data-pipelines/transform-examples/remapping-the-output.md index 08ecaa8881..25ec744e4a 100644 --- a/content/integrate/redis-data-integration/data-pipelines/transform-examples/remapping-the-output.md +++ b/content/integrate/redis-data-integration/data-pipelines/transform-examples/remapping-the-output.md @@ -16,7 +16,7 @@ type: integration weight: 40 --- -Sometimes, you may want to remap the fields in the output of a data pipeline. This can be done by defining a `mapping` section in the output configuration. +Sometimes, you may want to remap the fields in the output of a data pipeline. You can do this by defining a `mapping` section in the output configuration. ```yaml source: