diff --git a/content/integrate/redis-data-integration/data-pipelines/transform-examples/redis-row-format.md b/content/integrate/redis-data-integration/data-pipelines/transform-examples/redis-row-format.md index 9278ee0a51..227641bf16 100644 --- a/content/integrate/redis-data-integration/data-pipelines/transform-examples/redis-row-format.md +++ b/content/integrate/redis-data-integration/data-pipelines/transform-examples/redis-row-format.md @@ -102,3 +102,48 @@ output: expression: concat(['addresses-full', '#', values(key)[0]]) language: jmespath ``` + +### Important notes when using `row_format: full` + +- The `before` object will be `null` for `insert` and `create` operations, and the `after` object will be `null` for `delete` operations. If you are building the output key manually, you should account for this and ensure that you are not trying to access fields from a `null` object, as shown in the example below: + + ```yaml + source: + table: addresses + row_format: full + + output: + - uses: redis.write + with: + key: + language: jmespath + # The following pattern will fail for delete operations. In those cases `after` is null, the resulting key will + # be 'addresses:None' and the key won't be removed from the target + # expression: concat(['addresses:', after.ID]) + + # This pattern works for all operations, by using the ID from the `after` object if it is available, + # and falling back to the ID from the `before` object if not. + expression: concat(['addresses:', after.ID || before.ID]) + + # Another option is to use the ID from the `key` object + # expression: concat(['addresses:', values(key)[0]]) + ``` + + Please note that you should not use `key` in combination with `row_format: full` and more than one output, as the `key` object will be overwritten by the previous output. This is a known limitation of the current implementation and is subject to change in future versions. + + +- The final result of the processing (which is what will be stored in the output) is the value of the `after` object. This means you must reference the fields using the `after` prefix unless you change the output structure in a transformation step. Also, when you add new fields, you must prefix them with `after.` to ensure that they are added to the correct part of the output: + + ```yaml + source: + table: addresses + row_format: full + + transform: + - uses: add_field + with: + field: after.city_state # use this to add the new field to the final output + # field: city_state # use this if you need a temporary field in the transformation steps, but not in the final output + expression: concat([after.CITY, ', ', after.STATE]) + language: jmespath + ```