diff --git a/content/integrate/redis-data-integration/data-pipelines/transform-examples/redis-opcode-example.md b/content/integrate/redis-data-integration/data-pipelines/transform-examples/redis-opcode-example.md index ecec6bdf90..203e154e41 100644 --- a/content/integrate/redis-data-integration/data-pipelines/transform-examples/redis-opcode-example.md +++ b/content/integrate/redis-data-integration/data-pipelines/transform-examples/redis-opcode-example.md @@ -1,5 +1,5 @@ --- -Title: Add the opcode to the Redis output +Title: Using the operation code alwaysopen: false categories: - docs @@ -8,43 +8,90 @@ categories: - rdi description: null group: di -linkTitle: Add the opcode to the Redis output +linkTitle: Using the operation code summary: Redis Data Integration keeps Redis in sync with the primary database in near real time. type: integration weight: 100 --- -In the example below, the data is captured from the source table named `employee` and is written to the Redis database as a JSON document. When you specify the `data_type` parameter for the job, it overrides the system-wide setting `target_data_type` defined in `config.yaml`. +The operation code (`opcode`) is a metadata field that indicates the type of operation that generated the change in the source database. It can be useful for tracking changes and understanding the context of the data being processed. + +The opcode is only available in the [full row format]({{< relref "/integrate/redis-data-integration/data-pipelines/transform-examples/redis-row-format#full" >}}), and can be accessed in the `transform` and `output` sections of the job file. + +It has one of the following values: + +- r - Read (applies to only snapshots) +- c - Create +- u - Update +- d - Delete +- t = Truncate (PostgreSQL specific) +- m = Message (PostgreSQL specific) + + +You can add the value of the operation code to the output, and also use it in a conditional expression to modify the behavior of the job. The following examples demonstrate the different use-cases. + +### Adding the operation code to the output + +Use the `add_field` transformation to add a new field that contains the value of the `opcode` field from the source data. Note that the fields must be prefixed with `after` to be included in the output. -Here, the result will be Redis JSON documents with fields captured from the source table -(`employeeid`, `firstname`, `lastname`) and also with -an extra field `my_opcode` added using the `merge` update strategy (see the -[JSON job example]({{< relref "/integrate/redis-data-integration/data-pipelines/transform-examples/redis-json-example" >}}) -for more information). The `opcode` expression refers to the operation code captured from -the source. This is a database-specific value that indicates which type of operation generated -the change (insert, update, etc). ```yaml source: schema: public table: employee row_format: full + transform: + # add the operation code to the data - uses: add_field with: - field: after.my_opcode + field: after.operation_code expression: opcode language: jmespath -output: - - uses: redis.write +``` + + +### Filtering operation by output code. + +In some cases you may want to ignore certain operations (for example, you may not be interested in deletions). Use the `filter` transformation to filter out any operations you don't need to process. + +```yaml +source: + schema: public + table: employee + row_format: full + +transform: + - uses: filter + with: + expression: opcode != 'd' + language: jmespath +``` + +### Modifying the output based on the operation code + +The previous example filters out specific operations, but you can also modify the output based on the operation code. For example, you can add a new field that tracks the status of the record based on the operation code. + +Note that when a source record is deleted, you must modify the value of the `opcode` field if you want to prevent the corresponding record in the target database from being removed automatically. + +```yaml +source: + schema: public + table: employee + row_format: full + +transform: + - uses: add_field with: - data_type: json - mapping: - - employeeid - - firstname - - lastname - - my_opcode - connection: target - on_update: merge -``` \ No newline at end of file + fields: + # Here you set the value of the field based on the value of the opcode field + - field: after.status + expression: opcode == 'd' && 'inactive' || 'active' + language: jmespath + + # You have to change the value of the opcode field to prevent deletion + - field: opcode + expression: opcode == 'd' && 'u' || opcode + language: jmespath +``` diff --git a/content/integrate/redis-data-integration/data-pipelines/transform-examples/redis-row-format.md b/content/integrate/redis-data-integration/data-pipelines/transform-examples/redis-row-format.md index 7e659b5f54..1c500f738a 100644 --- a/content/integrate/redis-data-integration/data-pipelines/transform-examples/redis-row-format.md +++ b/content/integrate/redis-data-integration/data-pipelines/transform-examples/redis-row-format.md @@ -63,7 +63,7 @@ With `row_format: full` the input value is a JSON object with the following stru - `key` - An object containing the attributes of the primary key. For example, `key.id` will give you the value of the `id` column as long as it is part of the primary key. - `before` - An object containing the previous value of the row. - `after` - An object containing the current value of the row. -- `opcode` - The operation code. Different databases use different values for the operation code. See [operation code values]({{< relref "#operation-codes" >}}) below for more information. +- `opcode` - The operation code. See [Using the operation code]({{< relref "/integrate/redis-data-integration/data-pipelines/transform-examples/redis-opcode-example" >}}) for more information about the possible opcode values and how to use them. - `db` - The database name. - `table` - The table name. - `schema` - The schema name. @@ -102,12 +102,3 @@ output: expression: concat(['addresses-full', '#', values(key)[0]]) language: jmespath ``` - -## Operation code values {#operation-codes} - -- r - Read (applies to only snapshots) -- c - Create -- u - Update -- d - Delete -- t = truncate (PostgreSQL specific) -- m = message (PostgreSQL specific)