Address code review comments

Author: ZdravkoDonev-redis

content/integrate/redis-data-integration/reference/config-yaml-reference.md

@@ -53,7 +53,7 @@ Configuration settings that control how data is processed, including batch sizes
 | **dlq_max_messages**<br/>(DLQ message limit)                         | `integer`, `string` | Maximum number of messages to store in dead letter queue per stream<br/>Default: `1000`<br/>Pattern: `^\${.*}$`<br/>Minimum: `1`<br/>                                                                                                |          |
 | **target_data_type**<br/>(Target Redis data type)                    | `string`            | Data type to use in Redis: hash for Redis Hash, json for RedisJSON (requires RedisJSON module)<br/>Default: `"hash"`<br/>Pattern: `^\${.*}$\|hash\|json`<br/>                                                                        |          |
 | **json_update_strategy**                                             | `string`            | (DEPRECATED)<br/>Property 'json_update_strategy' will be deprecated in future releases. Use 'on_update' job-level property to define the json update strategy.<br/>Default: `"replace"`<br/>Pattern: `^\${.*}$\|replace\|merge`<br/> |          |
-| **use_native_json_merge**<br/>(Use native JSON merge)               | `boolean`           | Controls whether to use the native `JSON.MERGE` command (when `true`) or Lua scripts (when `false`) for JSON merge operations. Introduced in RDI 1.15.0. The native command provides 2x performance improvement but handles null values differently:<br/><br/>**Previous behavior (Lua merge)**: When merging `{"field1": "value1", "field2": "value2"}` with `{"field2": null, "field3": "value3"}`, the result was `{"field1": "value1", "field2": null, "field3": "value3"}` (null value is preserved)<br/><br/>**New behavior (JSON.MERGE)**: The same merge produces `{"field1": "value1", "field3": "value3"}` (null value removes the field, following [RFC 7396](https://datatracker.ietf.org/doc/html/rfc7396))<br/><br/>**Impact**: If your application logic distinguishes between a field with a `null` value and a missing field, you may need to adjust your data handling. This follows the JSON Merge Patch RFC standard but differs from the previous Lua implementation. Set to `false` to revert to the previous Lua-based merge behavior if needed.<br/>Default: `true`<br/>                                                                                                                                      |          |
+| **use_native_json_merge**<br/>(Use native JSON merge)               | `boolean`           | Controls whether to use the native `JSON.MERGE` command (when `true`) or Lua scripts (when `false`) for JSON merge operations. Introduced in RDI 1.15.0. The native command provides 2x performance improvement but handles null values differently:<br/><br/>**Previous behavior (Lua merge)**: When merging `{"field1": "value1", "field2": "value2"}` with `{"field2": null, "field3": "value3"}`, the result was `{"field1": "value1", "field2": null, "field3": "value3"}` (null value is preserved)<br/><br/>**New behavior (JSON.MERGE)**: The same merge produces `{"field1": "value1", "field3": "value3"}` (null value removes the field, following [RFC 7396](https://datatracker.ietf.org/doc/html/rfc7396))<br/><br/>**Note**: The native `JSON.MERGE` command requires RedisJSON 2.6.0 or higher. If the target database has an older version of RedisJSON, RDI will automatically fall back to using Lua-based merge operations regardless of this setting.<br/><br/>**Impact**: If your application logic distinguishes between a field with a `null` value and a missing field, you may need to adjust your data handling. This follows the JSON Merge Patch RFC standard but differs from the previous Lua implementation. Set to `false` to revert to the previous Lua-based merge behavior if needed.<br/>Default: `true`<br/>                                                                                                                                      |          |
 | **initial_sync_processes**                                           | `integer`, `string` | Number of parallel processes for performing initial data synchronization<br/>Default: `4`<br/>Pattern: `^\${.*}$`<br/>Minimum: `1`<br/>Maximum: `32`<br/>                                                                            |          |
 | **idle_sleep_time_ms**<br/>(Idle sleep interval)                     | `integer`, `string` | Time in milliseconds to sleep between processing batches when idle<br/>Default: `200`<br/>Pattern: `^\${.*}$`<br/>Minimum: `1`<br/>Maximum: `999999`<br/>                                                                            |          |
 | **idle_streams_check_interval_ms**<br/>(Idle streams check interval) | `integer`, `string` | Time in milliseconds between checking for new streams when processor is idle<br/>Default: `1000`<br/>Pattern: `^\${.*}$`<br/>Minimum: `1`<br/>Maximum: `999999`<br/>                                                                 |          |

content/integrate/redis-data-integration/release-notes/rdi-1-15-0.md

@@ -6,9 +6,9 @@ categories:
 - operate
 - rs
 description: |
-  Flink collector enabled by default for improved performance.
-  Enhanced high availability with leader election and standby mode.
-  Support for sharded Redis databases.
+  Flink collector for Spanner enabled by default for improved user experience.
+  Enhanced high availability with configurable leader election and standby mode.
+  Support for sharded RDI Redis databases.
   Improved configuration validation and monitoring capabilities.
   Better resource management and security enhancements.
 linkTitle: 1.15.0 (October 2025)
@@ -40,28 +40,30 @@ RDI now uses the native `JSON.MERGE` command instead of Lua scripts for JSON mer
 **Configuration**: You can control this behavior using the `use_native_json_merge` property in the processors section of your configuration. Set it to `false` to revert to the previous Lua-based merge behavior if needed.
 {{</warning>}}
 
-- **Native JSON merge for improved performance**: RDI now automatically uses the native `JSON.MERGE` command from RedisJSON 2.6.0+ instead of Lua scripts for JSON merge operations, providing 2x performance improvement. This feature is enabled by default and can be controlled via the `use_native_json_merge` property in the processors section of the configuration.
-- **Support for sharded Redis databases**: RDI now supports writing to multi-sharded Redis Enterprise databases for the RDI database, resolving cross-slot violations when saving statistics.
+- **Native JSON merge for improved performance**: RDI now automatically uses the native `JSON.MERGE` command from RedisJSON 2.6.0+ instead of Lua scripts for JSON merge operations, providing 2x performance improvement. This feature is enabled by default and can be controlled via the `use_native_json_merge` property in the processors section of the configuration. **Note**: If the target Redis database has RedisJSON version lower than 2.6.0, the processor will automatically revert to using the Lua-based merge implementation.
+- **Support for sharded Redis databases**: RDI now supports writing to multi-sharded Redis Enterprise databases for the RDI database, resolving cross-slot violations when reading from streams.
 - **Enhanced processor performance metrics**: Detailed performance metrics are now exposed through the metrics exporter and statistics endpoint, with separate tracking for transformation time and write time.
 - **Resource management improvements**: Collector and processor pods now support configurable resource requests, limits, and node affinity/tolerations for better cluster resource utilization.
   - The `collector` defaults to 1 CPU and 1024Mi memory (requests), with limits of 4 CPUs and 4096Mi memory.
   - The `processor` defaults to 1 CPU and 512Mi memory (requests), with limits of 4 CPUs and 3072Mi memory.
 
 - **Leadership status monitoring**: New metrics expose leadership status and pipeline phase information for better monitoring of HA deployments.
   - The `rdi_operator_is_leader` metric tracks the current leadership status of the operator: `1` indicates the instance is the leader, `0` indicates it is not the leader.
+  - The `rdi_operator_pipeline_phase` metric tracks the current phase of the pipeline. Phase indicates the current pipeline phase, must be one of `Active`, `Inactive`, `Resetting`, `Pending`, or `Error`.
 - **Improved configuration validation**: More rigid validation for `config.yaml` and `jobs.yaml` files helps catch configuration errors earlier in the deployment process.
 - **Custom K3s installation options**: The installer now supports passing custom arguments to K3s installation for more flexible on-premises deployments.
   - Example: `sudo INSTALL_K3S_EXEC='--write-kubeconfig-mode=644' ./install.sh`
-- **Workload Identity authentication**: Added support for Google Cloud Workload Identity authentication to Google Cloud Storage (GCS), eliminating the need for service account JSON files in many scenarios.
-- Processor performance metrics showing transformation and write times separately
+- **Workload Identity authentication**: Added support for Google Cloud Workload Identity authentication to Google Cloud Storage (GCS), eliminating the need for service account JSON files if using GSC-based leader election.
+- Exposed new processor performance metrics showing transformation and write times separately
   - `{namespace}_processor_process_time_ms_total` - Total time spent in the processor (transform + write)
   - `{namespace}_processor_transform_time_ms_total` - Time spent transforming data
   - `{namespace}_processor_write_time_ms_total` - Time spent writing data to Redis
+- Exposed all existing processor metrics that were only available through the RDI CLI status and the API statistics endpoint.
 - Enhanced statistics endpoint with new metrics for transform and process time
 
 ### Bug Fixes and Stability Improvements
 
-- **Fixed task reconciliation errors**: Resolved "The ID argument cannot be a complete ID because xadd-id-uniqueness-mode is strict" errors during task reconciliation.
+- **Fixed task reconciliation errors**: Resolved "The ID argument cannot be a complete ID because xadd-id-uniqueness-mode is strict" errors during task reconciliation when using an RDI Redis database with strict XADD id uniqueness mode.
 - **Fixed Debezium unavailable values**: Addressed issues where `__debezium_unavailable_value` was appearing in Redis data.
 - **Improved operator stability**: Disabled operator webhooks by default to simplify deployments and reduce potential issues.