Skip to content

Commit 7f3ea19

Browse files
ZdravkoDonev-redisZdravkoDonev-redis
committed
Update alerting strategy
1 parent d6be677 commit 7f3ea19

File tree

1 file changed

+63
-53
lines changed

1 file changed

+63
-53
lines changed

‎content/integrate/redis-data-integration/observability.md‎

Lines changed: 63 additions & 53 deletions
Original file line numberDiff line numberDiff line change
@@ -52,38 +52,38 @@ The following table lists all collector metrics and their descriptions:
5252
| Metric | Type | Description | Alerting Recommendations |
5353
|:--|:--|:--|:--|
5454
| **Schema History Metrics** | | | |
55-
| ChangesApplied | Counter | Total number of schema changes applied during recovery and runtime | Monitor for unexpected spikes (rate > 10/hour) |
56-
| ChangesRecovered | Counter | Number of changes that were read during the recovery phase | Alert if recovery fails (value stops increasing during recovery) |
57-
| MilliSecondsSinceLastAppliedChange | Gauge | Number of milliseconds since the last change was applied | Alert if > 300,000ms (5 minutes) during active schema changes |
58-
| MilliSecondsSinceLastRecoveredChange | Gauge | Number of milliseconds since the last change was recovered from the history store | Alert if > 600,000ms (10 minutes) during recovery |
59-
| RecoveryStartTime | Gauge | Time in epoch milliseconds when recovery started (-1 if not applicable) | Monitor for prolonged recovery (> 30 minutes) |
55+
| ChangesApplied | Counter | Total number of schema changes applied during recovery and runtime | Informational - monitor for trends |
56+
| ChangesRecovered | Counter | Number of changes that were read during the recovery phase | Informational - monitor for trends |
57+
| MilliSecondsSinceLastAppliedChange | Gauge | Number of milliseconds since the last change was applied | Informational - monitor for trends |
58+
| MilliSecondsSinceLastRecoveredChange | Gauge | Number of milliseconds since the last change was recovered from the history store | Informational - monitor for trends |
59+
| RecoveryStartTime | Gauge | Time in epoch milliseconds when recovery started (-1 if not applicable) | Informational - monitor for trends |
6060
| **Connection and State Metrics** | | | |
6161
| Connected | Gauge | Whether the connector is currently connected to the database (1=connected, 0=disconnected) | **Critical Alert**: Alert if value = 0 (disconnected) |
6262
| **Queue Metrics** | | | |
63-
| CurrentQueueSizeInBytes | Gauge | Current size of the connector's internal queue in bytes | Alert if > 80% of MaxQueueSizeInBytes |
63+
| CurrentQueueSizeInBytes | Gauge | Current size of the connector's internal queue in bytes | Informational - monitor for trends |
6464
| MaxQueueSizeInBytes | Gauge | Maximum configured size of the connector's internal queue in bytes | Informational - use for capacity planning |
65-
| QueueRemainingCapacity | Gauge | Remaining capacity of the connector's internal queue | **High Priority**: Alert if < 20% of total capacity |
65+
| QueueRemainingCapacity | Gauge | Remaining capacity of the connector's internal queue | Informational - monitor for trends |
6666
| QueueTotalCapacity | Gauge | Total capacity of the connector's internal queue | Informational - use for capacity planning |
6767
| **Streaming Performance Metrics** | | | |
68-
| MilliSecondsBehindSource | Gauge | Number of milliseconds the connector is behind the source database (-1 if not applicable) | **High Priority**: Alert if > 60,000ms (1 minute) behind source |
69-
| MilliSecondsSinceLastEvent | Gauge | Number of milliseconds since the connector processed the most recent event (-1 if not applicable) | **Critical Alert**: Alert if > 300,000ms (5 minutes) in active systems |
70-
| NumberOfCommittedTransactions | Counter | Number of committed transactions processed by the connector | Monitor rate - alert if drops to 0 for > 10 minutes in active systems |
71-
| NumberOfEventsFiltered | Counter | Number of events filtered by include/exclude list rules | Monitor rate for unexpected increases (> 50% of total events) |
68+
| MilliSecondsBehindSource | Gauge | Number of milliseconds the connector is behind the source database (-1 if not applicable) | Informational - monitor for trends and business SLA requirements |
69+
| MilliSecondsSinceLastEvent | Gauge | Number of milliseconds since the connector processed the most recent event (-1 if not applicable) | Informational - monitor for trends in active systems |
70+
| NumberOfCommittedTransactions | Counter | Number of committed transactions processed by the connector | Informational - monitor for trends |
71+
| NumberOfEventsFiltered | Counter | Number of events filtered by include/exclude list rules | Informational - monitor for trends |
7272
| **Event Counters** | | | |
73-
| TotalNumberOfCreateEventsSeen | Counter | Total number of CREATE (INSERT) events seen by the connector | Monitor rate for business logic validation |
74-
| TotalNumberOfDeleteEventsSeen | Counter | Total number of DELETE events seen by the connector | Monitor rate for business logic validation |
75-
| TotalNumberOfEventsSeen | Counter | Total number of events seen by the connector | **High Priority**: Alert if rate drops to 0 for > 10 minutes in active systems |
76-
| TotalNumberOfUpdateEventsSeen | Counter | Total number of UPDATE events seen by the connector | Monitor rate for business logic validation |
77-
| NumberOfErroneousEvents | Counter | Number of events that caused errors during processing | **Critical Alert**: Alert if > 0 (any errors) |
73+
| TotalNumberOfCreateEventsSeen | Counter | Total number of CREATE (INSERT) events seen by the connector | Informational - monitor for trends |
74+
| TotalNumberOfDeleteEventsSeen | Counter | Total number of DELETE events seen by the connector | Informational - monitor for trends |
75+
| TotalNumberOfEventsSeen | Counter | Total number of events seen by the connector | Informational - monitor for trends |
76+
| TotalNumberOfUpdateEventsSeen | Counter | Total number of UPDATE events seen by the connector | Informational - monitor for trends |
77+
| NumberOfErroneousEvents | Counter | Number of events that caused errors during processing | **Critical Alert**: Alert if > 0 (indicates processing failures) |
7878
| **Snapshot Metrics** | | | |
79-
| RemainingTableCount | Gauge | Number of tables remaining to be processed during snapshot | Monitor for stuck snapshots (no change for > 30 minutes) |
80-
| RowsScanned | Counter | Number of rows scanned per table during snapshot (reported per table) | Monitor rate for progress tracking |
81-
| SnapshotAborted | Gauge | Whether the snapshot was aborted (1=aborted, 0=not aborted) | **Critical Alert**: Alert if value = 1 (aborted) |
82-
| SnapshotCompleted | Gauge | Whether the snapshot completed successfully (1=completed, 0=not completed) | Monitor for successful completion |
83-
| SnapshotDurationInSeconds | Gauge | Total duration of the snapshot process in seconds | Alert if exceeds expected duration (> 4 hours for large datasets) |
84-
| SnapshotPaused | Gauge | Whether the snapshot is currently paused (1=paused, 0=not paused) | Alert if paused unexpectedly (value = 1) |
85-
| SnapshotPausedDurationInSeconds | Gauge | Total time the snapshot was paused in seconds | Alert if paused > 1800 seconds (30 minutes) |
86-
| SnapshotRunning | Gauge | Whether a snapshot is currently running (1=running, 0=not running) | Monitor for unexpected state changes |
79+
| RemainingTableCount | Gauge | Number of tables remaining to be processed during snapshot | Informational - monitor snapshot progress |
80+
| RowsScanned | Counter | Number of rows scanned per table during snapshot (reported per table) | Informational - monitor snapshot progress |
81+
| SnapshotAborted | Gauge | Whether the snapshot was aborted (1=aborted, 0=not aborted) | **Critical Alert**: Alert if value = 1 (snapshot failed) |
82+
| SnapshotCompleted | Gauge | Whether the snapshot completed successfully (1=completed, 0=not completed) | Informational - monitor snapshot completion |
83+
| SnapshotDurationInSeconds | Gauge | Total duration of the snapshot process in seconds | Informational - monitor for performance trends |
84+
| SnapshotPaused | Gauge | Whether the snapshot is currently paused (1=paused, 0=not paused) | Informational - monitor snapshot state |
85+
| SnapshotPausedDurationInSeconds | Gauge | Total time the snapshot was paused in seconds | Informational - monitor snapshot state |
86+
| SnapshotRunning | Gauge | Whether a snapshot is currently running (1=running, 0=not running) | Informational - monitor snapshot state |
8787
| TotalTableCount | Gauge | Total number of tables included in the snapshot | Informational - use for progress calculation |
8888

8989
{{< note >}}
@@ -102,17 +102,17 @@ RDI reports with their descriptions.
102102

103103
| Metric Name | Metric Type | Metric Description | Alerting Recommendations |
104104
|-------------|-------------|--------------------|-----------------------|
105-
| `incoming_records_total` | Counter | Total number of incoming records processed by the system | **High Priority**: Alert if rate drops to 0 for > 10 minutes in active systems |
105+
| `incoming_records_total` | Counter | Total number of incoming records processed by the system | Informational - monitor for trends |
106106
| `incoming_records_created` | Gauge | Timestamp when the incoming records counter was created | Informational - no alerting needed |
107-
| `processed_records_total` | Counter | Total number of records that have been successfully processed | Monitor processing rate - alert if significantly slower than incoming rate |
108-
| `rejected_records_total` | Counter | Total number of records that were rejected during processing | **Critical Alert**: Alert if > 0 (any rejections indicate data quality issues) |
109-
| `filtered_records_total` | Counter | Total number of records that were filtered out during processing | Monitor rate - alert if > 50% of incoming records are filtered |
110-
| `rdi_engine_state` | Gauge | Current state of the RDI engine with labels for `state` (e.g., STARTED, RUNNING) and `sync_mode` (e.g., SNAPSHOT, STREAMING) | **Critical Alert**: Alert if state != "RUNNING" for > 5 minutes |
107+
| `processed_records_total` | Counter | Total number of records that have been successfully processed | Informational - monitor for trends |
108+
| `rejected_records_total` | Counter | Total number of records that were rejected during processing | **Critical Alert**: Alert if > 0 (indicates processing failures) |
109+
| `filtered_records_total` | Counter | Total number of records that were filtered out during processing | Informational - monitor for trends |
110+
| `rdi_engine_state` | Gauge | Current state of the RDI engine with labels for `state` (e.g., STARTED, RUNNING) and `sync_mode` (e.g., SNAPSHOT, STREAMING) | **Critical Alert**: Alert if state indicates failure or error condition |
111111
| `rdi_version_info` | Gauge | Version information for RDI components with labels for `cli` and `engine` versions | Informational - use for version tracking |
112112
| `monitor_time_elapsed_total` | Counter | Total time elapsed (in seconds) since monitoring started | Informational - use for uptime tracking |
113113
| `monitor_time_elapsed_created` | Gauge | Timestamp when the monitor time elapsed counter was created | Informational - no alerting needed |
114-
| `rdi_incoming_entries` | Gauge | Count of incoming events by `data_source` and `operation` type (pending, inserted, updated, deleted, filtered, rejected) | **High Priority**: Alert if "rejected" > 0 or "pending" accumulates without processing |
115-
| `rdi_stream_event_latency_ms` | Gauge | Latency in milliseconds of the oldest event in each data stream, labeled by `data_source` | **High Priority**: Alert if > 60,000ms (1 minute) for real-time use cases |
114+
| `rdi_incoming_entries` | Gauge | Count of incoming events by `data_source` and `operation` type (pending, inserted, updated, deleted, filtered, rejected) | Informational - monitor for trends, alert only on "rejected" > 0 |
115+
| `rdi_stream_event_latency_ms` | Gauge | Latency in milliseconds of the oldest event in each data stream, labeled by `data_source` | Informational - monitor based on business SLA requirements |
116116

117117
{{< note >}}
118118
**Additional information about stream processor metrics:**
@@ -125,32 +125,42 @@ RDI reports with their descriptions.
125125

126126
## Recommended alerting strategy
127127

128-
Based on operational experience, the following metrics require immediate attention:
128+
The following alerting strategy focuses on system failures and data integrity issues that require immediate attention. Most metrics are informational and should be monitored for trends rather than triggering alerts.
129129

130130
### Critical alerts (immediate response required)
131-
- **`Connected = 0`**: Database connectivity lost
132-
- **`NumberOfErroneousEvents > 0`**: Data processing errors occurring
133-
- **`rejected_records_total > 0`**: Records being rejected (data quality issues)
134-
- **`SnapshotAborted = 1`**: Snapshot process failed
135-
- **`rdi_engine_state != "RUNNING"`**: RDI engine not in expected state
136-
137-
### High priority alerts (response within 15 minutes)
138-
- **`MilliSecondsBehindSource > 60000`**: Replication lag exceeding 1 minute
139-
- **`MilliSecondsSinceLastEvent > 300000`**: No events processed for 5+ minutes
140-
- **`QueueRemainingCapacity < 20%`**: Queue capacity critically low
141-
- **`rdi_stream_event_latency_ms > 60000`**: Event processing latency too high
142-
- **`TotalNumberOfEventsSeen` rate = 0**: No events flowing for 10+ minutes
143-
144-
### Medium priority alerts (response within 1 hour)
145-
- **Queue utilization > 80%**: Approaching capacity limits
146-
- **Snapshot duration > expected baseline**: Performance degradation
147-
- **High filtering rate (> 50%)**: Potential configuration issues
131+
132+
These are the only alerts that should wake someone up or require immediate action:
133+
134+
- **`Connected = 0`**: Database connectivity lost - RDI cannot function without database connection
135+
- **`NumberOfErroneousEvents > 0`**: Data processing errors occurring - indicates data corruption or processing failures
136+
- **`rejected_records_total > 0`**: Records being rejected - indicates data quality issues or processing failures
137+
- **`SnapshotAborted = 1`**: Snapshot process failed - initial sync is incomplete
138+
- **`rdi_engine_state`**: Alert only if the state indicates a clear failure condition (not just "not running")
139+
140+
### Important monitoring (but not alerts)
141+
142+
These metrics should be monitored on dashboards and reviewed regularly, but do not require automated alerts:
143+
144+
- **Queue metrics**: Queue utilization can vary widely and hitting 0% or 100% capacity may be normal during certain operations
145+
- **Latency metrics**: Lag and processing times depend heavily on business requirements and normal operational patterns
146+
- **Event counters**: Event rates naturally vary based on application usage patterns
147+
- **Snapshot progress**: Snapshot duration and progress depend on data size and are typically monitored manually
148+
- **Schema changes**: Schema change frequency is highly application-dependent
149+
150+
### Key principles for RDI alerting
151+
152+
1. **Alert on failures, not performance**: Focus alerts on system failures rather than performance degradation
153+
2. **Business context matters**: Latency and throughput requirements vary significantly between organizations
154+
3. **Establish baselines first**: Monitor metrics for weeks before setting any threshold-based alerts
155+
4. **Avoid alert fatigue**: Too many alerts reduce response to truly critical issues
156+
5. **Use dashboards for trends**: Most metrics are better suited for dashboard monitoring than alerting
148157

149158
### Monitoring best practices
150-
- Set up alerting rules in your monitoring system (Prometheus Alertmanager, Grafana, etc.)
151-
- Use rate() functions for counter metrics to detect changes in processing patterns
152-
- Establish baseline values for your specific workload before setting thresholds
153-
- Consider business hours and maintenance windows when configuring alert schedules
159+
160+
- **Dashboard-first approach**: Use Grafana dashboards to visualize trends and patterns
161+
- **Baseline establishment**: Monitor your specific workload for 2-4 weeks before considering additional alerts
162+
- **Business SLA alignment**: Only create alerts for metrics that directly impact your business SLA requirements
163+
- **Manual review**: Regularly review metric trends during business reviews rather than automated alerting
154164

155165
## RDI logs
156166

0 commit comments

Comments
 (0)