feat(csharp): implement CircuitBreakerManager (WI-3.5)#168
feat(csharp): implement CircuitBreakerManager (WI-3.5)#168jadewang-db wants to merge 8 commits intomainfrom
Conversation
ee7a037 to
4d0aa4a
Compare
4d0aa4a to
6b66d37
Compare
## 🥞 Stacked PR Use this [link](https://github.com/adbc-drivers/databricks/pull/161/files) to review incremental changes. - [**stack/wi-1.2-tag-definition-system**](#161) [[Files changed](https://github.com/adbc-drivers/databricks/pull/161/files)] - [stack/wi-2.1-telemetry-data-models](#162) [[Files changed](https://github.com/adbc-drivers/databricks/pull/162/files/ab7fa964ff62f3fc9884034e17a7e57630fa8037..a566292aec78d19717c92e28f135535b09f25c80)] - [stack/wi-2.1-exception-classifier](#163) [[Files changed](https://github.com/adbc-drivers/databricks/pull/163/files/a566292aec78d19717c92e28f135535b09f25c80..baa7a2ae32662fddc65272e0264e8bb7d1644716)] - [stack/wi-3.1-circuit-breaker](#164) [[Files changed](https://github.com/adbc-drivers/databricks/pull/164/files/baa7a2ae32662fddc65272e0264e8bb7d1644716..03f7027e6731efe032c15555afe517ba49de3651)] - [stack/wi-3.1-feature-flag-cache](#165) [[Files changed](https://github.com/adbc-drivers/databricks/pull/165/files/03f7027e6731efe032c15555afe517ba49de3651..1d6e3d5b1c4c31ec91361337e574e6e5411fbbb6)] - [stack/wi-3.4-databricks-telemetry-exporter](#166) [[Files changed](https://github.com/adbc-drivers/databricks/pull/166/files/1d6e3d5b1c4c31ec91361337e574e6e5411fbbb6..eb382cb291c120a5f3cc3a1c38e0975b99c1369f)] - [stack/wi-3.5-metrics-aggregator](#167) [[Files changed](https://github.com/adbc-drivers/databricks/pull/167/files/eb382cb291c120a5f3cc3a1c38e0975b99c1369f..67723fabe6f62d7ed16591c3e88e96aa269daddd)] - [stack/wi-3.5-circuit-breaker-manager](#168) [[Files changed](https://github.com/adbc-drivers/databricks/pull/168/files/67723fabe6f62d7ed16591c3e88e96aa269daddd..6b66d37e9d97ca621d88c48a58ac60b2487425ea)] - [stack/e2e-feature-flag-cache-tests](#169) [[Files changed](https://github.com/adbc-drivers/databricks/pull/169/files/6b66d37e9d97ca621d88c48a58ac60b2487425ea..2a6fff2b9b91c7fd6cff7558d1d3b3596c0fa3c2)] - [stack/databricks-activity-listener](#170) [[Files changed](https://github.com/adbc-drivers/databricks/pull/170/files/2a6fff2b9b91c7fd6cff7558d1d3b3596c0fa3c2..39f6aed55278a533390e9aadf655f80dc11159c2)] - [stack/circuit-breaker-telemetry-exporter](#171) [[Files changed](https://github.com/adbc-drivers/databricks/pull/171/files/39f6aed55278a533390e9aadf655f80dc11159c2..4473de5ca3cfca8579818e6d58f8a2b12e869a47)] - [stack/telemetry-client-manager-wi-3.2](#172) [[Files changed](https://github.com/adbc-drivers/databricks/pull/172/files/4473de5ca3cfca8579818e6d58f8a2b12e869a47..94b678636d76a6d41a6612f76d00b4caccdab48a)] - [stack/telemetry-client-wi-5.5](#173) [[Files changed](https://github.com/adbc-drivers/databricks/pull/173/files/94b678636d76a6d41a6612f76d00b4caccdab48a..ce00998cbd0372d94303ad1d69e9711e4489fe96)] - [stack/telemetry-client-manager-e2e-wi-7](#174) [[Files changed](https://github.com/adbc-drivers/databricks/pull/174/files/ce00998cbd0372d94303ad1d69e9711e4489fe96..2646e86223ff1e7706b20d5970e556ec2f17867b)] - [stack/telemetry-client-e2e-tests-wi-7-standalone](#175) [[Files changed](https://github.com/adbc-drivers/databricks/pull/175/files/2646e86223ff1e7706b20d5970e556ec2f17867b..0b9ebd3867250d92d0d8007cb17d6ce471d5560a)] - [stack/wi-6.1-databricks-connection-telemetry-integration](#176) [[Files changed](https://github.com/adbc-drivers/databricks/pull/176/files/0b9ebd3867250d92d0d8007cb17d6ce471d5560a..4f553284c30eb7efcf67369c58dddd56675cd0be)] - [stack/wi-6.2-telemetry-tags-driver-activities](#177) [[Files changed](https://github.com/adbc-drivers/databricks/pull/177/files/4f553284c30eb7efcf67369c58dddd56675cd0be..1f7cde0c5642072b06588665b16ee3a30a90d256)] - [stack/wi-9-full-integration-e2e-tests](#178) [[Files changed](https://github.com/adbc-drivers/databricks/pull/178/files/1f7cde0c5642072b06588665b16ee3a30a90d256..c65e9fea7c65fa456f0114e95c867ee15f21bd87)] --------- --------- Co-authored-by: Jade Wang <jade.wang+data@databricks.com> Co-authored-by: Claude <noreply@anthropic.com>
6b66d37 to
de8757a
Compare
## 🥞 Stacked PR Use this [link](https://github.com/adbc-drivers/databricks/pull/162/files) to review incremental changes. - [**stack/wi-2.1-telemetry-data-models**](#162) [[Files changed](https://github.com/adbc-drivers/databricks/pull/162/files)] - [stack/wi-2.1-exception-classifier](#163) [[Files changed](https://github.com/adbc-drivers/databricks/pull/163/files/1e58d3c3785fa7ec1b83da01f80ddea1f6167851..0dac01831e7d9d313c67dc31e4aacceb17e74298)] - [stack/wi-3.1-circuit-breaker](#164) [[Files changed](https://github.com/adbc-drivers/databricks/pull/164/files/0dac01831e7d9d313c67dc31e4aacceb17e74298..59b0221cb4c9262d80a35041a2f1098376f6e19e)] - [stack/wi-3.1-feature-flag-cache](#165) [[Files changed](https://github.com/adbc-drivers/databricks/pull/165/files/59b0221cb4c9262d80a35041a2f1098376f6e19e..8c30fc0649b09bc38e09cfd4d6875d66963ff6c0)] - [stack/wi-3.4-databricks-telemetry-exporter](#166) [[Files changed](https://github.com/adbc-drivers/databricks/pull/166/files/8c30fc0649b09bc38e09cfd4d6875d66963ff6c0..a6e926c8017e9a3b3b6de31bbbafb367adaba884)] - [stack/wi-3.5-metrics-aggregator](#167) [[Files changed](https://github.com/adbc-drivers/databricks/pull/167/files/a6e926c8017e9a3b3b6de31bbbafb367adaba884..c53df5d3c0124c490b920e1e1a611dd9c24e02a4)] - [stack/wi-3.5-circuit-breaker-manager](#168) [[Files changed](https://github.com/adbc-drivers/databricks/pull/168/files/c53df5d3c0124c490b920e1e1a611dd9c24e02a4..de8757a697dd023628011d1aff9961896560bc95)] - [stack/e2e-feature-flag-cache-tests](#169) [[Files changed](https://github.com/adbc-drivers/databricks/pull/169/files/de8757a697dd023628011d1aff9961896560bc95..0b77f8373958342da429c20f7e30c02105402331)] - [stack/databricks-activity-listener](#170) [[Files changed](https://github.com/adbc-drivers/databricks/pull/170/files/0b77f8373958342da429c20f7e30c02105402331..9090bdefba63d6c7fbff45bf60c2c63668f3884e)] - [stack/circuit-breaker-telemetry-exporter](#171) [[Files changed](https://github.com/adbc-drivers/databricks/pull/171/files/9090bdefba63d6c7fbff45bf60c2c63668f3884e..0a0159524a429726078bd7340057672d6927d1cd)] - [stack/telemetry-client-manager-wi-3.2](#172) [[Files changed](https://github.com/adbc-drivers/databricks/pull/172/files/0a0159524a429726078bd7340057672d6927d1cd..75039c6574c2dc437f5d670e71b938b98719c06f)] - [stack/telemetry-client-wi-5.5](#173) [[Files changed](https://github.com/adbc-drivers/databricks/pull/173/files/75039c6574c2dc437f5d670e71b938b98719c06f..254cdc75487f3e9344d3df6fb9b9cbf49fd03228)] - [stack/telemetry-client-manager-e2e-wi-7](#174) [[Files changed](https://github.com/adbc-drivers/databricks/pull/174/files/254cdc75487f3e9344d3df6fb9b9cbf49fd03228..7371da59309d109e8d457f4c27edd13adfa38a2c)] - [stack/telemetry-client-e2e-tests-wi-7-standalone](#175) [[Files changed](https://github.com/adbc-drivers/databricks/pull/175/files/7371da59309d109e8d457f4c27edd13adfa38a2c..5ff7e96827faa69e8bae1d5b5da06a9f95b91a8c)] - [stack/wi-6.1-databricks-connection-telemetry-integration](#176) [[Files changed](https://github.com/adbc-drivers/databricks/pull/176/files/5ff7e96827faa69e8bae1d5b5da06a9f95b91a8c..7757345889dbfd0b1dcb22556e2e6c746d7fa0f0)] - [stack/wi-6.2-telemetry-tags-driver-activities](#177) [[Files changed](https://github.com/adbc-drivers/databricks/pull/177/files/7757345889dbfd0b1dcb22556e2e6c746d7fa0f0..2364122ad5402c9205008f39acaec6a400a4db98)] - [stack/wi-9-full-integration-e2e-tests](#178) [[Files changed](https://github.com/adbc-drivers/databricks/pull/178/files/2364122ad5402c9205008f39acaec6a400a4db98..698f3ea13f65a17b62385be8e8e4032497f88993)] --------- --------- Co-authored-by: Jade Wang <jade.wang+data@databricks.com> Co-authored-by: Claude <noreply@anthropic.com>
de8757a to
61758b9
Compare
61758b9 to
295510e
Compare
f25510c to
d0385b2
Compare
## 🥞 Stacked PR Use this [link](https://github.com/adbc-drivers/databricks/pull/165/files) to review incremental changes. - [**stack/wi-3.1-feature-flag-cache**](#165) [[Files changed](https://github.com/adbc-drivers/databricks/pull/165/files)] - [stack/wi-3.4-databricks-telemetry-exporter](#166) [[Files changed](https://github.com/adbc-drivers/databricks/pull/166/files/2262e9d512776719d02355190bd8824522537c6a..109f1af5faf78b5fbbdf4620a321b811266f63af)] - [stack/wi-3.5-metrics-aggregator](#167) [[Files changed](https://github.com/adbc-drivers/databricks/pull/167/files/109f1af5faf78b5fbbdf4620a321b811266f63af..74af04194edcbce64f66efdf45944db1ed384727)] - [stack/wi-3.5-circuit-breaker-manager](#168) [[Files changed](https://github.com/adbc-drivers/databricks/pull/168/files/74af04194edcbce64f66efdf45944db1ed384727..d0385b2145cda82c5fe0fc8311b0d64e9a6b05bc)] - [stack/e2e-feature-flag-cache-tests](#169) [[Files changed](https://github.com/adbc-drivers/databricks/pull/169/files/d0385b2145cda82c5fe0fc8311b0d64e9a6b05bc..9017bc0626121281aa0bc9fc097edd3e9d98e1c0)] - [stack/databricks-activity-listener](#170) [[Files changed](https://github.com/adbc-drivers/databricks/pull/170/files/9017bc0626121281aa0bc9fc097edd3e9d98e1c0..200014e9b7557de490944e5889fb197dc00edb41)] - [stack/circuit-breaker-telemetry-exporter](#171) [[Files changed](https://github.com/adbc-drivers/databricks/pull/171/files/200014e9b7557de490944e5889fb197dc00edb41..e886588830c528ab644b38004337e7bfc1f12c31)] - [stack/telemetry-client-manager-wi-3.2](#172) [[Files changed](https://github.com/adbc-drivers/databricks/pull/172/files/e886588830c528ab644b38004337e7bfc1f12c31..7d1e0741f4942c15e8b68f64a8199ebdae678758)] - [stack/telemetry-client-wi-5.5](#173) [[Files changed](https://github.com/adbc-drivers/databricks/pull/173/files/7d1e0741f4942c15e8b68f64a8199ebdae678758..d2a7725cd098a05ac0b9b6a0135c34823b18c1ab)] - [stack/telemetry-client-manager-e2e-wi-7](#174) [[Files changed](https://github.com/adbc-drivers/databricks/pull/174/files/d2a7725cd098a05ac0b9b6a0135c34823b18c1ab..66a44b1a372e4c27c924002a47ef755a10e7eaec)] - [stack/telemetry-client-e2e-tests-wi-7-standalone](#175) [[Files changed](https://github.com/adbc-drivers/databricks/pull/175/files/66a44b1a372e4c27c924002a47ef755a10e7eaec..276cbbd79b55ed423ce772d98901a14c4dd7054d)] - [stack/wi-6.1-databricks-connection-telemetry-integration](#176) [[Files changed](https://github.com/adbc-drivers/databricks/pull/176/files/276cbbd79b55ed423ce772d98901a14c4dd7054d..e76a4a88dfb70a0bae460d4f976bb8332295bba5)] - [stack/wi-6.2-telemetry-tags-driver-activities](#177) [[Files changed](https://github.com/adbc-drivers/databricks/pull/177/files/e76a4a88dfb70a0bae460d4f976bb8332295bba5..b647e8c57b676691f60defa79b160945894e5900)] - [stack/wi-9-full-integration-e2e-tests](#178) [[Files changed](https://github.com/adbc-drivers/databricks/pull/178/files/b647e8c57b676691f60defa79b160945894e5900..b664d4cbe3b284e4415729bd34ccbc66e3f3ca67)] --------- --------- Co-authored-by: Jade Wang <jade.wang+data@databricks.com> Co-authored-by: Claude <noreply@anthropic.com>
d0385b2 to
09a3ee9
Compare
## 🥞 Stacked PR Use this [link](https://github.com/adbc-drivers/databricks/pull/166/files) to review incremental changes. - [**stack/wi-3.4-databricks-telemetry-exporter**](#166) [[Files changed](https://github.com/adbc-drivers/databricks/pull/166/files)] - [stack/wi-3.5-metrics-aggregator](#167) [[Files changed](https://github.com/adbc-drivers/databricks/pull/167/files/90b20fcc9c10e7e7168957568dd2bb2b9bc233e5..33406c1ba7e85bc6e1840b42de1a1d4e57c4b84a)] - [stack/wi-3.5-circuit-breaker-manager](#168) [[Files changed](https://github.com/adbc-drivers/databricks/pull/168/files/33406c1ba7e85bc6e1840b42de1a1d4e57c4b84a..09a3ee956ad1423fdc8f181da667b305d9e3483f)] - [stack/e2e-feature-flag-cache-tests](#169) [[Files changed](https://github.com/adbc-drivers/databricks/pull/169/files/09a3ee956ad1423fdc8f181da667b305d9e3483f..4fe0016513d084b72c82a5195b0bf6f1e50f2f80)] - [stack/databricks-activity-listener](#170) [[Files changed](https://github.com/adbc-drivers/databricks/pull/170/files/4fe0016513d084b72c82a5195b0bf6f1e50f2f80..f8bb1099aa0e7e086e52eb8306f9edd18f2b8578)] - [stack/circuit-breaker-telemetry-exporter](#171) [[Files changed](https://github.com/adbc-drivers/databricks/pull/171/files/f8bb1099aa0e7e086e52eb8306f9edd18f2b8578..383d3efd09800816b7610563dd91d6686bcb1484)] - [stack/telemetry-client-manager-wi-3.2](#172) [[Files changed](https://github.com/adbc-drivers/databricks/pull/172/files/383d3efd09800816b7610563dd91d6686bcb1484..cda26685fbce49c0c96c365f9dd26c70bcc65c13)] - [stack/telemetry-client-wi-5.5](#173) [[Files changed](https://github.com/adbc-drivers/databricks/pull/173/files/cda26685fbce49c0c96c365f9dd26c70bcc65c13..695232b24ea01751613750f9fd5c9ad6717616eb)] - [stack/telemetry-client-manager-e2e-wi-7](#174) [[Files changed](https://github.com/adbc-drivers/databricks/pull/174/files/695232b24ea01751613750f9fd5c9ad6717616eb..017930e03ae1fa36b7b36a0ca2e9c81fcecd7fa6)] - [stack/telemetry-client-e2e-tests-wi-7-standalone](#175) [[Files changed](https://github.com/adbc-drivers/databricks/pull/175/files/017930e03ae1fa36b7b36a0ca2e9c81fcecd7fa6..abfc895fb1566a5f38b144a1bbbd506a90374560)] - [stack/wi-6.1-databricks-connection-telemetry-integration](#176) [[Files changed](https://github.com/adbc-drivers/databricks/pull/176/files/abfc895fb1566a5f38b144a1bbbd506a90374560..9ae97e732ed4bbd9d0667be7d9a724dee7380d6b)] - [stack/wi-6.2-telemetry-tags-driver-activities](#177) [[Files changed](https://github.com/adbc-drivers/databricks/pull/177/files/9ae97e732ed4bbd9d0667be7d9a724dee7380d6b..3f776cecad4c8deded27d877aed857d6da826baf)] - [stack/wi-9-full-integration-e2e-tests](#178) [[Files changed](https://github.com/adbc-drivers/databricks/pull/178/files/3f776cecad4c8deded27d877aed857d6da826baf..197a9ee8a9d746b68848ea8006f9669ecee875e3)] --------- --------- Co-authored-by: Jade Wang <jade.wang+data@databricks.com> Co-authored-by: Claude <noreply@anthropic.com>
09a3ee9 to
1b84ad1
Compare
Code Review Summary🔴 Critical Issues (Must Fix)1. Missing The
The existing public CircuitBreaker(int failureThreshold = 5, TimeSpan? timeout = null)Required fix: Create the
🟡 Medium Priority Issues2. Use structured tracing instead of Debug.WriteLine (lines 84, 118, 186, 199) Per C#/.NET best practices, use Example fix: // Instead of:
Debug.WriteLine($"[DEBUG] CircuitBreakerManager: Creating circuit breaker for host '{host}'");
// Use:
Activity.Current?.AddEvent(new ActivityEvent("CircuitBreakerCreated", tags: new ActivityTagsCollection
{
{ "host", host }
}));✅ Positive Observations
📋 Recommendations (Priority Order)
|
|
✅ All issues addressed: 🔴 Critical Issues - FIXED1. Missing
🟡 Medium Priority Issues - FIXED2. Replaced
✅ VerificationAll 24 CircuitBreaker tests pass. |
1b84ad1 to
5b50744
Compare
| /// Gets or sets the number of consecutive successes required in half-open state to close the circuit. | ||
| /// </summary> | ||
| /// <value>Default is 2.</value> | ||
| public int SuccessThreshold { get; set; } = DefaultSuccessThreshold; |
samikshya-db
left a comment
There was a problem hiding this comment.
Minor comments, LGTM otherwise
Implement MetricsAggregator that aggregates Activity data by statement_id and handles exception buffering with terminal vs retryable classification. Key features: - ProcessActivity extracts tags and aggregates by statement_id using ConcurrentDictionary<string, StatementTelemetryContext> - CompleteStatement emits aggregated TelemetryEvent - RecordException flushes terminal exceptions immediately - RecordException buffers retryable exceptions until CompleteStatement - FlushAsync exports when batch size or time interval reached - Uses TelemetryTagRegistry to filter tags - Creates TelemetryFrontendLog wrapper with workspace_id - All exceptions swallowed and logged at TRACE level Implementation details: - Connection events emit immediately (no aggregation needed) - Statement events aggregate until CompleteStatement is called - Timer-based periodic flush using System.Threading.Timer - Thread-safe aggregation using ConcurrentDictionary - Nested StatementTelemetryContext holds aggregated metrics and buffered exceptions per statement Test coverage: - 29 unit tests covering all exit criteria - Tests for exception handling, tag filtering, frontend log wrapping - End-to-end statement lifecycle tests Co-Authored-By: Claude <noreply@anthropic.com>
5b50744 to
428aa6f
Compare
Implement CircuitBreakerManager as a singleton that manages circuit breakers per host. Each host gets its own circuit breaker instance for isolation, preventing one failing endpoint from affecting others. Key features: - Singleton pattern with GetInstance() method - Per-host circuit breaker isolation using ConcurrentDictionary - Thread-safe concurrent access - Case-insensitive host matching - Support for both default and custom configurations This follows the JDBC driver pattern in CircuitBreakerManager.java. Co-Authored-By: Claude <noreply@anthropic.com>
Update all CircuitBreaker tests to use the new constructor that takes a CircuitBreakerConfig object instead of individual parameters. Changes: - Add helper methods CreateDefaultConfig() and CreateConfig() for cleaner test setup - Replace `new CircuitBreaker()` with `new CircuitBreaker(CreateDefaultConfig())` - Replace `new CircuitBreaker(failureThreshold: N)` with `new CircuitBreaker(CreateConfig(failureThreshold: N))` - Replace `new CircuitBreaker(failureThreshold: N, timeout: T)` with `new CircuitBreaker(CreateConfig(failureThreshold: N, timeout: T))` - Add new tests for constructor with null config and config storage Co-Authored-By: Claude <noreply@anthropic.com>
428aa6f to
b9270bc
Compare
🥞 Stacked PR
Use this link to review incremental changes.