Skip to content

Commit d2d4c54

Browse files
jadewang-dbJade Wangclaude
authored
feat(csharp): implement Tag Definition System (WI-1.2) (#161)
## 🥞 Stacked PR Use this [link](https://github.com/adbc-drivers/databricks/pull/161/files) to review incremental changes. - [**stack/wi-1.2-tag-definition-system**](#161) [[Files changed](https://github.com/adbc-drivers/databricks/pull/161/files)] - [stack/wi-2.1-telemetry-data-models](#162) [[Files changed](https://github.com/adbc-drivers/databricks/pull/162/files/ab7fa964ff62f3fc9884034e17a7e57630fa8037..a566292aec78d19717c92e28f135535b09f25c80)] - [stack/wi-2.1-exception-classifier](#163) [[Files changed](https://github.com/adbc-drivers/databricks/pull/163/files/a566292aec78d19717c92e28f135535b09f25c80..baa7a2ae32662fddc65272e0264e8bb7d1644716)] - [stack/wi-3.1-circuit-breaker](#164) [[Files changed](https://github.com/adbc-drivers/databricks/pull/164/files/baa7a2ae32662fddc65272e0264e8bb7d1644716..03f7027e6731efe032c15555afe517ba49de3651)] - [stack/wi-3.1-feature-flag-cache](#165) [[Files changed](https://github.com/adbc-drivers/databricks/pull/165/files/03f7027e6731efe032c15555afe517ba49de3651..1d6e3d5b1c4c31ec91361337e574e6e5411fbbb6)] - [stack/wi-3.4-databricks-telemetry-exporter](#166) [[Files changed](https://github.com/adbc-drivers/databricks/pull/166/files/1d6e3d5b1c4c31ec91361337e574e6e5411fbbb6..eb382cb291c120a5f3cc3a1c38e0975b99c1369f)] - [stack/wi-3.5-metrics-aggregator](#167) [[Files changed](https://github.com/adbc-drivers/databricks/pull/167/files/eb382cb291c120a5f3cc3a1c38e0975b99c1369f..67723fabe6f62d7ed16591c3e88e96aa269daddd)] - [stack/wi-3.5-circuit-breaker-manager](#168) [[Files changed](https://github.com/adbc-drivers/databricks/pull/168/files/67723fabe6f62d7ed16591c3e88e96aa269daddd..6b66d37e9d97ca621d88c48a58ac60b2487425ea)] - [stack/e2e-feature-flag-cache-tests](#169) [[Files changed](https://github.com/adbc-drivers/databricks/pull/169/files/6b66d37e9d97ca621d88c48a58ac60b2487425ea..2a6fff2b9b91c7fd6cff7558d1d3b3596c0fa3c2)] - [stack/databricks-activity-listener](#170) [[Files changed](https://github.com/adbc-drivers/databricks/pull/170/files/2a6fff2b9b91c7fd6cff7558d1d3b3596c0fa3c2..39f6aed55278a533390e9aadf655f80dc11159c2)] - [stack/circuit-breaker-telemetry-exporter](#171) [[Files changed](https://github.com/adbc-drivers/databricks/pull/171/files/39f6aed55278a533390e9aadf655f80dc11159c2..4473de5ca3cfca8579818e6d58f8a2b12e869a47)] - [stack/telemetry-client-manager-wi-3.2](#172) [[Files changed](https://github.com/adbc-drivers/databricks/pull/172/files/4473de5ca3cfca8579818e6d58f8a2b12e869a47..94b678636d76a6d41a6612f76d00b4caccdab48a)] - [stack/telemetry-client-wi-5.5](#173) [[Files changed](https://github.com/adbc-drivers/databricks/pull/173/files/94b678636d76a6d41a6612f76d00b4caccdab48a..ce00998cbd0372d94303ad1d69e9711e4489fe96)] - [stack/telemetry-client-manager-e2e-wi-7](#174) [[Files changed](https://github.com/adbc-drivers/databricks/pull/174/files/ce00998cbd0372d94303ad1d69e9711e4489fe96..2646e86223ff1e7706b20d5970e556ec2f17867b)] - [stack/telemetry-client-e2e-tests-wi-7-standalone](#175) [[Files changed](https://github.com/adbc-drivers/databricks/pull/175/files/2646e86223ff1e7706b20d5970e556ec2f17867b..0b9ebd3867250d92d0d8007cb17d6ce471d5560a)] - [stack/wi-6.1-databricks-connection-telemetry-integration](#176) [[Files changed](https://github.com/adbc-drivers/databricks/pull/176/files/0b9ebd3867250d92d0d8007cb17d6ce471d5560a..4f553284c30eb7efcf67369c58dddd56675cd0be)] - [stack/wi-6.2-telemetry-tags-driver-activities](#177) [[Files changed](https://github.com/adbc-drivers/databricks/pull/177/files/4f553284c30eb7efcf67369c58dddd56675cd0be..1f7cde0c5642072b06588665b16ee3a30a90d256)] - [stack/wi-9-full-integration-e2e-tests](#178) [[Files changed](https://github.com/adbc-drivers/databricks/pull/178/files/1f7cde0c5642072b06588665b16ee3a30a90d256..c65e9fea7c65fa456f0114e95c867ee15f21bd87)] --------- --------- Co-authored-by: Jade Wang <jade.wang+data@databricks.com> Co-authored-by: Claude <noreply@anthropic.com>
1 parent 0bdd6fd commit d2d4c54

File tree

8 files changed

+969
-0
lines changed

8 files changed

+969
-0
lines changed

csharp/doc/telemetry-sprint-plan.md

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -92,11 +92,14 @@ Implement the core telemetry infrastructure including feature flag management, p
9292
#### WI-1.2: Tag Definition System
9393
**Description**: Create centralized tag definitions with export scope annotations.
9494

95+
**Status**: ✅ **COMPLETED**
96+
9597
**Location**: `csharp/src/Telemetry/TagDefinitions/`
9698

9799
**Files**:
98100
- `TelemetryTag.cs` - Attribute and enums for export scope
99101
- `TelemetryTagRegistry.cs` - Central registry
102+
- `TelemetryEventType.cs` - Event type enum
100103
- `ConnectionOpenEvent.cs` - Connection event tags
101104
- `StatementExecutionEvent.cs` - Statement execution tags
102105
- `ErrorEvent.cs` - Error event tags
@@ -118,6 +121,20 @@ Implement the core telemetry infrastructure including feature flag management, p
118121
| Unit | `TelemetryTagRegistry_ShouldExportToDatabricks_SafeTag_ReturnsTrue` | EventType.StatementExecution, "statement.id" | true |
119122
| Unit | `ConnectionOpenEvent_GetDatabricksExportTags_ExcludesServerAddress` | N/A | Set does NOT contain "server.address" |
120123

124+
**Implementation Notes**:
125+
- Used `HashSet<string>` instead of `IReadOnlySet<string>` for netstandard2.0 compatibility
126+
- All tag definitions use the `[TelemetryTag]` attribute for metadata
127+
- Sensitive tags (server.address, db.statement, error.stack_trace) are marked with `ExportScope = TagExportScope.ExportLocal`
128+
- Comprehensive test coverage with 31 unit tests in `TelemetryTagRegistryTests.cs`
129+
- Test file location: `csharp/test/Unit/Telemetry/TagDefinitions/TelemetryTagRegistryTests.cs`
130+
131+
**Key Design Decisions**:
132+
1. **Flag-based enum**: `TagExportScope` uses `[Flags]` attribute to support combinations (ExportAll = ExportLocal | ExportDatabricks)
133+
2. **Privacy by design**: Sensitive tags are explicitly marked as local-only and excluded from `GetDatabricksExportTags()`
134+
3. **Static helper methods**: Each event class has `GetDatabricksExportTags()` to return the set of exportable tags
135+
4. **Registry pattern**: `TelemetryTagRegistry.ShouldExportToDatabricks()` provides centralized filtering logic
136+
5. **Unknown tags dropped**: Tags not in the registry are silently dropped for Databricks export (returns false)
137+
121138
---
122139

123140
### Phase 2: Per-Host Management
Lines changed: 123 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,123 @@
1+
/*
2+
* Copyright (c) 2025 ADBC Drivers Contributors
3+
*
4+
* Licensed under the Apache License, Version 2.0 (the "License");
5+
* you may not use this file except in compliance with the License.
6+
* You may obtain a copy of the License at
7+
*
8+
* http://www.apache.org/licenses/LICENSE-2.0
9+
*
10+
* Unless required by applicable law or agreed to in writing, software
11+
* distributed under the License is distributed on an "AS IS" BASIS,
12+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
* See the License for the specific language governing permissions and
14+
* limitations under the License.
15+
*/
16+
17+
using System.Collections.Generic;
18+
19+
namespace AdbcDrivers.Databricks.Telemetry.TagDefinitions
20+
{
21+
/// <summary>
22+
/// Tag definitions for Connection.Open events.
23+
/// </summary>
24+
public static class ConnectionOpenEvent
25+
{
26+
/// <summary>
27+
/// The event name for connection open operations.
28+
/// </summary>
29+
public const string EventName = "Connection.Open";
30+
31+
/// <summary>
32+
/// Databricks workspace ID.
33+
/// Exported to Databricks telemetry service.
34+
/// </summary>
35+
[TelemetryTag("workspace.id",
36+
ExportScope = TagExportScope.ExportDatabricks,
37+
Description = "Databricks workspace ID",
38+
Required = true)]
39+
public const string WorkspaceId = "workspace.id";
40+
41+
/// <summary>
42+
/// Connection session ID.
43+
/// Exported to Databricks telemetry service.
44+
/// </summary>
45+
[TelemetryTag("session.id",
46+
ExportScope = TagExportScope.ExportDatabricks,
47+
Description = "Connection session ID",
48+
Required = true)]
49+
public const string SessionId = "session.id";
50+
51+
/// <summary>
52+
/// ADBC driver version.
53+
/// Exported to all destinations.
54+
/// </summary>
55+
[TelemetryTag("driver.version",
56+
ExportScope = TagExportScope.ExportAll,
57+
Description = "ADBC driver version")]
58+
public const string DriverVersion = "driver.version";
59+
60+
/// <summary>
61+
/// Operating system information.
62+
/// Exported to all destinations.
63+
/// </summary>
64+
[TelemetryTag("driver.os",
65+
ExportScope = TagExportScope.ExportAll,
66+
Description = "Operating system")]
67+
public const string DriverOS = "driver.os";
68+
69+
/// <summary>
70+
/// .NET runtime version.
71+
/// Exported to all destinations.
72+
/// </summary>
73+
[TelemetryTag("driver.runtime",
74+
ExportScope = TagExportScope.ExportAll,
75+
Description = ".NET runtime version")]
76+
public const string DriverRuntime = "driver.runtime";
77+
78+
/// <summary>
79+
/// Whether CloudFetch is enabled.
80+
/// Exported to Databricks telemetry service.
81+
/// </summary>
82+
[TelemetryTag("feature.cloudfetch",
83+
ExportScope = TagExportScope.ExportDatabricks,
84+
Description = "CloudFetch enabled")]
85+
public const string FeatureCloudFetch = "feature.cloudfetch";
86+
87+
/// <summary>
88+
/// Whether LZ4 compression is enabled.
89+
/// Exported to Databricks telemetry service.
90+
/// </summary>
91+
[TelemetryTag("feature.lz4",
92+
ExportScope = TagExportScope.ExportDatabricks,
93+
Description = "LZ4 compression enabled")]
94+
public const string FeatureLz4 = "feature.lz4";
95+
96+
/// <summary>
97+
/// Workspace host address.
98+
/// Only exported to local diagnostics (contains potentially sensitive data).
99+
/// </summary>
100+
[TelemetryTag("server.address",
101+
ExportScope = TagExportScope.ExportLocal,
102+
Description = "Workspace host (local diagnostics only)")]
103+
public const string ServerAddress = "server.address";
104+
105+
/// <summary>
106+
/// Gets all tags that should be exported to Databricks telemetry service.
107+
/// </summary>
108+
/// <returns>A set of tag names for Databricks export.</returns>
109+
public static HashSet<string> GetDatabricksExportTags()
110+
{
111+
return new HashSet<string>
112+
{
113+
WorkspaceId,
114+
SessionId,
115+
DriverVersion,
116+
DriverOS,
117+
DriverRuntime,
118+
FeatureCloudFetch,
119+
FeatureLz4
120+
};
121+
}
122+
}
123+
}
Lines changed: 102 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,102 @@
1+
/*
2+
* Copyright (c) 2025 ADBC Drivers Contributors
3+
*
4+
* Licensed under the Apache License, Version 2.0 (the "License");
5+
* you may not use this file except in compliance with the License.
6+
* You may obtain a copy of the License at
7+
*
8+
* http://www.apache.org/licenses/LICENSE-2.0
9+
*
10+
* Unless required by applicable law or agreed to in writing, software
11+
* distributed under the License is distributed on an "AS IS" BASIS,
12+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
* See the License for the specific language governing permissions and
14+
* limitations under the License.
15+
*/
16+
17+
using System.Collections.Generic;
18+
19+
namespace AdbcDrivers.Databricks.Telemetry.TagDefinitions
20+
{
21+
/// <summary>
22+
/// Tag definitions for Error events.
23+
/// </summary>
24+
public static class ErrorEvent
25+
{
26+
/// <summary>
27+
/// The event name for error events.
28+
/// </summary>
29+
public const string EventName = "Error";
30+
31+
/// <summary>
32+
/// Error type or exception type name.
33+
/// Exported to Databricks telemetry service.
34+
/// </summary>
35+
[TelemetryTag("error.type",
36+
ExportScope = TagExportScope.ExportDatabricks,
37+
Description = "Error type or exception type",
38+
Required = true)]
39+
public const string ErrorType = "error.type";
40+
41+
/// <summary>
42+
/// Error message (sanitized, no PII).
43+
/// Exported to Databricks telemetry service.
44+
/// </summary>
45+
[TelemetryTag("error.message",
46+
ExportScope = TagExportScope.ExportDatabricks,
47+
Description = "Error message (sanitized)")]
48+
public const string ErrorMessage = "error.message";
49+
50+
/// <summary>
51+
/// HTTP status code if applicable.
52+
/// Exported to Databricks telemetry service.
53+
/// </summary>
54+
[TelemetryTag("error.http_status",
55+
ExportScope = TagExportScope.ExportDatabricks,
56+
Description = "HTTP status code")]
57+
public const string ErrorHttpStatus = "error.http_status";
58+
59+
/// <summary>
60+
/// Statement ID associated with the error.
61+
/// Exported to Databricks telemetry service.
62+
/// </summary>
63+
[TelemetryTag("statement.id",
64+
ExportScope = TagExportScope.ExportDatabricks,
65+
Description = "Statement ID associated with error")]
66+
public const string StatementId = "statement.id";
67+
68+
/// <summary>
69+
/// Session ID associated with the error.
70+
/// Exported to Databricks telemetry service.
71+
/// </summary>
72+
[TelemetryTag("session.id",
73+
ExportScope = TagExportScope.ExportDatabricks,
74+
Description = "Session ID associated with error")]
75+
public const string SessionId = "session.id";
76+
77+
/// <summary>
78+
/// Full stack trace.
79+
/// Only exported to local diagnostics (may contain sensitive paths).
80+
/// </summary>
81+
[TelemetryTag("error.stack_trace",
82+
ExportScope = TagExportScope.ExportLocal,
83+
Description = "Full stack trace (local diagnostics only)")]
84+
public const string ErrorStackTrace = "error.stack_trace";
85+
86+
/// <summary>
87+
/// Gets all tags that should be exported to Databricks telemetry service.
88+
/// </summary>
89+
/// <returns>A set of tag names for Databricks export.</returns>
90+
public static HashSet<string> GetDatabricksExportTags()
91+
{
92+
return new HashSet<string>
93+
{
94+
ErrorType,
95+
ErrorMessage,
96+
ErrorHttpStatus,
97+
StatementId,
98+
SessionId
99+
};
100+
}
101+
}
102+
}
Lines changed: 133 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,133 @@
1+
/*
2+
* Copyright (c) 2025 ADBC Drivers Contributors
3+
*
4+
* Licensed under the Apache License, Version 2.0 (the "License");
5+
* you may not use this file except in compliance with the License.
6+
* You may obtain a copy of the License at
7+
*
8+
* http://www.apache.org/licenses/LICENSE-2.0
9+
*
10+
* Unless required by applicable law or agreed to in writing, software
11+
* distributed under the License is distributed on an "AS IS" BASIS,
12+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
* See the License for the specific language governing permissions and
14+
* limitations under the License.
15+
*/
16+
17+
using System.Collections.Generic;
18+
19+
namespace AdbcDrivers.Databricks.Telemetry.TagDefinitions
20+
{
21+
/// <summary>
22+
/// Tag definitions for Statement execution events.
23+
/// </summary>
24+
public static class StatementExecutionEvent
25+
{
26+
/// <summary>
27+
/// The event name for statement execution operations.
28+
/// </summary>
29+
public const string EventName = "Statement.Execute";
30+
31+
/// <summary>
32+
/// Statement execution ID.
33+
/// Exported to Databricks telemetry service.
34+
/// </summary>
35+
[TelemetryTag("statement.id",
36+
ExportScope = TagExportScope.ExportDatabricks,
37+
Description = "Statement execution ID",
38+
Required = true)]
39+
public const string StatementId = "statement.id";
40+
41+
/// <summary>
42+
/// Connection session ID.
43+
/// Exported to Databricks telemetry service.
44+
/// </summary>
45+
[TelemetryTag("session.id",
46+
ExportScope = TagExportScope.ExportDatabricks,
47+
Description = "Connection session ID",
48+
Required = true)]
49+
public const string SessionId = "session.id";
50+
51+
/// <summary>
52+
/// Result format (inline or cloudfetch).
53+
/// Exported to Databricks telemetry service.
54+
/// </summary>
55+
[TelemetryTag("result.format",
56+
ExportScope = TagExportScope.ExportDatabricks,
57+
Description = "Result format: inline, cloudfetch")]
58+
public const string ResultFormat = "result.format";
59+
60+
/// <summary>
61+
/// Number of CloudFetch chunks.
62+
/// Exported to Databricks telemetry service.
63+
/// </summary>
64+
[TelemetryTag("result.chunk_count",
65+
ExportScope = TagExportScope.ExportDatabricks,
66+
Description = "Number of CloudFetch chunks")]
67+
public const string ResultChunkCount = "result.chunk_count";
68+
69+
/// <summary>
70+
/// Total bytes downloaded.
71+
/// Exported to Databricks telemetry service.
72+
/// </summary>
73+
[TelemetryTag("result.bytes_downloaded",
74+
ExportScope = TagExportScope.ExportDatabricks,
75+
Description = "Total bytes downloaded")]
76+
public const string ResultBytesDownloaded = "result.bytes_downloaded";
77+
78+
/// <summary>
79+
/// Whether compression is enabled for results.
80+
/// Exported to Databricks telemetry service.
81+
/// </summary>
82+
[TelemetryTag("result.compression_enabled",
83+
ExportScope = TagExportScope.ExportDatabricks,
84+
Description = "Compression enabled for results")]
85+
public const string ResultCompressionEnabled = "result.compression_enabled";
86+
87+
/// <summary>
88+
/// Number of status poll requests.
89+
/// Exported to Databricks telemetry service.
90+
/// </summary>
91+
[TelemetryTag("poll.count",
92+
ExportScope = TagExportScope.ExportDatabricks,
93+
Description = "Number of status poll requests")]
94+
public const string PollCount = "poll.count";
95+
96+
/// <summary>
97+
/// Total polling latency in milliseconds.
98+
/// Exported to Databricks telemetry service.
99+
/// </summary>
100+
[TelemetryTag("poll.latency_ms",
101+
ExportScope = TagExportScope.ExportDatabricks,
102+
Description = "Total polling latency")]
103+
public const string PollLatencyMs = "poll.latency_ms";
104+
105+
/// <summary>
106+
/// SQL query text.
107+
/// Only exported to local diagnostics (sensitive data).
108+
/// </summary>
109+
[TelemetryTag("db.statement",
110+
ExportScope = TagExportScope.ExportLocal,
111+
Description = "SQL query text (local diagnostics only)")]
112+
public const string DbStatement = "db.statement";
113+
114+
/// <summary>
115+
/// Gets all tags that should be exported to Databricks telemetry service.
116+
/// </summary>
117+
/// <returns>A set of tag names for Databricks export.</returns>
118+
public static HashSet<string> GetDatabricksExportTags()
119+
{
120+
return new HashSet<string>
121+
{
122+
StatementId,
123+
SessionId,
124+
ResultFormat,
125+
ResultChunkCount,
126+
ResultBytesDownloaded,
127+
ResultCompressionEnabled,
128+
PollCount,
129+
PollLatencyMs
130+
};
131+
}
132+
}
133+
}

0 commit comments

Comments
 (0)