Skip to content

Conversation

@sreekanth-db
Copy link
Collaborator

@sreekanth-db sreekanth-db commented Nov 7, 2025

🥞 Stacked PR

Use this link to review incremental changes.


What's Changed

This PR implements Phase 1 of telemetry tag definitions (telemetry-activity-based-design.md) for the C# Databricks ADBC driver, establishing structured telemetry collection with privacy-aware export controls.

Changes

Introduces a declarative tag definition system across three telemetry event types:

1. Connection Events - Captures connection lifecycle and driver configuration

  • Identity: workspace ID, session ID
  • Driver metadata: version, OS, .NET runtime
  • Feature flags: CloudFetch, LZ4, direct results, multiple catalog, trace propagation
  • Local-only: server addresses

2. Statement Execution Events - Tracks query performance and resource usage

  • Result metrics: format, chunk count, bytes downloaded, row count
  • Polling metrics: poll count, latency
  • Operation metadata: operation type (SELECT, INSERT, etc.)
  • Local-only: SQL text, catalog/schema names

3. Error Events - Records error patterns and retry behavior

  • Classification: error type, HTTP status codes, SQL states
  • Retry context: retry count, operation context
  • Local-only: error messages, stack traces

Export Scope Design

Tags use TagExportScope enum to control export destinations:

  • ExportLocal - Sensitive data only (SQL, errors, addresses)
  • ExportDatabricks - Only to databricks service
  • ExportAll - Non-sensitive operational metrics (both local & Databricks)

TelemetryTagRegistry provides centralized privacy filtering via GetDatabricksExportTags() and ShouldExportToDatabricks().

Design Note: Added TODO comment acknowledging the current approach duplicates export scope between attributes and whitelist methods.

Testing

  • ✅ All 7 unit tests pass (TelemetryTagRegistryTests)
  • ✅ Privacy filters validated (sensitive tags excluded from Databricks export)
  • ✅ Unknown event types handled gracefully

Next Steps

Future phases will add ActivityListener, metrics aggregator, and exporter components.

@sreekanth-db sreekanth-db force-pushed the stack/telemetry-phase-1 branch from 1700d25 to 29e9867 Compare November 7, 2025 13:01
@sreekanth-db sreekanth-db changed the title feat(csharp): Implement telemetry tag definitions (phase 1) feat(csharp): implement telemetry tag definitions (phase 1) Nov 7, 2025
/// <summary>
/// Tag definitions for Connection.Open events.
/// </summary>
internal static class ConnectionOpenEvent
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we log all the property for openConnection? Like query tag, etc?
Does other driver output those properties, any PII info?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants