Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
134d8b8
docs: add fix-telemetry-gaps design document
Mar 10, 2026
2867946
docs: move E2E tests to Phase 1 (test-first approach)
Mar 11, 2026
425a554
docs: consolidate implementation into 2 phases (Thrift gaps first, SE…
Mar 12, 2026
b498f06
Build E2E test infrastructure with CapturingTelemetryExporter\n\nTask…
Mar 13, 2026
3d80f8d
Populate runtime_vendor and client_app_name in DriverSystemConfigurat…
Mar 13, 2026
9fdd789
Populate auth_type on root telemetry log\n\nTask ID: task-1.3-auth-ty…
Mar 13, 2026
12b28b2
Populate WorkspaceId in TelemetrySessionContext\n\nTask ID: task-1.4-…
Mar 13, 2026
5a6d1ef
Expand DriverConnectionParameters with additional fields\n\nTask ID: …
Mar 13, 2026
128a824
Add ChunkMetrics aggregation to CloudFetchDownloader\n\nTask ID: task…
Mar 13, 2026
2e676d4
Expose GetChunkMetrics() on CloudFetchReader interface\n\nTask ID: ta…
Mar 13, 2026
54e94e1
Call SetChunkDetails() in DatabricksStatement.EmitTelemetry()\n\nTask…
Mar 13, 2026
ac0a512
Track retry_count in SqlExecutionEvent\n\nTask ID: task-1.9-track-ret…
Mar 13, 2026
532e529
Mark internal calls with is_internal_call flag\n\nTask ID: task-1.10-…
Mar 13, 2026
a6b030a
Add telemetry for metadata operations (GetObjects, GetTableTypes)\n\n…
Mar 13, 2026
80477df
Verify all Phase 1 E2E tests pass\n\nTask ID: task-1.12-verify-phase1…
Mar 13, 2026
a811006
fix(csharp): remove accidentally committed demo submodule breaking CI
Mar 13, 2026
e33bee4
fix(csharp): fix lint issues - trailing whitespace, license headers, …
Mar 13, 2026
e953996
fix(csharp): remove extra trailing newline in telemetry-design.md
Mar 13, 2026
d504b0a
fix(csharp): address PR review feedback - test isolation, assertions,…
Mar 13, 2026
50c331f
Extract TelemetryHelper for shared telemetry logic\n\nTask ID: task-2…
Mar 13, 2026
d21300f
Wire InitializeTelemetry() into StatementExecutionConnection\n\nTask …
Mar 13, 2026
c6d4830
Add EmitTelemetry() to StatementExecutionStatement\n\nTask ID: task-2…
Mar 13, 2026
10cd160
Wire SetChunkDetails() in StatementExecutionStatement for SEA CloudFe…
Mar 13, 2026
5b65a42
Verify all Phase 2 SEA E2E tests pass\n\nTask ID: task-2.5-verify-pha…
Mar 13, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -292,3 +292,6 @@ generated_task_specs.json

# Git worktrees
.worktrees/

# Demo directory (local only)
demo/
80 changes: 80 additions & 0 deletions csharp/doc/telemetry-design.md
Original file line number Diff line number Diff line change
Expand Up @@ -2846,3 +2846,83 @@ This **direct object telemetry design (V3)** provides a simple approach to colle
4. **Deterministic emission**: Exactly one telemetry event per statement — on reader dispose (success) or catch block (error)
5. **Flush-before-close**: Connection dispose blocks until all pending telemetry is sent to Databricks
6. **JDBC-compatible**: snake_case JSON field names, same proto schema, same export endpoint

---

## Implementation Notes - E2E Test Infrastructure (2026-03-13)

### Files Implemented

1. **CapturingTelemetryExporter.cs** (`csharp/test/E2E/Telemetry/CapturingTelemetryExporter.cs`)
- Thread-safe telemetry event capture using `ConcurrentBag<TelemetryFrontendLog>`
- Export call counting for validation
- Reset capability for test cleanup

2. **TelemetryTestHelpers.cs** (`csharp/test/E2E/Telemetry/TelemetryTestHelpers.cs`)
- `CreateConnectionWithCapturingTelemetry()` - Uses `TelemetryClientManager.ExporterOverride` to inject test exporter
- `WaitForTelemetryEvents()` - Waits for expected telemetry events with timeout
- Proto field assertion helpers for session, system config, connection params, SQL operations, and errors

3. **TelemetryBaselineTests.cs** (`csharp/test/E2E/Telemetry/TelemetryBaselineTests.cs`)
- 10 baseline E2E tests validating all currently populated proto fields
- Tests against real Databricks workspace (no backend connectivity required)
- All tests passing ✅

### Test Coverage

Baseline tests validate:
- ✅ session_id population
- ✅ sql_statement_id population
- ✅ operation_latency_ms > 0
- ✅ system_configuration fields (driver_version, driver_name, os_name, runtime_name)
- ✅ driver_connection_params.mode is set
- ✅ sql_operation fields (statement_type, operation_type, result_latency)
- ✅ Multiple statements share session_id but have unique statement_ids
- ✅ Telemetry disabled when telemetry.enabled=false
- ✅ error_info populated on SQL errors
- ✅ UPDATE statement telemetry

### Implementation Patterns Discovered

1. **Exporter Override**: `TelemetryClientManager.ExporterOverride` provides global test exporter injection
2. **Proto Enums**: Use nested structure `Statement.Types.Type.Query`, `Operation.Types.Type.ExecuteStatement`, etc.
3. **Name Collision**: Proto `Statement` conflicts with `AdbcStatement` - resolved with type aliases:
```csharp
using ProtoStatement = AdbcDrivers.Databricks.Telemetry.Proto.Statement;
using ProtoOperation = AdbcDrivers.Databricks.Telemetry.Proto.Operation;
using ProtoDriverMode = AdbcDrivers.Databricks.Telemetry.Proto.DriverMode;
```
4. **QueryResult**: `ExecuteQuery()` returns `QueryResult` with `Stream` property (IDisposable)

### Test Pattern

```csharp
CapturingTelemetryExporter exporter = null!;
AdbcConnection? connection = null;

try
{
var properties = TestEnvironment.GetDriverParameters(TestConfiguration);
(connection, exporter) = TelemetryTestHelpers.CreateConnectionWithCapturingTelemetry(properties);

// Execute operation
using var statement = connection.CreateStatement();
statement.SqlQuery = "SELECT 1";
var result = statement.ExecuteQuery();
using var reader = result.Stream;

statement.Dispose();

// Wait for and validate telemetry
var logs = await TelemetryTestHelpers.WaitForTelemetryEvents(exporter, expectedCount: 1);
var protoLog = TelemetryTestHelpers.GetProtoLog(logs[0]);

Assert.False(string.IsNullOrEmpty(protoLog.SessionId));
// ... more assertions
}
finally
{
connection?.Dispose();
TelemetryTestHelpers.ClearExporterOverride();
}
```
Loading
Loading