feat(csharp): sEA connection metadata — GetObjects, GetTableSchema, GetTableTypes, GetInfo, SQL commands by msrathore-db · Pull Request #257 · adbc-drivers/databricks

msrathore-db · 2026-02-27T17:38:51Z

🥞 Stacked PR

feat/sea-metadata-part1 [Files changed]
- feat/sea-metadata-part2 [Files changed]
  - feat/sea-metadata-part3 [Files changed]
    - feat/sea-metadata-part4 [Files changed]
      - feat/sea-metadata-part5 [Files changed]
        
        feat/sea-metadata-part6 [Files changed]

Summary

Core SEA metadata infrastructure and connection-level operations.

Connection-level metadata

GetObjects: async IGetObjectsDataProvider with shared CancellationToken per operation
GetTableSchema: via SHOW COLUMNS + shared HiveServer2Connection.GetArrowType
GetTableTypes: returns TABLE, VIEW
GetInfo: via shared MetadataSchemaFactory.BuildGetInfoResult
GetObjects xdbc_column_size/xdbc_decimal_digits: nulled for non-DECIMAL/CHAR to match Thrift

SQL command builders (`MetadataCommands/`)

MetadataCommandBase with ConvertPattern, QuoteIdentifier, AppendCatalogScope
ShowCatalogs, ShowSchemas, ShowTables, ShowColumns, ShowKeys, ShowForeignKeys

HTTP & protocol

x-databricks-sea-can-run-fully-sync header on metadata requests via IsMetadata flag
ExecuteMetadataSqlAsync — async SQL execution for metadata
CreateMetadataTimeoutCts — disposable CTS with configurable timeout

Utilities

MetadataUtilities: NormalizeSparkCatalog, IsInvalidPKFKCatalog, BuildQualifiedTableName
DescTableExtendedResult: REAL→FLOAT via ColumnMetadataHelper.GetDataTypeCode directly

Also includes

DatabricksStatement wired to shared MetadataSchemaFactory
sea-metadata-design.md architecture documentation

🤖 Generated with Claude Code

Copilot

Pull request overview

Implements core SEA metadata support for the C# Databricks driver when using the Statement Execution API, adding SQL-command-based metadata retrieval and refactoring schema/result construction to shared helpers.

Changes:

Implement GetObjects, GetTableSchema, GetTableTypes, and GetInfo for StatementExecutionConnection using SHOW ... SQL commands.
Add metadata SQL command builders (MetadataCommandBase, ShowCatalogs/Schemas/Tables/Columns/Keys/ForeignKeys).
Refactor/centralize metadata-related utilities and schema/result construction; update column metadata handling and related tests.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
csharp/test/Unit/Result/DescTableExtendedResultTest.cs	Updates expected column-size behavior for BINARY to align with new metadata computation.
csharp/src/StatementExecution/StatementExecutionConnection.cs	Implements SEA metadata APIs and `IGetObjectsDataProvider` via SQL execution + tracing.
csharp/src/StatementExecution/ShowTablesCommand.cs	Adds `SHOW TABLES` SQL builder with catalog scope + pattern filtering.
csharp/src/StatementExecution/ShowSchemasCommand.cs	Adds `SHOW SCHEMAS` SQL builder with optional catalog scoping + pattern filtering.
csharp/src/StatementExecution/ShowKeysCommand.cs	Adds `SHOW KEYS` and `SHOW FOREIGN KEYS` SQL builders for PK/FK metadata.
csharp/src/StatementExecution/ShowColumnsCommand.cs	Adds `SHOW COLUMNS` SQL builder with catalog/schema/table/column pattern filtering.
csharp/src/StatementExecution/ShowCatalogsCommand.cs	Adds `SHOW CATALOGS` SQL builder with pattern filtering.
csharp/src/StatementExecution/MetadataCommandBase.cs	Introduces shared quoting + pattern conversion helpers for metadata SQL builders.
csharp/src/Result/DescTableExtendedResult.cs	Refactors column metadata derivation to use shared `ColumnMetadataHelper`, with interval sizing special-case.
csharp/src/MetadataUtilities.cs	Adds shared catalog normalization and PK/FK gating helpers plus qualified table name builder.
csharp/src/DatabricksStatement.cs	Switches to shared metadata schema/result factories and simplifies empty-result construction.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

csharp/src/StatementExecution/StatementExecutionConnection.cs

csharp/src/StatementExecution/MetadataCommands/MetadataCommandBase.cs

csharp/src/StatementExecution/MetadataCommands/ShowColumnsCommand.cs

csharp/src/Result/DescTableExtendedResult.cs

csharp/src/StatementExecution/StatementExecutionConnection.cs

…tTableTypes Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

eric-wang-1990 · 2026-03-04T22:55:20Z

I am not seeing you are referring to new hiveserver2 branch which contain the refactor code?

csharp/src/StatementExecution/StatementExecutionConnection.cs

csharp/src/DatabricksStatement.cs

eric-wang-1990 · 2026-03-05T09:07:04Z

Can you try to use the plugin command /AddressPR so claude would post response to each of the comment? Otherwise it is not clear whether those are accepted or rejected.

Copilot

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated 3 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

csharp/src/StatementExecution/StatementExecutionConnection.cs

Copilot · 2026-03-05T09:18:19Z

csharp/src/StatementExecution/StatementExecutionConnection.cs

+        internal async Task<List<RecordBatch>> ExecuteMetadataSqlAsync(string sql, CancellationToken cancellationToken = default)
+        {
+            var batches = new List<RecordBatch>();
+            using var stmt = (StatementExecutionStatement)CreateStatement();
+            stmt.SqlQuery = sql;
+            stmt.IsMetadataExecution = true;
+            var result = await stmt.ExecuteQueryAsync(cancellationToken).ConfigureAwait(false);
+            using var stream = result.Stream;
+            if (stream == null) return batches;
+            while (true)
+            {
+                cancellationToken.ThrowIfCancellationRequested();
+                var batch = await stream.ReadNextRecordBatchAsync(cancellationToken).ConfigureAwait(false);
+                if (batch == null) break;
+                batches.Add(batch);
+            }
+            return batches;
+        }


ExecuteMetadataSqlAsync accumulates RecordBatch instances into a list but never disposes them. Since RecordBatch holds unmanaged Arrow buffers (and elsewhere in the codebase batches are explicitly disposed when no longer needed), this can leak memory during metadata operations. Consider processing batches in a streaming fashion (dispose each batch after extracting needed values) or ensure callers dispose all returned batches.

The returned batches are consumed immediately by callers that extract column values and discard the batch references. The batches then become eligible for GC. Adding explicit disposal would require a streaming API change. Noted as follow-up tech debt.

csharp/src/StatementExecution/StatementExecutionConnection.cs

…iew comments - Update hiveserver2 submodule to merged upstream main (e42efb4) with both PR #21 and #22 - Use shared DatabricksDriverName constant instead of hardcoded string in SEA GetInfo (addresses eric comment on line 425) - Restore inline comment on ShouldReturnEmptyPkFkResult (addresses eric comment on line 623) - Update Thrift parity comment to list all 8 preserved types (addresses Copilot comment on line 637) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

csharp/src/MetadataUtilities.cs

Implement metadata operations for the Statement Execution API (REST) protocol, achieving parity with the existing Thrift implementation. ## Core Implementation - IGetObjectsDataProvider async implementation for SEA in StatementExecutionConnection (GetCatalogs, GetSchemas, GetTables, GetColumns via SHOW commands) - Statement-level metadata routing in StatementExecutionStatement (GetColumns, GetColumnsExtended, GetPrimaryKeys, GetCrossReference, GetTableSchema, GetTableTypes, GetInfo) - SQL command builders (ShowCatalogs/Schemas/Tables/Columns/Keys/ ForeignKeys) with pattern conversion (ADBC % → Databricks *) - ColumnMetadataHelper for computing column metadata from type name strings (SEA-only, Thrift gets these from server) - FlatColumnsResultBuilder for building flat GetColumns results - MetadataUtilities for shared catalog/pattern helpers ## Connection Parameters - EnablePKFK: gate PK/FK queries (default: true) - EnableMultipleCatalogSupport: single vs multi-catalog mode (default: true), matching Thrift DatabricksConnection behavior - DatabricksStatement-specific options silently accepted in SEA ## Design - Shared MetadataSchemaFactory (hiveserver2) for Arrow schemas - Shared GetObjectsResultBuilder for nested GetObjects structure - Async internally, blocks once at ADBC sync boundary - x-databricks-sea-can-run-fully-sync header for metadata queries - Design doc: csharp/doc/sea-metadata-design.md ## Testing - Unit tests for ShowCommands and MetadataUtilities (146 tests) - E2E tests comparing Thrift and SEA parity (17 tests) - Metadata comparator tool with schema type comparison ## Reviewed in - hiveserver2: #21, #22 (merged) - databricks: #257, #258, #259, #260, #261, #282 (reviewed) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

## What's Changed Implement metadata operations for the Statement Execution API (REST) protocol, achieving parity with the existing Thrift implementation. ## Core Implementation - IGetObjectsDataProvider async implementation for SEA in StatementExecutionConnection (GetCatalogs, GetSchemas, GetTables, GetColumns via SHOW commands) - Statement-level metadata routing in StatementExecutionStatement (GetColumns, GetColumnsExtended, GetPrimaryKeys, GetCrossReference, GetTableSchema, GetTableTypes, GetInfo) - SQL command builders (ShowCatalogs/Schemas/Tables/Columns/Keys/ ForeignKeys) with pattern conversion (ADBC % → Databricks *) - ColumnMetadataHelper for computing column metadata from type name strings (SEA-only, Thrift gets these from server) - FlatColumnsResultBuilder for building flat GetColumns results - MetadataUtilities for shared catalog/pattern helpers ## Connection Parameters - EnablePKFK: gate PK/FK queries (default: true) - EnableMultipleCatalogSupport: single vs multi-catalog mode (default: true), matching Thrift DatabricksConnection behavior - DatabricksStatement-specific options silently accepted in SEA ## Design - Shared MetadataSchemaFactory (hiveserver2) for Arrow schemas - Shared GetObjectsResultBuilder for nested GetObjects structure - Async internally, blocks once at ADBC sync boundary - x-databricks-sea-can-run-fully-sync header for metadata queries - Design doc: csharp/doc/sea-metadata-design.md ## Testing - Unit tests for ShowCommands and MetadataUtilities (146 tests) - E2E tests comparing Thrift and SEA parity (17 tests) - Metadata comparator tool with schema type comparison ## Reviewed in - hiveserver2: #21, #22 (merged) - databricks: #257, #258, #259, #260, #261, #282 (reviewed) --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

msrathore-db had a problem deploying to azure-prod February 27, 2026 17:38 — with GitHub Actions Error

eric-wang-1990 requested a review from Copilot March 2, 2026 07:16

Copilot started reviewing on behalf of eric-wang-1990 March 2, 2026 07:17 View session

Copilot AI reviewed Mar 2, 2026

View reviewed changes

csharp/src/StatementExecution/StatementExecutionConnection.cs Outdated Show resolved Hide resolved

csharp/src/StatementExecution/MetadataCommands/MetadataCommandBase.cs Show resolved Hide resolved

msrathore-db force-pushed the feat/sea-metadata-part1 branch from 4110133 to 2efb879 Compare March 2, 2026 07:40

msrathore-db had a problem deploying to azure-prod March 2, 2026 07:40 — with GitHub Actions Error

eric-wang-1990 reviewed Mar 2, 2026

View reviewed changes

csharp/src/StatementExecution/MetadataCommands/ShowColumnsCommand.cs Show resolved Hide resolved

csharp/src/Result/DescTableExtendedResult.cs Outdated Show resolved Hide resolved

msrathore-db force-pushed the feat/sea-metadata-part1 branch from 2efb879 to 7b9b26a Compare March 2, 2026 08:23

msrathore-db had a problem deploying to azure-prod March 2, 2026 08:23 — with GitHub Actions Error

eric-wang-1990 reviewed Mar 2, 2026

View reviewed changes

csharp/src/StatementExecution/StatementExecutionConnection.cs Outdated Show resolved Hide resolved

csharp/src/StatementExecution/StatementExecutionConnection.cs Outdated Show resolved Hide resolved

feat(csharp): implement SEA metadata — GetObjects, GetTableSchema, Ge…

3c9e2bf

…tTableTypes Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

msrathore-db force-pushed the feat/sea-metadata-part1 branch from 7b9b26a to 3c9e2bf Compare March 2, 2026 09:46

msrathore-db requested a deployment to azure-prod March 2, 2026 09:46 — with GitHub Actions Waiting

msrathore-db mentioned this pull request Mar 2, 2026

feat(csharp): add connection parameter support — EnablePKFK, EnableMultipleCatalogSupport #282

Open

msrathore-db changed the title ~~feat(csharp): implement SEA metadata — GetObjects, GetTableSchema, GetTableTypes~~ feat(csharp): SEA connection metadata — GetObjects, GetTableSchema, GetTableTypes, GetInfo, SQL commands Mar 2, 2026

eric-wang-1990 approved these changes Mar 5, 2026

View reviewed changes

csharp/src/StatementExecution/StatementExecutionConnection.cs Show resolved Hide resolved

csharp/src/DatabricksStatement.cs Show resolved Hide resolved

eric-wang-1990 requested a review from Copilot March 5, 2026 09:08

Copilot started reviewing on behalf of eric-wang-1990 March 5, 2026 09:08 View session

Copilot AI reviewed Mar 5, 2026

View reviewed changes

eric-wang-1990 changed the title ~~feat(csharp): SEA connection metadata — GetObjects, GetTableSchema, GetTableTypes, GetInfo, SQL commands~~ feat(csharp): sEA connection metadata — GetObjects, GetTableSchema, GetTableTypes, GetInfo, SQL commands Mar 6, 2026

eric-wang-1990 reviewed Mar 6, 2026

View reviewed changes

csharp/src/MetadataUtilities.cs Show resolved Hide resolved

This was referenced Mar 9, 2026

feat(csharp): implement SEA metadata operations #326

Closed

feat(csharp): implement SEA metadata operations #327

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(csharp): sEA connection metadata — GetObjects, GetTableSchema, GetTableTypes, GetInfo, SQL commands#257

feat(csharp): sEA connection metadata — GetObjects, GetTableSchema, GetTableTypes, GetInfo, SQL commands#257
msrathore-db wants to merge 1 commit intomainfrom
feat/sea-metadata-part1

msrathore-db commented Feb 27, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

eric-wang-1990 commented Mar 4, 2026

Uh oh!

Uh oh!

Uh oh!

eric-wang-1990 commented Mar 5, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI Mar 5, 2026

Uh oh!

msrathore-db Mar 6, 2026

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

msrathore-db commented Feb 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🥞 Stacked PR

Summary

Connection-level metadata

SQL command builders (MetadataCommands/)

HTTP & protocol

Utilities

Also includes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

eric-wang-1990 commented Mar 4, 2026

Uh oh!

Uh oh!

Uh oh!

eric-wang-1990 commented Mar 5, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI Mar 5, 2026

Choose a reason for hiding this comment

Uh oh!

msrathore-db Mar 6, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

msrathore-db commented Feb 27, 2026 •

edited

Loading

SQL command builders (`MetadataCommands/`)