Skip to content

fix(csharp): normalize TypeName once in TryGetSchemaFromManifest #337

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
eric-wang-1990 merged 4 commits into from
Mar 12, 2026
Merged

fix(csharp): normalize TypeName once in TryGetSchemaFromManifest #337

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
47ba26e
fix(csharp): normalize TypeName once in TryGetSchemaFromManifest; add…
eric-wang-1990 Mar 12, 2026
cc71688
test(csharp): remove redundant Spark:DataType:SqlName assertion from …
eric-wang-1990 Mar 12, 2026
4254d7c
test(csharp): remove ComplexAndDecimalTypesAreMappedCorrectly test
eric-wang-1990 Mar 12, 2026
3860e05
test(csharp): move null TypeName coverage to ColumnMetadataHelperTests
eric-wang-1990 Mar 12, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions csharp/src/StatementExecution/StatementExecutionStatement.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -482,7 +482,8 @@ private Schema GetSchemaFromManifest(ResultManifest manifest)
var fields = new List<Field>();
foreach (var column in manifest.Schema.Columns)
{
var arrowType = MapDatabricksTypeToArrowType(column.TypeName);
var typeName = column.TypeName ?? string.Empty;
var arrowType = MapDatabricksTypeToArrowType(typeName);
// Embed the SQL type name as Arrow field metadata so that consumers
// (e.g. the PowerBI connector's AdjustNativeTypes) can read it via
// the "Spark:DataType:SqlName" key — the same metadata the Databricks
Expand All @@ -494,7 +495,7 @@ private Schema GetSchemaFromManifest(ResultManifest manifest)
// it here for now. Add it if a consumer requires it (PECO-2950).
var metadata = new Dictionary<string, string>
{
["Spark:DataType:SqlName"] = ColumnMetadataHelper.GetSparkSqlName(column.TypeName ?? string.Empty)
["Spark:DataType:SqlName"] = ColumnMetadataHelper.GetSparkSqlName(typeName)
};
fields.Add(new Field(column.Name, arrowType, true, metadata));
}
Expand Down
2 changes: 2 additions & 0 deletions csharp/test/ColumnMetadataHelperTests.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,7 @@ public void GetDataTypeCode_ReturnsCorrectCode(string typeName, short expectedCo
[InlineData("STRING", "STRING")]
[InlineData("BOOLEAN", "BOOLEAN")]
[InlineData("DOUBLE", "DOUBLE")]
[InlineData("", "")]
public void GetBaseTypeName_ReturnsCorrectName(string typeName, string expectedBase)
{
Assert.Equal(expectedBase, ColumnMetadataHelper.GetBaseTypeName(typeName));
Expand Down Expand Up @@ -220,6 +221,7 @@ public void GetDataTypeCode_HandlesTrimAndCase(string typeName, short expectedCo
[InlineData("STRUCT<a:INT>", "STRUCT")]
[InlineData("INTERVAL DAY TO SECOND", "INTERVAL")]
[InlineData("VARIANT", "VARIANT")]
[InlineData("", "")]
public void GetSparkSqlName_ReturnsCorrectValues(string typeName, string expected)
{
Assert.Equal(expected, ColumnMetadataHelper.GetSparkSqlName(typeName));
Expand Down
Loading