Timestamp DataType does not work in schema

[rangp]

I am trying to define a schema with a field of Timestamp Data type

var builder = new Apache.Arrow.Schema.Builder();
builder
    .Field(fb =>
    {
        fb.Name("id");
        fb.DataType(StringType.Default);
        fb.Nullable(false);
    })
    .Field(fb =>
    {
        fb.Name("time");
        fb.DataType(TimestampType.Default);
        fb.Nullable(false);
    });

var schema = builder.Build();

using var table = await engine.CreateTableAsync(
    new TableCreateOptions(baseUri.ToString(), schema)
    {
        PartitionBy = ["id"],
        StorageOptions = storageOptions,
    },
    CancellationToken.None);

But creating the table does not work with error:
Schema error: Invalid data type for Delta Lake: Timestamp(Millisecond, Some("+00:00"))'

Using just a string type, the table can be created successfully

[CurtHagenlocher]

Delta Lake supports only Microsecond resolution for timestamps. Use new TimestampType(TimeUnit.Microsecond, "+00:00").

[rangp]

Nice, thank you! Using new TimestampType(TimeUnit.Microsecond, default(string?)) I can get it to work. But shouldn't TimeStampType.Default be using microsecond resolution then if it's the only supported resolution?

[CurtHagenlocher]

TimestampType is part of the Arrow project, which both predates Delta Lake and is not at all tied to it. Changing the definition of TimestampType.Default would be a breaking change for Arrow.

[rangp]

I see. Seems like a very common pitfall, if you do not know the specifics. TimeUnit.Microsecond needs to be set not only in the schema definition, but also in the RecordBatchBuilder and in both cases time zone offsets do not appear to be supported, so they need to be set to null explicitly.

[mightyshazam]

@CurtHagenlocher got to it before me, but this is another one for issue #128 .
Timestamp is a thing that has vexed everyone, including the delta-rs project because almost all arrow types are 1:1 with their delta equivalent, but timestamp is not.