Add support for null values in `Datum`

[gbrgr]

In #1824 we use instances of Datum to represent constants which can be added either by partition data or by user request as columns in the record batch transformer. However, Datum does not support null values. See this discussion thread.

[mbutrovich]

I was testing my iceberg-rust DF 51 branch with my DataFusion Comet DF 51 branch, and it looks like this broke the DataFusion Comet integration. Specifically, we're now failing 45 tests in the Iceberg Java suite for NULL identity-partitioned columns. I will try to work around it in Comet.

[liurenjie1024]

I was testing my iceberg-rust DF 51 branch with my DataFusion Comet DF 51 branch, and it looks like this broke the DataFusion Comet integration. Specifically, we're now failing 45 tests in the Iceberg Java suite for NULL identity-partitioned columns. I will try to work around it in Comet.

Hi, @mbutrovich this should not be a large feature to implement, so we quickly fix it if necessary. What I want to confirm is that we should return null if identity-partitioned columns return null, right?

[mbutrovich]

Hi, @mbutrovich this should not be a large feature to implement, so we quickly fix it if necessary. What I want to confirm is that we should return null if identity-partitioned columns return null, right?

Hey @liurenjie1024!, yep that's my understanding. I think we fell through into that scenario before #1824.

We're trying to work around this in Comet by skipping NULL partition values during serialization and relying on the Iceberg spec rule 4 to correctly resolve missing fields as NULL:

https://github.com/mbutrovich/datafusion-comet/blob/df51/spark/src/main/scala/org/apache/comet/serde/operator/CometIcebergNativeScan.scala#L391-L398

But I still have one failing test related to null identity-partitioned tables.

[mbutrovich]

The NULL Datum support has a huge blast radius, but I think I can get correct behavior by just skipping NULL in constants_map generation like it was before. I have opened draft #1922 and am testing with Iceberg Java tests and Comet to see if it resolves our issue with NULL identity-partitioned columns.