Skip to content

Commit b24a6d4

Browse files
authored
chore: Reenable nested types for CometFuzzTestSuite with int96 (#1761)
1 parent 8f14c72 commit b24a6d4

File tree

3 files changed

+1
-8
lines changed

3 files changed

+1
-8
lines changed

docs/source/user-guide/compatibility.md

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -62,7 +62,6 @@ logical types. Arrow-based readers, such as DataFusion and Comet do respect thes
6262
rather than signed. By default, Comet will fall back to Spark when scanning Parquet files containing `byte` or `short`
6363
types (regardless of the logical type). This behavior can be disabled by setting
6464
`spark.comet.scan.allowIncompatible=true`.
65-
- Reading legacy INT96 timestamps contained within complex types can produce different results to Spark
6665
- There is a known performance issue when pushing filters down to Parquet. See the [Comet Tuning Guide] for more
6766
information.
6867
- There are failures in the Spark SQL test suite when enabling these new scans (tracking issues: [#1542] and [#1545]).

docs/templates/compatibility-template.md

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -62,7 +62,6 @@ The new scans currently have the following limitations:
6262
rather than signed. By default, Comet will fall back to Spark when scanning Parquet files containing `byte` or `short`
6363
types (regardless of the logical type). This behavior can be disabled by setting
6464
`spark.comet.scan.allowIncompatible=true`.
65-
- Reading legacy INT96 timestamps contained within complex types can produce different results to Spark
6665
- There is a known performance issue when pushing filters down to Parquet. See the [Comet Tuning Guide] for more
6766
information.
6867
- There are failures in the Spark SQL test suite when enabling these new scans (tracking issues: [#1542] and [#1545]).

spark/src/test/scala/org/apache/comet/CometFuzzTestSuite.scala

Lines changed: 1 addition & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -247,12 +247,7 @@ class CometFuzzTestSuite extends CometTestBase with AdaptiveSparkPlanHelper {
247247
}
248248

249249
test("Parquet temporal types written as INT96") {
250-
// int96 coercion in DF does not work with nested types yet
251-
// https://github.com/apache/datafusion/issues/15763
252-
testParquetTemporalTypes(
253-
ParquetOutputTimestampType.INT96,
254-
generateArray = false,
255-
generateStruct = false)
250+
testParquetTemporalTypes(ParquetOutputTimestampType.INT96)
256251
}
257252

258253
test("Parquet temporal types written as TIMESTAMP_MICROS") {

0 commit comments

Comments
 (0)