Commit deaec6f
docs: document datetime rebasing and V2 API limitations for DataFusion-based scans (apache#3259)
Add two new limitations to the shared limitations section for
native_datafusion and native_iceberg_compat scan implementations:
1. No support for datetime rebasing detection or the
spark.comet.exceptionOnDatetimeRebase configuration. When reading
Parquet files with dates/timestamps written before Spark 3.0
(hybrid Julian/Gregorian calendar), these implementations cannot
detect legacy values and may produce incorrect results for dates
before October 15, 1582.
2. No support for Spark's Datasource V2 API. When V2 is enabled,
Comet falls back to native_comet.
Co-authored-by: Claude Opus 4.5 <[email protected]>1 parent 1a000ab commit deaec6f
1 file changed
+9
-0
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
50 | 50 | | |
51 | 51 | | |
52 | 52 | | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
53 | 62 | | |
54 | 63 | | |
55 | 64 | | |
| |||
0 commit comments