apache · andygrove · Nov 3, 2025 · Oct 31, 2025 · Oct 31, 2025 · Oct 31, 2025
diff --git a/docs/source/contributor-guide/index.md b/docs/source/contributor-guide/index.md
@@ -26,6 +26,7 @@ under the License.
 Getting Started <contributing>
 Comet Plugin Overview <plugin_overview>
 Arrow FFI <ffi>
+Parquet Scans <parquet_scans>
 Development Guide <development>
 Debugging Guide <debugging>
 Benchmarking Guide <benchmarking>

diff --git a/docs/source/contributor-guide/parquet_scans.md b/docs/source/contributor-guide/parquet_scans.md
@@ -0,0 +1,62 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Comet Parquet Scan Implementations
+
+Comet currently has three distinct implementations of the Parquet scan operator. The configuration property
+`spark.comet.scan.impl` is used to select an implementation. The default setting is `spark.comet.scan.impl=auto`, and
+Comet will choose the most appropriate implementation based on the Parquet schema and other Comet configuration
+settings. Most users should not need to change this setting. However, it is possible to force Comet to try and use
+a particular implementation for all scan operations by setting this configuration property to one of the following
+implementations.
+
+| Implementation          | Description                                                                                                                                                                          |
+| ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| `native_comet`          | This implementation provides strong compatibility with Spark but does not support complex types. This is the original scan implementation in Comet and may eventually be removed.    |
+| `native_iceberg_compat` | This implementation delegates to DataFusion's `DataSourceExec` but uses a hybrid approach of JVM and native code. This scan is designed to be integrated with Iceberg in the future. |
+| `native_datafusion`     | This experimental implementation delegates to DataFusion's `DataSourceExec` for full native execution. There are known compatibility issues when using this scan.                    |
+
+The `native_datafusion` and `native_iceberg_compat` scans provide the following benefits over the `native_comet`
+implementation:
+
+- Leverages the DataFusion community's ongoing improvements to `DataSourceExec`
+- Provides support for reading complex types (structs, arrays, and maps)
+- Removes the use of reusable mutable-buffers in Comet, which is complex to maintain
+- Improves performance
+
+The `native_datafusion` and `native_iceberg_compat` scans share the following limitations:
+
+- When reading Parquet files written by systems other than Spark that contain columns with the logical types `UINT_8`
+  or `UINT_16`, Comet will produce different results than Spark because Spark does not preserve or understand these
+  logical types. Arrow-based readers, such as DataFusion and Comet do respect these types and read the data as unsigned
+  rather than signed. By default, Comet will fall back to `native_comet` when scanning Parquet files containing `byte` or `short`
+  types (regardless of the logical type). This behavior can be disabled by setting
+  `spark.comet.scan.allowIncompatible=true`.
+- No support for default values that are nested types (e.g., maps, arrays, structs). Literal default values are supported.
+
+The `native_datafusion` scan has some additional limitations:
+
+- Bucketed scans are not supported
+- No support for row indexes
+- `PARQUET_FIELD_ID_READ_ENABLED` is not respected [#1758]
+- There are failures in the Spark SQL test suite [#1545]
+- Setting Spark configs `ignoreMissingFiles` or `ignoreCorruptFiles` to `true` is not compatible with Spark
+
+[#1545]: https://github.com/apache/datafusion-comet/issues/1545
+[#1758]: https://github.com/apache/datafusion-comet/issues/1758
diff --git a/docs/source/user-guide/latest/compatibility.md b/docs/source/user-guide/latest/compatibility.md
@@ -25,59 +25,11 @@ This guide offers information about areas of functionality where there are known
 
 ## Parquet
 
-### Data Type Support
+Comet has the following limitations when reading Parquet files:
 
-Comet does not support reading decimals encoded in binary format.
-
-### Parquet Scans
-
-Comet currently has three distinct implementations of the Parquet scan operator. The configuration property
-`spark.comet.scan.impl` is used to select an implementation. The default setting is `spark.comet.scan.impl=auto`, and
-Comet will choose the most appropriate implementation based on the Parquet schema and other Comet configuration
-settings. Most users should not need to change this setting. However, it is possible to force Comet to try and use
-a particular implementation for all scan operations by setting this configuration property to one of the following
-implementations.
-
-| Implementation          | Description                                                                                                                                                                          |
-| ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
-| `native_comet`          | This implementation provides strong compatibility with Spark but does not support complex types. This is the original scan implementation in Comet and may eventually be removed.    |
-| `native_iceberg_compat` | This implementation delegates to DataFusion's `DataSourceExec` but uses a hybrid approach of JVM and native code. This scan is designed to be integrated with Iceberg in the future. |
-| `native_datafusion`     | This experimental implementation delegates to DataFusion's `DataSourceExec` for full native execution. There are known compatibility issues when using this scan.                    |
-
-The `native_datafusion` and `native_iceberg_compat` scans provide the following benefits over the `native_comet`
-implementation:
-
-- Leverages the DataFusion community's ongoing improvements to `DataSourceExec`
-- Provides support for reading complex types (structs, arrays, and maps)
-- Removes the use of reusable mutable-buffers in Comet, which is complex to maintain
-- Improves performance
-
-The `native_datafusion` and `native_iceberg_compat` scans share the following limitations:
-
-- When reading Parquet files written by systems other than Spark that contain columns with the logical types `UINT_8`
-  or `UINT_16`, Comet will produce different results than Spark because Spark does not preserve or understand these
-  logical types. Arrow-based readers, such as DataFusion and Comet do respect these types and read the data as unsigned
-  rather than signed. By default, Comet will fall back to `native_comet` when scanning Parquet files containing `byte` or `short`
-  types (regardless of the logical type). This behavior can be disabled by setting
-  `spark.comet.scan.allowIncompatible=true`.
+- Comet does not support reading decimals encoded in binary format.
 - No support for default values that are nested types (e.g., maps, arrays, structs). Literal default values are supported.
 
-The `native_datafusion` scan has some additional limitations:
-
-- Bucketed scans are not supported
-- No support for row indexes
-- `PARQUET_FIELD_ID_READ_ENABLED` is not respected [#1758]
-- There are failures in the Spark SQL test suite [#1545]
-- Setting Spark configs `ignoreMissingFiles` or `ignoreCorruptFiles` to `true` is not compatible with Spark
-
-[#1545]: https://github.com/apache/datafusion-comet/issues/1545
-[#1758]: https://github.com/apache/datafusion-comet/issues/1758
-
-### S3 Support with `native_iceberg_compat`
-
-- When using the default AWS S3 endpoint (no custom endpoint configured), a valid region is required. Comet
-  will attempt to resolve the region if it is not provided.
-
 ## ANSI Mode
 
 Comet will fall back to Spark for the following expressions when ANSI mode is enabled, unless
@@ -101,18 +53,14 @@ Sorting on floating-point data types (or complex types containing floating-point
 Spark if the data contains both zero and negative zero. This is likely an edge case that is not of concern for many users
 and sorting on floating-point data can be enabled by setting `spark.comet.expression.SortOrder.allowIncompatible=true`.
 
-There is a known bug with using count(distinct) within aggregate queries, where each NaN value will be counted
-separately [#1824](https://github.com/apache/datafusion-comet/issues/1824).
-
 ## Incompatible Expressions
 
-Some Comet native expressions are not 100% compatible with Spark and are disabled by default. These expressions
-will fall back to Spark but can be enabled by setting `spark.comet.expression.allowIncompatible=true`.
-
-## Array Expressions
+Expressions that are not 100% Spark-compatible will fall back to Spark by default and can be enabled by setting
+`spark.comet.expression.EXPRNAME.allowIncompatible=true`, where `EXPRNAME` is the Spark expression class name. See 
+the [Comet Supported Expressions Guide](expressions.md) for more information on this configuration setting.  
 
-Comet has experimental support for a number of array expressions. These are experimental and currently marked
-as incompatible and can be enabled by setting `spark.comet.expression.allowIncompatible=true`.
+It is also possible to specify `spark.comet.expression.allowIncompatible=true` to enable all
+incompatible expressions.
 
 ## Regular Expressions
 
@@ -127,7 +75,7 @@ Cast operations in Comet fall into three levels of support:
 - **Compatible**: The results match Apache Spark
 - **Incompatible**: The results may match Apache Spark for some inputs, but there are known issues where some inputs
   will result in incorrect results or exceptions. The query stage will fall back to Spark by default. Setting
-  `spark.comet.expression.allowIncompatible=true` will allow all incompatible casts to run natively in Comet, but this is not
+  `spark.comet.expression.Cast.allowIncompatible=true` will allow all incompatible casts to run natively in Comet, but this is not
   recommended for production use.
 - **Unsupported**: Comet does not provide a native version of this cast expression and the query stage will fall back to
   Spark.

diff --git a/docs/source/user-guide/latest/datasources.md b/docs/source/user-guide/latest/datasources.md
@@ -163,7 +163,7 @@ Or use `spark-shell` with HDFS support as described [above](#using-experimental-
 
 ## S3
 
-DataFusion Comet has [multiple Parquet scan implementations](./compatibility.md#parquet-scans) that use different approaches to read data from S3.
+DataFusion Comet has [multiple Parquet scan implementations](../../contributor-guide/parquet_scans.md) that use different approaches to read data from S3.
 
 ### `native_comet`
-Original file line number
+Diff line change
@@ Expand Up @@
     ## S3
-    DataFusion Comet has [multiple Parquet scan implementations](./compatibility.md#parquet-scans) that use different approaches to read data from S3.
+    DataFusion Comet has [multiple Parquet scan implementations](../../contributor-guide/parquet_scans.md) that use different approaches to read data from S3.
     ### `native_comet`
@@ Expand Down @@