Skip to content

Commit dba523d

Browse files
andygroveandygrove
authored
docs: Various documentation updates (#2674)
1 parent ff975c0 commit dba523d

File tree

4 files changed

+149
-129
lines changed

4 files changed

+149
-129
lines changed

‎docs/source/contributor-guide/index.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,7 @@ under the License.
2626
Getting Started <contributing>
2727
Comet Plugin Overview <plugin_overview>
2828
Arrow FFI <ffi>
29+
Parquet Scans <parquet_scans>
2930
Development Guide <development>
3031
Debugging Guide <debugging>
3132
Benchmarking Guide <benchmarking>

‎docs/source/contributor-guide/parquet_scans.md‎

Lines changed: 137 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,137 @@
1+
<!--
2+
Licensed to the Apache Software Foundation (ASF) under one
3+
or more contributor license agreements. See the NOTICE file
4+
distributed with this work for additional information
5+
regarding copyright ownership. The ASF licenses this file
6+
to you under the Apache License, Version 2.0 (the
7+
"License"); you may not use this file except in compliance
8+
with the License. You may obtain a copy of the License at
9+
10+
http://www.apache.org/licenses/LICENSE-2.0
11+
12+
Unless required by applicable law or agreed to in writing,
13+
software distributed under the License is distributed on an
14+
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
KIND, either express or implied. See the License for the
16+
specific language governing permissions and limitations
17+
under the License.
18+
-->
19+
20+
# Comet Parquet Scan Implementations
21+
22+
Comet currently has three distinct implementations of the Parquet scan operator. The configuration property
23+
`spark.comet.scan.impl` is used to select an implementation. The default setting is `spark.comet.scan.impl=auto`, and
24+
Comet will choose the most appropriate implementation based on the Parquet schema and other Comet configuration
25+
settings. Most users should not need to change this setting. However, it is possible to force Comet to try and use
26+
a particular implementation for all scan operations by setting this configuration property to one of the following
27+
implementations.
28+
29+
| Implementation | Description |
30+
| ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
31+
| `native_comet` | This implementation provides strong compatibility with Spark but does not support complex types. This is the original scan implementation in Comet and may eventually be removed. |
32+
| `native_iceberg_compat` | This implementation delegates to DataFusion's `DataSourceExec` but uses a hybrid approach of JVM and native code. This scan is designed to be integrated with Iceberg in the future. |
33+
| `native_datafusion` | This experimental implementation delegates to DataFusion's `DataSourceExec` for full native execution. There are known compatibility issues when using this scan. |
34+
35+
The `native_datafusion` and `native_iceberg_compat` scans provide the following benefits over the `native_comet`
36+
implementation:
37+
38+
- Leverages the DataFusion community's ongoing improvements to `DataSourceExec`
39+
- Provides support for reading complex types (structs, arrays, and maps)
40+
- Removes the use of reusable mutable-buffers in Comet, which is complex to maintain
41+
- Improves performance
42+
43+
The `native_datafusion` and `native_iceberg_compat` scans share the following limitations:
44+
45+
- When reading Parquet files written by systems other than Spark that contain columns with the logical types `UINT_8`
46+
or `UINT_16`, Comet will produce different results than Spark because Spark does not preserve or understand these
47+
logical types. Arrow-based readers, such as DataFusion and Comet do respect these types and read the data as unsigned
48+
rather than signed. By default, Comet will fall back to `native_comet` when scanning Parquet files containing `byte` or `short`
49+
types (regardless of the logical type). This behavior can be disabled by setting
50+
`spark.comet.scan.allowIncompatible=true`.
51+
- No support for default values that are nested types (e.g., maps, arrays, structs). Literal default values are supported.
52+
53+
The `native_datafusion` scan has some additional limitations:
54+
55+
- Bucketed scans are not supported
56+
- No support for row indexes
57+
- `PARQUET_FIELD_ID_READ_ENABLED` is not respected [#1758]
58+
- There are failures in the Spark SQL test suite [#1545]
59+
- Setting Spark configs `ignoreMissingFiles` or `ignoreCorruptFiles` to `true` is not compatible with Spark
60+
61+
## S3 Support
62+
63+
There are some
64+
65+
### `native_comet`
66+
67+
The default `native_comet` Parquet scan implementation reads data from S3 using the [Hadoop-AWS module](https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html), which
68+
is identical to the approach commonly used with vanilla Spark. AWS credential configuration and other Hadoop S3A
69+
configurations works the same way as in vanilla Spark.
70+
71+
### `native_datafusion` and `native_iceberg_compat`
72+
73+
The `native_datafusion` and `native_iceberg_compat` Parquet scan implementations completely offload data loading
74+
to native code. They use the [`object_store` crate](https://crates.io/crates/object_store) to read data from S3 and
75+
support configuring S3 access using standard [Hadoop S3A configurations](https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html#General_S3A_Client_configuration) by translating them to
76+
the `object_store` crate's format.
77+
78+
This implementation maintains compatibility with existing Hadoop S3A configurations, so existing code will
79+
continue to work as long as the configurations are supported and can be translated without loss of functionality.
80+
81+
#### Additional S3 Configuration Options
82+
83+
Beyond credential providers, the `native_datafusion` implementation supports additional S3 configuration options:
84+
85+
| Option | Description |
86+
|--------|-------------|
87+
| `fs.s3a.endpoint` | The endpoint of the S3 service |
88+
| `fs.s3a.endpoint.region` | The AWS region for the S3 service. If not specified, the region will be auto-detected. |
89+
| `fs.s3a.path.style.access` | Whether to use path style access for the S3 service (true/false, defaults to virtual hosted style) |
90+
| `fs.s3a.requester.pays.enabled` | Whether to enable requester pays for S3 requests (true/false) |
91+
92+
All configuration options support bucket-specific overrides using the pattern `fs.s3a.bucket.{bucket-name}.{option}`.
93+
94+
#### Examples
95+
96+
The following examples demonstrate how to configure S3 access with the `native_datafusion` Parquet scan implementation using different authentication methods.
97+
98+
**Example 1: Simple Credentials**
99+
100+
This example shows how to access a private S3 bucket using an access key and secret key. The `fs.s3a.aws.credentials.provider` configuration can be omitted since `org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider` is included in Hadoop S3A's default credential provider chain.
101+
102+
```shell
103+
$SPARK_HOME/bin/spark-shell \
104+
...
105+
--conf spark.comet.scan.impl=native_datafusion \
106+
--conf spark.hadoop.fs.s3a.access.key=my-access-key \
107+
--conf spark.hadoop.fs.s3a.secret.key=my-secret-key
108+
...
109+
```
110+
111+
**Example 2: Assume Role with Web Identity Token**
112+
113+
This example demonstrates using an assumed role credential to access a private S3 bucket, where the base credential for assuming the role is provided by a web identity token credentials provider.
114+
115+
```shell
116+
$SPARK_HOME/bin/spark-shell \
117+
...
118+
--conf spark.comet.scan.impl=native_datafusion \
119+
--conf spark.hadoop.fs.s3a.aws.credentials.provider=org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider \
120+
--conf spark.hadoop.fs.s3a.assumed.role.arn=arn:aws:iam::123456789012:role/my-role \
121+
--conf spark.hadoop.fs.s3a.assumed.role.session.name=my-session \
122+
--conf spark.hadoop.fs.s3a.assumed.role.credentials.provider=com.amazonaws.auth.WebIdentityTokenCredentialsProvider
123+
...
124+
```
125+
126+
#### Limitations
127+
128+
The S3 support of `native_datafusion` has the following limitations:
129+
130+
1. **Partial Hadoop S3A configuration support**: Not all Hadoop S3A configurations are currently supported. Only the configurations listed in the tables above are translated and applied to the underlying `object_store` crate.
131+
132+
2. **Custom credential providers**: Custom implementations of AWS credential providers are not supported. The implementation only supports the standard credential providers listed in the table above. We are planning to add support for custom credential providers through a JNI-based adapter that will allow calling Java credential providers from native code. See [issue #1829](https://github.com/apache/datafusion-comet/issues/1829) for more details.
133+
134+
135+
136+
[#1545]: https://github.com/apache/datafusion-comet/issues/1545
137+
[#1758]: https://github.com/apache/datafusion-comet/issues/1758

‎docs/source/user-guide/latest/compatibility.md‎

Lines changed: 8 additions & 60 deletions
Original file line numberDiff line numberDiff line change
@@ -25,59 +25,11 @@ This guide offers information about areas of functionality where there are known
2525

2626
## Parquet
2727

28-
### Data Type Support
28+
Comet has the following limitations when reading Parquet files:
2929

30-
Comet does not support reading decimals encoded in binary format.
31-
32-
### Parquet Scans
33-
34-
Comet currently has three distinct implementations of the Parquet scan operator. The configuration property
35-
`spark.comet.scan.impl` is used to select an implementation. The default setting is `spark.comet.scan.impl=auto`, and
36-
Comet will choose the most appropriate implementation based on the Parquet schema and other Comet configuration
37-
settings. Most users should not need to change this setting. However, it is possible to force Comet to try and use
38-
a particular implementation for all scan operations by setting this configuration property to one of the following
39-
implementations.
40-
41-
| Implementation | Description |
42-
| ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
43-
| `native_comet` | This implementation provides strong compatibility with Spark but does not support complex types. This is the original scan implementation in Comet and may eventually be removed. |
44-
| `native_iceberg_compat` | This implementation delegates to DataFusion's `DataSourceExec` but uses a hybrid approach of JVM and native code. This scan is designed to be integrated with Iceberg in the future. |
45-
| `native_datafusion` | This experimental implementation delegates to DataFusion's `DataSourceExec` for full native execution. There are known compatibility issues when using this scan. |
46-
47-
The `native_datafusion` and `native_iceberg_compat` scans provide the following benefits over the `native_comet`
48-
implementation:
49-
50-
- Leverages the DataFusion community's ongoing improvements to `DataSourceExec`
51-
- Provides support for reading complex types (structs, arrays, and maps)
52-
- Removes the use of reusable mutable-buffers in Comet, which is complex to maintain
53-
- Improves performance
54-
55-
The `native_datafusion` and `native_iceberg_compat` scans share the following limitations:
56-
57-
- When reading Parquet files written by systems other than Spark that contain columns with the logical types `UINT_8`
58-
or `UINT_16`, Comet will produce different results than Spark because Spark does not preserve or understand these
59-
logical types. Arrow-based readers, such as DataFusion and Comet do respect these types and read the data as unsigned
60-
rather than signed. By default, Comet will fall back to `native_comet` when scanning Parquet files containing `byte` or `short`
61-
types (regardless of the logical type). This behavior can be disabled by setting
62-
`spark.comet.scan.allowIncompatible=true`.
30+
- Comet does not support reading decimals encoded in binary format.
6331
- No support for default values that are nested types (e.g., maps, arrays, structs). Literal default values are supported.
6432

65-
The `native_datafusion` scan has some additional limitations:
66-
67-
- Bucketed scans are not supported
68-
- No support for row indexes
69-
- `PARQUET_FIELD_ID_READ_ENABLED` is not respected [#1758]
70-
- There are failures in the Spark SQL test suite [#1545]
71-
- Setting Spark configs `ignoreMissingFiles` or `ignoreCorruptFiles` to `true` is not compatible with Spark
72-
73-
[#1545]: https://github.com/apache/datafusion-comet/issues/1545
74-
[#1758]: https://github.com/apache/datafusion-comet/issues/1758
75-
76-
### S3 Support with `native_iceberg_compat`
77-
78-
- When using the default AWS S3 endpoint (no custom endpoint configured), a valid region is required. Comet
79-
will attempt to resolve the region if it is not provided.
80-
8133
## ANSI Mode
8234

8335
Comet will fall back to Spark for the following expressions when ANSI mode is enabled, unless
@@ -101,18 +53,14 @@ Sorting on floating-point data types (or complex types containing floating-point
10153
Spark if the data contains both zero and negative zero. This is likely an edge case that is not of concern for many users
10254
and sorting on floating-point data can be enabled by setting `spark.comet.expression.SortOrder.allowIncompatible=true`.
10355

104-
There is a known bug with using count(distinct) within aggregate queries, where each NaN value will be counted
105-
separately [#1824](https://github.com/apache/datafusion-comet/issues/1824).
106-
10756
## Incompatible Expressions
10857

109-
Some Comet native expressions are not 100% compatible with Spark and are disabled by default. These expressions
110-
will fall back to Spark but can be enabled by setting `spark.comet.expression.allowIncompatible=true`.
111-
112-
## Array Expressions
58+
Expressions that are not 100% Spark-compatible will fall back to Spark by default and can be enabled by setting
59+
`spark.comet.expression.EXPRNAME.allowIncompatible=true`, where `EXPRNAME` is the Spark expression class name. See
60+
the [Comet Supported Expressions Guide](expressions.md) for more information on this configuration setting.
11361

114-
Comet has experimental support for a number of array expressions. These are experimental and currently marked
115-
as incompatible and can be enabled by setting `spark.comet.expression.allowIncompatible=true`.
62+
It is also possible to specify `spark.comet.expression.allowIncompatible=true` to enable all
63+
incompatible expressions.
11664

11765
## Regular Expressions
11866

@@ -127,7 +75,7 @@ Cast operations in Comet fall into three levels of support:
12775
- **Compatible**: The results match Apache Spark
12876
- **Incompatible**: The results may match Apache Spark for some inputs, but there are known issues where some inputs
12977
will result in incorrect results or exceptions. The query stage will fall back to Spark by default. Setting
130-
`spark.comet.expression.allowIncompatible=true` will allow all incompatible casts to run natively in Comet, but this is not
78+
`spark.comet.expression.Cast.allowIncompatible=true` will allow all incompatible casts to run natively in Comet, but this is not
13179
recommended for production use.
13280
- **Unsupported**: Comet does not provide a native version of this cast expression and the query stage will fall back to
13381
Spark.

0 commit comments

Comments
 (0)