@@ -68,6 +68,7 @@ Expressions that are not Spark-compatible will fall back to Spark by default and
6868| Contains | Yes | |
6969| EndsWith | Yes | |
7070| InitCap | No | Behavior is different in some cases, such as hyphenated names. |
71+ | Left | Yes | Length argument must be a literal value |
7172| Length | Yes | |
7273| Like | Yes | |
7374| Lower | No | Results can vary depending on locale and character set. Requires ` spark.comet.caseConversion.enabled=true ` |
@@ -94,15 +95,20 @@ Expressions that are not Spark-compatible will fall back to Spark by default and
9495| Expression | SQL | Spark-Compatible? | Compatibility Notes |
9596| -------------- | ---------------------------- | ----------------- | -------------------------------------------------------------------------------------------------------------------- |
9697| DateAdd | ` date_add ` | Yes | |
98+ | DateDiff | ` datediff ` | Yes | |
99+ | DateFormat | ` date_format ` | Yes | Partial support. Only specific format patterns are supported. |
97100| DateSub | ` date_sub ` | Yes | |
98101| DatePart | ` date_part(field, source) ` | Yes | Supported values of ` field ` : ` year ` /` month ` /` week ` /` day ` /` dayofweek ` /` dayofweek_iso ` /` doy ` /` quarter ` /` hour ` /` minute ` |
99102| Extract | ` extract(field FROM source) ` | Yes | Supported values of ` field ` : ` year ` /` month ` /` week ` /` day ` /` dayofweek ` /` dayofweek_iso ` /` doy ` /` quarter ` /` hour ` /` minute ` |
100103| FromUnixTime | ` from_unixtime ` | No | Does not support format, supports only -8334601211038 <= sec <= 8210266876799 |
101104| Hour | ` hour ` | Yes | |
105+ | LastDay | ` last_day ` | Yes | |
102106| Minute | ` minute ` | Yes | |
103107| Second | ` second ` | Yes | |
104108| TruncDate | ` trunc ` | Yes | |
105- | TruncTimestamp | ` trunc_date ` | Yes | |
109+ | TruncTimestamp | ` date_trunc ` | Yes | |
110+ | UnixDate | ` unix_date ` | Yes | |
111+ | UnixTimestamp | ` unix_timestamp ` | Yes | |
106112| Year | ` year ` | Yes | |
107113| Month | ` month ` | Yes | |
108114| DayOfMonth | ` day ` /` dayofmonth ` | Yes | |
@@ -163,6 +169,7 @@ Expressions that are not Spark-compatible will fall back to Spark by default and
163169| ----------- | ----------------- |
164170| Md5 | Yes |
165171| Murmur3Hash | Yes |
172+ | Sha1 | Yes |
166173| Sha2 | Yes |
167174| XxHash64 | Yes |
168175
@@ -256,12 +263,13 @@ Comet supports using the following aggregate functions within window contexts wi
256263
257264## Struct Expressions
258265
259- | Expression | Spark-Compatible? |
260- | -------------------- | ----------------- |
261- | CreateNamedStruct | Yes |
262- | GetArrayStructFields | Yes |
263- | GetStructField | Yes |
264- | StructsToJson | Yes |
266+ | Expression | Spark-Compatible? | Compatibility Notes |
267+ | -------------------- | ----------------- | ------------------------------------------ |
268+ | CreateNamedStruct | Yes | |
269+ | GetArrayStructFields | Yes | |
270+ | GetStructField | Yes | |
271+ | JsonToStructs | No | Partial support. Requires explicit schema. |
272+ | StructsToJson | Yes | |
265273
266274## Conversion Expressions
267275
0 commit comments