Description

Here listed spark functions still not supported by Gluten Velox backend. Please leave a comment if you'd like to pick some. In the below list, [√] means someone is working in progress for the corresponding function.
You can find all functions' support status from this gluten doc.

To avoid duplicate work, before starting, please check whether a PR has been submitted in Velox community or whether it has already been implemented in Velox who holds most sql functions in its sparksql folder & prestosql folder.

Reference:

• spark sql expression
• spark built-in functions

• percentile_approx/approx_percentile (WIP, guangxin)
• concat_ws (PR ready, feat: Add Spark concat_ws function facebookincubator/velox#8854)
• unix_timestamp: "Only supports string type, with session timezone considered, todo: support date type"
• locate
• parse_url (PR drafted, not merged)
• urldecoder: "UDF, supported by spark as a built-in function since 3.4.0."
• normalizenanandzero
• arrayintersects
• default.json_split (udf, no need to impl.): "external UDF"
• parsejsonarray: "external UDF"
• struct
• percentile (@Yohahaha)
• first/first_value (@JkSelf)
• last/last_value (@JkSelf)
• posexplode (WIP, @marin-ma)
• trunc (WIP, HannanKan)
• months_between (PR ready)
• stack
• grouping_id
• printf (@Surbhi-Vijay)
• space (WIP, rhh777)
• inline (WIP, @marin-ma)
• to_unix_timestamp: "Only supports string type, with session timezone considered. todo: support date type"
• from_csv (@mudit-97)
• from_json (feat: Add Spark from_json function facebookincubator/velox#11709)
• to_json (@wecharyu)
• json_object_keys (@dcoliversun)
• json_tuple
• schema_of_csv
• schema_of_json
• to_csv
• make_ym_interval (WIP, @marin-ma)
• make_timestamp (WIP, @marin-ma)
• make_interval
• make_dt_interval
• from_utc_timestamp (@acvictor)
• extract
• exists (@lyy-pineapple)
• date_part
• zip_with
• transform (@Yohahaha)
• transform_keys
• transform_values
• map_from_entries (WIP, MaYan)
• map_filter (WIP, MaYan)
• map_entries (Done, by MaYan)
• map_concat
• forall (@lyy-pineapple)
• flatten (@ivoson)
• filter
• filter (array) (@ivoson)
• width_bucket
• array_sort (@boneanxs)
• xpath
• xpath_boolean
• xpath_double
• xpath_float
• xpath_int
• xpath_long
• xpath_number
• xpath_short
• xpath_string
• unbase64 (WIP, @fyp711)
• decode (partially supported if translated to caseWhen. WIP Cody)
• initcap (WIP, velox PR: 8676)
• unix_date (velox PR 8725, completed)
• count_min_sketch
• bool_and/every (@mskapilks)
• bool_or/any/some (@mskapilks)
• shuffle (completed)
• bround (@xumingming)
• format_string (@gaoyangxiaozhu)
• format_number (@gaoyangxiaozhu)
• soundex (@zhli1142015)
• levenshtein (@zhli1142015)
• cot (@honeyhexin)
• expm1 (@Donvi)
• stack (generator function, @xumingming)
• randn (@Donvi)
• empty2null (internal function, @jinchengchenghh)
• toprettystring (internal function, @jinchengchenghh)
• AtLeastNNonNulls (internal funciton, @zhli1142015)
• GetStructField (internal funciton)

• Since Spark-3.3 (related to ML, low priority)

• regr_count
• regr_avgx
• regr_avgy
• regr_r2
• regr_sxx
• regr_sxy
• regr_syy
• regr_slope
• regr_intercept

• Since Spark-3.3

• Since Spark-3.4

• mode
• get (@Yohahaha)
• array_append (@ivoson)
• array_insert (@ivoson)
• mode (@zhli1142015)