Skip to content

feat: Add support for RegExpExtract/RegExpExtractAll #2831

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
danielhumanmod wants to merge 14 commits into
base: main
Choose a base branch
from
Open

feat: Add support for RegExpExtract/RegExpExtractAll #2831

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
68f74ae
prototype
danielhumanmod Nov 30, 2025
4dbed77
refactor strings.scala
danielhumanmod Nov 30, 2025
f101362
test, format and configs
danielhumanmod Dec 1, 2025
ff1ebd6
make regexp_extract more align with spark's behavior
danielhumanmod Dec 1, 2025
87dfed4
Solve comments (test not yet fixed)
danielhumanmod Dec 3, 2025
d83cac5
fix regexp_extract_all test failure
danielhumanmod Dec 3, 2025
f2d82b3
refactor udf impl
danielhumanmod Dec 7, 2025
84c0132
fix rust lint
danielhumanmod Dec 7, 2025
a55263f
minor updates
danielhumanmod Dec 7, 2025
5de2413
Merge remote-tracking branch 'apache/main' into regexp-extract-impl
andygrove Dec 22, 2025
4b599d9
fix escape issue in test
danielhumanmod Dec 29, 2025
65b2390
Merge remote-tracking branch 'apache/main' into regexp-extract-impl
danielhumanmod Dec 29, 2025
e11a3ef
Merge branch 'main' into regexp-extract-impl
danielhumanmod Dec 29, 2025
9e95bf5
fix format
danielhumanmod Jan 8, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions docs/source/user-guide/latest/configs.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -294,6 +294,8 @@ These settings can be used to determine which parts of the plan are accelerated
| `spark.comet.expression.RLike.enabled` | Enable Comet acceleration for `RLike` | true |
| `spark.comet.expression.Rand.enabled` | Enable Comet acceleration for `Rand` | true |
| `spark.comet.expression.Randn.enabled` | Enable Comet acceleration for `Randn` | true |
| `spark.comet.expression.RegExpExtract.enabled` | Enable Comet acceleration for `RegExpExtract` | true |
| `spark.comet.expression.RegExpExtractAll.enabled` | Enable Comet acceleration for `RegExpExtractAll` | true |
Copy link
Contributor

@coderfender coderfender Jan 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps the default configs can be false ? (given that COMET_REGEXP_ALLOW_INCOMPATIBLE is defaulted to false)

| `spark.comet.expression.RegExpReplace.enabled` | Enable Comet acceleration for `RegExpReplace` | true |
| `spark.comet.expression.Remainder.enabled` | Enable Comet acceleration for `Remainder` | true |
| `spark.comet.expression.Reverse.enabled` | Enable Comet acceleration for `Reverse` | true |
Expand Down
6 changes: 4 additions & 2 deletions native/spark-expr/src/comet_scalar_funcs.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,8 +22,8 @@ use crate::math_funcs::modulo_expr::spark_modulo;
use crate::{
spark_array_repeat, spark_ceil, spark_decimal_div, spark_decimal_integral_div, spark_floor,
spark_isnan, spark_lpad, spark_make_decimal, spark_read_side_padding, spark_round, spark_rpad,
spark_unhex, spark_unscaled_value, EvalMode, SparkBitwiseCount, SparkDateTrunc, SparkSizeFunc,
SparkStringSpace,
spark_unhex, spark_unscaled_value, EvalMode, SparkBitwiseCount, SparkDateTrunc,
SparkRegExpExtract, SparkRegExpExtractAll, SparkSizeFunc, SparkStringSpace,
};
use arrow::datatypes::DataType;
use datafusion::common::{DataFusionError, Result as DataFusionResult};
Expand Down Expand Up @@ -195,6 +195,8 @@ fn all_scalar_functions() -> Vec<Arc<ScalarUDF>> {
Arc::new(ScalarUDF::new_from_impl(SparkDateTrunc::default())),
Arc::new(ScalarUDF::new_from_impl(SparkStringSpace::default())),
Arc::new(ScalarUDF::new_from_impl(SparkSizeFunc::default())),
Arc::new(ScalarUDF::new_from_impl(SparkRegExpExtract::default())),
Arc::new(ScalarUDF::new_from_impl(SparkRegExpExtractAll::default())),
]
}

Expand Down
2 changes: 2 additions & 0 deletions native/spark-expr/src/string_funcs/mod.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,10 @@
// specific language governing permissions and limitations
// under the License.

mod regexp_extract;
mod string_space;
mod substring;

pub use regexp_extract::{SparkRegExpExtract, SparkRegExpExtractAll};
pub use string_space::SparkStringSpace;
pub use substring::SubstringExpr;
Loading