Skip to content

Feat : Bringing in support for map_filter expression. #2236

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
+587 −10
Open

Feat : Bringing in support for map_filter expression. #2236

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
28eabf4
Feat : Introducing support for map_filter expression.
codetyri0n Aug 26, 2025
6fe9e9b
Remove duplicated line.
codetyri0n Aug 26, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 3 additions & 6 deletions native/spark-expr/src/comet_scalar_funcs.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,12 +18,8 @@
use crate::hash_funcs::*;
use crate::math_funcs::checked_arithmetic::{checked_add, checked_div, checked_mul, checked_sub};
use crate::math_funcs::modulo_expr::spark_modulo;
use crate::{
spark_array_repeat, spark_ceil, spark_date_add, spark_date_sub, spark_decimal_div,
spark_decimal_integral_div, spark_floor, spark_hex, spark_isnan, spark_make_decimal,
spark_read_side_padding, spark_round, spark_rpad, spark_unhex, spark_unscaled_value,
SparkBitwiseCount, SparkBitwiseGet, SparkBitwiseNot, SparkDateTrunc, SparkStringSpace,
};
use crate::{spark_array_repeat, spark_ceil, spark_date_add, spark_date_sub, spark_decimal_div, spark_decimal_integral_div, spark_floor, spark_hex, spark_isnan, spark_make_decimal, spark_read_side_padding, spark_round, spark_rpad, spark_unhex, spark_unscaled_value, SparkBitwiseCount, SparkBitwiseGet, SparkBitwiseNot, SparkDateTrunc, SparkStringSpace};
use crate::map_funcs::map_filter::SparkMapFilter;
use arrow::datatypes::DataType;
use datafusion::common::{DataFusionError, Result as DataFusionResult};
use datafusion::execution::FunctionRegistry;
Expand Down Expand Up @@ -172,6 +168,7 @@ fn all_scalar_functions() -> Vec<Arc<ScalarUDF>> {
Arc::new(ScalarUDF::new_from_impl(SparkBitwiseGet::default())),
Arc::new(ScalarUDF::new_from_impl(SparkDateTrunc::default())),
Arc::new(ScalarUDF::new_from_impl(SparkStringSpace::default())),
Arc::new(ScalarUDF::new_from_impl(SparkMapFilter::default())),
]
}

Expand Down
6 changes: 2 additions & 4 deletions native/spark-expr/src/lib.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,19 +42,16 @@ mod array_funcs;
mod bitwise_funcs;
mod comet_scalar_funcs;
pub mod hash_funcs;

mod string_funcs;

mod datetime_funcs;
pub use agg_funcs::*;

pub use cast::{spark_cast, Cast, SparkCastOptions};

mod bloom_filter;
pub use bloom_filter::{BloomFilterAgg, BloomFilterMightContain};

mod conditional_funcs;
mod conversion_funcs;
mod map_funcs;
mod math_funcs;
mod nondetermenistic_funcs;

Expand All @@ -78,6 +75,7 @@ pub use math_funcs::{
spark_unhex, spark_unscaled_value, CheckOverflow, NegativeExpr, NormalizeNaNAndZero,
};
pub use string_funcs::*;
pub use map_funcs::*;

/// Spark supports three evaluation modes when evaluating expressions, which affect
/// the behavior when processing input values that are invalid or would result in an
Expand Down
Loading
Loading