Skip to content

Conversation

goldmedal
Copy link
Contributor

@goldmedal
Copy link
Contributor Author

@alamb, What do you think about this? I think this change is related to the default behavior of DataFusion.

@alamb alamb changed the title enable supports_filter_during_aggregation for Generic enable supports_filter_during_aggregation for Generic dialect Apr 17, 2025
@alamb
Copy link
Contributor

alamb commented Apr 17, 2025

Makes sense to me @goldmedal -- I think the intent is that the GenericDialect should support as many SQL constructs as possible

The only thing I think this PR needs is a test and it should be good to go

@goldmedal
Copy link
Contributor Author

The only thing I think this PR needs is a test and it should be good to go

oops , I forgot to mention the tests. I found it will be covered by the original case:

fn test_selective_aggregation() {
let sql = concat!(
"SELECT ",
"ARRAY_AGG(name) FILTER (WHERE name IS NOT NULL), ",
"ARRAY_AGG(name) FILTER (WHERE name LIKE 'a%') AS agg2 ",
"FROM region"
);
assert_eq!(
all_dialects_where(|d| d.supports_filter_during_aggregation())
.verified_only_select(sql)
.projection,
vec![

I guess it's enough?

@alamb
Copy link
Contributor

alamb commented Apr 17, 2025

oops , I forgot to mention the tests. I found it will be covered by the original case:

I was thinking a test that would prevent against regressions in a future refactor -- namely a test that will fail if this code change is reverted,

I think as it stands, if the code change is reverted all the tests will still pass 🤔

@goldmedal
Copy link
Contributor Author

oops , I forgot to mention the tests. I found it will be covered by the original case:

I was thinking a test that would prevent against regressions in a future refactor -- namely a test that will fail if this code change is reverted,

I think as it stands, if the code change is reverted all the tests will still pass 🤔

I see. Make sense. I'll add another test to protect it.

Comment on lines +11645 to +11657
let testing_dialects = all_dialects_where(|d| d.supports_filter_during_aggregation());
let expected_dialects: Vec<Box<dyn Dialect>> = vec![
Box::new(PostgreSqlDialect {}),
Box::new(DatabricksDialect {}),
Box::new(HiveDialect {}),
Box::new(SQLiteDialect {}),
Box::new(DuckDbDialect {}),
Box::new(GenericDialect {}),
];
assert_eq!(testing_dialects.dialects.len(), expected_dialects.len());
expected_dialects
.into_iter()
.for_each(|d| assert!(d.supports_filter_during_aggregation()));
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I list all the dialects expected to support filtering during aggregation.

@goldmedal goldmedal requested a review from alamb April 18, 2025 13:34
Copy link
Contributor

@alamb alamb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @goldmedal

FYI @iffyio

@alamb alamb merged commit 3ec80e1 into apache:main Apr 19, 2025
9 checks passed
@goldmedal goldmedal deleted the feat/generic-enable-filter-agg branch April 28, 2025 13:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants