Skip to content

fix: Restore pyarrow predicate conversion for is_in#26811

Open
erandagan wants to merge 2 commits intopola-rs:mainfrom
erandagan:fix/is-in-pyarrow-predicate-pushdown
Open

fix: Restore pyarrow predicate conversion for is_in#26811
erandagan wants to merge 2 commits intopola-rs:mainfrom
erandagan:fix/is-in-pyarrow-predicate-pushdown

Conversation

@erandagan
Copy link

Closes #26802

AI DIsclosure:

  1. I used Claude Code while working on this pull request.
  2. I confirm that I have reviewed all changes myself, and I believe they are relevant and correct.

@github-actions github-actions bot added fix Bug fix python Related to Python Polars rust Related to Rust Polars first-contribution First contribution by user labels Mar 5, 2026
@codecov
Copy link

codecov bot commented Mar 5, 2026

Codecov Report

❌ Patch coverage is 69.56522% with 14 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.66%. Comparing base (9f1a742) to head (d491925).
⚠️ Report is 46 commits behind head on main.

Files with missing lines Patch % Lines
crates/polars-plan/src/plans/python/pyarrow.rs 69.56% 14 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #26811      +/-   ##
==========================================
+ Coverage   81.30%   81.66%   +0.35%     
==========================================
  Files        1802     1805       +3     
  Lines      246972   248038    +1066     
  Branches     3086     3132      +46     
==========================================
+ Hits       200810   202549    +1739     
+ Misses      45371    44683     -688     
- Partials      791      806      +15     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copy link
Collaborator

@nameexhaustion nameexhaustion left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doesn't seem right.. we cannot apply the current conversion on AnyValue::List as we would lose a nesting level on list conversion.

Could you re-work this to ensure the conversion happens as an exception only for the RHS of the is_in() operation?

@erandagan
Copy link
Author

erandagan commented Mar 10, 2026

This doesn't seem right.. we cannot apply the current conversion on AnyValue::List as we would lose a nesting level on list conversion.

Could you re-work this to ensure the conversion happens as an exception only for the RHS of the is_in() operation?

@nameexhaustion Apologies for missing this, I'm not too familiar with the intricacies of the codebase. Pushed a fix.

let mut args = args;
args.allow_literal_series = true;
let values = predicate_to_pa(input.get(1)?.node(), expr_arena, args)?;
let values = predicate_to_pa(rhs_node, expr_arena, args)
Copy link
Collaborator

@nameexhaustion nameexhaustion Mar 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we make this only do the conversion if we have Literal(List(s)) (i.e. remove the leading predicate_to_pa().or_else. It's otherwise not correct to convert the RHS since the semantics are different

@nameexhaustion nameexhaustion changed the title fix: Restore serialization of is_in pushdown fix: Restore pyarrow predicate conversion for is_in Mar 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

first-contribution First contribution by user fix Bug fix python Related to Python Polars rust Related to Rust Polars

Projects

None yet

Development

Successfully merging this pull request may close these issues.

is_in predicate pushdown silently broken for scan_pyarrow_dataset since 1.28

2 participants