refactor(expr-ir): Make `arrow.functions` a package #3384

dangotbanned · 2026-01-02T12:15:40Z

Description

The size (2K+ LOC!) of (https://github.com/narwhals-dev/narwhals/blob/8d6101cdc56fd9d7d89e8a4665afbfdee8b6e9ec/narwhals/_plan/arrow/functions.py) really got out of hand and was long-overdue a cleanup.

The bad

Difficult to remember everything that's there
- Not having docs made this part worse, while only really making the LOC appear less-bad
Having all this typing in one module degrades LSP performance
Unclear what dependencies there are between functions
Shadowing builtins gets more complicated the larger a module is

The (new) good

New modules have an __all__
Most exports have at-least a 1-line docstring
LSP is pretty fast now that you'd need to have ~20 modules open to have same number of symbols 😅
No modules have intra-(sub)package inline dependencies
- Quite nice seeing all the relationships up top now
Shadowing (any, all, min, max, str, list) is a non-issue
list and str functions can freely use the same names

Related issues

Child of feat(RFC): A richer Expr IR #2572

Will gradually split things into modules, then import those modules as aliases

What a cute lil module we have there

This is such a badly named method! `polars` doesn't have an equivalent AFAIK

narwhals/_plan/arrow/functions/_vector.py

- Was avoiding using `dtypes` before as *at some point* the `dtypes` module was imported into `functions`. - No longer an issue here though - Found a couple more uses for `dtype_native` too

That was a lot more involved than I'd hoped for! Kinda forced my hand on addressing some circular imports from #3384 (comment)

for more information, see https://pre-commit.ci

#3384 (comment)

In #3384, I started tweaking the impl but realized I had only covered it indirectly Merging ahead of that so I can be sure I don't break it

- Not planning to do this *everywhere* - in `_vector` I want everything except `hist` - They're all good for composing other functions - So if we can skip re-wrapping as a `ChunkedArray`, then I think we should

Quite a lot of inconsistency, but inheriting from polars so idk

Will stick with the name for now Makes a bit more sense without `drop_nulls` - and that didn't need to exist in the first place

#3384 (comment)

dangotbanned added 28 commits December 31, 2025 21:47

refactor(expr-ir): Make arrow.functions.py a package

a23077e

Will gradually split things into modules, then import those modules as aliases

refactor(expr-ir): Split _dtypes

14c4fec

refactor(expr-ir): Split _categorical

3e99ee6

What a cute lil module we have there

refactor(expr-ir): Split _construction

e5e7158

refactor(expr-ir): Split _ranges

154029d

docs: covert enough for me to miss!

a2b3f91

refactor: Split _repeat

58759f7

refactor(expr-ir): Split _bin_op, _boolean, _common

c442641

refactor(expr-ir): Finish migrating to _boolean

05f343e

refactor: Just use flatten instead

06b42b8

This is such a badly named method! `polars` doesn't have an equivalent AFAIK

refactor(expr-ir): Split _struct

23a4e3e

refactor(expr-ir): Split _strings

be1f23f

refactor: Move concat_tables re-export

bd56eaa

refactor: Split _cumulative

cf17cb9

refactor(expr-ir): Split _multiplex

2a9de00

update notes

b5edd39

chore: Add some missing exports

042cf1f

refactor: Align dtype names with polars

040d9af

refactor(expr-ir): Split _sort

81838fb

typo

e34f0c3

refactor(expr-ir): Split _lists

24c8687

refactor(expr-ir): Split _aggregation

c769732

refactor(expr-ir): Split _arithmetic, _horizontal, _round

4afee53

refactor(expr-ir): Split _vector

5328d17

distribute the stragglers

0cd4e06

refactor: Un-suffix export modules

7c536bb

refactor: Add arrow.functions.__all__

df28b38

refactor: any_ , all_ -> any, all

1a2ef02

dangotbanned added internal pyarrow Issue is related to pyarrow backend labels Jan 2, 2026