Skip to content

fix(pyarrow): handle non-positive width in str.zfill#3770

Open
mkzung wants to merge 1 commit into
narwhals-dev:mainfrom
mkzung:fix/zfill-nonpositive-width
Open

fix(pyarrow): handle non-positive width in str.zfill#3770
mkzung wants to merge 1 commit into
narwhals-dev:mainfrom
mkzung:fix/zfill-nonpositive-width

Conversation

@mkzung

@mkzung mkzung commented Jul 10, 2026

Copy link
Copy Markdown

Description

On the PyArrow backend, Series.str.zfill(width) and Expr.str.zfill(width) crash with pyarrow.lib.ArrowInvalid: Negative buffer resize whenever width <= 0.

The result is built with pc.case_when(...), and one of the branches is pc.utf8_lpad(remaining_chars, width - 1, padding="0"). pc.case_when evaluates every branch eagerly, so for width <= 0 the width - 1 pad length is negative and PyArrow raises before any row is selected.

This handles the two non-positive cases the way the other backends do. A width of 0 is a no-op, since every string is already at least 0 long, so it returns the input unchanged. A negative width raises InvalidOperationError, matching Polars, whose zfill argument is unsigned. PyArrow was the only backend that crashed on either one.

Reproduction (before this change):

import narwhals as nw, pyarrow as pa
s = nw.from_native(pa.table({"a": ["1", "-5", "abc"]}), eager_only=True)["a"]
s.str.zfill(0)  # pyarrow.lib.ArrowInvalid: Negative buffer resize

The regression tests are scoped to PyArrow, since that is the backend that crashed.

What type of PR is this? (check all applicable)

  • 💾 Refactor
  • ✨ Feature
  • 🐛 Bug Fix
  • 🔧 Optimization
  • 📝 Documentation
  • ✅ Test
  • 🐳 Other

Related issues

  • No existing issue; found while exercising the string namespace on the PyArrow backend.

AI assistance

  • No AI tools were used for this PR.
  • AI tools were used.

I used an AI coding assistant to help locate the root cause and draft the regression test. I have read, run, and verified every line, and I take full responsibility for the change.

Checklist

  • Code follows style guide (ruff)
  • Tests added
  • Documented the changes (no doc change needed; behaviour matches the documented str.zfill)

@MarcoGorelli

Copy link
Copy Markdown
Member

thanks @mkzung !

If you would rather the Arrow backend raise on a negative width to match Polars
instead of returning unchanged, I am happy to switch.

i think i'd prefer that, could we do that instead please? thanks 🙏

@mkzung mkzung force-pushed the fix/zfill-nonpositive-width branch from ce7c603 to 0363a10 Compare July 10, 2026 21:30
@mkzung mkzung changed the title fix(pyarrow): don't crash on str.zfill with non-positive width fix(pyarrow): handle non-positive width in str.zfill Jul 10, 2026
@mkzung

mkzung commented Jul 10, 2026

Copy link
Copy Markdown
Author

Done. The Arrow backend now raises InvalidOperationError on a negative width, matching Polars, instead of returning unchanged, and width 0 stays a no-op. I also moved the regression tests to pyarrow only, since Polars' own zfill(0) raises on the older CI versions.

# Match that here rather than letting pyarrow's utf8_lpad(width - 1)
# crash later with a negative pad width.
msg = f"zfill does not support a negative width, got {width}."
raise InvalidOperationError(msg)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@MarcoGorelli should we push this check up to narwhals/expr_str.py. If we do that then

  1. The error can be raised as early as possible for all backends
  2. All implementations will have a uniform error message

If not, then I'm happy with the implementation here.

@mkzung

mkzung commented Jul 14, 2026

Copy link
Copy Markdown
Author

Worth adding that the inconsistency is wider than zfill. With a negative width today, pandas returns the string unchanged, Polars raises conversion from 'i128' to 'u64' failed, and pyarrow crashes inside utf8_lpad. pad_start and pad_end do the same thing - on pyarrow they leak ArrowInvalid: Negative buffer resize.

So moving the check up to expr_str/series_str fixes three methods at once and gives every backend the same error. I have it on a branch: validation in Expr.str and Series.str, the Arrow negative-width raise removed (the width == 0 no-op stays, that one is a genuine pyarrow quirk), tests parametrised across backends, full suite green.

mkzung/narwhals@main...mkzung:narwhals:fix/zfill-validate-in-public-layer

Happy to fold it into this PR, or to leave it until you and @MarcoGorelli have settled where the check belongs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants