Skip to content

list.eval with scalar aggregation expressions produces wrong results for empty inner lists #26850

@gautamvarmadatla

Description

@gautamvarmadatla

Checks

  • I have checked that this issue has not already been reported.
  • I have confirmed this bug exists on the latest version of Polars.

Reproducible example

import polars as pl

print("=== Slow path: empty list becomes a phantom scalar result ===")
s = pl.Series([[1, 2], [], [3, 4]])

print("sum  got:   ", s.list.eval(pl.element().sum()).to_list())
print("sum  expect:", [[3], [], [7]])

print("max  got:   ", s.list.eval(pl.element().max()).to_list())
print("max  expect:", [[2], [], [4]])

print("mean got:   ", s.list.eval(pl.element().mean()).to_list())
print("mean expect:", [[1.5], [], [3.5]])

print("\n=== Fast path: adjacent row corruption ===")
s2 = pl.Series([[1, 2], [], [3]])
print("sum  got:   ", s2.list.eval(pl.element().sum()).to_list())
print("sum  expect:", [[3], [], [3]])

print("\n=== Elementwise control (works correctly) ===")
print("got:   ", s.list.eval(pl.element() * 2).to_list())
print("expect:", [[2, 4], [], [6, 8]])

Log output

=== Slow path: empty list becomes a phantom scalar result ===
sum  got:    [[3], [0], [7]]
sum  expect: [[3], [], [7]]
max  got:    [[2], [None], [4]]
max  expect: [[2], [], [4]]
mean got:    [[1.5], [None], [3.5]]
mean expect: [[1.5], [], [3.5]]

=== Fast path: adjacent row corruption ===
sum  got:    [[3, 0], [], [3]]
sum  expect: [[3], [], [3]]

=== Elementwise control (works correctly) ===
got:    [[2, 4], [], [6, 8]]
expect: [[2, 4], [], [6, 8]]

Issue description

list.eval with scalar aggregation expressions can silently return incorrect results when at least one valid empty inner list is present. Two failure modes exist. If the scalar output length happens to equal the original flattened inner-element count, the fast path can reuse the original list offsets with the new aggregated values, causing row-boundary corruption (for example [[3,0],[],[3]] instead of [[3],[],[3]]). Otherwise, the slow path converts each scalar result into a one-element list, so the empty inner list incorrectly contributes a phantom singleton (for example [[3],[0],[7]] instead of [[3],[],[7]]).

This is caused because evaluate_on_list_chunked builds groups directly from list offsets for every valid outer row, so valid empty inner lists become zero-length groups and are still evaluated. The scalar result produced for that empty group is then either reinterpreted under the original offsets in the fast path or emitted as a singleton list in the slow path.

Expected behavior

A valid empty inner list should remain empty in the output list structure. It should not produce a phantom scalar singleton and should not affect adjacent rows

Installed versions

Details
--------Version info---------
Polars:              1.38.1
Index type:          UInt32
Platform:            Linux-6.6.87.2-microsoft-standard-WSL2-x86_64-with-glibc2.39
Python:              3.12.12 | packaged by conda-forge | (main, Oct 22 2025, 23:25:55) [GCC 14.3.0]
Runtime:             rt32

----Optional dependencies----
Azure CLI            <not installed>
adbc_driver_manager  1.10.0
altair               6.0.0
azure.identity       1.25.2
boto3                1.42.49
cloudpickle          3.1.2
connectorx           0.4.5
deltalake            1.4.2
fastexcel            0.19.0
fsspec               2026.2.0
gevent               25.9.1
google.auth          2.48.0
great_tables         0.20.0
matplotlib           3.10.8
numpy                2.4.2
openpyxl             3.1.5
pandas               3.0.1
polars_cloud         0.5.0
pyarrow              23.0.1
pydantic             2.12.5
pyiceberg            0.11.0
sqlalchemy           2.0.46
torch                <not installed>
xlsx2csv             0.8.6
xlsxwriter           3.2.9

Metadata

Metadata

Assignees

No one assigned

    Labels

    A-dtype-list/arrayArea: list/array data typeP-highPriority: highacceptedReady for implementationbugSomething isn't workingpythonRelated to Python PolarsregressionIssue introduced by a new release

    Type

    No type

    Projects

    Status

    Ready

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions