GH-45457: [Python] Add `pyarrow.ArrayStatistics` by kou · Pull Request #45550 · apache/arrow

kou · 2025-02-17T06:06:13Z

Rationale for this change

Apache Arrow C++ can attach statistics read from Apache Parquet data to arrow::Array. If we have the bindings of the feature in Python, Python users can also use attached statistics.

What changes are included in this PR?

Add pyarrow.ArrayStatistics
Add pyarrow.Array.statistics().

Are these changes tested?

Yes.

Are there any user-facing changes?

Yes.

GitHub Issue: [Python] Add arrow::ArrayStatistics bindings #45457

kou · 2025-02-17T06:06:37Z

@github-actions crossbow submit -g python

github-actions · 2025-02-17T06:06:38Z

⚠️ GitHub issue #45457 has been automatically assigned in GitHub to PR creator.

kou · 2025-02-20T20:32:41Z

@pitrou @jorisvandenbossche Could you take a look at this?

kou · 2025-02-24T06:10:36Z

I'll merge this in a few days if nobody objects it.

pitrou

Thanks a lot @kou ! Some minor comments below, but LGTM in general.

python/pyarrow/array.pxi

pitrou · 2025-02-24T14:07:39Z

python/pyarrow/array.pxi

For the record, I've opened a Cython feature request to make this more automatic.

Thanks! I've added a comment that refers the issue.

pitrou · 2025-02-24T14:09:05Z

python/pyarrow/array.pxi

uint64_t isn't handled below, should the docstring or the code be fixed?

Oh... The code was wrong... I've added the uint64_t case.

pitrou · 2025-02-24T14:10:26Z

python/pyarrow/array.pxi

except * means it could raise Python exceptions, but it doesn't here, so perhaps you can remove that annotation (though it's not really a problem either).

Thanks! I didn't know much about except in Cython...

python/pyarrow/array.pxi

It's the bindings of `arrow::ArrayStatistics`. You can get it by `pyarrow.Array.statistics()`.

Co-authored-by: Antoine Pitrou <pitrou@free.fr>

kou · 2025-02-25T05:16:50Z

@github-actions crossbow submit -g python

kou · 2025-02-25T06:57:12Z

@github-actions crossbow submit -g python

pitrou · 2025-02-25T08:58:20Z

python/pyarrow/tests/parquet/test_parquet_file.py

+    assert statistics.min == -1
+    assert statistics.is_min_exact
+    assert statistics.max == 3
+    assert statistics.is_max_exact


Can we have a test for repr(statistics) to make sure that the string representation works?

It's a good idea. I've added it.

kou · 2025-02-25T11:10:08Z

@github-actions crossbow submit -g python

github-actions · 2025-02-25T11:12:50Z

Revision: e3a20b5

Submitted crossbow builds: ursacomputing/crossbow @ actions-747dbaddf2

Task	Status
example-python-minimal-build-fedora-conda
example-python-minimal-build-ubuntu-venv
test-conda-python-3.10
test-conda-python-3.10-hdfs-2.9.2
test-conda-python-3.10-hdfs-3.2.1
test-conda-python-3.10-pandas-latest-numpy-latest
test-conda-python-3.11
test-conda-python-3.11-dask-latest
test-conda-python-3.11-dask-upstream_devel
test-conda-python-3.11-hypothesis
test-conda-python-3.11-pandas-latest-numpy-1.26
test-conda-python-3.11-pandas-latest-numpy-latest
test-conda-python-3.11-pandas-nightly-numpy-nightly
test-conda-python-3.11-pandas-upstream_devel-numpy-nightly
test-conda-python-3.11-spark-master
test-conda-python-3.12
test-conda-python-3.12-cpython-debug
test-conda-python-3.13
test-conda-python-3.9
test-conda-python-3.9-pandas-1.1.3-numpy-1.19.5
test-conda-python-emscripten
test-cuda-python-ubuntu-22.04-cuda-11.7.1
test-debian-12-python-3-amd64
test-debian-12-python-3-i386
test-fedora-39-python-3
test-ubuntu-22.04-python-3
test-ubuntu-22.04-python-313-freethreading
test-ubuntu-24.04-python-3

conbench-apache-arrow · 2025-02-25T20:07:58Z

After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit 631fa0a.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 12 possible false positives for unstable benchmarks that are known to sometimes produce them.

github-actions bot added Component: Python awaiting committer review Awaiting committer review labels Feb 17, 2025

This comment was marked as outdated.

Sign in to view

kou requested review from jorisvandenbossche and pitrou and removed request for pitrou February 18, 2025 00:45

pitrou reviewed Feb 24, 2025

View reviewed changes

github-actions bot added awaiting changes Awaiting changes and removed awaiting committer review Awaiting committer review labels Feb 25, 2025

kou requested review from assignUser, jonkeane, raulcd, thisisnic, westonpace and wgtmac as code owners February 25, 2025 05:14

github-actions bot added Component: R Component: Parquet Component: C++ Component: Documentation awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Feb 25, 2025

kou and others added 4 commits February 25, 2025 14:15

apacheGH-45457: [Python] Add pyarrow.ArrayStatistics

cd2aa85

It's the bindings of `arrow::ArrayStatistics`. You can get it by `pyarrow.Array.statistics()`.

Use f-string

49bf5ef

Co-authored-by: Antoine Pitrou <pitrou@free.fr>

Use f-string

5837501

Co-authored-by: Antoine Pitrou <pitrou@free.fr>

Format

d5a15f5

kou removed request for assignUser, jonkeane, raulcd, thisisnic, westonpace and wgtmac February 25, 2025 05:16

Fix wrong change

bca14f5

This comment was marked as outdated.

Sign in to view

github-actions bot added awaiting changes Awaiting changes and removed awaiting change review Awaiting change review labels Feb 25, 2025

Fix English

fd6d18e

github-actions bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Feb 25, 2025

This comment was marked as outdated.

Sign in to view

pitrou reviewed Feb 25, 2025

View reviewed changes

Assert repr() too

e3a20b5

github-actions bot added awaiting changes Awaiting changes and removed awaiting change review Awaiting change review labels Feb 25, 2025

pitrou approved these changes Feb 25, 2025

View reviewed changes

kou merged commit 631fa0a into apache:main Feb 25, 2025
15 checks passed

kou removed the awaiting changes Awaiting changes label Feb 25, 2025

kou deleted the python-array-statistics branch February 25, 2025 13:25

kou mentioned this pull request Feb 25, 2025

[Python] Add arrow::ArrayStatistics bindings #45457

Closed

kou mentioned this pull request Mar 3, 2025

[CI][Python] test_cython_api is failing for Windows wheel #45651

Closed

Conversation

kou commented Feb 17, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

kou commented Feb 17, 2025

Uh oh!

github-actions bot commented Feb 17, 2025

Uh oh!

This comment was marked as outdated.

kou commented Feb 20, 2025

Uh oh!

kou commented Feb 24, 2025

Uh oh!

pitrou left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kou commented Feb 25, 2025

Uh oh!

This comment was marked as outdated.

kou commented Feb 25, 2025

Uh oh!

This comment was marked as outdated.

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kou commented Feb 25, 2025

Uh oh!

github-actions bot commented Feb 25, 2025

Uh oh!

Uh oh!

conbench-apache-arrow bot commented Feb 25, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

kou commented Feb 17, 2025 •

edited by github-actions bot

Loading