Skip to content

Commit afe4699

Browse files
rokrok
committed
Add GitHub Actions workflow for type checking (mypy, pyright, ty). Add script for including docstrings into stubfiles before building wheels.
1 parent 3049131 commit afe4699

File tree

15 files changed

+296
-7
lines changed

15 files changed

+296
-7
lines changed

‎.github/workflows/python.yml‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -239,6 +239,11 @@ jobs:
239239
- name: Test
240240
shell: bash
241241
run: ci/scripts/python_test.sh $(pwd) $(pwd)/build
242+
- name: Test annotations
243+
shell: bash
244+
env:
245+
PYARROW_TEST_ANNOTATIONS: "ON"
246+
run: ci/scripts/python_test_type_annotations.sh $(pwd)/python
242247

243248
windows:
244249
name: AMD64 Windows 2022 Python 3.13
@@ -296,3 +301,7 @@ jobs:
296301
shell: cmd
297302
run: |
298303
call "ci\scripts\python_test.bat" %cd%
304+
- name: Test annotations
305+
shell: cmd
306+
run: |
307+
call "ci\scripts\python_test_type_annotations.bat" %cd%\python

‎ci/scripts/python_test_type_annotations.bat‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
@rem Licensed to the Apache Software Foundation (ASF) under one
2+
@rem or more contributor license agreements. See the NOTICE file
3+
@rem distributed with this work for additional information
4+
@rem regarding copyright ownership. The ASF licenses this file
5+
@rem to you under the Apache License, Version 2.0 (the
6+
@rem "License"); you may not use this file except in compliance
7+
@rem with the License. You may obtain a copy of the License at
8+
@rem
9+
@rem http://www.apache.org/licenses/LICENSE-2.0
10+
@rem
11+
@rem Unless required by applicable law or agreed to in writing,
12+
@rem software distributed under the License is distributed on an
13+
@rem "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
@rem KIND, either express or implied. See the License for the
15+
@rem specific language governing permissions and limitations
16+
@rem under the License.
17+
18+
@echo on
19+
20+
set PYARROW_DIR=%1
21+
22+
echo Annotation testing on Windows ...
23+
24+
@REM Install library stubs
25+
%PYTHON_CMD% -m pip install pandas-stubs scipy-stubs sphinx types-cffi types-psutil types-requests types-python-dateutil || exit /B 1
26+
27+
@REM Install other dependencies for type checking
28+
%PYTHON_CMD% -m pip install fsspec || exit /B 1
29+
30+
@REM Install type checkers
31+
%PYTHON_CMD% -m pip install mypy pyright ty || exit /B 1
32+
33+
@REM Run type checkers
34+
pushd %PYARROW_DIR%
35+
36+
mypy
37+
pyright
38+
ty check

‎ci/scripts/python_test_type_annotations.sh‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
#!/usr/bin/env bash
2+
#
3+
# Licensed to the Apache Software Foundation (ASF) under one
4+
# or more contributor license agreements. See the NOTICE file
5+
# distributed with this work for additional information
6+
# regarding copyright ownership. The ASF licenses this file
7+
# to you under the Apache License, Version 2.0 (the
8+
# "License"); you may not use this file except in compliance
9+
# with the License. You may obtain a copy of the License at
10+
#
11+
# http://www.apache.org/licenses/LICENSE-2.0
12+
#
13+
# Unless required by applicable law or agreed to in writing,
14+
# software distributed under the License is distributed on an
15+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
16+
# KIND, either express or implied. See the License for the
17+
# specific language governing permissions and limitations
18+
# under the License.
19+
20+
set -ex
21+
pyarrow_dir=${1}
22+
23+
if [ "${PYARROW_TEST_ANNOTATIONS}" == "ON" ]; then
24+
# Install library stubs
25+
pip install pandas-stubs scipy-stubs sphinx types-cffi types-psutil types-requests types-python-dateutil
26+
27+
# Install type checkers
28+
pip install mypy pyright ty
29+
30+
# Install other dependencies for type checking
31+
pip install fsspec
32+
33+
# Run type checkers
34+
pushd ${pyarrow_dir}
35+
mypy
36+
pyright
37+
ty check;
38+
else
39+
echo "Skipping type annotation tests";
40+
fi

‎ci/scripts/python_wheel_macos_build.sh‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -175,6 +175,11 @@ export CMAKE_PREFIX_PATH=${build_dir}/install
175175
export SETUPTOOLS_SCM_PRETEND_VERSION=${PYARROW_VERSION}
176176

177177
pushd ${source_dir}/python
178+
# We first populate stub docstrings and then build the wheel
179+
python setup.py build_ext --inplace
180+
python -m pip install griffe libcst
181+
python ../dev/update_stub_docstrings.py pyarrow-stubs
182+
178183
python setup.py bdist_wheel
179184
popd
180185

‎ci/scripts/python_wheel_validate_contents.py‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,11 @@ def validate_wheel(path):
3535
assert not outliers, f"Unexpected contents in wheel: {sorted(outliers)}"
3636
print(f"The wheel: {wheels[0]} seems valid.")
3737

38+
candidates = [info for info in f.filelist if info.filename.endswith('compute.pyi')]
39+
assert candidates, "compute.pyi not found in wheel"
40+
content = f.read(candidates[0]).decode('utf-8', errors='replace')
41+
assert '"""' in content, "compute.pyi missing docstrings (no triple quotes found)"
42+
3843

3944
def main():
4045
parser = argparse.ArgumentParser()

‎ci/scripts/python_wheel_windows_build.bat‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -135,6 +135,11 @@ pushd C:\arrow\python
135135
@REM Build wheel
136136
%PYTHON_CMD% setup.py bdist_wheel || exit /B 1
137137

138+
@REM We first populate stub docstrings and then build the wheel
139+
%PYTHON_CMD% setup.py build_ext --inplace
140+
%PYTHON_CMD% -m pip install griffe libcst
141+
%PYTHON_CMD% ..\dev\update_stub_docstrings.py pyarrow-stubs
142+
138143
@REM Repair the wheel with delvewheel
139144
@REM
140145
@REM Since we bundled the Arrow C++ libraries ourselves, we only need to

‎ci/scripts/python_wheel_xlinux_build.sh‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -167,6 +167,11 @@ export ARROW_HOME=/tmp/arrow-dist
167167
export CMAKE_PREFIX_PATH=/tmp/arrow-dist
168168

169169
pushd /arrow/python
170+
# We first populate stub docstrings and then build the wheel
171+
python setup.py build_ext --inplace
172+
python -m pip install griffe libcst
173+
python ../dev/update_stub_docstrings.py pyarrow-stubs
174+
170175
python setup.py bdist_wheel
171176

172177
echo "=== Strip symbols from wheel ==="

‎compose.yaml‎

Lines changed: 12 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -919,12 +919,14 @@ services:
919919
environment:
920920
<<: [*common, *ccache, *sccache]
921921
PYTEST_ARGS: # inherit
922+
PYARROW_TEST_ANNOTATIONS: "ON"
922923
volumes: *conda-volumes
923924
command: &python-conda-command
924925
["
925926
/arrow/ci/scripts/cpp_build.sh /arrow /build &&
926927
/arrow/ci/scripts/python_build.sh /arrow /build &&
927-
/arrow/ci/scripts/python_test.sh /arrow"]
928+
/arrow/ci/scripts/python_test.sh /arrow &&
929+
/arrow/ci/scripts/python_test_type_annotations.sh /arrow/python"]
928930

929931
conda-python-emscripten:
930932
# Usage:
@@ -1001,14 +1003,16 @@ services:
10011003
ARROW_S3: "OFF"
10021004
ARROW_SUBSTRAIT: "OFF"
10031005
ARROW_WITH_OPENTELEMETRY: "OFF"
1006+
PYARROW_TEST_ANNOTATIONS: "ON"
10041007
SETUPTOOLS_SCM_PRETEND_VERSION:
10051008
volumes: *ubuntu-volumes
10061009
deploy: *cuda-deploy
10071010
command: &python-command >
10081011
/bin/bash -c "
10091012
/arrow/ci/scripts/cpp_build.sh /arrow /build &&
10101013
/arrow/ci/scripts/python_build.sh /arrow /build &&
1011-
/arrow/ci/scripts/python_test.sh /arrow"
1014+
/arrow/ci/scripts/python_test.sh /arrow &&
1015+
/arrow/ci/scripts/python_test_type_annotations.sh /arrow/python"
10121016

10131017
debian-python:
10141018
# Usage:
@@ -1500,6 +1504,7 @@ services:
15001504
python: ${PYTHON}
15011505
shm_size: *shm-size
15021506
environment:
1507+
PYARROW_TEST_ANNOTATIONS: "ON"
15031508
<<: [*common, *ccache, *sccache]
15041509
PARQUET_REQUIRE_ENCRYPTION: # inherit
15051510
HYPOTHESIS_PROFILE: # inherit
@@ -1510,7 +1515,8 @@ services:
15101515
/arrow/ci/scripts/cpp_build.sh /arrow /build &&
15111516
/arrow/ci/scripts/python_build.sh /arrow /build &&
15121517
mamba uninstall -y numpy &&
1513-
/arrow/ci/scripts/python_test.sh /arrow"]
1518+
/arrow/ci/scripts/python_test.sh /arrow &&
1519+
/arrow/ci/scripts/python_test_type_annotations.sh /arrow/python"]
15141520

15151521
conda-python-docs:
15161522
# Usage:
@@ -1530,13 +1536,15 @@ services:
15301536
BUILD_DOCS_CPP: "ON"
15311537
BUILD_DOCS_PYTHON: "ON"
15321538
PYTEST_ARGS: "--doctest-modules --doctest-cython"
1539+
PYARROW_TEST_ANNOTATIONS: "ON"
15331540
volumes: *conda-volumes
15341541
command:
15351542
["/arrow/ci/scripts/cpp_build.sh /arrow /build &&
15361543
/arrow/ci/scripts/python_build.sh /arrow /build &&
15371544
pip install -e /arrow/dev/archery[numpydoc] &&
15381545
archery numpydoc --allow-rule GL10,PR01,PR03,PR04,PR05,PR10,RT03,YD01 &&
1539-
/arrow/ci/scripts/python_test.sh /arrow"]
1546+
/arrow/ci/scripts/python_test.sh /arrow &&
1547+
/arrow/ci/scripts/python_test_type_annotations.sh /arrow/python"]
15401548

15411549
conda-python-dask:
15421550
# Possible $DASK parameters:

‎docs/source/developers/python/development.rst‎

Lines changed: 69 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -42,7 +42,7 @@ Unit Testing
4242
============
4343

4444
We are using `pytest <https://docs.pytest.org/en/latest/>`_ to develop our unit
45-
test suite. After `building the project <build_pyarrow>`_ you can run its unit tests
45+
test suite. After `building the project <building.html>`_ you can run its unit tests
4646
like so:
4747

4848
.. code-block::
@@ -101,6 +101,74 @@ The test groups currently include:
101101
* ``s3``: Tests for Amazon S3
102102
* ``tensorflow``: Tests that involve TensorFlow
103103

104+
Type Checking
105+
=============
106+
107+
PyArrow provides type stubs (``*.pyi`` files) for static type checking. These
108+
stubs are located in the ``pyarrow-stubs/`` directory and are automatically
109+
included in the distributed wheel packages.
110+
111+
Running Type Checkers
112+
---------------------
113+
114+
We support multiple type checkers. Their configurations are in
115+
``pyproject.toml``.
116+
117+
**mypy**
118+
119+
To run mypy on the PyArrow codebase:
120+
121+
.. code-block::
122+
123+
$ cd arrow/python
124+
$ mypy
125+
126+
The mypy configuration is in the ``[tool.mypy]`` section of ``pyproject.toml``.
127+
128+
**pyright**
129+
130+
To run pyright:
131+
132+
.. code-block::
133+
134+
$ cd arrow/python
135+
$ pyright
136+
137+
The pyright configuration is in the ``[tool.pyright]`` section of ``pyproject.toml``.
138+
139+
**ty**
140+
141+
To run ty (note: currently only partially configured):
142+
143+
.. code-block::
144+
145+
$ cd arrow/python
146+
$ ty check
147+
148+
Maintaining Type Stubs
149+
-----------------------
150+
151+
Type stubs for PyArrow are maintained in the ``pyarrow-stubs/``
152+
directory. These stubs mirror the structure of the main ``pyarrow/`` package.
153+
154+
When adding or modifying public APIs:
155+
156+
1. **Update the corresponding ``.pyi`` stub file** in ``pyarrow-stubs/``
157+
to reflect the new or changed function/class signatures.
158+
159+
2. **Include type annotations** where possible. For Cython modules or
160+
dynamically generated APIs such as compute kernels add the corresponding
161+
stub in ``pyarrow-stubs/``.
162+
163+
3. **Run type checkers** to ensure the stubs are correct and complete.
164+
165+
The stub files are automatically copied into the built wheel during the build
166+
process and will be included when users install PyArrow, enabling type checking
167+
in downstream projects and for users' IDEs.
168+
169+
Note: ``py.typed`` marker file in the ``pyarrow/`` directory indicates to type
170+
checkers that PyArrow supports type checking according to :pep:`561`.
171+
104172
Doctest
105173
=======
106174

‎python/MANIFEST.in‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@ include ../NOTICE.txt
44

55
global-include CMakeLists.txt
66
graft pyarrow
7+
graft pyarrow-stubs
78
graft cmake_modules
89

910
global-exclude *.so

0 commit comments

Comments
 (0)