GH-32609: [CI] Add type checking infrastructure and CI workflow for type annotations #48618

rok · 2025-12-22T00:15:41Z

Rationale for this change

This is the first in series of PRs adding type annotations to pyarrow and resolving #32609.

What changes are included in this PR?

This PR establishes infrastructure for type checking:

Adds CI workflow for running mypy, pyright, and ty type checkers on linux, macos and windows
Configures type checkers to validate stub files (excluding source files for now)
Adds PEP 561 py.typed marker to enable type checking
Updates wheel build scripts to include stub files in distributions
Creates initial minimal stub directory structure
Updates developer documentation with type checking workflow

Are these changes tested?

No. This is mostly a CI change.

Are there any user-facing changes?

This does not add any actual annotations (only py.typed marker) so user should not be affected.

GitHub Issue: [Python] Type checking support #32609

AlenkaF

Thank you @rok for working on this! Splitting up the main PR sounds like a great idea to me.

I only have one small question, otherwise the changes look good to me.

ci/scripts/python_test_type_annotations.sh

rok · 2025-12-23T10:26:37Z

Thanks for review @AlenkaF ! I think we probably want @raulcd to look at this too before merging?

AlenkaF · 2025-12-23T10:32:04Z

Yes, let's wait for Raul 👍

raulcd

Thanks for the PR @rok sorry it took me a while to review with holidays, release, etcetera

ci/scripts/python_test_type_annotations.bat

ci/scripts/python_test_type_annotations.sh

docs/source/developers/python/development.rst

ci/scripts/python_wheel_validate_contents.py

ci/scripts/python_wheel_xlinux_build.sh

raulcd · 2026-01-13T19:27:53Z

@github-actions crossbow submit wheel-manylinux-2-28-cp311-cp311-amd64

github-actions · 2026-01-13T19:30:08Z

Revision: 28fba3a

Submitted crossbow builds: ursacomputing/crossbow @ actions-8a459250fe

Task	Status
wheel-manylinux-2-28-cp311-cp311-amd64

rok · 2026-01-14T10:41:02Z

compose.yaml

      ARROW_S3: "OFF"
      ARROW_SUBSTRAIT: "OFF"
      ARROW_WITH_OPENTELEMETRY: "OFF"
+      PYARROW_TEST_ANNOTATIONS: "ON"


@raulcd do you think we actually want to be able to toggle these? I'm thinking we don't really need them

I think is worth it if instead of using a separate ci/scripts/python_test_type_annotations.sh we use ci/scripts/python_test.sh and we use the PYARROW_TEST_ANNOTATIONS variable to manage whether we want annotations to be tested or not as part of python testing.
If we decide we want to have two separate scripts I think this env var should be removed.
I lean towards having a single python test script that can allow us to also test annotations but I can find arguments for both so not an issue.

Ok, agreed with your reasoning. I also introduced PYARROW_TEST_ANNOTATIONS to the windows script.

rok · 2026-01-14T10:41:33Z

Thanks for review @raulcd ! I've responded to your points.

raulcd · 2026-01-14T11:41:13Z

compose.yaml

      ARROW_S3: "OFF"
      ARROW_SUBSTRAIT: "OFF"
      ARROW_WITH_OPENTELEMETRY: "OFF"
+      PYARROW_TEST_ANNOTATIONS: "ON"


I think is worth it if instead of using a separate ci/scripts/python_test_type_annotations.sh we use ci/scripts/python_test.sh and we use the PYARROW_TEST_ANNOTATIONS variable to manage whether we want annotations to be tested or not as part of python testing.
If we decide we want to have two separate scripts I think this env var should be removed.
I lean towards having a single python test script that can allow us to also test annotations but I can find arguments for both so not an issue.

raulcd · 2026-01-14T11:42:42Z

ci/scripts/python_wheel_macos_build.sh

+# We first populate stub docstrings and then build the wheel
+python setup.py build_ext --inplace
+python -m pip install griffe libcst
+python ../dev/update_stub_docstrings.py pyarrow-stubs


this is not necessary, right?

Same as #48618 (comment). It's a noop and we don't have to introduce python logic in stub PRs if we do here.

ci/scripts/python_wheel_validate_contents.py

raulcd · 2026-01-14T11:44:04Z

ci/scripts/python_wheel_windows_build.bat

+@REM We first populate stub docstrings and then build the wheel
+%PYTHON_CMD% setup.py build_ext --inplace
+%PYTHON_CMD% -m pip install griffe libcst


Until we have real annotations this isn't necessary, right?

I added running dev\update_stub_docstrings.py, so I suppose it's a noop now and we can keep it to make sure it doesn't fail before introducing actual stubs? This will keep CI logic out of stub PRs.

Co-authored-by: Raúl Cumplido <raulcumplido@gmail.com>

…g check

change bat lint add a popd and nicer logging for windows ReplaceElipsis -> DocstringInserter simplify remove sphinx

Co-authored-by: Sutou Kouhei <kou@cozmixng.org>

rok · 2026-01-26T22:05:57Z

@github-actions crossbow submit wheel-*

github-actions · 2026-01-26T22:09:28Z

Revision: de7883d

Submitted crossbow builds: ursacomputing/crossbow @ actions-fbb5798826

Task	Status
wheel-macos-monterey-cp310-cp310-amd64
wheel-macos-monterey-cp310-cp310-arm64
wheel-macos-monterey-cp311-cp311-amd64
wheel-macos-monterey-cp311-cp311-arm64
wheel-macos-monterey-cp312-cp312-amd64
wheel-macos-monterey-cp312-cp312-arm64
wheel-macos-monterey-cp313-cp313-amd64
wheel-macos-monterey-cp313-cp313-arm64
wheel-macos-monterey-cp313-cp313t-amd64
wheel-macos-monterey-cp313-cp313t-arm64
wheel-macos-monterey-cp314-cp314-amd64
wheel-macos-monterey-cp314-cp314-arm64
wheel-macos-monterey-cp314-cp314t-amd64
wheel-macos-monterey-cp314-cp314t-arm64
wheel-manylinux-2-28-cp310-cp310-amd64
wheel-manylinux-2-28-cp310-cp310-arm64
wheel-manylinux-2-28-cp311-cp311-amd64
wheel-manylinux-2-28-cp311-cp311-arm64
wheel-manylinux-2-28-cp312-cp312-amd64
wheel-manylinux-2-28-cp312-cp312-arm64
wheel-manylinux-2-28-cp313-cp313-amd64
wheel-manylinux-2-28-cp313-cp313-arm64
wheel-manylinux-2-28-cp313-cp313t-amd64
wheel-manylinux-2-28-cp313-cp313t-arm64
wheel-manylinux-2-28-cp314-cp314-amd64
wheel-manylinux-2-28-cp314-cp314-arm64
wheel-manylinux-2-28-cp314-cp314t-amd64
wheel-manylinux-2-28-cp314-cp314t-arm64
wheel-musllinux-1-2-cp310-cp310-amd64
wheel-musllinux-1-2-cp310-cp310-arm64
wheel-musllinux-1-2-cp311-cp311-amd64
wheel-musllinux-1-2-cp311-cp311-arm64
wheel-musllinux-1-2-cp312-cp312-amd64
wheel-musllinux-1-2-cp312-cp312-arm64
wheel-musllinux-1-2-cp313-cp313-amd64
wheel-musllinux-1-2-cp313-cp313-arm64
wheel-musllinux-1-2-cp313-cp313t-amd64
wheel-musllinux-1-2-cp313-cp313t-arm64
wheel-musllinux-1-2-cp314-cp314-amd64
wheel-musllinux-1-2-cp314-cp314-arm64
wheel-musllinux-1-2-cp314-cp314t-amd64
wheel-musllinux-1-2-cp314-cp314t-arm64
wheel-windows-cp310-cp310-amd64
wheel-windows-cp311-cp311-amd64
wheel-windows-cp312-cp312-amd64
wheel-windows-cp313-cp313-amd64
wheel-windows-cp313-cp313t-amd64
wheel-windows-cp314-cp314-amd64
wheel-windows-cp314-cp314t-amd64

rok requested review from AlenkaF, assignUser, jonkeane, kou and raulcd as code owners December 22, 2025 00:15

github-actions bot added Component: Python Component: Documentation awaiting committer review Awaiting committer review labels Dec 22, 2025

rok force-pushed the pyarrow-stubs-pr1-infrastructure branch from afe4699 to 30017ff Compare December 22, 2025 00:20

This was referenced Dec 22, 2025

GH-32609: [Python] Add type annotations to PyArrow #47609

Open

GH-32609: [Python] Add internal type system stubs (_types, error, _stubs_typing) #48622

Draft

AlenkaF approved these changes Dec 23, 2025

View reviewed changes

ci/scripts/python_test_type_annotations.sh Outdated Show resolved Hide resolved

github-actions bot added awaiting changes Awaiting changes and removed awaiting committer review Awaiting committer review labels Dec 23, 2025

github-actions bot added awaiting change review Awaiting change review awaiting changes Awaiting changes and removed awaiting changes Awaiting changes awaiting change review Awaiting change review labels Dec 23, 2025

raulcd reviewed Jan 13, 2026

View reviewed changes

github-actions bot added awaiting change review Awaiting change review awaiting changes Awaiting changes and removed awaiting changes Awaiting changes awaiting change review Awaiting change review labels Jan 14, 2026

rok commented Jan 14, 2026

View reviewed changes

raulcd reviewed Jan 14, 2026

View reviewed changes

rok and others added 20 commits January 26, 2026 20:05

Apply suggestion from @raulcd

0083d79

Co-authored-by: Raúl Cumplido <raulcumplido@gmail.com>

review feedback

74a8b10

include dev/update_stub_docstrings.py

73a1d5d

Use PYARROW_TEST_ANNOTATIONS in windows build, disable wheel docstrin…

f1aad41

…g check

work on dev/update_stub_docstrings.py

9f56c7c

further work on dev/update_stub_docstrings.py

bd990aa

lint

d3ba211

add click for docstring population

d936e7e

Remove dependencies on click and griffe

d7674c5

fix import paths

c6a182f

add PYARROW_TEST_ANNOTATIONS to AMD64 Windows 2022 Python 3.13

dc44409

change bat lint add a popd and nicer logging for windows ReplaceElipsis -> DocstringInserter simplify remove sphinx

move check to pre-commit

3507664

change pre-commit, add note

72a5f91

change pre-commit script

4ff8c14

Revert from pre-commit to ci/scripts/python_test_type_annotations.sh

a1c48da

Apply suggestions from code review

c5ca97a

Co-authored-by: Sutou Kouhei <kou@cozmixng.org>

apply review suggestion

465438a

fix shellcheck

3f7f726

Try single build

21e72ad

lint

de7883d

rok force-pushed the pyarrow-stubs-pr1-infrastructure branch from ec1c497 to de7883d Compare January 26, 2026 19:10

github-actions bot added awaiting change review Awaiting change review awaiting changes Awaiting changes and removed awaiting changes Awaiting changes awaiting change review Awaiting change review labels Jan 26, 2026

apache deleted a comment from github-actions bot Jan 26, 2026

GH-32609: [CI] Add type checking infrastructure and CI workflow for type annotations #48618

Are you sure you want to change the base?

GH-32609: [CI] Add type checking infrastructure and CI workflow for type annotations #48618

Conversation

rok commented Dec 22, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

AlenkaF left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

rok commented Dec 23, 2025

Uh oh!

AlenkaF commented Dec 23, 2025

Uh oh!

raulcd left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

raulcd commented Jan 13, 2026

Uh oh!

github-actions bot commented Jan 13, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rok commented Jan 14, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rok commented Jan 26, 2026

Uh oh!

github-actions bot commented Jan 26, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

rok commented Dec 22, 2025 •

edited by github-actions bot

Loading