From b70298ebf4af682664d03b908843e60a84a3602a Mon Sep 17 00:00:00 2001 From: jrmccluskey Date: Thu, 13 Nov 2025 10:35:11 -0500 Subject: [PATCH 1/6] Bring the Python Version update docs into the main repo --- .../updating-supported-python-versions.md | 83 +++++++++++++++++++ 1 file changed, 83 insertions(+) create mode 100644 contributor-docs/updating-supported-python-versions.md diff --git a/contributor-docs/updating-supported-python-versions.md b/contributor-docs/updating-supported-python-versions.md new file mode 100644 index 000000000000..820bef4ce7a8 --- /dev/null +++ b/contributor-docs/updating-supported-python-versions.md @@ -0,0 +1,83 @@ + + +# Adding/Removing Python Versions in Apache Beam + +Python releases are now on an annual cadence, with new versions being released (and an old version reaching end-of-life) in October of a given year. The means that at any given time, Beam could be supporting up to five different versions of Python. Removing EOL versions is a higher priorty than adding new versions, as EOL Python versions may not get vulnerability fixes when dependencies fix them. + +## Adding a Python Version + +1. Upgrade Beam direct dependencies to versions that support the new Python versions. Complex libraries, like pyarrow or numpy need to provide wheels for the new Python version. Infrastructure libraries, such as Beam build dependencies, cibuildwheel, and other libraries with a hardcoded version, may have to be upgraded as well. + * Some dependency versions may not support both the minimum and maximum Python version for Beam and will require version-specific dependencies.s + +1. Add a Beam Python container for the new python version. + * https://github.com/apache/beam/tree/master/sdks/python/container + +1. Add a new Python version to different test suites: + * [Tox test suties](https://github.com/apache/beam/blob/master/sdks/python/tox.ini) + * Gradle tasks such as pre-commits, post-commits etc. + * Runner-specific versioning checks + * Fix any tests that fail on the new Python version. + * Typically, a new Python version requires updating Beam Type Inference code. See https://github.com/apache/beam/issues/31047 + +1. Add the GitHub actions workflows for the new Python version. + * Example: https://github.com/apache/beam/blob/master/.github/workflows/python_tests.yml + * The minimum and maximum Python versions are defined in a number of workflows and the [test-properties.json](https://github.com/apache/beam/blob/ce1b1dcbc596d1e7c914ee0f7b0d48f2d2bf87e1/.github/actions/setup-default-test-properties/test-properties.json) file, there will be potentially hundreds of changes for this step. + +1. Add support for building wheels for the new Python version. + * https://github.com/apache/beam/blob/master/.github/workflows/build_wheels.yml + +1. Update the upper limit in [__init__.py](https://github.com/apache/beam/blob/0ef5d3a185c1420da118208353ceb0b40b3a27c9/sdks/python/apache_beam/__init__.py#L78) with the next major Python version. + +1. Add the new Python version in release validation scripts: https://github.com/apache/beam/pull/31415 + +* If there is a new feature update or there is a regression when adding a new Python version, please file an [issue](https://github.com/apache/beam/issues). + * **All the unit tests and Integration tests must pass before merging the new version.** + * If you are a non-committer, please ask the committers to run a seed job on your PR to test all the new changes. + +For an example, see PRs associated with https://github.com/apache/beam/issues/29149, and commits on https://github.com/apache/beam/pull/30828 which add Python 3.12 support. + +## Removing a Python Version + +1. Bump the Python version in [setup.py](https://github.com/apache/beam/blob/0ef5d3a185c1420da118208353ceb0b40b3a27c9/sdks/python/setup.py#L152) and update the Python version warning in [__init__.py](https://github.com/apache/beam/blob/0ef5d3a185c1420da118208353ceb0b40b3a27c9/sdks/python/apache_beam/__init__.py#L78). + +1. Remove test suites for the unsupported Python version: + * Migrate GitHub actions workflows from the deprecated Python version to the next one + * Example PR: https://github.com/apache/beam/pull/32429 + * Make these changes on a branch in the main Beam repository if possible so you can execute the new workflows directly for testing. + * Some workflows only run on the minimum supported Python version (like the linting and coverage precommits.) These may utilize libraries that need updates to run on the next Python version. + * Remove the unsuppported Python version from the following files/directories: + * sdks/python/test-suites/gradle.properties + * apache_beam/testing/tox + Move any workflows that exist only for the minimum Python version from tox/py3X to the next minimum Python version's folder + * apache_beam/testing/dataflow + * apache_beam/testing/direct + * apache_beam/testing/portable + * Remove the unsupported python version gradle tasks from + * build.gradle.kts + * settings.gradle.kts + * buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy + * Remove the support for building wheels and source distributions for the unsupported Python version from [.github/workflows/build_wheels.yml](https://github.com/apache/beam/blob/ce1b1dcbc596d1e7c914ee0f7b0d48f2d2bf87e1/.github/workflows/build_wheels.yml) + * Remove the unsupported Python version from [sdks/python/tox.ini](https://github.com/apache/beam/blob/master/sdks/python/tox.ini) + + +1. Delete the unsupported Python version containers from [sdks/python/container](https://github.com/apache/beam/tree/master/sdks/python/container) + +1. Clean up any code that applies to the removed Python version. + * This will usually be version-specific dependencies in setup.py or branches in the typehinting module. \ No newline at end of file From f37ec0beec619c66e9b50c58fb5094de8518462e Mon Sep 17 00:00:00 2001 From: jrmccluskey Date: Thu, 13 Nov 2025 10:35:34 -0500 Subject: [PATCH 2/6] fix typo --- contributor-docs/updating-supported-python-versions.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/contributor-docs/updating-supported-python-versions.md b/contributor-docs/updating-supported-python-versions.md index 820bef4ce7a8..39b834f21dd6 100644 --- a/contributor-docs/updating-supported-python-versions.md +++ b/contributor-docs/updating-supported-python-versions.md @@ -24,7 +24,7 @@ Python releases are now on an annual cadence, with new versions being released ( ## Adding a Python Version 1. Upgrade Beam direct dependencies to versions that support the new Python versions. Complex libraries, like pyarrow or numpy need to provide wheels for the new Python version. Infrastructure libraries, such as Beam build dependencies, cibuildwheel, and other libraries with a hardcoded version, may have to be upgraded as well. - * Some dependency versions may not support both the minimum and maximum Python version for Beam and will require version-specific dependencies.s + * Some dependency versions may not support both the minimum and maximum Python version for Beam and will require version-specific dependenciess. 1. Add a Beam Python container for the new python version. * https://github.com/apache/beam/tree/master/sdks/python/container From b34fb9059e9b335e066e89ddc2b8aaf55945fd22 Mon Sep 17 00:00:00 2001 From: Jack McCluskey <34928439+jrmccluskey@users.noreply.github.com> Date: Thu, 13 Nov 2025 10:42:33 -0500 Subject: [PATCH 3/6] Apply suggestions from code review Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> --- .../updating-supported-python-versions.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/contributor-docs/updating-supported-python-versions.md b/contributor-docs/updating-supported-python-versions.md index 39b834f21dd6..1c5ea4ed3891 100644 --- a/contributor-docs/updating-supported-python-versions.md +++ b/contributor-docs/updating-supported-python-versions.md @@ -19,21 +19,21 @@ # Adding/Removing Python Versions in Apache Beam -Python releases are now on an annual cadence, with new versions being released (and an old version reaching end-of-life) in October of a given year. The means that at any given time, Beam could be supporting up to five different versions of Python. Removing EOL versions is a higher priorty than adding new versions, as EOL Python versions may not get vulnerability fixes when dependencies fix them. +Python releases are now on an annual cadence, with new versions being released (and an old version reaching end-of-life) in October of a given year. This means that at any given time, Beam could be supporting up to five different versions of Python. Removing EOL versions is a higher priority than adding new versions, as EOL Python versions may not get vulnerability fixes when dependencies fix them. ## Adding a Python Version 1. Upgrade Beam direct dependencies to versions that support the new Python versions. Complex libraries, like pyarrow or numpy need to provide wheels for the new Python version. Infrastructure libraries, such as Beam build dependencies, cibuildwheel, and other libraries with a hardcoded version, may have to be upgraded as well. - * Some dependency versions may not support both the minimum and maximum Python version for Beam and will require version-specific dependenciess. + * Some dependency versions may not support both the minimum and maximum Python version for Beam and will require version-specific dependencies. -1. Add a Beam Python container for the new python version. +1. Add a Beam Python container for the new Python version. * https://github.com/apache/beam/tree/master/sdks/python/container 1. Add a new Python version to different test suites: - * [Tox test suties](https://github.com/apache/beam/blob/master/sdks/python/tox.ini) + * [Tox test suites](https://github.com/apache/beam/blob/master/sdks/python/tox.ini) * Gradle tasks such as pre-commits, post-commits etc. * Runner-specific versioning checks - * Fix any tests that fail on the new Python version. + * Fix any tests that fail on the new Python version. * Typically, a new Python version requires updating Beam Type Inference code. See https://github.com/apache/beam/issues/31047 1. Add the GitHub actions workflows for the new Python version. @@ -69,7 +69,7 @@ For an example, see PRs associated with https://github.com/apache/beam/issues/29 * apache_beam/testing/dataflow * apache_beam/testing/direct * apache_beam/testing/portable - * Remove the unsupported python version gradle tasks from + * Remove the unsupported Python version gradle tasks from * build.gradle.kts * settings.gradle.kts * buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy From a3ec78f10f2554cf5b1e22b824d519c56bc3cfec Mon Sep 17 00:00:00 2001 From: Jack McCluskey <34928439+jrmccluskey@users.noreply.github.com> Date: Thu, 13 Nov 2025 10:42:51 -0500 Subject: [PATCH 4/6] Update contributor-docs/updating-supported-python-versions.md Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> --- contributor-docs/updating-supported-python-versions.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/contributor-docs/updating-supported-python-versions.md b/contributor-docs/updating-supported-python-versions.md index 1c5ea4ed3891..e3b89aa3c80f 100644 --- a/contributor-docs/updating-supported-python-versions.md +++ b/contributor-docs/updating-supported-python-versions.md @@ -62,7 +62,7 @@ For an example, see PRs associated with https://github.com/apache/beam/issues/29 * Example PR: https://github.com/apache/beam/pull/32429 * Make these changes on a branch in the main Beam repository if possible so you can execute the new workflows directly for testing. * Some workflows only run on the minimum supported Python version (like the linting and coverage precommits.) These may utilize libraries that need updates to run on the next Python version. - * Remove the unsuppported Python version from the following files/directories: + * Remove the unsupported Python version from the following files/directories: * sdks/python/test-suites/gradle.properties * apache_beam/testing/tox Move any workflows that exist only for the minimum Python version from tox/py3X to the next minimum Python version's folder From 58e54c5eca90a4abc837802fa2f6530463a17176 Mon Sep 17 00:00:00 2001 From: jrmccluskey Date: Thu, 13 Nov 2025 10:43:20 -0500 Subject: [PATCH 5/6] remove extra line --- contributor-docs/updating-supported-python-versions.md | 1 - 1 file changed, 1 deletion(-) diff --git a/contributor-docs/updating-supported-python-versions.md b/contributor-docs/updating-supported-python-versions.md index e3b89aa3c80f..08eec34a2591 100644 --- a/contributor-docs/updating-supported-python-versions.md +++ b/contributor-docs/updating-supported-python-versions.md @@ -76,7 +76,6 @@ For an example, see PRs associated with https://github.com/apache/beam/issues/29 * Remove the support for building wheels and source distributions for the unsupported Python version from [.github/workflows/build_wheels.yml](https://github.com/apache/beam/blob/ce1b1dcbc596d1e7c914ee0f7b0d48f2d2bf87e1/.github/workflows/build_wheels.yml) * Remove the unsupported Python version from [sdks/python/tox.ini](https://github.com/apache/beam/blob/master/sdks/python/tox.ini) - 1. Delete the unsupported Python version containers from [sdks/python/container](https://github.com/apache/beam/tree/master/sdks/python/container) 1. Clean up any code that applies to the removed Python version. From 3e69b395ba0841797e50fdc83bc6a21d14776d26 Mon Sep 17 00:00:00 2001 From: jrmccluskey Date: Thu, 13 Nov 2025 10:59:05 -0500 Subject: [PATCH 6/6] trailing whitespace --- contributor-docs/updating-supported-python-versions.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/contributor-docs/updating-supported-python-versions.md b/contributor-docs/updating-supported-python-versions.md index 08eec34a2591..829ccb58d103 100644 --- a/contributor-docs/updating-supported-python-versions.md +++ b/contributor-docs/updating-supported-python-versions.md @@ -55,7 +55,7 @@ For an example, see PRs associated with https://github.com/apache/beam/issues/29 ## Removing a Python Version -1. Bump the Python version in [setup.py](https://github.com/apache/beam/blob/0ef5d3a185c1420da118208353ceb0b40b3a27c9/sdks/python/setup.py#L152) and update the Python version warning in [__init__.py](https://github.com/apache/beam/blob/0ef5d3a185c1420da118208353ceb0b40b3a27c9/sdks/python/apache_beam/__init__.py#L78). +1. Bump the Python version in [setup.py](https://github.com/apache/beam/blob/0ef5d3a185c1420da118208353ceb0b40b3a27c9/sdks/python/setup.py#L152) and update the Python version warning in [__init__.py](https://github.com/apache/beam/blob/0ef5d3a185c1420da118208353ceb0b40b3a27c9/sdks/python/apache_beam/__init__.py#L78). 1. Remove test suites for the unsupported Python version: * Migrate GitHub actions workflows from the deprecated Python version to the next one