From aa2180bcaeddf15522cbf4e45ab9bef7227cd410 Mon Sep 17 00:00:00 2001 From: Mika Naylor Date: Fri, 8 Aug 2025 16:38:00 +0200 Subject: [PATCH] [FLINK-38231][python] Standardise use of uv for PyFlink building, testing & linting --- docs/content/docs/flinkDev/building.md | 43 +++- .../downloads/setup-pyflink-virtual-env.sh | 6 +- .../test_kubernetes_pyflink_application.sh | 8 +- .../test-scripts/test_pyflink.sh | 14 +- flink-python/README.md | 1 - .../{setup.cfg => pyproject.toml} | 29 ++- flink-python/apache-flink-libraries/setup.py | 16 -- flink-python/dev/build-wheels.sh | 52 ++-- flink-python/dev/install_command.sh | 49 ---- flink-python/dev/lint-python.sh | 231 ++---------------- flink-python/docs/Makefile | 6 +- flink-python/pyproject.toml | 31 ++- flink-python/setup.py | 16 -- flink-python/tox.ini | 16 +- 14 files changed, 152 insertions(+), 366 deletions(-) rename flink-python/apache-flink-libraries/{setup.cfg => pyproject.toml} (57%) delete mode 100755 flink-python/dev/install_command.sh diff --git a/docs/content/docs/flinkDev/building.md b/docs/content/docs/flinkDev/building.md index e2c7059eb824b..498d9bed8af35 100644 --- a/docs/content/docs/flinkDev/building.md +++ b/docs/content/docs/flinkDev/building.md @@ -79,6 +79,15 @@ The `fast` and `skip-webui-build` profiles have a significant impact on the buil # the version printed here must be 3.9, 3.10, 3.11 or 3.12 ``` +3. Uv for building, installing and managing Python environments (optional) + + The tool [uv](https://docs.astral.sh/uv/) is a Python package and project manager that can make building and installing PyFlink from source much easier. + To verify that it is installed: + + ```shell + $ uv --version + ``` + 3. Build PyFlink with Cython extension support (optional) To build PyFlink with Cython extension support, you’ll need a C compiler. It's a little different on how to install the C compiler on different operating systems: @@ -87,7 +96,13 @@ The `fast` and `skip-webui-build` profiles have a significant impact on the buil * **Mac OS X** To install GCC on Mac OS X, you need to download and install "[Command Line Tools for Xcode](https://developer.apple.com/downloads/index.action)", which is available in Apple’s developer page. - You also need to install the dependencies with following command: + You also need to install the dependencies with following commands. Either with `uv`: + + ```shell + $ uv pip install --group flink-python/pyproject.toml:dev + ``` + + or with `python` and `pip`: ```shell $ python -m pip install --group flink-python/pyproject.toml:dev @@ -95,13 +110,35 @@ The `fast` and `skip-webui-build` profiles have a significant impact on the buil #### Installation -Then go to the root directory of flink source code and run this command to build the sdist package and wheel package of `apache-flink` and `apache-flink-libraries`: +##### With `uv` + +To build and install the `apache-flink` and `apache-flink-libraries` packages with `uv`, go to the root directory of flink source code and run the following: + +```bash +cd flink-python; uv build; uv build apache-flink-libraries --sdist; +``` + +The sdist package of `apache-flink-libraries` will be found under `./flink-python/apache-flink-libraries/dist/`. It can be installed as follows: + +```bash +uv pip install apache-flink-libraries/dist/*.tar.gz +``` + +The sdist and wheel packages of `apache-flink` will be found under `./flink-python/dist/`. Either of them could be used for installation, such as: + +```bash +uv pip install dist/*.whl +``` + +##### With `python` and `pip` + +To build and install the `apache-flink` and `apache-flink-libraries` packages with `python` and `pip`, go to the root directory of flink source code and run the following: ```bash cd flink-python; python setup.py sdist bdist_wheel; cd apache-flink-libraries; python setup.py sdist; cd ..; ``` -The sdist package of `apache-flink-libraries` will be found under `./flink-python/apache-flink-libraries/dist/`. It could be installed as following: +The sdist package of `apache-flink-libraries` will be found under `./flink-python/apache-flink-libraries/dist/`. It can be installed as follows: ```bash python -m pip install apache-flink-libraries/dist/*.tar.gz diff --git a/docs/static/downloads/setup-pyflink-virtual-env.sh b/docs/static/downloads/setup-pyflink-virtual-env.sh index 8aadb6768646e..862018f7adcae 100755 --- a/docs/static/downloads/setup-pyflink-virtual-env.sh +++ b/docs/static/downloads/setup-pyflink-virtual-env.sh @@ -20,7 +20,7 @@ echo "Detected OS: ${sys_os}" sys_machine=$(uname -m) echo "Detected machine: ${sys_machine}" -wget "https://github.com/astral-sh/uv/releases/download/0.5.23/uv-installer.sh" -O "uv-installer.sh" +wget "https://github.com/astral-sh/uv/releases/download/0.8.9/uv-installer.sh" -O "uv-installer.sh" chmod +x uv-installer.sh UV_UNMANAGED_INSTALL="uv-bin" ./uv-installer.sh @@ -34,10 +34,10 @@ source venv/bin/activate "" # install PyFlink dependency if [[ $1 = "" ]]; then # install the latest version of pyflink - pip install apache-flink + uv pip install apache-flink else # install the specified version of pyflink - pip install "apache-flink==$1" + uv pip install "apache-flink==$1" fi # deactivate the python virtual environment diff --git a/flink-end-to-end-tests/test-scripts/test_kubernetes_pyflink_application.sh b/flink-end-to-end-tests/test-scripts/test_kubernetes_pyflink_application.sh index 389ee33b79277..27fb447f729ad 100755 --- a/flink-end-to-end-tests/test-scripts/test_kubernetes_pyflink_application.sh +++ b/flink-end-to-end-tests/test-scripts/test_kubernetes_pyflink_application.sh @@ -54,13 +54,9 @@ cd "${FLINK_PYTHON_DIR}" if [[ -d "dist" ]]; then rm -Rf dist; fi -pushd apache-flink-libraries +uv build apache-flink-libraries --sdist -python setup.py sdist - -popd - -python setup.py sdist +uv build cd dev diff --git a/flink-end-to-end-tests/test-scripts/test_pyflink.sh b/flink-end-to-end-tests/test-scripts/test_pyflink.sh index 20fd2640553c0..f92c4c73de835 100755 --- a/flink-end-to-end-tests/test-scripts/test_pyflink.sh +++ b/flink-end-to-end-tests/test-scripts/test_pyflink.sh @@ -97,19 +97,7 @@ cd "${FLINK_PYTHON_DIR}" rm -rf dist -uv pip install --group dev - -python setup.py sdist - -pushd apache-flink-libraries - -python setup.py sdist - -uv pip install dist/* - -popd - -uv pip install dist/* +uv pip install --group dev . cd dev diff --git a/flink-python/README.md b/flink-python/README.md index 32f5a95ce59a7..fee7a0a98e8ce 100644 --- a/flink-python/README.md +++ b/flink-python/README.md @@ -67,6 +67,5 @@ Then you can activate your environment and run tests, for example: ```shell source pyflink_39/bin/activate -uv pip install --group dev ./dev/lint-python.sh ``` diff --git a/flink-python/apache-flink-libraries/setup.cfg b/flink-python/apache-flink-libraries/pyproject.toml similarity index 57% rename from flink-python/apache-flink-libraries/setup.cfg rename to flink-python/apache-flink-libraries/pyproject.toml index 138d2bd2c1505..9cd414ec29ad5 100644 --- a/flink-python/apache-flink-libraries/setup.cfg +++ b/flink-python/apache-flink-libraries/pyproject.toml @@ -15,9 +15,30 @@ # See the License for the specific language governing permissions and # limitations under the License. ################################################################################ +[build-system] +# Minimum requirements for the build system to execute. +requires = [ + "setuptools>=75.3", + "wheel", +] -[bdist_wheel] -universal = 1 +[project] +name = "apache-flink-libraries" +description = "Apache Flink Libraries" +readme = "README.md" +license = "Apache-2.0" +authors = [ + {name = "Apache Software Foundation", email = "dev@flink.apache.org"} +] +requires-python = ">=3.9" +classifiers = [ + "Development Status :: 5 - Production/Stable", + "Programming Language :: Python :: 3.9", + "Programming Language :: Python :: 3.10", + "Programming Language :: Python :: 3.11", + "Programming Language :: Python :: 3.12", +] +dynamic = ["version", "dependencies"] -[metadata] -description-file = README.md +[project.urls] +homepage = "https://flink.apache.org" diff --git a/flink-python/apache-flink-libraries/setup.py b/flink-python/apache-flink-libraries/setup.py index 2119b032bfb09..ced977690d36f 100644 --- a/flink-python/apache-flink-libraries/setup.py +++ b/flink-python/apache-flink-libraries/setup.py @@ -215,27 +215,11 @@ def find_file_path(pattern): PACKAGE_DATA['pyflink.licenses'] = ['*'] setup( - name='apache-flink-libraries', version=VERSION, packages=PACKAGES, include_package_data=True, package_dir=PACKAGE_DIR, package_data=PACKAGE_DATA, - url='https://flink.apache.org', - license='https://www.apache.org/licenses/LICENSE-2.0', - author='Apache Software Foundation', - author_email='dev@flink.apache.org', - python_requires='>=3.9', - description='Apache Flink Libraries', - long_description=long_description, - long_description_content_type='text/markdown', - classifiers=[ - 'Development Status :: 5 - Production/Stable', - 'License :: OSI Approved :: Apache Software License', - 'Programming Language :: Python :: 3.9', - 'Programming Language :: Python :: 3.10', - 'Programming Language :: Python :: 3.11', - 'Programming Language :: Python :: 3.12'], ) finally: if in_flink_source: diff --git a/flink-python/dev/build-wheels.sh b/flink-python/dev/build-wheels.sh index 39180452450cd..0da2911b2f8fe 100755 --- a/flink-python/dev/build-wheels.sh +++ b/flink-python/dev/build-wheels.sh @@ -15,41 +15,35 @@ # limitations under the License. set -e -x -## 1. install python env -dev/lint-python.sh -s py_env - -PY_ENV_DIR=`pwd`/dev/.uv/envs -py_env=("3.9" "3.10" "3.11" "3.12") -## 2. install dependency -for ((i=0;i<${#py_env[@]};i++)) do - source `pwd`/dev/.uv/envs/${py_env[i]}/bin/activate - echo "Installing dependencies for environment: ${py_env[i]}" - uv pip install --group dev - deactivate -done +## 1. Setup virtual environment with uv +dev/lint-python.sh -s basic + +python_versions=("3.9" "3.10" "3.11" "3.12") + +if [[ "$(uname)" != "Darwin" ]]; then + # force the linker to use the older glibc version in Linux + export CFLAGS="-I. -include dev/glibc_version_fix.h" +fi + +source "$(pwd)"/dev/.uv/bin/activate -## 3. build wheels -for ((i=0;i<${#py_env[@]};i++)) do - echo "Building wheel for environment: ${py_env[i]}" - if [[ "$(uname)" != "Darwin" ]]; then - # force the linker to use the older glibc version in Linux - export CFLAGS="-I. -include dev/glibc_version_fix.h" - fi - ${PY_ENV_DIR}/${py_env[i]}/bin/python setup.py bdist_wheel +## 2. Build wheels for each Python version +for ((i=0;i<${#python_versions[@]};i++)) do + echo "Building wheel for Python: ${python_versions[i]}" + uv build --wheel --python "${python_versions[i]}" done -## 4. convert linux_x86_64 wheel to manylinux1 wheel in Linux +## 3. Convert linux_x86_64 wheel to manylinux1 wheel in Linux if [[ "$(uname)" != "Darwin" ]]; then echo "Converting linux_x86_64 wheel to manylinux1" - source `pwd`/dev/.uv/bin/activate - # 4.1 install patchelf and auditwheel - uv pip install patchelf==0.17.2.1 auditwheel==3.2.0 - # 4.2 convert Linux wheel for wheel_file in dist/*.whl; do - auditwheel repair ${wheel_file} -w dist - rm -f ${wheel_file} + uv run --group auditwheel auditwheel repair "${wheel_file}" -w dist + rm -f "${wheel_file}" done - deactivate fi -## see the result + +deactivate + +## 4. Output the result +echo "Build finished - created the following wheels:" ls -al dist/ diff --git a/flink-python/dev/install_command.sh b/flink-python/dev/install_command.sh deleted file mode 100755 index b422b20a17934..0000000000000 --- a/flink-python/dev/install_command.sh +++ /dev/null @@ -1,49 +0,0 @@ -#!/usr/bin/env bash -################################################################################ -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -################################################################################ -if [[ "$@" =~ 'apache-flink-libraries' ]]; then - # As of Python 3.12, setuptools is no longer a seed package. - # We should ensure its existence. - python -m pip install setuptools - pushd apache-flink-libraries - python setup.py sdist - pushd dist - python -m pip install * - popd - popd -fi - -if [[ `uname -s` == "Darwin" && `uname -m` == "arm64" ]]; then - echo "Adding MacOS arm64 GRPC pip install fix" - export GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=1 - export GRPC_PYTHON_BUILD_SYSTEM_ZLIB=1 -fi - -retry_times=3 -install_command="python -m pip install $@" -${install_command} -status=$? -while [[ ${status} -ne 0 ]] && [[ ${retry_times} -gt 0 ]]; do - retry_times=$((retry_times-1)) - # sleep 3 seconds and then reinstall. - sleep 3 - ${install_command} - status=$? -done - -exit ${status} diff --git a/flink-python/dev/lint-python.sh b/flink-python/dev/lint-python.sh index 90fceed46ecea..fa82cc229d0d1 100755 --- a/flink-python/dev/lint-python.sh +++ b/flink-python/dev/lint-python.sh @@ -137,7 +137,7 @@ function parse_component_args() { if [[ `contains_element "${SUPPORTED_INSTALLATION_COMPONENTS[*]}" "${component}"` = true ]]; then REAL_COMPONENTS+=(${component}) else - echo "unknown install component ${component}, currently we only support installing basic,py_env,tox,flake8,sphinx,mypy,all." + echo "unknown install component ${component}, currently we only support installing basic,all." exit 1 fi done @@ -215,140 +215,6 @@ function install_uv() { fi } -# Create different Python virtual environments for different Python versions -function install_py_env() { - py_env=("3.9" "3.10" "3.11" "3.12") - for ((i=0;i<${#py_env[@]};i++)) do - if [ -d "$CURRENT_DIR/.uv/envs/${py_env[i]}" ]; then - rm -rf "$CURRENT_DIR/.uv/envs/${py_env[i]}" - if [ $? -ne 0 ]; then - echo "rm -rf $CURRENT_DIR/.uv/envs/${py_env[i]} failed, please \ - rm -rf $CURRENT_DIR/.uv/envs/${py_env[i]} manually.\ - Then retry to exec the script." - exit 1 - fi - fi - print_function "STEP" "installing python${py_env[i]}..." - max_retry_times=3 - retry_times=0 - install_command="$UV_PATH venv $CURRENT_DIR/.uv/envs/${py_env[i]} -q --python=${py_env[i]} --seed" - ${install_command} 2>&1 >/dev/null - status=$? - while [[ ${status} -ne 0 ]] && [[ ${retry_times} -lt ${max_retry_times} ]]; do - retry_times=$((retry_times+1)) - # sleep 3 seconds and then reinstall. - sleep 3 - echo "uv venv ${py_env[i]} retrying ${retry_times}/${max_retry_times}" - ${install_command} 2>&1 >/dev/null - status=$? - done - if [[ ${status} -ne 0 ]]; then - echo "uv venv ${py_env[i]} failed after retrying ${max_retry_times} times.\ - You can retry to execute the script again." - exit 1 - fi - - $CURRENT_DIR/.uv/envs/${py_env[i]}/bin/pip install -q uv==${UV_VERSION} - print_function "STEP" "install python${py_env[i]}... [SUCCESS]" - done -} - -# Install tox. -# In some situations,you need to run the script with "sudo". e.g. sudo ./lint-python.sh -function install_tox() { - source $ENV_HOME/bin/activate - if [ -f "$TOX_PATH" ]; then - $UV_PATH pip uninstall tox -q 2>&1 >/dev/null - if [ $? -ne 0 ]; then - echo "uv pip uninstall tox failed \ - please try to exec the script again.\ - if failed many times, you can try to exec in the form of sudo ./lint-python.sh -f" - exit 1 - fi - fi - - $CURRENT_DIR/install_command.sh -q --group "${PYPROJECT_PATH}:tox" 2>&1 >/dev/null - if [ $? -ne 0 ]; then - echo "uv pip install tox failed \ - please try to exec the script again.\ - if failed many times, you can try to exec in the form of sudo ./lint-python.sh -f" - exit 1 - fi - deactivate -} - -# Install flake8. -# In some situations,you need to run the script with "sudo". e.g. sudo ./lint-python.sh -function install_flake8() { - source $UV_HOME/bin/activate - if [ -f "$FLAKE8_PATH" ]; then - $UV_PATH pip uninstall flake8 -q 2>&1 >/dev/null - if [ $? -ne 0 ]; then - echo "uv pip uninstall flake8 failed \ - please try to exec the script again.\ - if failed many times, you can try to exec in the form of sudo ./lint-python.sh -f" - exit 1 - fi - fi - - $CURRENT_DIR/install_command.sh -q --group "${PYPROJECT_PATH}:flake8" 2>&1 >/dev/null - if [ $? -ne 0 ]; then - echo "uv pip install flake8 failed \ - please try to exec the script again.\ - if failed many times, you can try to exec in the form of sudo ./lint-python.sh -f" - exit 1 - fi - deactivate -} - -# Install sphinx. -# In some situations,you need to run the script with "sudo". e.g. sudo ./lint-python.sh -function install_sphinx() { - source $UV_HOME/bin/activate - if [ -f "$SPHINX_PATH" ]; then - $UV_PATH pip uninstall Sphinx -q 2>&1 >/dev/null - if [ $? -ne 0 ]; then - echo "uv pip uninstall sphinx failed \ - please try to exec the script again.\ - if failed many times, you can try to exec in the form of sudo ./lint-python.sh -f" - exit 1 - fi - fi - - $CURRENT_DIR/install_command.sh -q --group "${PYPROJECT_PATH}:sphinx" 2>&1 >/dev/null - if [ $? -ne 0 ]; then - echo "uv pip install sphinx failed \ - please try to exec the script again.\ - if failed many times, you can try to exec in the form of sudo ./lint-python.sh -f" - exit 1 - fi - deactivate -} - - -# Install mypy. -# In some situations, you need to run the script with "sudo". e.g. sudo ./lint-python.sh -function install_mypy() { - source ${UV_HOME}/bin/activate - if [[ -f "$MYPY_PATH" ]]; then - ${UV_PATH} pip uninstall mypy -q 2>&1 >/dev/null - if [[ $? -ne 0 ]]; then - echo "uv pip uninstall mypy failed \ - please try to exec the script again.\ - if failed many times, you can try to exec in the form of sudo ./lint-python.sh -f" - exit 1 - fi - fi - ${CURRENT_DIR}/install_command.sh -q --group "${PYPROJECT_PATH}:mypy" 2>&1 >/dev/null - if [[ $? -ne 0 ]]; then - echo "uv pip install mypy failed \ - please try to exec the script again.\ - if failed many times, you can try to exec in the form of sudo ./lint-python.sh -f" - exit 1 - fi - deactivate -} - function need_install_component() { if [[ `contains_element "${SUPPORTED_INSTALLATION_COMPONENTS[*]}" "$1"` = true ]]; then echo true @@ -377,52 +243,6 @@ function install_environment() { print_function "STEP" "install uv... [SUCCESS]" fi - # step-2 install python environment which includes - # 3.9 3.10 3.11 3.12 - if [ $STEP -lt 2 ] && [ `need_install_component "py_env"` = true ]; then - print_function "STEP" "installing python environment..." - install_py_env - STEP=2 - checkpoint_stage $STAGE $STEP - print_function "STEP" "install python environment... [SUCCESS]" - fi - - # step-3 install tox - if [ $STEP -lt 3 ] && [ `need_install_component "tox"` = true ]; then - print_function "STEP" "installing tox..." - install_tox - STEP=3 - checkpoint_stage $STAGE $STEP - print_function "STEP" "install tox... [SUCCESS]" - fi - - # step-4 install flake8 - if [ $STEP -lt 4 ] && [ `need_install_component "flake8"` = true ]; then - print_function "STEP" "installing flake8..." - install_flake8 - STEP=4 - checkpoint_stage $STAGE $STEP - print_function "STEP" "install flake8... [SUCCESS]" - fi - - # step-5 install sphinx - if [ $STEP -lt 5 ] && [ `need_install_component "sphinx"` = true ]; then - print_function "STEP" "installing sphinx..." - install_sphinx - STEP=5 - checkpoint_stage $STAGE $STEP - print_function "STEP" "install sphinx... [SUCCESS]" - fi - - # step-5 install mypy - if [[ ${STEP} -lt 6 ]] && [[ `need_install_component "mypy"` = true ]]; then - print_function "STEP" "installing mypy..." - install_mypy - STEP=6 - checkpoint_stage ${STAGE} ${STEP} - print_function "STEP" "install mypy... [SUCCESS]" - fi - print_function "STAGE" "install environment... [SUCCESS]" } @@ -559,20 +379,20 @@ function check_stage() { ######################### # Tox check function tox_check() { + local TOX="$UV_PATH run --group tox tox" LATEST_PYTHON="py312" print_function "STAGE" "tox checks" - # Set created py-env in $PATH for tox's creating virtual env - activate + # Ensure the permission of the scripts set correctly chmod +x $FLINK_PYTHON_DIR/../build-target/bin/* chmod +x $FLINK_PYTHON_DIR/dev/* if [[ ${BUILD_REASON} = 'IndividualCI' ]]; then # Only run test in latest python version triggered by a Git push - $TOX_PATH -vv -c $FLINK_PYTHON_DIR/tox.ini -e ${LATEST_PYTHON} --recreate 2>&1 | tee -a $LOG_FILE + $TOX -vv -c $FLINK_PYTHON_DIR/tox.ini -e ${LATEST_PYTHON} --recreate 2>&1 | tee -a $LOG_FILE else # Only run random selected python version in nightly CI. - ENV_LIST_STRING=`$TOX_PATH -l -c $FLINK_PYTHON_DIR/tox.ini` + ENV_LIST_STRING=`$TOX -l -c $FLINK_PYTHON_DIR/tox.ini` _OLD_IFS=$IFS IFS=$'\n' ENV_LIST=(${ENV_LIST_STRING}) @@ -580,7 +400,7 @@ function tox_check() { ENV_LIST_SIZE=${#ENV_LIST[@]} index=$(($RANDOM % ENV_LIST_SIZE)) - $TOX_PATH -vv -c $FLINK_PYTHON_DIR/tox.ini -e ${ENV_LIST[$index]} --recreate 2>&1 | tee -a $LOG_FILE + $TOX -vv -c $FLINK_PYTHON_DIR/tox.ini -e ${ENV_LIST[$index]} --recreate 2>&1 | tee -a $LOG_FILE fi TOX_RESULT=$((grep -c "congratulations :)" "$LOG_FILE") 2>&1) @@ -589,8 +409,6 @@ function tox_check() { else print_function "STAGE" "tox checks... [SUCCESS]" fi - # Reset the $PATH - deactivate # If check failed, stop the running script. if [ $TOX_RESULT -eq '0' ]; then @@ -600,13 +418,10 @@ function tox_check() { # Flake8 check function flake8_check() { + local FLAKE8="$UV_PATH run --group flake8 flake8" local PYTHON_SOURCE="$(find . \( -path ./dev -o -path ./.tox \) -prune -o -type f -name "*.py" -print )" print_function "STAGE" "flake8 checks" - if [ ! -f "$FLAKE8_PATH" ]; then - echo "For some unknown reasons, the flake8 package is not complete,\ - you should exec the script with the parameter: -f" - fi if [[ ! "$PYTHON_SOURCE" ]]; then echo "No python files found! Something is wrong exiting." @@ -616,7 +431,7 @@ function flake8_check() { # the return value of a pipeline is the status of the last command to exit # with a non-zero status or zero if no command exited with a non-zero status set -o pipefail - ($FLAKE8_PATH --config=tox.ini $PYTHON_SOURCE) 2>&1 | tee -a $LOG_FILE + ($FLAKE8 --config=tox.ini $PYTHON_SOURCE) 2>&1 | tee -a $LOG_FILE PYCODESTYLE_STATUS=$? if [ $PYCODESTYLE_STATUS -ne 0 ]; then @@ -630,7 +445,9 @@ function flake8_check() { # Sphinx check function sphinx_check() { - export SPHINXBUILD=$SPHINX_PATH + export SPHINXBUILD="$UV_PATH run --group sphinx sphinx-build" + print_function "STAGE" "sphinx checks" + # cd to $FLINK_PYTHON_DIR pushd "$FLINK_PYTHON_DIR"/docs &> /dev/null make clean @@ -653,13 +470,14 @@ function sphinx_check() { # mypy check function mypy_check() { + local MYPY="$UV_PATH run --group mypy mypy" print_function "STAGE" "mypy checks" # the return value of a pipeline is the status of the last command to exit # with a non-zero status or zero if no command exited with a non-zero status set -o pipefail - (${MYPY_PATH} --install-types --non-interactive --config-file tox.ini) 2>&1 | tee -a ${LOG_FILE} + ($MYPY --install-types --non-interactive --config-file tox.ini) 2>&1 | tee -a ${LOG_FILE} TYPE_HINT_CHECK_STATUS=$? if [ ${TYPE_HINT_CHECK_STATUS} -ne 0 ]; then print_function "STAGE" "mypy checks... [FAILED]" @@ -694,18 +512,6 @@ UV_PATH=$UV_HOME/bin/uv # pip path PIP_PATH=$ENV_HOME/bin/pip -# tox path -TOX_PATH=$ENV_HOME/bin/tox - -# flake8 path -FLAKE8_PATH=$ENV_HOME/bin/flake8 - -# sphinx path -SPHINX_PATH=$ENV_HOME/bin/sphinx-build - -# mypy path -MYPY_PATH=$ENV_HOME/bin/mypy - _OLD_PATH="$PATH" SUPPORT_OS=("Darwin" "Linux") @@ -739,7 +545,7 @@ UV_VERSION=0.7.20 UV_INSTALL_SH=$CURRENT_DIR/download/uv.sh # stage "install" includes the num of steps. -STAGE_INSTALL_STEPS=6 +STAGE_INSTALL_STEPS=1 # whether force to restart the script. FORCE_START=0 @@ -768,11 +574,12 @@ USAGE=" usage: $0 [options] -h print this help message and exit -f force to exec from the progress of installing environment --s [basic,py_env,tox,flake8,sphinx,mypy,all] +-s [basic,all] install environment with specified components which split by comma(,) note: This option is used to install environment components and will skip all subsequent checks, so do not use this option with -e,-i simultaneously. + Tools like tox, flake8, sphinx, mypy are run via 'uv run' and don't need separate installation. -e [tox,flake8,sphinx,mypy] exclude checks which split by comma(,) -i [tox,flake8,sphinx,mypy] @@ -780,13 +587,11 @@ usage: $0 [options] -l list all checks supported. Examples: ./lint-python.sh -s basic => install environment with basic components. - ./lint-python.sh -s all => install environment with all components such as python env,tox,flake8,sphinx,mypy etc. - ./lint-python.sh -s tox,flake8 => install environment with tox,flake8. - ./lint-python.sh -s tox -f => reinstall environment with tox. + ./lint-python.sh -s all => install environment with all components (uv). ./lint-python.sh -e tox,flake8 => exclude checks tox,flake8. ./lint-python.sh -i flake8 => include checks flake8. ./lint-python.sh => exec all checks. - ./lint-python.sh -f => reinstall environment with all components and exec all checks. + ./lint-python.sh -f => reinstall environment and exec all checks. ./lint-python.sh -l => list all checks supported. ./lint-python.sh -r => clean up python environment. " diff --git a/flink-python/docs/Makefile b/flink-python/docs/Makefile index d0debac5d781e..6299930225d79 100644 --- a/flink-python/docs/Makefile +++ b/flink-python/docs/Makefile @@ -27,9 +27,9 @@ endif # If SPHINXBUILD is set, the command is 'sphinx-build'. # Elseif SPHINXPYTHON is set, the command is 'python -m sphinx'. ifdef SPHINXBUILD -# Check whether sphinx-build is installed -ifeq ($(shell hash $(SPHINXBUILD) >/dev/null 2>&1; echo $$?), 1) -$(error The command '$(SPHINXBUILD)' was not found.If you don't have Sphinx installed, grab it from http://sphinx-doc.org/) +# Check whether sphinx-build command works (supports compound commands like "uv run sphinx-build") +ifeq ($(shell $(SPHINXBUILD) --help >/dev/null 2>&1; echo $$?), 1) +$(error The command '$(SPHINXBUILD)' was not found or failed. If you don't have Sphinx installed, grab it from http://sphinx-doc.org/) endif else ifeq ($(shell $(SPHINXPYTHON) -c 'import sphinx' >/dev/null 2>&1; echo $$?), 1) diff --git a/flink-python/pyproject.toml b/flink-python/pyproject.toml index eaa5149a9e433..be0352567c536 100644 --- a/flink-python/pyproject.toml +++ b/flink-python/pyproject.toml @@ -27,6 +27,27 @@ requires = [ "cython>=0.29.24" ] +[project] +name = "apache-flink" +description = "Apache Flink Python API" +readme = "README.md" +license = "Apache-2.0" +authors = [ + {name = "Apache Software Foundation", email = "dev@flink.apache.org"} +] +requires-python = ">=3.9" +classifiers = [ + "Development Status :: 5 - Production/Stable", + "Programming Language :: Python :: 3.9", + "Programming Language :: Python :: 3.10", + "Programming Language :: Python :: 3.11", + "Programming Language :: Python :: 3.12", +] +dynamic = ["version", "scripts", "dependencies"] + +[project.urls] +homepage = "https://flink.apache.org" + [dependency-groups] dev = [ "pip>=20.3", @@ -52,7 +73,8 @@ dev = [ "ruamel.yaml>=0.18.4", ] tox = [ - "tox==3.14.0" + "tox==4.28.4", + "tox-uv==1.27.0" ] flake8 = [ "flake8==7.2.0" @@ -75,6 +97,10 @@ mypy = [ "types-pytz", "types-python-dateutil" ] +auditwheel = [ + "patchelf==0.17.2.1", + "auditwheel==3.2.0" +] [tool.cibuildwheel] @@ -85,3 +111,6 @@ archs = ["x86_64", "arm64"] [tool.cibuildwheel.linux] archs = ["x86_64"] + +[tool.uv.sources] +apache-flink-libraries = { path = "apache-flink-libraries/" } diff --git a/flink-python/setup.py b/flink-python/setup.py index 2526f62cd9577..b68cd7ec5e172 100644 --- a/flink-python/setup.py +++ b/flink-python/setup.py @@ -330,31 +330,15 @@ def extracted_output_files(base_dir, file_path, output_directory): apache_flink_libraries_dependency] setup( - name='apache-flink', version=VERSION, packages=PACKAGES, include_package_data=True, package_dir=PACKAGE_DIR, package_data=PACKAGE_DATA, scripts=scripts, - url='https://flink.apache.org', - license='https://www.apache.org/licenses/LICENSE-2.0', - author='Apache Software Foundation', - author_email='dev@flink.apache.org', - python_requires='>=3.9', install_requires=install_requires, cmdclass={'build_ext': build_ext}, - description='Apache Flink Python API', - long_description=long_description, - long_description_content_type='text/markdown', zip_safe=False, - classifiers=[ - 'Development Status :: 5 - Production/Stable', - 'License :: OSI Approved :: Apache Software License', - 'Programming Language :: Python :: 3.9', - 'Programming Language :: Python :: 3.10', - 'Programming Language :: Python :: 3.11', - 'Programming Language :: Python :: 3.12'], ext_modules=extensions ) finally: diff --git a/flink-python/tox.ini b/flink-python/tox.ini index 6f57a22dc26c0..134f6307c867b 100644 --- a/flink-python/tox.ini +++ b/flink-python/tox.ini @@ -21,29 +21,27 @@ # in multiple virtualenvs. This configuration file will run the # test suite on all supported python versions. # new environments will be excluded by default unless explicitly added to envlist. -envlist = {py39, py310, py311, py312}-cython +env_list = {py39, py310, py311, py312}-cython [testenv] -whitelist_externals= - /bin/bash +allowlist_externals = + bash +dependency_groups = + dev deps = - --group dev - apache-flink-libraries + ./apache-flink-libraries passenv = * commands = python --version bash ./dev/integration_test.sh bash ./dev/run_pip_test.sh -# Replace the default installation command with a custom retry installation script, because on high-speed -# networks, downloading a package may raise a ConnectionResetError: [Errno 104] Peer reset connection. -install_command = {toxinidir}/dev/install_command.sh {opts} {packages} [flake8] # We follow PEP 8 (https://www.python.org/dev/peps/pep-0008/) with one exception: lines can be # up to 100 characters in length, not 79. ignore=E226,E241,E305,E402,E722,E731,E741,W503,W504,E721 max-line-length=100 -exclude=.tox/*,dev/*,lib/*,target/*,build/*,dist/*,pyflink/shell.py,.eggs/*,pyflink/fn_execution/tests/process_mode_test_data.py,pyflink/fn_execution/*_pb2.py*,pyflink/examples/table/basic_operations.py +exclude=.venv/*,.tox/*,dev/*,lib/*,target/*,build/*,dist/*,pyflink/shell.py,.eggs/*,pyflink/fn_execution/tests/process_mode_test_data.py,pyflink/fn_execution/*_pb2.py*,pyflink/examples/table/basic_operations.py [mypy] files=pyflink/common/*.py,pyflink/table/*.py,pyflink/datastream/*.py,pyflink/metrics/*.py