chore: migrate build system from pip/setup.py to uv workspaces (#12686)

[sh4shv4t]

Description
This PR implements a complete migration from fragmented pip-based dependency management to a unified uv workspace, addressing The migration modernizes the build system while maintaining full backward compatibility with existing workflows.

What does this PR do?
Replaces 7 setup.py files, 18 requirements.txt files, and 46+ scattered pip install commands across the repository with a single unified uv workspace containing one lockfile (uv.lock) with 195 resolved packages.

Key Changes
Build System Migration:

Created root pyproject.toml defining workspace with 4 member packages
Generated single uv.lock (634KB, 195 packages) replacing all requirements files
Centralized constraint-dependencies (protobuf>=6.31.1,<7.0, kubernetes>=8.0.0,<31, google-auth>=1.6.1,<3, etc.)
Configured workspace member resolution via [tool.uv.sources] for local development
Package Conversions (PEP 621 + hatchling):

kfp (sdk/python): Dynamic versioning from kfp/version.py, 20+ dependencies with optional extras
kfp-kubernetes (kubernetes_platform/python): Dynamic versioning from init.py, workspace source links
kfp-pipeline-spec (api/v2alpha1/python): Proto-generated package with hardcoded version
kfp-server-api (backend/api/v2beta1/python_http_client): Swagger-generated client with inline readme
CI/CD Infrastructure:

Added .github/actions/setup-python/action.yml composite action using astral-sh/setup-uv
Updated 6 workflows: sdk-yapf, kfp-sdk-unit-tests, kfp-sdk-client-tests, publish-packages, readthedocs-builds, gcpc-modules-tests
Replaced 46+ scattered pip install commands with standardized uv sync/uv run
Added .github/workflows/check-uv-lock.yml for automated lockfile sync verification
Configured .github/dependabot.yml for pip ecosystem dependency updates
Development Tooling:

Updated .pre-commit-config.yaml: Python linting hooks converted to use uv run (yapf, isort, pycln, docformatter)
Updated Makefiles in api/, kubernetes_platform/, sdk/ to use uv build --package
Proto generation Makefiles install uv in Docker container, use ~/.local/bin/uv build
Documentation:

Updated AGENTS.md: Modernized local testing, linting, and quick reference sections
Updated sdk/CONTRIBUTING.md: Complete rewrite with uv-based setup instructions
Updated .readthedocs.yml: Configured to use uv for Sphinx documentation builds
Cleanup:

Removed 7 setup.py files from all workspace packages
Removed 18 requirements.txt/requirements.in/requirements-dev.txt files
Removed 4 MANIFEST.in files
Removed hack/update-all-requirements.sh and hack/update-requirements.sh
Verification
✅ All 4 packages build successfully:

uv build --package kfp-pipeline-spec # → 2.15.2 (.tar.gz + .whl)
uv build --package kfp-server-api # → 2.15.2 (.tar.gz + .whl)
uv build --package kfp # → 2.15.2 (.tar.gz + .whl)
uv build --package kfp-kubernetes # → 2.15.2 (.tar.gz + .whl)
✅ Proto generation working:

make -C api python # Generates pipeline_spec_pb2.py
make -C kubernetes_platform python # Generates kubernetes_executor_config_pb2.py
✅ Dependency resolution:

uv lock → Resolved 195 packages in <10s
uv lock --check → Lockfile in sync
All constraint-dependencies applied correctly across workspace
✅ Test suite:

uv run pytest sdk/python/kfp/dsl/types/ → 263 tests collected and passing
uv run pytest sdk/python/kfp/compiler/ --co → 256 tests collected successfully
✅ Package imports:

import kfp # ✓ KFP 2.15.2
from kfp.pipeline_spec import pipeline_spec_pb2 # ✓ Proto imports working
from kfp.dsl import component, pipeline # ✓ All main modules import
✅ CLI tools:

uv run kfp --version → kfp 2.15.2
uv run yapf --version → yapf 0.43.0
uv run isort --version → isort 5.10.1
Benefits
Single source of truth: One uv.lock replaces 18 fragmented requirements files
Reproducible builds: Constraint-dependencies enforce version bounds across all packages
Faster CI: uv resolves dependencies 10-100x faster than pip
Simplified workflow: uv sync/uv run/uv build replaces complex pip/virtualenv/build commands
Atomic operations: All packages resolved together, eliminates version conflicts between workspace members
Migration Path for Contributors
Existing workflows remain compatible:

Before (still works in legacy areas):

pip install -r requirements.txt

Fixes #12686

After (new workspace):

uv sync # Install all dependencies
uv sync --extra dev # Install with dev dependencies
uv run pytest sdk/python/kfp # Run tests
uv build --package kfp # Build package
Checklist:

Signed off commits with git commit --signoff
PR title follows convention: chore: migrate build system from pip/setup.py to uv workspaces (#12686)
All packages buildable and tests passing
CI workflows updated and validated
Documentation updated for new workflow
No breaking changes to published package APIs

[google-oss-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign chensun for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[github-actions]

🎉 Welcome to the Kubeflow Pipelines repo! 🎉

Thanks for opening your first PR! We're excited to have you onboard 🚀

Next steps:

• Check out the Contributing Guide for repo-specific guidelines and the Kubeflow Contributor Guide for general community standards
• Join our #kubeflow-pipelines Slack channel
• Attend the Kubeflow Pipelines WG meetings

Feel free to ask questions in the comments.

[google-oss-prow]

Hi @sh4shv4t. Thanks for your PR.

I'm waiting for a kubeflow member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[Copilot]

Pull request overview

This PR migrates the KFP repository from fragmented pip-based dependency management to a unified uv workspace, replacing 7 setup.py files, 18 requirements.txt files, and 46+ scattered pip install commands with a single uv.lock file containing 195 resolved packages.

Changes:

• Created root pyproject.toml defining a workspace with 4 member packages (kfp, kfp-kubernetes, kfp-pipeline-spec, kfp-server-api)
• Migrated all packages from setup.py to PEP 621-compliant pyproject.toml files with hatchling build backend
• Updated CI/CD workflows and test scripts to use uv commands (uv sync, uv run, uv build)
• Added new composite action .github/actions/setup-python for consistent Python/uv setup
• Updated documentation (AGENTS.md, sdk/CONTRIBUTING.md, .readthedocs.yml) for uv-based workflows
• Removed all legacy setup.py, requirements.txt, requirements.in, and MANIFEST.in files

Reviewed changes

Copilot reviewed 58 out of 59 changed files in this pull request and generated 17 comments.

Show a summary per file

File	Description
pyproject.toml	Root workspace configuration with constraint-dependencies and optional extras groups (lint, test, dev, docs, ci)
sdk/python/pyproject.toml	KFP SDK package configuration with dynamic versioning from kfp/version.py
kubernetes_platform/python/pyproject.toml	kfp-kubernetes package with workspace source links to kfp
api/v2alpha1/python/pyproject.toml	kfp-pipeline-spec proto-generated package
backend/api/v2beta1/python_http_client/pyproject.toml	kfp-server-api swagger-generated client
.github/actions/setup-python/action.yml	New composite action for Python + uv setup with dependency installation
.github/workflows/*.yml	Updated 10+ workflows to use new setup-python action and uv commands
test/presubmit-*.sh	Updated test scripts to use uv sync and uv run commands
api/Makefile, sdk/Makefile, kubernetes_platform/Makefile	Updated to use uv build commands in Docker containers
.pre-commit-config.yaml	Converted Python linting hooks to use uv run with local repo
.readthedocs.yml	Configured to use uv for documentation builds
.github/dependabot.yml	Added dependency update configuration for pip ecosystem and GitHub Actions
.github/workflows/check-uv-lock.yml	New workflow to verify uv.lock stays in sync with pyproject.toml files

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

The twine check dist/* command should be executed as uv run twine check dist/* for consistency with the rest of the codebase and to ensure the correct environment is used.

[Copilot]

The twine check dist/* command should be executed as uv run twine check dist/* for consistency with the rest of the codebase and to ensure the correct environment is used.

[Copilot]

The twine check dist/* command should be executed as uv run twine check dist/* for consistency with the rest of the codebase and to ensure the correct environment is used.

Suggested change

	twine check dist/*
	uv run twine check dist/*

[Copilot]

The uv build --package kfp command is executed from within the sdk/python directory, but --package expects to be run from the workspace root. Since the command changes to sdk/python first, it should use uv build without the --package flag, or stay at the workspace root and use the full package name. This will likely fail at build time.

[Copilot]

The same issue exists here: uv build --package kfp-kubernetes is executed from within the kubernetes_platform/python directory. The --package flag expects to be run from the workspace root. This should either use uv build without the --package flag, or execute from the workspace root with the package name.

[Copilot]

This script is using a mix of pip and uv commands. Lines 19-20 use pip to install kfp from PyPI, but lines 31-33 use uv build. Lines 36-38 then use pip to install the built wheels. While this might be intentional for testing upgrade scenarios, it creates an inconsistent environment. Consider whether uv should be used for all package installations, or document why this mixing is necessary for the upgrade test scenario.

[Copilot]

The uv build --package kfp command is executed from within the sdk/python directory. The --package flag expects to be run from the workspace root. This should either use uv build without the --package flag, or stay at the workspace root and use the full package name.

[Copilot]

This build target installs the uv build tool via pip install --user --break-system-packages uv without any version pinning or integrity verification, then uses it to produce release artifacts with uv build --package kfp. If the uv package on PyPI or the network path to it is compromised, an attacker could execute arbitrary code in the build container and tamper with the generated kfp distributions. To reduce supply-chain risk, pin uv to a specific trusted version (or vendor it locally) and, where feasible, add an integrity check for the downloaded artifact before using it in the release pipeline.

[Copilot]

This build rule installs the uv build tool from PyPI with pip install --user --break-system-packages uv and immediately uses it to build the kfp-kubernetes wheel, but the dependency is not pinned or integrity-checked. A malicious or compromised uv release could execute in this container and alter the generated wheel or otherwise tamper with build outputs. Pin uv to a known-good version (or vendor it) and prefer an installation method that supports integrity verification for the tool used in your packaging pipeline.

[Copilot]

This Docker-based build path installs uv via pip install --user --break-system-packages uv without a version pin or integrity verification and then uses it to build the kfp-pipeline-spec package. Because uv directly controls how the wheel is constructed, a compromised or malicious uv package from PyPI could tamper with the generated artifacts or run arbitrary code in the build environment. Mitigate this supply-chain risk by pinning uv to a specific trusted version (or vendoring it) and, where possible, validating the downloaded tool before using it in the release process.

[hbelmiro]

/ok-to-test