Skip to content

Add conda/mamba CI workflows#227

Merged
aktech merged 5 commits intogoogle:masterfrom
czgdp1807:de-dock-ci
Jan 23, 2026
Merged

Add conda/mamba CI workflows#227
aktech merged 5 commits intogoogle:masterfrom
czgdp1807:de-dock-ci

Conversation

@czgdp1807
Copy link
Collaborator

Summary

This PR modernizes the CI pipeline by adding builds with conda/mamba GitHub Actions workflows, enabling builds with better reproducibility and manylinux2014 compatibility. In future, Docker based based, build.yml and test.yml will be removed.

Changes

New Files

  • ci/environment.yml: Conda environment with GCC 8.5.0, sysroot_linux-64=2.17, and all build/test dependencies
  • .github/workflows/conda-build.yml: Workflow for building manylinux2014 wheels across Python 3.9-3.11
  • .github/workflows/conda-test.yml: Workflow for running pytest tests on built wheels

Key Improvements

  1. manylinux2014 Compatibility: Uses GCC 8.5.0 + sysroot_linux-64=2.17 to match CentOS 7 devtoolset-8 (equivalent to the Docker environment)
  2. Simplified Maintenance: Declarative environment.yml replaces complex Dockerfile configuration
  3. Native Execution: Runs directly on GitHub-hosted ubuntu-latest runners instead of inside Docker containers

Technical Details

  • Compiler Toolchain: GCC 8.5.0 with glibc 2.17 headers ensures binary compatibility with manylinux2014
  • Dependency Management: Uses mamba-org/setup-micromamba@v1 for environment setup
  • Wheel Repair: auditwheel repair --plat manylinux2014_x86_64 ensures PyPI compatibility
  • Bazel Management: Uses Bazelisk 1.20.0 for consistent Bazel 6.1.0 version management

Removed Dependencies (to-be-done after merging this PR)

  • No longer requires Docker or docker-compose for local development
  • Eliminates need for maintaining Dockerfiles and container configurations

Benefits

  • Developer Experience: Local builds easily reproducible with micromamba env create -f ci/environment.yml
  • Maintenance: Single environment.yml file vs multiple Docker configuration files
  • Debugging: Easier to troubleshoot builds running natively on runners
  • Compatibility: Maintains exact manylinux2014 compliance for PyPI distribution

Testing

Build workflow produces manylinux2014_x86_64 wheels for Python 3.9-3.11
Test workflow validates wheels install and pass all pytest tests
Wheels are uploadable to PyPI with automated trusted publishing

This is ready to merge because all CI tests are passing. Refer - czgdp1807#4

Migration Notes

This intends to replace the existing Docker-based build process. Future PRs can add macOS support by:

  1. Creating ci/environment-macos.yml with Clang compilers
  2. Expanding the matrix.os to include macos-latest
  3. Using delocate for macOS wheel repair

@czgdp1807
Copy link
Collaborator Author

I will address this tomorrow morning.

Add conda/micromamba-based GitHub Actions workflows for building and
testing ml-metadata wheels. Uses GCC 8.5.0 to match manylinux2014
compatibility, supports Python 3.9-3.11, and includes automated PyPI
publishing.
- Bazel: Raise minimum to 6.5.0 and update repo version
- CI: Use Bazel 6.5.0 in conda-build/test GitHub Actions
- Docker: Switch manylinux image to bazel-6.5.0 tag
- Toolchain: Use devtoolset-10 for wheel builds
- Scope: Build/CI-only; no MLMD runtime or API changes

Motivation: align CI/local builds with Bazel 6.5.0 for stability and modernize the wheel build toolchain.
…lity

CMake 3.29+ removed support for CMake < 3.5 syntax, breaking libmysqlclient
(mariadb-connector-c v3.0.8) builds. Pin to CMake 3.24.4 to maintain backward
compatibility while supporting current toolchains.
@czgdp1807
Copy link
Collaborator Author

@aktech I think we should remove Build ml-metadata / Upload to PyPI since Build ml-metadata with Conda / Upload to PyPI (pull_request) will take priority.

The upload_to_pypi job is already present in conda-build.yml, which will
supersede the Docker-based build.yml workflow.
@czgdp1807
Copy link
Collaborator Author

@aktech This is ready to go. Let's merge.

- Use python -m build instead of setup.py bdist_wheel for PEP 517 compliance
- Pin setuptools<69.3 to avoid Metadata-Version 2.4 (PyPI only supports up to 2.3)
- Update numpy requirement to >=1.23,<2.0 for Python 3.11 compatibility
- Use auditwheel --plat auto for automatic manylinux tag detection
@aktech aktech merged commit 9357425 into google:master Jan 23, 2026
17 checks passed
vkarampudi pushed a commit to vkarampudi/ml-metadata that referenced this pull request Jan 29, 2026
vkarampudi added a commit that referenced this pull request Feb 4, 2026
* feat: Replace Docker builds with conda-based CI workflows

Add conda/micromamba-based GitHub Actions workflows for building and
testing ml-metadata wheels. Uses GCC 8.5.0 to match manylinux2014
compatibility, supports Python 3.9-3.11, and includes automated PyPI
publishing.

* build(ci): bump Bazel to 6.5.0 and update manylinux toolchain

- Bazel: Raise minimum to 6.5.0 and update repo version
- CI: Use Bazel 6.5.0 in conda-build/test GitHub Actions
- Docker: Switch manylinux image to bazel-6.5.0 tag
- Toolchain: Use devtoolset-10 for wheel builds
- Scope: Build/CI-only; no MLMD runtime or API changes

Motivation: align CI/local builds with Bazel 6.5.0 for stability and modernize the wheel build toolchain.

* fix: Pin CMake to 3.24.4 in Docker build for libmysqlclient compatibility

CMake 3.29+ removed support for CMake < 3.5 syntax, breaking libmysqlclient
(mariadb-connector-c v3.0.8) builds. Pin to CMake 3.24.4 to maintain backward
compatibility while supporting current toolchains.

* Fix conda build workflow for PyPI-compatible wheels

- Use python -m build instead of setup.py bdist_wheel for PEP 517 compliance
- Pin setuptools<69.3 to avoid Metadata-Version 2.4 (PyPI only supports up to 2.3)
- Update numpy requirement to >=1.23,<2.0 for Python 3.11 compatibility
- Use auditwheel --plat auto for automatic manylinux tag detection

* Merge pull request #227 from czgdp1807/de-dock-ci

Add conda/mamba CI workflows

* Fix: Update libmysqlclient.BUILD

* fix: correct mkdocstrings import config for v2.0 compatibility

---------

Co-authored-by: Gagandeep Singh <gdp.1807@gmail.com>
Co-authored-by: Amit Kumar <dtu.amit@gmail.com>
nsingla pushed a commit to nsingla/ml-metadata that referenced this pull request Feb 5, 2026
Signed-off-by: konflux-internal-p02 <170854209+konflux-internal-p02[bot]@users.noreply.github.com>
Co-authored-by: konflux-internal-p02[bot] <170854209+konflux-internal-p02[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants