PR Description: Upgrade Docker Build Infrastructure and Fix s390x and AMD Build Failures

Overview

This PR performs a comprehensive upgrade of the Docker build infrastructure, including base images, compilers, build tools, and Python packages. The upgrade process revealed and fixed multiple build failures, particularly on s390x architecture.

Work Performed: Infrastructure Upgrades

1. Ubuntu Base Image Upgrade

• From: Ubuntu Jammy (22.04)
• To: Ubuntu Noble (24.04) - ghcr.io/onnxmlir/ubuntu:noble
• Reason: Access to newer toolchain versions and better s390x support
• Impact: Python 3.12.3, updated system libraries

2. RHEL/UBI Image Upgrade

• From: UBI8
• To: UBI9
• Reason: Align with enterprise Linux lifecycle and security updates
• Impact: Different package names and availability

3. CMake Upgrade

• From: Manually installed specific version
• To: Latest from Ubuntu Noble repositories
• Reason: Better support for modern C++ features and build systems

4. Clang Compiler Upgrade

• From: clang-19 (unavailable in UBI9)
• To: clang-20 (all architectures)
• Reason:
  • clang-19 not available in UBI9 base images
  • Better SystemZ (s390x) backend support
  • Improved multiply-with-overflow handling
  • Fixes Abseil duration.cc compilation on s390x

5. Setuptools Downgrade

• From: Latest (80.x)
• To: 77.0.1 (pinned)
• Reason: setuptools >= 70.x creates conflicts with pip packaging versions
• Impact: Stable builds with --no-build-isolation flag

6. Java Version for Protobuf Build

• Version: Java 11 (JDK 11)
• Reason: Java module system access restrictions
• Details: Newer JDK versions (17+) have stricter module encapsulation that causes Bazel build failures with "unnamed module" errors
• Solution: Use Java 11 which has more permissive module access

7. Python Package Management

• Strategy: Consistent --user flag usage
• Reason: Ensures pip and setuptools are in the same location for --no-build-isolation builds

Issues Discovered and Fixed

Issue 1: Protobuf Build Failures

Problem 1.1: Java Module Access Restrictions

Symptoms:

ERROR: unnamed module cannot access java.base packages

Root Cause: Ubuntu Noble ships with JDK 21 by default, which has strict module encapsulation. Bazel's build process requires access to internal Java packages that are restricted in newer JDK versions.

Commits:

• 95b24e8b - Fix the bazel build for unnamed module found issue in ubuntu noble
• 5fadf25b - Add buildbot for amd64 image and update bazel build to add java options
• 2284e318 - Update jdk
• 51077caf - Try with java11 similar to ubuntu:jammy

Solution: Install and use Java 11 (JDK 11) for Protobuf Bazel builds, which has more permissive module access policies.

Problem 1.2: Bazel Not Respecting Compiler Environment Variables

Symptoms:

Abseil duration.cc compilation failure on s390x
Multiply-with-overflow operations failing

Root Cause: Bazel ignores CC/CXX environment variables by default. It needs explicit flags to force compiler selection during both repository fetch (for dependencies like Abseil) and compilation phases.

Commits:

• 33be29fe - Fix clang compiler version for s390x for bazel build
• f2eb2b8d - Force bazel to use clang-20 compiler
• 459251b5 - Add action-env to bazel build and not startup

Solution:

• Use --repo_env=CC=clang-20 --repo_env=CXX=clang++-20 for repository rules (Abseil fetch)
• Use --action_env=CC=clang-20 --action_env=CXX=clang++-20 for compilation actions

Problem 1.3: Clang-19 Unavailable in UBI9

Symptoms:

Package clang-19 not found

Root Cause: UBI9 (RHEL 9) repositories don't provide clang-19 packages.

Commits:

• 3be92696 - Clang-19 Fix Applied for s390x Protobuf Build
• 931e9d38 - Use clang-20 for s390x shared and static builds and for amd64 shared and static builds
• becdead9 - Add clang-20 for all the onnx-mlir dev build images

Solution: Migrate to clang-20 which is available in Ubuntu Noble repositories for all architectures.

Issue 2: ONNX Build Failures

Problem 2.1: protoc-gen-mypy Segfault on s390x

Symptoms:

Segmentation fault during type stub generation
protoc-gen-mypy crashes on s390x architecture

Root Cause: The protobuf type stub generator plugin has architecture-specific bugs on s390x that cause segmentation faults.

Commits:

• 35a70672 - protoc-gen-mypy: Not invoked on s390x (bypassed entirely)
• f2dc81d3 - Create empty stubs to satisfy setup.py's assertion check

Solution:

• Disable stub generation with CMake flag: -DONNX_GEN_PB_TYPE_STUBS=OFF
• Create empty stub files (onnx_ml_pb2.pyi, onnx_data_pb2.pyi, onnx_operators_ml_pb2.pyi) to satisfy ONNX setup.py's assertion checks

Problem 2.2: Python Package Installation Path Conflicts

Symptoms:

ModuleNotFoundError: No module named 'setuptools'
pip cannot find setuptools during --no-build-isolation builds

Root Cause: Inconsistent use of --user flag caused pip and setuptools to be installed in different locations (system vs user site-packages), breaking --no-build-isolation builds.

Commits:

• 72761db6 - Fix python package installation paths to --user path
• 9c93882c - Add --user flag to pip3 install
• 6ae22f45 - Remove --user
• c8e8b690 - Remove --user
• 0bea7dc6 - Add back --user flag
• 1565dc9b - Add back --user flag and sed the assert check in setup.py for onnx build
• 7c051d27 - Use no-build-isolation flag for resolving the issue

Solution:

• Use --user flag consistently for all Python package installations
• Pin setuptools to 77.0.1 to avoid conflicts with pip packaging
• Use --no-build-isolation for ONNX build to ensure pip finds setuptools

Problem 2.3: Setup.py Assertion Failures

Symptoms:

AssertionError: Type stub files not found

Root Cause: ONNX setup.py asserts that type stub files exist, even when stub generation is disabled.

Commits:

• a2d4b7ff - Update sed for assertion error
• 1565dc9b - Add back --user flag and sed the assert check in setup.py for onnx build

Solution: Create empty stub files to satisfy the assertion without actually generating stubs (which would segfault on s390x).

Version Summary

Technical Implementation Details

Protobuf Build Configuration

# Use Java 11 to avoid module access restrictions
export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-$(dpkg --print-architecture)

# Use clang-20 for all architectures
export CC_COMPILER="clang-20" CXX_COMPILER="clang++-20"

# Bazel fetch with --repo_env for repository rules (Abseil)
bazel fetch --repo_env=CC=$CC_COMPILER --repo_env=CXX=$CXX_COMPILER //python/dist:binary_wheel

# Bazel build with --action_env for compilation actions
bazel build --action_env=CC=$CC_COMPILER --action_env=CXX=$CXX_COMPILER //python/dist:binary_wheel

ONNX Build Configuration (s390x)

# Detect architecture and set flags
EXTRA_CMAKE_ARGS=""
if [ "$(uname -m)" = "s390x" ]; then
    # Disable type stub generation to avoid protoc-gen-mypy segfault
    EXTRA_CMAKE_ARGS="-DONNX_GEN_PB_TYPE_STUBS=OFF"
    # Create empty stub files to satisfy setup.py's assertion check
    mkdir -p onnx && touch onnx/onnx_ml_pb2.pyi onnx/onnx_data_pb2.pyi onnx/onnx_operators_ml_pb2.pyi
fi

# Build with clang-20 and no-build-isolation
CC=clang-20 CXX=clang++-20 \
CMAKE_ARGS="-DCMAKE_INSTALL_LIBDIR=lib \
            -Dprotobuf_DIR=/usr/local/lib/cmake/protobuf \
            -Dabsl_DIR=/usr/local/lib/cmake/absl \
            ${EXTRA_CMAKE_ARGS}" \
python3 -m pip install --no-build-isolation .

Files Modified

1. docker/Dockerfile.llvm-project - Base image, clang-20, Java 11, Bazel compiler flags
2. docker/Dockerfile.onnx-mlir-dev - Setuptools 77.0.1, ONNX s390x workaround, clang-20
3. docker/Dockerfile.onnx-mlir - Same changes as onnx-mlir-dev

Testing

All three Docker images build successfully on all architectures:

# s390x
docker build -f docker/Dockerfile.llvm-project -t llvm-project:s390x .
docker build -f docker/Dockerfile.onnx-mlir-dev -t onnx-mlir-dev:s390x .
docker build -f docker/Dockerfile.onnx-mlir -t onnx-mlir:s390x .

# x86_64, ppc64le, aarch64
# Same commands work on all architectures

Impact

• ✅ Fixes s390x build failures
• ✅ Upgrades infrastructure to modern versions
• ✅ Maintains compatibility with all architectures (x86_64, ppc64le, aarch64, s390x)
• ✅ Ensures consistent ABI across all C++ components
• ✅ Well-documented with inline comments
• ✅ Iterative debugging approach preserved in commit history
• ✅ Resolves Java module access issues for Protobuf builds
• ✅ Handles architecture-specific issues (protoc-gen-mypy on s390x)