Skip to content

Stage the OpenBLAS install in the builder and just COPY the files #2466

@coderabbitai

Description

@coderabbitai

Problem Description

During ppc64le builds of the jupyter-datascience image, OpenBLAS installation fails with permission errors when attempting to install directly into /usr/local during the final stage. Build logs from PR #2460 show:

  • cc1: fatal error: can't open 'ctest1.s' for writing: Permission denied
  • /usr/bin/ld: cannot open output file getarch: Permission denied
  • mkdir: cannot create directory '/usr/local/lib/cmake': Permission denied
  • Writing /usr/local/include/openblas_config.h failed

The current implementation in jupyter/datascience/ubi9-python-3.12/Dockerfile.cpu (lines 206-221) builds OpenBLAS in the openblas-builder stage but attempts installation in the final jupyter-datascience stage, causing permission conflicts.

Root Cause Analysis

  1. Multi-stage permission mismatch: OpenBLAS build artifacts are copied as source code, then make install runs in final stage
  2. Runtime permission conflicts: Final stage USER 1001 cannot write to /usr/local system paths
  3. Architecture-specific complexity: Issue only affects ppc64le where OpenBLAS compilation is required

Proposed Solution

Recommended Approach: Stage the complete installation within the builder stage using DESTDIR pattern:

# In openblas-builder stage
source /opt/rh/gcc-toolset-13/enable
make -j$(nproc) TARGET=POWER9 BINARY=64 USE_OPENMP=1 USE_THREAD=1 NUM_THREADS=120 DYNAMIC_ARCH=1 INTERFACE64=0
make -j$(nproc) PREFIX=/usr/local DESTDIR=/openblas-root install

# In final stage  
COPY --from=openblas-builder /openblas-root/usr/local /usr/local

Alternative Approaches

  1. Custom prefix approach: Install to builder-specific path and copy selectively
  2. Archive approach: Create tar archive in builder, extract in final stage

Acceptance Criteria

  • OpenBLAS builds successfully in openblas-builder stage with DESTDIR staging
  • Installation artifacts are copied cleanly to final stage without permission errors
  • ppc64le builds complete successfully without OpenBLAS-related failures
  • Non-ppc64le architectures continue to work with placeholder/skip logic
  • Build time remains reasonable (no significant performance regression)
  • Runtime functionality verified - OpenBLAS libraries accessible to Python packages
  • Multi-architecture CI builds pass consistently

Implementation Guidance

  1. Modify openblas-builder stage to use DESTDIR pattern for installation staging
  2. Update final stage COPY commands to transfer staged artifacts from builder
  3. Ensure proper file ownership/permissions during COPY operation
  4. Verify PKG_CONFIG_PATH environment variable remains functional
  5. Test integration with NumPy, SciPy and other OpenBLAS-dependent packages

Context

This issue addresses systematic infrastructure reliability improvements for Power architecture support in ODH notebook images.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

Status

📋 Backlog

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions