RHOAISTRAT-169: add s390x testing to CI using QEMU and podman build --platform (#1167)

* Add s390x testing to CI using QEMU and --platform

This commit introduces CI testing for the s390x architecture by
leveraging QEMU user-mode emulation and Podman's --platform flag.
This revised approach avoids running Podman-in-Docker.

Key changes:
- Added `docker/setup-qemu-action` to the main build job in
  `.github/workflows/build-notebooks-TEMPLATE.yaml` to enable
  the runner to execute s390x binaries.
- Modified `ci/cached-builds/gen_gha_matrix_jobs.py` to generate
  a matrix including an `s390x` boolean flag.
- Updated the `build` job in
  `.github/workflows/build-notebooks-TEMPLATE.yaml`:
    - It now conditionally sets `--platform=linux/s390x` for
      `podman build` commands via `CONTAINER_BUILD_CACHE_ARGS`
      when `matrix.s390x` is true.
    - PyTest execution is conditional:
        - Runs tests marked `s390x` for s390x builds.
        - Runs tests marked `not s390x` for amd64 builds.
    - OpenShift and Playwright tests are skipped for s390x builds.
- The main workflow files (`build-notebooks-pr.yaml` and
  `build-notebooks-push.yaml`) pass the `--s390x-images`
  argument to the matrix generation script to control s390x builds
  (excluded for PRs, included for pushes).
- s390x-specific tests for `pyzmq` and `mongocli` are included and
  marked with `@pytest.mark.s390x`.

This approach provides a cleaner way to build and test for s390x
by directly using Podman's multi-architecture capabilities, facilitated
by QEMU on the runner.

* RHOAIENG-26279: include s390x platform in build workflows and update matrix configuration

* RHOAIENG-26279: streamline s390x platform configuration in build workflows

* RHOAIENG-26279: remove s390x-specific conditions from workflows and streamline testing logic

* RHOAIENG-26279: refine container build cache arguments to explicitly pass platform configuration

* RHOAIENG-24348: remove s390x marker from mongocli binary test in JupyterLab

* RHOAIENG-26279: update s390x platform conditions in matrix generation and fix JupyterLab test assertion formatting

* RHOAIENG-26279: fix matrix generation by updating key from `targets` to `targets_with_platform` in `has_jobs` calculation

* RHOAIENG-26279: fix matrix generation by updating key from `targets` to `targets_with_platform` in `has_jobs` calculation

* RHOAIENG-26279: add explicit `choices` constraints for `--rhel-images` and `--s390x-images` arguments in matrix generation script

* RHOAIENG-26279: remove unused `--s390x-images` validation in matrix generation script

* RHOAIENG-26279: centralize s390x-compatible target list into `S390X_COMPATIBLE` set in matrix generation script

* RHOAIENG-26279: rename JupyterLab test method for pyzmq to improve clarity

* RHOAIENG-26279: refactor JupyterLab pyzmq test to use encoded function execution logic for cleaner implementation

* RHOAIENG-26279: update JupyterLab pyzmq test to reorder imports and add comments for ruff and pyright noqa annotations

* RHOAIENG-26279: add runtime image fixture and move pyzmq test to runtime-specific test module

* RHOAIENG-26279: simplify runtime image validation logic, reorder imports, and improve socket variable naming

* Update tests/containers/runtimes/runtime_test.py

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>

* RHOAIENG-26279: add workaround for s390x podman build issue by passing extra build args

* RHOAIENG-26279: update runtime image fixture to use yield instead of return

* RHOAIENG-26279: update runtime_image fixture scope from session to function

* RHOAIENG-26279: update platform check from architecture to platform for s390x podman build workaround

* RHOAIENG-26279: refactor runtime test to use context manager for container lifecycle management

* RHOAIENG-26279: refactor runtime test to use context manager for container lifecycle management

# Conflicts:
#	tests/containers/runtimes/runtime_test.py

* RHOAIENG-26279: update s390x podman build workaround env variable to use undefined flag

* RHOAIENG-26279: fix Python interpreter path in runtime test to use user venv

* RHOAIENG-26279: remove unused WorkbenchContainer import and cleanup test logic

* RHOAIENG-26279: update runtime test to handle exec output as bytes instead of string

* RHOAIENG-26279: remove redundant failure case in runtime test cleanup logic

* RHOAIENG-26279: skip image tests on s390x due to unresolvable dependency installation issue

---------

Co-authored-by: google-labs-jules[bot] <161369871+google-labs-jules[bot]@users.noreply.github.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>

Author: 3 people

.github/workflows/build-notebooks-TEMPLATE.yaml

@@ -15,6 +15,10 @@ name: Build & Publish Notebook Servers (TEMPLATE)
         required: true
         description: "top workflow's `github`"
         type: string
+      platform:
+        required: true
+        description: "platform to build, podman build --platform="
+        type: string
       subscription:
         required: false
         default: false
@@ -57,6 +61,12 @@ jobs:
         with:
           ref: "refs/pull/${{ fromJson(inputs.github).event.number }}/merge"
 
+      - name: Set up QEMU to build linux/s390x
+        if: ${{ inputs.platform == 'linux/s390x' }}
+        uses: docker/setup-qemu-action@v3
+        with:
+          platforms: s390x
+
       - run: mkdir -p $TMPDIR
 
       # do this early because it's fast and why not
@@ -289,6 +299,19 @@ jobs:
 
       # region Image build
 
+      - name: Compute extra podman build args
+        id: extra-podman-build-args
+        run: |
+          set -Eeuxo pipefail
+
+          EXTRA_PODMAN_BUILD_ARGS=""
+          if [[ "${{ inputs.platform }}" == "linux/s390x" ]]; then
+            # workaround for known issue https://github.com/zeromq/libzmq/pull/4486
+            # In qemu-user, CACHELINE_SIZE probe is undefined
+            EXTRA_PODMAN_BUILD_ARGS+='--env=CXXFLAGS=-Dundefined=64'
+          fi
+          echo "EXTRA_PODMAN_BUILD_ARGS=$EXTRA_PODMAN_BUILD_ARGS" >> $GITHUB_OUTPUT
+
       # https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#push
       # https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#pull_request
 
@@ -303,7 +326,8 @@ jobs:
           fromJson(inputs.github).event_name == 'workflow_dispatch' }}
         env:
           IMAGE_TAG: "${{ steps.calculated_vars.outputs.IMAGE_TAG }}"
-          CONTAINER_BUILD_CACHE_ARGS: "--cache-from ${{ env.CACHE }} --cache-to ${{ env.CACHE }}"
+          # Configuring `podman build --platform=` through CONTAINER_BUILD_CACHE_ARGS is a venerable hack on this project
+          CONTAINER_BUILD_CACHE_ARGS: "${{ steps.extra-podman-build-args.outputs.EXTRA_PODMAN_BUILD_ARGS }} --platform=${{ inputs.platform }} --cache-from ${{ env.CACHE }} --cache-to ${{ env.CACHE }}"
       - name: "pull_request: make ${{ inputs.target }}"
         run: |
           # print running stats on disk occupancy
@@ -314,7 +338,8 @@ jobs:
           fromJson(inputs.github).event_name == 'pull_request_target' }}"
         env:
           IMAGE_TAG: "${{ steps.calculated_vars.outputs.IMAGE_TAG }}"
-          CONTAINER_BUILD_CACHE_ARGS: "--cache-from ${{ env.CACHE }}"
+          # Configuring `podman build --platform=` through CONTAINER_BUILD_CACHE_ARGS is a venerable hack on this project
+          CONTAINER_BUILD_CACHE_ARGS: "${{ steps.extra-podman-build-args.outputs.EXTRA_PODMAN_BUILD_ARGS }} --platform=${{ inputs.platform }} --cache-from ${{ env.CACHE }}"
           # We don't have access to image registry, so disable pushing
           PUSH_IMAGES: "no"
 
@@ -504,7 +529,9 @@ jobs:
           kubectl patch storageclass local-path -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
 
       - name: "Run image tests"
-        if: ${{ steps.have-tests.outputs.tests == 'true' }}
+        # skip on s390x because we are unable to install requirements-elyra.txt that's installed by runtime image tests
+        # https://raw.githubusercontent.com/opendatahub-io/elyra/refs/heads/main/etc/generic/requirements-elyra.txt
+        if: ${{ steps.have-tests.outputs.tests == 'true' && !contains(fromJSON('["linux/s390x"]'), inputs.platform) }}
         run: python3 ci/cached-builds/make_test.py --target ${{ inputs.target }}
         env:
           IMAGE_TAG: "${{ steps.calculated_vars.outputs.IMAGE_TAG }}"

.github/workflows/build-notebooks-pr-rhel.yaml

@@ -71,5 +71,6 @@ jobs:
     with:
       target: "${{ matrix.target }}"
       github: "${{ toJSON(github) }}"
+      platform: "${{ matrix.platform }}"
       subscription: "${{ matrix.subscription }}"
     secrets: inherit

.github/workflows/build-notebooks-pr.yaml

@@ -40,7 +40,8 @@ jobs:
           python3 ci/cached-builds/gen_gha_matrix_jobs.py \
             --from-ref 'origin/${{ github.event.pull_request.base.ref }}' \
             --to-ref '${{ github.event.pull_request.head.ref }}' \
-            --rhel-images exclude
+            --rhel-images exclude \
+            --s390x-images include
         id: gen
         env:
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
@@ -56,5 +57,6 @@ jobs:
     with:
       target: "${{ matrix.target }}"
       github: "${{ toJSON(github) }}"
+      platform: "${{ matrix.platform }}"
       subscription: "${{ matrix.subscription }}"
     secrets: inherit

.github/workflows/build-notebooks-push.yaml

@@ -29,7 +29,8 @@ jobs:
       - name: Determine targets to build (we want to build everything on push)
         run: |
           set -x
-          python3 ci/cached-builds/gen_gha_matrix_jobs.py
+          python3 ci/cached-builds/gen_gha_matrix_jobs.py \
+            --s390x-images include
         id: gen
         env:
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
@@ -45,5 +46,6 @@ jobs:
     with:
       target: "${{ matrix.target }}"
       github: "${{ toJSON(github) }}"
+      platform: "${{ matrix.platform }}"
       subscription: "${{ matrix.subscription }}"
     secrets: inherit

ci/cached-builds/gen_gha_matrix_jobs.py

@@ -21,6 +21,11 @@
 
 project_dir = pathlib.Path(__file__).parent.parent.parent.absolute()
 
+S390X_COMPATIBLE = {
+    "runtime-minimal-ubi9-python-3.11",
+    # add more here
+}
+
 
 def extract_image_targets(makefile_dir: pathlib.Path | str | None = None) -> list[str]:
     if makefile_dir is None:
@@ -48,6 +53,12 @@ class RhelImages(enum.Enum):
     INCLUDE_ONLY = "include-only"
 
 
+class S390xImages(enum.Enum):
+    EXCLUDE = "exclude"
+    INCLUDE = "include"
+    ONLY = "only"
+
+
 def main() -> None:
     logging.basicConfig(level=logging.DEBUG, stream=sys.stderr)
 
@@ -61,11 +72,21 @@ def main() -> None:
     argparser.add_argument(
         "--rhel-images",
         type=RhelImages,
+        choices=list(RhelImages),
         required=False,
         default=RhelImages.INCLUDE,
         nargs="?",
         help="Whether to `include` rhel images or `exclude` them or `include-only` them",
     )
+    argparser.add_argument(
+        "--s390x-images",
+        type=S390xImages,
+        choices=list(S390xImages),
+        required=False,
+        default=S390xImages.INCLUDE,
+        nargs="?",
+        help="Whether to include, exclude, or only include s390x images",
+    )
     args = argparser.parse_args()
 
     targets = extract_image_targets()
@@ -84,19 +105,35 @@ def main() -> None:
     else:
         raise Exception(f"Unknown value for --rhel-images: {args.rhel_images}")
 
+    targets_with_platform: list[tuple[str, str]] = []
+    for target in targets:
+        if args.s390x_images != S390xImages.ONLY:
+            targets_with_platform.append((target, "linux/amd64"))
+        if args.s390x_images != S390xImages.EXCLUDE:
+            # NOTE: hardcode the list of s390x-compatible Makefile targets in S390X_COMPATIBLE
+            if target in S390X_COMPATIBLE:
+                targets_with_platform.append((target, "linux/s390x"))
+
     # https://stackoverflow.com/questions/66025220/paired-values-in-github-actions-matrix
     output = [
         "matrix="
         + json.dumps(
             {
-                "include": [{"target": target, "subscription": "rhel" in target} for target in targets],
+                "include": [
+                    {
+                        "target": target,
+                        "platform": platform,
+                        "subscription": "rhel" in target,
+                    }
+                    for (target, platform) in targets_with_platform
+                ],
             },
             separators=(",", ":"),
         ),
-        "has_jobs=" + json.dumps(len(targets) > 0, separators=(",", ":")),
+        "has_jobs=" + json.dumps(len(targets_with_platform) > 0, separators=(",", ":")),
     ]
 
-    print("targets", targets)
+    print("targets", targets_with_platform)
     print(*output, sep="\n")
 
     if "GITHUB_ACTIONS" in os.environ:

tests/containers/conftest.py

@@ -115,6 +115,16 @@ def image(request):
     yield request.param
 
 
+@pytest.fixture(scope="function")
+def runtime_image(image: str):
+    image_metadata = get_image_metadata(image)
+
+    if "-runtime-" not in image_metadata.labels["name"]:
+        pytest.skip(f"Image {image} does not have any of '-runtime-' in {image_metadata.labels['name']=}'")
+
+    yield image_metadata
+
+
 @pytest.fixture(scope="function")
 def workbench_image(image: str):
     skip_if_not_workbench_image(image)

tests/containers/runtimes/runtime_test.py

@@ -0,0 +1,54 @@
+from __future__ import annotations
+
+import contextlib
+
+import allure
+import pytest
+import testcontainers.core.container
+
+from tests.containers import base_image_test, conftest, docker_utils
+
+
+class TestRuntimeImage:
+    """Tests for runtime images in this repository."""
+
+    @allure.description("Check that pyzmq library works correctly, important to check especially on s390x.")
+    def test_pyzmq_import(self, runtime_image: conftest.Image) -> None:
+        def check_zmq():
+            # ruff: noqa: PLC0415 `import` should be at the top-level of a file
+            import zmq  # pyright: ignore reportMissingImports
+
+            context = zmq.Context()
+            socket = None
+            try:
+                socket = context.socket(zmq.PAIR)
+                print("pyzmq imported and socket created successfully")
+            finally:
+                if socket is not None:
+                    socket.close(0)  # linger=0
+                context.term()
+
+        with running_image(runtime_image.name) as container:
+            exit_code, output_bytes = container.exec(
+                # NOTE: /usr/bin/python3 would not find zmq, we need python3 in user's venv
+                base_image_test.encode_python_function_execution_command_interpreter("python3", check_zmq)
+            )
+
+        assert exit_code == 0, f"Python script execution failed. Output: {output_bytes}"
+        assert b"pyzmq imported and socket created successfully" in output_bytes, (
+            f"Expected success message not found in output. Output: {output_bytes}"
+        )
+
+
+@contextlib.contextmanager
+def running_image(image: str):
+    """Usage: with running_image("quay.io/...") as container:"""
+    container = testcontainers.core.container.DockerContainer(image=image, user=23456, group_add=[0])
+    container.with_command("/bin/sh -c 'sleep infinity'")
+    try:
+        container.start()
+        yield container
+    except Exception as e:
+        pytest.fail(f"Unexpected exception in test: {e}")
+    finally:
+        docker_utils.NotebookContainer(container).stop(timeout=0)