Umbrella Issue: Porting Kubeflow to IBM Power (ppc64le)

[lehrig]

/kind feature

Enable builds & releases for IBM Power (ppc64le architecture). This proposal was presented with these slides at the 2022-10-25 Kubeflow community call with positive community feedback. We also created this design documentation: https://docs.google.com/document/d/1nGUvLonahoLogfWCHsoUOZl-s77YtPEiCjWBVlZjJHo/edit?usp=sharing

Why you need this feature:

• Widen scope of possible on-premises deployments (vanilla Kubernetes & OpenShift on Power)
• More general independence regarding processor architecture (x86, ppc64le, arm, …)
• Unified container builds

Describe the solution you'd like:

• Upstreaming changes that allow to build Dockerfiles on multiple architecture (starting with x86 & ppc64le)
• Upstreaming CI integration for multi-arch builds (starting with x86 & ppc64le)

We currently plan to divide our efforts into multiply phases:

1. low-hanging "easy" integrations where no or minor code changes are needed; excluding KFP; Kubeflow 1.7 release scope (✅ done),
2. same as 1. but now including additional KServe components for model serving; Kubeflow 1.8 release scope,
3. same as 1. but now including KFP; Kubeflow 1.9 release scope,
4. more complex integrations where external dependencies to python wheels exist.

Below is a detailed overview of each required integration, including links to associated PRs if those already exist.

Phase 1 Integrations (Kubeflow 1.7 scope)

• Poddefaults (Admission) Webhook: updated compatible base images & removed arch dependencies in different components for multiple arch support kubeflow#6650, Adding support for linux-ppc64le in CI for admission-webhook multi-arch docker image kubeflow#6803
  🚀 https://hub.docker.com/r/kubeflownotebookswg/poddefaults-webhook/tags
• Central Dashboard: Updating central-board Dockerfile for multi-arch support kubeflow#6861, Adding support for linux/ppc64le in CI for centraldashboard multi-arc… kubeflow#6923
  🚀 https://hub.docker.com/r/kubeflownotebookswg/centraldashboard/tags
• Jupyter Web App: updated compatible base images & removed arch dependencies in different components for multiple arch support kubeflow#6650, Adding support for linux/ppc64le in CI for jupyter-web-app multi-arch… kubeflow#6800
  🚀 https://hub.docker.com/r/kubeflownotebookswg/jupyter-web-app/tags
• KServe: Agent: removed arch dependency for multiarc support kserve/kserve#2476, Adding support for linux/ppc64le arch in github action for kserve-agent kserve/kserve#2549
  🚀 https://hub.docker.com/r/kserve/agent/tags
• KServe: Controller: removed arch dependency for multiarc support kserve/kserve#2476, Adding support for linux/ppc64le in github action for kserve-controller kserve/kserve#2550
  🚀 https://hub.docker.com/r/kserve/kserve-controller/tags
• KServe: Models Web App: updated base images for multiple arch support kserve/models-web-app#45, Adding support for linux-ppc64le in CI for models-web-app kserve/models-web-app#55
  🚀 https://hub.docker.com/r/kserve/models-web-app/tags
• KServe: QPExt: Adding multi-arch support for linux-ppc64le for qpext  kserve/kserve#2604
  🚀 https://hub.docker.com/r/kserve/qpext/tags
• KServe: Router: Adding multi-arch support for linux-ppc64le for router  kserve/kserve#2605
  🚀 https://hub.docker.com/r/kserve/router/tags
• MPI Operator: Adding mpi-operator workflow to release multi-arch docker image mpi-operator#489
  🚀 https://hub.docker.com/r/mpioperator/mpi-operator/tags
• Notebook Controller: updated compatible base images & removed arch dependencies in different components for multiple arch support kubeflow#6650, Adding multi-arch support for linux-ppc64le in CI for notebook-controller kubeflow#6771
  🚀 https://hub.docker.com/r/kubeflownotebookswg/notebook-controller/tags
• Profiles + KFAM: updated compatible base images & removed arch dependencies in different components for multiple arch support kubeflow#6650, Adding support for linux-ppc64le in CI for kfam multi-arch docker image kubeflow#6785, Adding support for linux-ppc64le in CI for profile-controller kubeflow#6809
  🚀 https://hub.docker.com/r/kubeflownotebookswg/profile-controller/tags
  🚀 https://hub.docker.com/r/kubeflownotebookswg/kfam/tags
• Tensorboard Controller: updated compatible base images & removed arch dependencies in different components for multiple arch support kubeflow#6650, Adding support for linux/ppc64le in CI for tensorboard-controller multi-arch docker images. kubeflow#6805
  🚀 https://hub.docker.com/r/kubeflownotebookswg/notebook-controller/tags
• Tensorboard Web App: updated compatible base images & removed arch dependencies in different components for multiple arch support kubeflow#6650, Adding support for linux/ppc64le in CI for tensorboard-web-app multi-arch docker images. kubeflow#6810
  🚀 https://hub.docker.com/r/kubeflownotebookswg/tensorboards-web-app/tags
• Training Operator: Removed GOARCH dependency for multiarch support trainer#1674, Adding support for linux/ppc64le in github actions for training-operator trainer#1692
  🚀 https://hub.docker.com/r/kubeflow/training-operator/tags
• Volumes Web App: updated compatible base images & removed arch dependencies in different components for multiple arch support kubeflow#6650, Adding support for linux-ppc64le in CI to release multi-arch docker image volumes-web-app  kubeflow#6811
  🚀 https://hub.docker.com/r/kubeflownotebookswg/volumes-web-app/tags

Phase 2 Integrations (Kubeflow 1.9 scope)

• KServe: PMML Server
• KServe: AIX
• KServe: Alibi
• KServe: Art
• Triton Inference Server (external): feat: Added power support for python backend on ubuntu. triton-inference-server/server#8329
• Seldon: ML Server (external)
• PyTorch: TorchServe (external)

Phase 3 Integrations (Kubeflow 1.10 scope)

Note: KFP is currently blocked by kubeflow/pipelines#8660 / GoogleCloudPlatform/oss-test-infra#1972

• KFP: Application-CRD-Controller
• KFP: Argoexec
• KFP: Cache-Server
• KFP: Frontend: feat: Updated dockerfile to support Power pipelines#12125
• KFP: Metadata Envoy
• KFP: Persistence Agent
• KFP: Scheduled Workflow
• KFP: Workflow Controller
• KFP: Viewer-CRD-Controller
• KServe: LGB Server: blocked by buildx and ppc64le wheel pyca/cryptography#7723
• KServe: Paddle Server: blocked by buildx and ppc64le wheel pyca/cryptography#7723
• KServe: SKLearn Server: blocked by buildx and ppc64le wheel pyca/cryptography#7723
• KServe: XGB Server: blocked by buildx and ppc64le wheel pyca/cryptography#7723
• Katib: controller, db-manager, ui
• Katib: file-metrics-collector
• Katib: tfevent-metrics-collector
• Katib: suggestion-hyperopt: updated dockerfiles for grpc builds for powerPC compilation~ katib#2262, Updated dockerfiles for grpc installation for ppc64le katib#2290
• Katib: suggestion-chocolate
• Katib: suggestion-hyperband: updated dockerfiles for grpc builds for powerPC compilation~ katib#2262, Updated dockerfiles for grpc installation for ppc64le katib#2290
• Katib: suggestion-skopt: updated dockerfiles for grpc builds for powerPC compilation~ katib#2262, Updated dockerfiles for grpc installation for ppc64le katib#2290
• Katib: suggestion-goptuna
• Katib: suggestion-optuna: updated dockerfiles for grpc builds for powerPC compilation~ katib#2262, Updated dockerfiles for grpc installation for ppc64le katib#2290
• Katib: suggestion-enas
• Katib: suggestion-darts: updated dockerfiles for grpc builds for powerPC compilation~ katib#2262, Updated dockerfiles for grpc installation for ppc64le katib#2290
• Katib: suggestion-pbt: updated dockerfiles for grpc builds for powerPC compilation~ katib#2262, Updated dockerfiles for grpc installation for ppc64le katib#2290
• Katib: earlystopping-medianstop: Updated dockerfiles for grpc installation for ppc64le katib#2290

Phase 4 Integrations (Post Kubeflow 1.11 scope)

• KFP: Api Server
• KFP: Metadata Writer
• KFP: Visualization Server
• ml-metadata (KFP wheel dep.): Adding multi-arch support for linux/ppc64le in ml-metadata google/ml-metadata#171, [ppc64le] Added GCC-11 and Power support google/ml-metadata#218
• KServe: Storage Initializer: blocked by buildx and ppc64le wheel pyca/cryptography#7723
• OIDC Auth (external): Enable oidc-authservice repository CI for power(ppc64le) architecture. arrikto/oidc-authservice#104; on-hold as potentially irrelevant as of Kubeflow v1.8 (Move away from AuthService manifests#2469)

[kimwnasptd]

Thanks for creating this tracking issue @lehrig!

I'm onboard with adding support for ppc64le, since this will greatly help KF adoption. The proposed plan makes sense.

My initial question at this time is whether we need to build different executables for this platform, which means we need a new set of images. I see in the PRs that the only needed change is to actually not set a specific platform, but I might be missing something.

Could you provide some more context on this one?

[lehrig]

@kimwnasptd, thanks for your support!

There are essentially 2 options for publishing images:

1. Multi-arch images, where we publish only 1 "virtual" image with support for multiple architectures. A pull command will then only fetch the concrete container image for the required platform. To do so, I'd recommend using buildx (e.g., see https://www.docker.com/blog/multi-arch-build-and-images-the-simple-way/) because it is easier to use / more automated compared to manually creating a container manifest file for multiple architectures.
2. Separate images per architecture.

IMO 1. should be the preferred solution. A challenge here will be that builds over all Kubeflow components are quite inconsistent. For example, some projects already use buildx while some others don't. I'd opt for implementing more consistency in the scope of this endeavor, e.g., by migrating builds towards buildx where feasible. My team would be willing to drive this, if this sounds good.

[kimwnasptd]

@lehrig I agree with the first approach as well, if it's viable to avoid having multiple manifests.

Docker's buildx seems promising. I hadn't used it the past though, but it seems quite straight forward. I don't have a hard preference on using buildx as long as we don't lock in ourselves and end up with Dockerfiles that need specific Docker features and can only be build with docker.

[lehrig]

As wished by @kimwnasptd, quoting myself from kubeflow/kubeflow#6650 (comment) to clarify how we envision multi-arch builds:

Yes, it's good to let GO determine the arch, so we don't have to maintain an arch list explicitly here or wrap around boiler-plate code with arch-specific if/else statements.

Instead, we now shift the control which arch is actually build to the build system. If doing nothing special, the arch of the build machine is simply used (and as Kubeflow is currently build on amd64, we are backwards-compatible to the current behavior).

In further PRs, we will additionally modify docker-based builds using buildx, where you can, for instance, do something like this:
docker buildx build --platform linux/amd64,linux/ppc64le ...

Here, docker will automatically run actually 2 builds: one for amd64 and one for ppc64le. When coming to above GO-code, GO will acknowledge the external platform configuration and build it correctly. In case no native hardware is available for the given platform, docker will emulate the architecture using QEMU - so you can also build for different archs on amd64.

The final outcome is a single multi-arch image with support of all archs listed in Docker's platform statement.

[kimwnasptd]

@lehrig @adilhusain-s! the first image in this repo with support for ppc64le is up! 🎉

https://hub.docker.com/layers/kubeflownotebookswg/notebook-controller/80f695e/images/sha256-2870219816f6be1153ca97eb604b4f20393c34afdb4eade83f0966ccf90f8018?context=explore

[kimwnasptd]

@lehrig @pranavpandit1 @adilhusain-s I realized that right now we've only implemented the logic for using docker buildx only for the Actions that run when a PR is merged, and not when a PR is opened.

Realized that the Centraldashboard image was not getting build, even though the PR checks were green:
28a24ffb170769a228d46a19892f7420b22a0816
74f020e0d9c3f58712a3b466f9d1bb86c4607beb
65e41bf28b8e79be4e1f822afe56e218c69db8a1

We fixed the issue for this in kubeflow/kubeflow#6960, but we should be able to catch errors for the multi-arch build when a PR is opened as well.

The fix should be straightforward. We'll just need to use the same build command in both types of actions. Referencing the relevant parts in one:
https://github.com/kubeflow/kubeflow/blob/master/.github/workflows/centraldb_intergration_test.yaml#L24
https://github.com/kubeflow/kubeflow/blob/master/.github/workflows/centraldb_docker_publish.yaml#L37-L41

[kimwnasptd]

Do you have cycles to help with this effort?

[lehrig]

@kimwnasptd let me confirm with the team but I think we can handle it