chore: Enforce strict backend version compatibility checks

[sameerdattav]

Fixes #221.

Summary

Adds a strict control-plane version compatibility check for the Trainer
Kubernetes backend. The check is fail-fast, user-friendly, opt-out, and
covered by unit tests.

Approach

• Runs from KubernetesBackend.__init__ after CoreV1Api creation
• Uses namespace from KUBEFLOW_SYSTEM_NAMESPACE (default: kubeflow)
• Reads kubeflow-trainer-public ConfigMap (kubeflow_trainer_api_version)
• Compares against local version from importlib.metadata
• Raises RuntimeError on mismatch or missing control plane
• Skippable via KUBEFLOW_TRAINER_SKIP_VERSION_CHECK

Tests

• Added coverage for success, mismatch, missing ConfigMap/data, and skip path
• Updated existing tests to account for verification during backend init
• pytest passes locally

Thanks @RavinduWeerakoon for the initial approach and discussion.

[google-oss-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign electronic-waste for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[github-actions]

🎉 Welcome to the Kubeflow SDK! 🎉

Thanks for opening your first PR! We're happy to have you as part of our community 🚀

Here's what happens next:

• If you haven't already, please check out our Contributing Guide for repo-specific guidelines and the Kubeflow Contributor Guide for general community standards
• Our team will review your PR soon! cc @kubeflow/kubeflow-sdk-team

Join the community:

• Slack: Join our #kubeflow-ml-experience and #kubeflow-trainer Slack channels
• Meetings: Attend the Kubeflow SDK and ML Experience bi-weekly meetings

Feel free to ask questions in the comments if you need any help or clarification!
Thanks again for contributing to Kubeflow! 🙏

[Copilot]

Pull request overview

This PR adds a strict version compatibility check between the Kubeflow Trainer SDK and the Kubernetes control plane. The check runs automatically when initializing the KubernetesBackend and compares the local kubeflow-trainer-api package version against the version published in a ConfigMap by the control plane. Users can opt out via the KUBEFLOW_TRAINER_SKIP_VERSION_CHECK environment variable.

Changes:

• Added verify_backend() method to KubernetesBackend that performs version compatibility checks
• Added abstract verify_backend() hook to the base RuntimeBackend class
• Comprehensive test coverage for success, mismatch, missing ConfigMap, and skip scenarios
• Updated existing test fixtures to mock the version check behavior

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 7 comments.

File	Description
kubeflow/trainer/backends/kubernetes/backend.py	Implements version verification logic that reads control-plane version from ConfigMap and compares with local SDK version
kubeflow/trainer/backends/base.py	Adds optional verify_backend() hook to the RuntimeBackend abstract class with default no-op implementation
kubeflow/trainer/backends/kubernetes/backend_test.py	Adds comprehensive test coverage for version check scenarios and updates existing fixtures to mock version check dependencies
kubeflow/trainer/api/trainer_client_test.py	Updates test mocks to include ConfigMap and version metadata mocking for backend initialization

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

[andreyvelich]

What is the reason to introduce this env variable?
I would suggest that we always check the version for now.

[andreyvelich]

For now, let's do logger.warning() if version of control plane is incompatible or config map is missing.
We need to release the Trainer v2.2 where we add this ConfigMap.

[andreyvelich]

You don't need this check, since this package is always installed.

[andreyvelich]

I guess, we should do this check:

1. For the minor version it should: server_version == local_version
2. For the patch version it should: server_version >= local_version

Since we don't introduce new features for the patch versions.

We might also want to introduce strict dependency for minor version here
(e.g. kubeflow-trainer-api~=2.1.0): https://github.com/kubeflow/sdk/blob/7797f35dfe8dce1993e968364c17c59a6a422697/pyproject.toml#L31C4-L31C24

What do you think @kubeflow/kubeflow-sdk-team @kubeflow/kubeflow-trainer-team ?

Let's talk about it more at the Kubeflow SDK call: https://docs.google.com/document/d/1jH2WAX2ePxOfI4JuiVK9nPlesDMiyg67xzLwhpR7wTQ/edit?tab=t.0

[andreyvelich]

@sameerdattav We briefly talked about it at the latest call: https://youtu.be/zwPVfB8tTuY?t=2254
Initially, let's just verify that ConfigMap with the appropriate info (e.g. kubeflow_trainer_version) exists.
In the future, we can discuss how we can verify the appropriate version in TrainerClient.

[andreyvelich]

If the server_version == dev, you should skip the check.

[sameerdattav]

@andreyvelich, there was some issue with git and line endings, so i went ahead and opened up a new pr with a clean slate.
this implements the minimal version that you were talking about on the call.
#250