chore(deps): Bump Go 1.25, k8s v1.35, and controller-runtime v0.23.1 by andreyvelich · Pull Request #3127 · kubeflow/trainer

andreyvelich · 2026-01-26T21:27:43Z

Updating Go to 1.25, k8s to v1.35, and controller-runtime to v0.23.1

Also, updated the validation webhook due to this breaking change: kubernetes-sigs/controller-runtime#3360

/assign @astefanutti @tenzen-y @akshaychitneni @robert-bell

This is needed for JobSet v0.11.0 upgrade
cc @kannon92

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

google-oss-prow · 2026-01-26T21:27:47Z

@andreyvelich: GitHub didn't allow me to assign the following users: robert-bell.

Note that only kubeflow members with read permissions, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time.
For more information please see the contributor guide

Details

In response to this:

Updating Go to 1.25, k8s to v1.35, and controller-runtime to v0.23.1

Also, updated the validation webhook due to this breaking change: kubernetes-sigs/controller-runtime#3360

/assign @astefanutti @tenzen-y @akshaychitneni @robert-bell

This is needed for JobSet v0.11.0 upgrade
cc @kannon92

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

andreyvelich · 2026-01-26T21:28:52Z

Fixes: #3104

Copilot

Pull request overview

This PR upgrades the project to Go 1.25, Kubernetes 1.35, and controller-runtime v0.23.1, and regenerates API clients/CRDs/OpenAPI artifacts to align with the new Kubernetes APIs and controller-runtime webhook changes.

Changes:

Bump Go toolchain version and key Kubernetes/controller-runtime dependencies (including kube-openapi, utils, structured-merge-diff, json, cobra, pflag, etc.).
Update admission webhooks to use the new typed validator interfaces (e.g., TrainJobValidator, TrainingRuntimeValidator, ClusterTrainingRuntimeValidator) and adjust webhook setup wiring accordingly.
Regenerate Go clientsets/informers/applyconfigurations, Python OpenAPI models, and CRDs/Swagger to match Kubernetes 1.35 schema updates (new fields, updated descriptions, toleration operators, Job/Pod/PVC semantics, etc.).

Reviewed changes

Copilot reviewed 66 out of 68 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
go.mod / go.sum	Bump Go version to 1.25.0, upgrade Kubernetes libraries to v0.35.0, controller-runtime to v0.23.1, and refresh related dependency versions.
pkg/webhooks/trainjob_webhook.go	Replace `TrainJobWebhook` with typed `TrainJobValidator`, adapt webhook manager construction to the new controller-runtime API, and keep TrainJob validation logic wired through the runtime registry.
pkg/webhooks/trainingruntime_webhook.go	Convert to `TrainingRuntimeValidator` with typed signatures and new `ctrl.NewWebhookManagedBy(mgr, &trainer.TrainingRuntime{})` usage; validation still delegates to `validateReplicatedJobs`.
pkg/webhooks/clustertrainingruntime_webhook.go	Convert to `ClusterTrainingRuntimeValidator` with typed signatures and updated webhook setup; maintains deprecation warning logic and reused `validateReplicatedJobs`.
pkg/webhooks/setup.go	Update webhook setup wiring so only `TrainJobValidator` receives the runtime registry; cluster and namespaced TrainingRuntime webhooks no longer depend on it.
pkg/webhooks/trainjob_webhook_test.go	Extend `TestValidateCreate` for TrainJob to cover unsupported and deprecated runtimes, and switch to the new `TrainJobValidator` type.
pkg/webhooks/clustertrainingruntime_webhook_test.go	Update tests to construct `ClusterTrainingRuntimeValidator` directly.
pkg/client/clientset/versioned/fake/clientset_generated.go	Regenerated fake clientset, including a marker for WatchList semantics support and watch reactor updates (auto-generated).
pkg/client/informers/externalversions/**	Swap raw `&cache.ListWatch{...}` with `cache.ToListWatcherWithWatchListSemantics` to align with new Kubernetes reflector semantics (auto-generated).
pkg/client/applyconfiguration/trainer/v1alpha1/**	Regenerated apply-configuration types with richer field-level documentation for Trainer/TrainJob/TrainingRuntime and related types (auto-generated).
manifests/base/crds/ & charts/kubeflow-trainer/crds/	Regenerated CRDs with Kubernetes 1.35 schema updates (toleration operators, volume resize semantics, PodCertificate userAnnotations, workloadRef, etc.).
api/python_api/kubeflow_trainer_api/models/**	Regenerated Python client models and added new types (e.g., WorkloadReference, InternalEvent, JobSet volume policies), plus docstring updates to track the new OpenAPI schema.
api/openapi-spec/swagger.json	Regenerated swagger spec to reflect the updated Kubernetes and trainer APIs, including new fields and updated descriptions.
CONTRIBUTING.md	Update documented minimum Go version requirement from 1.24 to 1.25.
.golangci.yaml / .golangci-kal.yml	Drop explicit Go version override and tweak a kal rule list format to remain valid YAML with the newer toolchain.

Copilot · 2026-01-26T21:31:20Z

pkg/webhooks/trainjob_webhook.go

+	runtimeRefGK := runtime.RuntimeRefToRuntimeRegistryKey(obj.Spec.RuntimeRef)
 	runtime, ok := w.runtimes[runtimeRefGK]
 	if !ok {
 		return nil, fmt.Errorf("unsupported runtime: %s", runtimeRefGK)
 	}
-	warnings, errors := runtime.ValidateObjects(ctx, nil, trainJob)
+	warnings, errors := runtime.ValidateObjects(ctx, nil, obj)


ValidateCreate returns a plain fmt.Errorf when the referenced runtime is not found, but the associated test (TestValidateCreate's "unsupported runtime" case) expects an aggregated field.ErrorList marking spec.RuntimeRef as invalid, so this implementation both breaks that test and surfaces a less precise validation error to API consumers.

Copilot · 2026-01-26T21:31:20Z

pkg/webhooks/trainjob_webhook.go

+	runtimeRefGK := runtime.RuntimeRefToRuntimeRegistryKey(newObj.Spec.RuntimeRef)
 	runtime, ok := w.runtimes[runtimeRefGK]
 	if !ok {
 		return nil, fmt.Errorf("unsupported runtime: %s", runtimeRefGK)
 	}
-	warnings, errors := runtime.ValidateObjects(ctx, oldTrainJob, newTrainJob)
+	warnings, errors := runtime.ValidateObjects(ctx, oldObj, newObj)


ValidateUpdate mirrors ValidateCreate by returning a plain fmt.Errorf for unsupported runtimes, but for consistency with TestValidateCreate and to provide field-scoped admission errors it should also return an aggregated field.ErrorList that points at spec.RuntimeRef instead of a generic error.

Copilot · 2026-01-26T21:31:20Z

api/python_api/kubeflow_trainer_api/models/io_k8s_api_core_v1_workload_reference.py

+
+from __future__ import annotations
+import pprint
+import re  # noqa: F401


Import of 're' is not used.

Suggested change

import re # noqa: F401

Copilot · 2026-01-26T21:31:21Z

...ython_api/kubeflow_trainer_api/models/io_k8s_apimachinery_pkg_apis_meta_v1_internal_event.py

+
+from __future__ import annotations
+import pprint
+import re  # noqa: F401


Import of 're' is not used.

Suggested change

import re # noqa: F401

Copilot · 2026-01-26T21:31:21Z

api/python_api/kubeflow_trainer_api/models/jobset_v1alpha2_volume_claim_policy.py

+
+from __future__ import annotations
+import pprint
+import re  # noqa: F401


Import of 're' is not used.

Suggested change

import re # noqa: F401

Copilot · 2026-01-26T21:31:21Z

api/python_api/kubeflow_trainer_api/models/jobset_v1alpha2_volume_retention_policy.py

+
+from __future__ import annotations
+import pprint
+import re  # noqa: F401


Import of 're' is not used.

Suggested change

import re # noqa: F401

coveralls · 2026-01-26T21:32:39Z

Pull Request Test Coverage Report for Build 21405922660

Details

11 of 45 (24.44%) changed or added relevant lines in 8 files are covered.
2 unchanged lines in 2 files lost coverage.
Overall coverage increased (+0.09%) to 51.217%

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
pkg/controller/trainjob_controller.go	0	2	0.0%
pkg/webhooks/setup.go	0	2	0.0%
pkg/controller/setup.go	0	3	0.0%
pkg/webhooks/trainjob_webhook.go	4	12	33.33%
pkg/webhooks/clustertrainingruntime_webhook.go	5	14	35.71%
pkg/webhooks/trainingruntime_webhook.go	0	10	0.0%

Files with Coverage Reduction	New Missed Lines	%
pkg/webhooks/clustertrainingruntime_webhook.go	1	48.0%
pkg/webhooks/trainingruntime_webhook.go	1	62.79%

Totals
Change from base Build 21369554754:	0.09%
Covered Lines:	1241
Relevant Lines:	2423

💛 - Coveralls

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

andreyvelich · 2026-01-26T23:01:13Z

pkg/controller/setup.go

 	runtimeRec := NewTrainingRuntimeReconciler(
 		mgr.GetClient(),
-		mgr.GetEventRecorderFor("trainer-trainingruntime-controller"),
+		mgr.GetEventRecorder("trainer-trainingruntime-controller"),


I also migrated to the new Events API due to this: kubernetes-sigs/controller-runtime#3262
Let me know if that looks good @tenzen-y @astefanutti.

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

tenzen-y · 2026-01-27T14:45:44Z

.golangci.yaml

 version: "2"

 run:
-  go: "1.24"


why do we need to remove this?

We don't have it in Kueue: https://github.com/kubernetes-sigs/kueue/blob/main/.golangci.yaml
I was thinking that it is better to maintain Go version in a go.mod file only.
WDYT @tenzen-y?

I have no preference here. I'm asking if you faced any problems or not.

It looks to be working fine, so I prefer to remove it from .golangci.yaml

It defaults to use Go version from the go.mod file so it's probably better to remove it.

tenzen-y · 2026-01-27T14:50:53Z

pkg/webhooks/trainingruntime_webhook.go


-// +kubebuilder:webhook:path=/validate-trainer-kubeflow-org-v1alpha1-trainingruntime,mutating=false,failurePolicy=fail,sideEffects=None,groups=trainer.kubeflow.org,resources=trainingruntimes,verbs=create;update,versions=v1alpha1,name=validator.trainingruntime.trainer.kubeflow.org,admissionReviewVersions=v1
-
-var _ webhook.CustomValidator = (*TrainingRuntimeWebhook)(nil)


It would be better to keep a type compliance check.

Sure, added it back.

tenzen-y · 2026-01-27T14:51:26Z

pkg/webhooks/clustertrainingruntime_webhook.go


-// +kubebuilder:webhook:path=/validate-trainer-kubeflow-org-v1alpha1-clustertrainingruntime,mutating=false,failurePolicy=fail,sideEffects=None,groups=trainer.kubeflow.org,resources=clustertrainingruntimes,verbs=create;update,versions=v1alpha1,name=validator.clustertrainingruntime.trainer.kubeflow.org,admissionReviewVersions=v1
-
-var _ webhook.CustomValidator = (*ClusterTrainingRuntimeWebhook)(nil)


Ditto about the type compliance check.

tenzen-y · 2026-01-27T14:51:48Z

pkg/webhooks/trainjob_webhook.go


-// +kubebuilder:webhook:path=/validate-trainer-kubeflow-org-v1alpha1-trainjob,mutating=false,failurePolicy=fail,sideEffects=None,groups=trainer.kubeflow.org,resources=trainjobs,verbs=create;update,versions=v1alpha1,name=validator.trainjob.trainer.kubeflow.org,admissionReviewVersions=v1
-
-var _ webhook.CustomValidator = (*TrainJobWebhook)(nil)


Ditto about the type compliance check.

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

robert-bell

Thanks @andreyvelich.

I’ve gone through the changes and tested it locally and everything looks solid on my end.

I had just one small comment, but lgtm otherwise.

robert-bell · 2026-01-27T15:21:28Z

pkg/controller/trainjob_controller.go

 				message = fmt.Sprintf("%s ...", message)
 			}
-			r.recorder.Event(&trainJob, corev1.EventTypeWarning, "TrainJobResourcesCreationFailed", message)
+			r.recorder.Eventf(&trainJob, nil, corev1.EventTypeWarning, "TrainJobResourcesCreationFailed", "TrainJobResourcesCreationFailed", message)


nit: should we use a different value for the action argument rather than duplicating the reason?

Maybe?

Suggested change

r.recorder.Eventf(&trainJob, nil, corev1.EventTypeWarning, "TrainJobResourcesCreationFailed", "TrainJobResourcesCreationFailed", message)

r.recorder.Eventf(&trainJob, nil, corev1.EventTypeWarning, "TrainJobResourcesCreationFailed", "Reconciling", message)

action is what action was taken/failed regarding to the regarding object. It is machine-readable. This field cannot
be empty for new Events and it can have at most 128 characters.

I think, Reconciling should be fine, since we didn't put TrainJob to the Failed state.
Thoughts @tenzen-y @astefanutti ?

Yes that sounds reasonable to me.

Updated, let me know if that looks good @astefanutti @tenzen-y @robert-bell

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

astefanutti · 2026-01-27T19:19:50Z

Thanks @andreyvelich!

/lgtm

andreyvelich

/approve

google-oss-prow · 2026-01-27T19:50:22Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: andreyvelich

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [andreyvelich]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

chore(deps): Bump Go 1.25, k8s to v1.35, and controller-runtime v0.23.1

75b7f34

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

Copilot AI review requested due to automatic review settings January 26, 2026 21:27

google-oss-prow bot assigned akshaychitneni, astefanutti and tenzen-y Jan 26, 2026

google-oss-prow bot requested review from jinchihe and kuizhiqing January 26, 2026 21:27

google-oss-prow bot added the size/XL label Jan 26, 2026

Copilot started reviewing on behalf of andreyvelich January 26, 2026 21:28 View session

andreyvelich changed the title ~~chore(deps): Bump Go 1.25, k8s to v1.35, and controller-runtime v0.23.1~~ chore(deps): Bump Go 1.25, k8s v1.35, and controller-runtime v0.23.1 Jan 26, 2026

andreyvelich added the dependencies Pull requests that update a dependency file label Jan 26, 2026

Copilot AI reviewed Jan 26, 2026

View reviewed changes

Migrate to new events API with EventRecorder

11796ac

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

andreyvelich commented Jan 26, 2026

View reviewed changes

Fix RBAC for events

4d314e8

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

andreyvelich force-pushed the bump-go-1.25 branch from 9ee9da5 to 4d314e8 Compare January 27, 2026 00:18

Bump k8s version in tests

cb1e102

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

andreyvelich force-pushed the bump-go-1.25 branch from 1eef167 to cb1e102 Compare January 27, 2026 14:45

tenzen-y reviewed Jan 27, 2026

View reviewed changes

Add type compliance check

05995d3

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

robert-bell reviewed Jan 27, 2026

View reviewed changes

Use Reconciling action for event

d8389db

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

google-oss-prow bot added the lgtm label Jan 27, 2026

andreyvelich commented Jan 27, 2026

View reviewed changes

google-oss-prow bot added the approved label Jan 27, 2026

google-oss-prow bot merged commit 5568ea2 into kubeflow:master Jan 27, 2026
31 checks passed

google-oss-prow bot added this to the v2.2 milestone Jan 27, 2026

andreyvelich deleted the bump-go-1.25 branch January 27, 2026 19:55

andreyvelich mentioned this pull request Feb 2, 2026

Bump the kubernetes group across 1 directory with 2 updates kubernetes-sigs/jobset#1145

Open


		// +kubebuilder:webhook:path=/validate-trainer-kubeflow-org-v1alpha1-trainingruntime,mutating=false,failurePolicy=fail,sideEffects=None,groups=trainer.kubeflow.org,resources=trainingruntimes,verbs=create;update,versions=v1alpha1,name=validator.trainingruntime.trainer.kubeflow.org,admissionReviewVersions=v1

		var _ webhook.CustomValidator = (*TrainingRuntimeWebhook)(nil)


		// +kubebuilder:webhook:path=/validate-trainer-kubeflow-org-v1alpha1-clustertrainingruntime,mutating=false,failurePolicy=fail,sideEffects=None,groups=trainer.kubeflow.org,resources=clustertrainingruntimes,verbs=create;update,versions=v1alpha1,name=validator.clustertrainingruntime.trainer.kubeflow.org,admissionReviewVersions=v1

		var _ webhook.CustomValidator = (*ClusterTrainingRuntimeWebhook)(nil)


		// +kubebuilder:webhook:path=/validate-trainer-kubeflow-org-v1alpha1-trainjob,mutating=false,failurePolicy=fail,sideEffects=None,groups=trainer.kubeflow.org,resources=trainjobs,verbs=create;update,versions=v1alpha1,name=validator.trainjob.trainer.kubeflow.org,admissionReviewVersions=v1

		var _ webhook.CustomValidator = (*TrainJobWebhook)(nil)

	r.recorder.Eventf(&trainJob, nil, corev1.EventTypeWarning, "TrainJobResourcesCreationFailed", "TrainJobResourcesCreationFailed", message)
	r.recorder.Eventf(&trainJob, nil, corev1.EventTypeWarning, "TrainJobResourcesCreationFailed", "Reconciling", message)

Conversation

andreyvelich commented Jan 26, 2026

Uh oh!

google-oss-prow bot commented Jan 26, 2026

Uh oh!

andreyvelich commented Jan 26, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Jan 26, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 26, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 26, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 26, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 26, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 26, 2026

Choose a reason for hiding this comment

Uh oh!

coveralls commented Jan 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Test Coverage Report for Build 21405922660

Details

💛 - Coveralls

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

robert-bell left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

astefanutti commented Jan 27, 2026

Uh oh!

andreyvelich left a comment

Choose a reason for hiding this comment

Uh oh!

google-oss-prow bot commented Jan 27, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

coveralls commented Jan 26, 2026 •

edited

Loading