[Feature]: Framework-level validation for conflicting environment variables across runtime plugins

[Goku2099]

What you would like to be added?

I would like to add framework-level validation to detect conflicts between environment variables managed by
different runtime plugins, and between plugin-reserved environment variables and user-provided envs in a TrainJob.

This follows the recent addition of validation for MPI-reserved environment variables and addresses a remaining TODO in the framework regarding cross-plugin env conflicts.

Why is this needed?

At the moment, each runtime plugin validates only its own reserved environment variables. This means conflicting or overlapping environment variables can still be configured across plugins or via user input, potentially leading to undefined behavior or hard-to-debug runtime failures.

As more runtime plugins are added, this risk increases.

##goal

Prevent users from configuring environment variables that conflict with environment variables managed by runtime plugins, and surface such issues early through validation with clear error messages.

##approach

• Runtime plugins expose the set of environment variables they reserve or manage.

• During TrainJob validation, detect conflicts between:

  1. plugin-reserved environment variables
  2. user-provided environment variables

• Fail validation early when conflicts are detected.

This is intended as an initial, conservative step without introducing plugin precedence or automatic conflict resolution.

Love this feature?

Give it a 👍 We prioritize the features with most 👍

[github-actions]

🎉 Welcome to the Kubeflow Trainer! 🎉

Thanks for opening your first issue! We're happy to have you as part of our community 🚀

Here's what happens next:

• Our team will review your issue soon! cc @kubeflow/kubeflow-trainer-team
• If you'd like to contribute to this issue, check out our Contributing Guide for repo-specific guidelines and the Kubeflow Contributor Guide for general community standards.

Join the community:

• Slack: Join our #kubeflow-trainer Slack channel.
• Meetings: Attend the Kubeflow AutoML and Training Working Group bi-weekly meetings.

Feel free to ask questions in the comments if you need any help or clarification!
Thanks again for contributing to Kubeflow! 🙏

[Goku2099]

@andreyevlich @akshaychitneni, I’d appreciate your feedback on the proposed direction for this issue. Happy to help with the implementation once there’s alignment on scope and approach.

[akshaychitneni]

@Goku2099 Could you share an example with conflict and how your approach solves?

[Goku2099]

@akshaychitneni Thanks for the question.

Consider a TrainJob that enables multiple runtime plugins (for example, MPI and Torch), or that combines plugin-managed environment variables with user-provided envs.

For instance, the MPI plugin reserves environment variables such as OMPI_MCA_orte_default_hostfile. A user can still manually set this variable under spec.trainer.env. Today, this configuration would be accepted unless the specific plugin explicitly validates it, and the resulting conflict would only become visible at runtime.

With the proposed approach, each plugin exposes the set of environment variables it reserves, and the framework validates that:

-" user-provided envs do not overlap with any plugin-reserved envs"
-"reserved env sets across enabled plugins do not overlap"

In this case, validation would fail early with a clear error indicating the conflicting env name, rather than allowing a misconfigured job to proceed and fail at runtime.

This keeps the initial scope limited to detection only, without introducing precedence rules or automatic conflict resolution.