Potential Race condition in LocalJob Executions

[RavinduWeerakoon]

In the LocalJob class, which inherits from threading.Thread, the run() method uses os.chdir() to change the working directory before launching a background process, and then changes it back in a finally block

 os.chdir(self.execution_dir)

  self._process = subprocess.Popen(
      self.command,
      stdout=subprocess.PIPE,
      stderr=subprocess.STDOUT,
      text=True,
      encoding="utf-8",
      bufsize=1,
      env=self.env,
  )

https://github.com/kubeflow/sdk/blob/57ac0138eefa2cac18729fa9da611bfbd8fc00ed/kubeflow/trainer/backends/localprocess/job.py#L76C1-L87C14

If multiple LocalJob threads run concurrently, they will mix up each other's working directories, leading to chaotic path-resolution failures.

So instead of changing the path using the os.chdir(), can't we provide the path as an argument for the subprocess creation time? since this prevents changing the main process's path.

 self._process = subprocess.Popen(
                        self.command,
                        cwd=self.execution_dir, 
                        ......
)

[github-actions]

🎉 Welcome to the Kubeflow SDK! 🎉

Thanks for opening your first issue! We're happy to have you as part of our community 🚀

Here's what happens next:

• Our team will review your issue soon! cc @kubeflow/kubeflow-sdk-team
• If you'd like to contribute to this issue, check out our Contributing Guide for repo-specific guidelines and the Kubeflow Contributor Guide for general community standards

Join the community:

• Slack: Join our #kubeflow-ml-experience and #kubeflow-trainer Slack channels
• Meetings: Attend the Kubeflow SDK and ML Experience bi-weekly meetings

Feel free to ask questions in the comments if you need any help or clarification!
Thanks again for contributing to Kubeflow! 🙏

[MansiSingh17]

I'd like to work on this. The fix would be to pass cwd=self.execution_dir directly to subprocess.Popen() and remove the os.chdir() call and its finally cleanup — this scopes the directory entirely to the subprocess rather than the thread. I'll also add a concurrent execution test to verify the fix. Will open a PR shortly.