Skip to content

Setting env in CommandFunction overrides PATH env varΒ #1781

@niansong1996

Description

@niansong1996

According to the examples here, the field env can be used to pass in env variables:

By default, the function hides stdout and returns it at the end:

import submitit
function = submitit.helpers.CommandFunction(["which", "python"])  # commands must be provided as a list!
print(function())  # This returns your python path (which you be inside your virtualenv)
# for me: /private/home/jrapin/.conda/envs/dfconda/bin/python
Some useful parameters of the CommandFunction class:

cwd: to choose from which directory the command is run.
env: to provide specific environment variables.
verbose: set to False if you do not want any logging.
As an experimental feature, you can also provide arguments when calling the instance:

However, it seems that adding this field would lead to the execution failure due to the job unable to find the executable.

After some digging, I found that because we directly pass in env for the subprocess.Popen as in here:

with subprocess.Popen(
            full_command,
            stdout=subprocess.PIPE,
            stderr=subprocess.PIPE,
            shell=False,
            cwd=self.cwd,
            env=self.env,
        ) as process:

But according to the docs of Python, this could easily lead to such error as env will override all the env variables, including PATH:

Resolving the path of executable (or the first item of args) is platform dependent. For POSIX, see [os.execvpe()]
(https://docs.python.org/3/library/os.html#os.execvpe), and note that when resolving or searching for the executable path, cwd 
overrides the current working directory and env can override the PATH environment variable.

So I would suggest instead of simply doing evn=self.env, we should dump all the current env vars and only override the ones that presented in env. Or at least adding a warning in the examples to note that this could lead to overridden of other env vars like PATH.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions