Setting environment variables in pixi tasks

[ashah03]

Hi! I'd like to be able to set environment variables from a file within pixi tasks, e.g.

[tool.pixi.tasks]
creds = { cmd = "export $(cat env/creds.env | xargs)" }
train = { cmd = "python train.py", depends-on = ["creds"]}

But when I do this, the files are not exported to the shell running train, so I'm having to do

[tool.pixi.tasks]
train = { cmd = "export $(cat env/creds.env | xargs) && python train.py"}

Is there any other workaround? Each of my many tasks will need credentials so don't want to have to copy paste that portion if it can be avoided.

[ruben-arts]

You can add a script to the activation that exports these variables.

activate.sh

export $(cat env/creds.env | xargs)

pyproject.toml

[tool.pixi.activation]
scripts = ["activate.sh"]

The environment isn't persisted over the tasks using the depends-on.

[pormr]

You can add a script to the activation that exports these variables.

activate.sh

export $(cat env/creds.env | xargs)

pyproject.toml

[tool.pixi.activation]
scripts = ["activate.sh"]

The environment isn't persisted over the tasks using the depends-on.

@ruben-arts
As far as I know, if an activation script is attached to a feature such as "default," the script will run whenever the associated environment is activated. However, it is sometimes preferable to modify environment variables only in certain tasks and keep the main environment "clean". For example, when installing additional R packages which is not yet available on Conda, I need to suppress automatic upgrades of existing packages, as well as setting the default repository for CRAN, by configuring environment variables accordingly.

A potential solutions is introducing pseudo-tasks (or meta-tasks), which are not actual tasks but instead define common environment variables (and other behaviors) for a group of tasks.

[tasks.foo-meta]
env = {
   ENV_FOO = "xxxxxxxxxxxx",
   ENV_BAR = "xxxxxxxxxxxx",
   ...
}
# clean-env = true

[tasks.foo]
cmd = "..."
inherits = ["foo-meta"]

[tasks.foo-alt]
cmd = "..."
inherits = ["foo-meta"]
env = { ENV_BAR = "foo" } # Possibly override some environment variables

By introducing inheritance here, we could reduces redundancy and ensure that all child tasks share a consistent set of environment variables or behaviors, with possibility of overrides. This feature could greatly enhance clarity and flexibility within complex workflows.

---

An alternative solution I could think of, without altering the existing behavior of pixi, is to set a flag variable within the tasks definition and check for the flag in the activation script, allowing conditional adjustments to environment variables based on the presence of the flag.

[leriel]

An alternative to permanently set env vars for an environment is if you have conda binary

conda env config vars set --prefix $PWD/.pixi/envs/default/ PYTHONPATH=/tmp/

This places simple file "state" in your pixi/env/conda-meta directory:

cat $PWD/.pixi/envs/default/conda-meta/state
{"env_vars": {"PYTHONPATH": "/tmp/"}}

This file can be placed manually, in the absence of conda binary.

Essentially this is the same solution as when using micromamba

test:

[tasks]
testenv = {cmd="echo $PYTHONPATH"}

pixi run testenv
/tmp/

The benefit here is those env vars are present for both tasks and any env commands

pixi run python
>>> import os
>>> os.environ['PYTHONPATH']
'/tmp/'
>>> 

[pormr]

An update to my previous comment^[1]:
Pixi has already allowing passing arguments to dependent tasks (with optional), so a better practice would be using a template task to handle all similar or related tasks:

[tasks.general-task]
cmd = "base_command {{ input }} --arg1 {{ arg1 }} --arg2 {{ arg2 }}"
env = {
   ENV_FOO = "xxxxxxxxxxxx", 
   ENV_BAR = "xxxxxxxxxxxx",
   ...
}
args = [
  { "arg" = "input" }, # path to input file like raw data or a Python script
  { "arg" = "arg1", "default" = "default1" }, # first argument with default
  { "arg" = "arg2", "default" = "default2" }, # second argument with default
]

[tasks.install-dependencies]
cmd = "echo Installing dependencies." # an NOP command or necessary clean-up procedures (can be left out in the latest version)
depends-on = [
  { "task" = "general-task", "args" = ["input_file1"] }, # with default value
  { "task" = "general-task", "args" = ["input_file2", "foo", "bar"] }, # arguments are positional
  # more tasks to be included here
]

A template task can be referred to multiple times in the dependency list.