[WIP] Add environment management to python package

[chillenzer]

• This is still WIP (at least a major clean-up). Mainly opening the PR to point @PrometheusPi, @steindev and @pordyna to this branch for testing.

Environment management for PIConGPU Python package

Effectively this PR attempts to make our Python package a self-contained installable code. Schematically, the following should work afterwards

pip install picongpu
python my_picmi_script.py

without any need for sourcing environments or the like. See below for a full script that I have run successfully on our local dev system. In practice, we all know the hassles of environment management, so this is on a best-effort basis at best. But it is very configurable, so theoretically it should work for every use case (with enough configuration).

This PR does not yet attempt to automatically install dependencies.

User experience

Interface

The environment management is based on a newly introduced runtime configuration dict-like object called picongpu.rc_params. This approach is strongly inspired by the matplotlib.rc interface. The user can fill this object with configuration parameters that will be used to compose completely self-contained scripts (i.e. including environment setup equivalent to sourcing a profile manually) for running pic-build, etc.

The user has multiple options for providing configuration parameters:

• The code looks for [.]picongpurc.toml files in various places (see matplotlib's precendence rules) to generate the default configuration.
• The user can (in a .picongpurc.toml or at runtime) set preset = ... to any of the available profiles from etc/picongpu which will generate a profile template from the example profile that the user can fill with a few pieces of information (the detailed values allowed are more permissive and user-friendly, see _rc_params.py for details).
• The user can at runtime in their script or live python session manipulate rc_params["my_key"] = my_value before executing a command. For temporarily altering rc_params it provides a context manager

with rc_params.set_temporarily(my_key=my_value):
    ...

Semantics

Effectively (see runner.py for details), the new approach amounts to compiling a script for running commands as

{rc_params.shebang}
{rc_params.preamble}
{rc_params.profile_content}
{commands}

Each of the rc_params.<property> accesses a property that typically returns a value with the following precedence (see _rc_params.py for details):

• The explicitly set value, e.g., rc_params["profile_content"] = "echo 'Hello World'" will return echo 'Hello World' as rc_params.profile_content.
• The content read from rc_params["profile_path"] if that was set.
• A rendering of the mustache template provided via rc_params["profile_template_content"] or profile_template_path.
• A default value.

Those scripts are put into setup_dir/workflow/scripts and they are self-contained, i.e., for debugging (or less-python-based workflows) they can be run directly from your preferred shell without any reference to Python. If you base your further workflow on Python as well, Runner.build() and Runner.run() will simply run them for you.

Setting picongpurc_path or preset at runtime amounts to resetting your rc_params and raises an exception by default if you have already modified rc_params in the same session. You can change this behaviour by

with rc_params.set_temporarily(dirty_reset_policy='ignore'):
    rc_params["preset"] = "bash"

Other values (more interesting for putting into .picongpurc.toml or setting non-temporarily) include "raise" (default), "warn" or a custom handler function.

Implementation details

The presets are generated from a crude parsing of the *.profile.example files. I consider this EXTREMELY BRITTLE! In the medium term we'll surely want a more robust way of handling this. In order to pull this off at all, I have made some minor changes (renamings/reformattings) to the example profiles to achieve a slightly more uniform parsing and interface.

RCParams mimics the major semantics of a dictionary. I originally had it inherit directly but always acting mutably onto itself became a major maintenance burden and I switched to managing an internal resource. That implies that the dict-interface might be incomplete. Just ask if you need more... that's probably easy to add.

I've added the templates and core sub-package. Their only python-side functionality is to provide the path to the template/C++ files for reference in other parts of the code. In addition, upon non-editable installation all relevant C++ files are copied into core and the core.path() just points to itself. In editable installation, nothing is copied and core.path() points to the top-level repo.

As a build backend I've switched to hatchling which I found on the internet to be recommended for non-trivial file arrangements and install hooks. The former is already heavily used, the latter will probably come in handy in the future. Only annoyance seems to be that it treats the sdist (source distribution) and wheel (binary distribution) targets completely independent, so configuration apparently has to be duplicated. But maybe I just overlooked an obvious switch here.

The not-too-coupled semantics of the interface are tested in unit tests. Please refer to them for more details on the interface semantics until we've written proper documentation.

Full scripts that worked for me

Running locally

cd lib/python
echo 'profile_template_path = "/path/to/my/profile_template"' > .picongpurc.toml
pip install -e '.[test]'
pytest -m "not slow"
pytest -m "end_to_end"
pytest -k "laser_wakefield"

with /path/to/my/profile_template containing something like

#!/usr/bin/env zsh
export PIC_PROFILE=$(cd $(dirname ${(%):-%N}) && pwd)"/"$(basename ${(%):-%N})

export PIC_BACKEND="omp2b:native" # running on cpu
export PIC_SYSTEM_TEMPLATE_PATH=${PIC_SYSTEM_TEMPLATE_PATH:-"etc/picongpu/zsh"}

export PICSRC="{{{pic_src_path}}}"

export PIC_EXAMPLES=$PICSRC/share/picongpu/examples
export PATH=$PICSRC/bin:$PATH
export PATH=$PICSRC/src/tools/bin:$PATH

# "tbg" default options #######################################################
export TBG_SUBMIT="zsh"
export TBG_TPLFILE="/home/lenz/profiles/mpirun.tpl"

source /home/lenz/opt/spack/share/spack/setup-env.sh
spack env activate picongpu --with-view=default

Running on HAL

ssh hal

# install picongpu
yes | mamba create -n tmp -c conda-forge python=3.13 && mamba activate tmp
pip install picongpu@git+https://github.com/chillenzer/picongpu@add-env-management-to-python-package#subdirectory=lib/python

# generate runtime config
echo 'preset = "bash-devServer-hzdr/gpu_a30_picongpu"
email = "my@email.de"
author = "My Name"' > .picongpurc.toml

# get something to run
wget https://raw.githubusercontent.com/ComputationalRadiationPhysics/picongpu/refs/heads/dev/lib/python/examples/warm_plasma/main.py -O main.py
sed -i '/write_input_file/d' main.py && echo "    sim.picongpu_run()" >> main.py

# run it
python main.py

This produces the results in a temporary directory, roughly /tmp/<username>/pypicongpu-...-{setup,run}....

Please test this!

For testing, I'd start with running locally. Always install in non-editable mode pip install lib/python in order to make sure you're not relying on your local clone of the repository anymore.

1. Compose a my_picongpu.profile that allows to run the pre-this-PR code, then check out the new code and put profile_path = "my_picongpu.profile" into .picongpurc.toml in your current working directory (or any parent of it, e.g., your home directory). There should be no need to source this profile from now on. This will not have gained you much because your profile still relies on your local copy of the source code.
2. Then, change the export PICSRC=... line in that profile to export PICSRC="{{{pic_src_path}}}" and change your .picongpurc.toml to profile_template_path = "my_picongpu.profile". This should now point the corresponding scripts at the installed code. You should now be independent of your original source location.
3. If your local profile is sufficiently close to our bash profile example, you could try to run this directly via preset = "bash" in your .picongpurc.toml. Otherwise, skip this step.

Afterwards, please go to as many systems as you can and try to run a small test job there. If we've got an example for this system, you can simply start from preset = "<system name>" in your .picongpurc.toml. It will raise an exception telling you about any missing information it needs (typically author = "<name>" and email = "<my email>"). Fill in this information in your .picongpurc.toml and you might be good to go. (If we have multiple *.profile.example files, you might need to be more precise: preset = "bash-devServer-hzdr/gpu_a30_picongpu". You'll figure out the details.)