Merge branch 'main' into update_policies

Author: lucamar

docs/software/container-engine/known-issue.md

@@ -52,3 +52,16 @@ Mounting individual home directories (usually located on the `/users` filesystem
 
 It is generally NOT recommended to mount home folders inside containers, due to the risk of exposing personal data to programs inside the container.
 Defining a mount related to `/users` in the EDF should only be done when there is a specific reason to do so, and the container image being deployed is trusted.
+
+[](){#ref-ce-why-no-sbatch-env}
+## Why `--environment` as `#SBATCH` is discouraged
+
+The use of `--environment` as `#SBATCH` is known to cause **unexpected behaviors** and is exclusively reserved for highly customized workflows. This is because `--environment` as `#SBATCH` puts the entire SBATCH script in a container from the EDF file. The following are a few known associated issues.
+
+ - **Slurm availability in a container**: Either Slurm components are not completely injected inside a container, or injected Slurm components do not function properly.
+
+ - **Non-host execution context**: Since the SBATCH script runs inside a container, most host resources are inaccessible by default unless EDF explicitly exposes them. Affected resources include: filesystems, devices, system resources, container hooks, etc.
+
+ - **Nested use of `--environment`**: running `srun --environment` in `#SBATCH --environment` results in double-entering EDF containers, causing unexpected errors in the underlying container runtime.
+
+To avoid any unexpected confusion, users are advised **not** to use `--environment` as `#SBATCH`. If users encounter a problem while using this, it's recommended to move `--environment` from `#SBATCH` to each `srun` and see if the problem disappears.

docs/software/container-engine/run.md

@@ -34,17 +34,30 @@ Use `--environment` with the Slurm command (e.g., `srun` or `salloc`):
     #SBATCH --job-name=edf-example
     #SBATCH --time=00:01:00
     ...
-
-    # Run job step
     srun --environment=ubuntu cat /etc/os-release
     ```
 
+Multiple Slurm commands may have different EDF environments; this is useful when a single environment is not feasible due to compatibility issues or keep EDF files modular.
+
+!!! example "`srun`s with different EDFs"
+    ```bash
+    #!/bin/bash
+    #SBATCH --job-name=edf-example
+    #SBATCH --time=00:01:00
+    ...
+    srun --environment=env1 ... # (1)! 
+    ...
+    srun --environment=env2 ... # (2)!
+    ```
+    
+    1. Assuming `env1.toml` is at `EDF_PATH`. See [EDF search path][ref-ce-edf-search-path] below.
+    2. Assuming `env2.toml` is at `EDF_PATH`. See [EDF search path][ref-ce-edf-search-path] below.
+
 Specifying the `--environment` option with an `#SBATCH` option is **experimental**. 
 Such usage is discouraged as it may result in unexpected behaviors.
 
-!!! note
-    Specifying `--environment` with `#SBATCH` will put the entire batch script inside the containerized environment, requiring the Slurm hook to use any Slurm commands within the batch script (e.g., `srun` or `scontrol`). 
-    The hook is controlled by the `ENROOT_SLURM_HOOK` environment variable and activated by default on most vClusters.
+!!! warning 
+    The use of `--environment` as an `#SBATCH` option is reserved for highly customized workflows, and it may result in several **counterintuitive, hard-to-diagnose failures**. See [Why `--environment` as `#SBATCH` is discouraged][ref-ce-why-no-sbatch-env] for details.
 
 [](){#ref-ce-edf-search-path}
 ### EDF search path