Updated deprecated Slurm variable name

docs/guides/mlp_tutorials/llm-nanotron-training.md

@@ -1,13 +1,13 @@
 [](){#ref-mlp-llm-nanotron-tutorial}
 
 # LLM Nanotron Training Tutorial
 
 In this tutorial, we will build a container image to run nanotron training jobs.
 We will train a 109M parameter model with ~100M wikitext tokens as a proof of concept.
 
 ### Prequisites 
 
 It is also recommended to follow the previous tutorials: [LLM Inference][ref-mlp-llm-inference-tutorial] and [LLM Finetuning][ref-mlp-llm-finetuning-tutorial], as this will build up from it.
 
 ### Set up Podman
 
@@ -61,7 +61,7 @@
 
 ### Set up an EDF
 
 See the previous tutorial for context. In this case, the edf will be at `$HOME/.edf/nanotron.toml` and will have the following contents:
 
 ```toml title="$HOME/.edf/nanotron.toml"
 image = "/capstor/scratch/cscs/<USER>/container-image/nanotron/nanotron-v1.0.sqsh"
@@ -81,7 +81,7 @@
 
 ### Preparing a Training Job
 
 Now let's download nanotron.
 In the login node run:
 
 ```console
@@ -89,7 +89,7 @@
 $ cd nanotron
 ```
 
 And with your favorite text editor, create the following nanotron configuration file in `$HOME/nanotron/examples/config_tiny_llama_wikitext.yaml`:
 
 ```yaml title="$HOME/nanotron/examples/config_tiny_llama_wikitext.yaml"
 general:
@@ -192,9 +192,9 @@
   log_level_replica: info
 ```
 
 This configuration file will train, as a proof of concept, a gpt-2-like (109M parameters) llama model with approximately 100M tokens of wikitext with settings `tp=4, dp=2, pp=1` (which means that it requires two nodes to train).
 This training job will require approximately 10 minutes to run.
 Now, create a batchfile in `$HOME/nanotron/run_tiny_llama.sh` with the contents:
 
 ```bash title="$HOME/nanotron/run_tiny_llama.sh"
 #!/bin/bash
@@ -229,7 +229,7 @@
    --master-addr=\${MASTER_ADDR} \
    --master-port=\${MASTER_PORT} \
    --nnodes=\${SLURM_NNODES} \
-   --nproc-per-node=\${SLURM_GPUS_PER_TASK} \
+   --nproc-per-node=\${SLURM_GPUS_ON_NODE} \
   \"
 
   torchrun \${TORCHRUN_ARGS} run_train.py --config-file examples/config_tiny_llama_wikitext.yaml
@@ -239,10 +239,10 @@
 A few comments:
 
 - The parts outside the srun command will be run on the first node of the Slurm allocation for this job. srun commands without further specifiers execute with the settings of the sbatch script (i.e. using all nodes allocated to the job). 
 - If you have a [wandb](https://wandb.ai/) API key and want to synchronize the training run, be sure to set the `WANDB_API_KEY` variable. Otherwise, set `WANDB_MODE=of​f​line` instead.
 - Note that we are setting `HF_HOME` in a directory in scratch. This is done to place the downloaded dataset in scratch, instead of your home directory.
 - The pip install command is only run once in every container (compute node).
 Note that this will only link the nanotron python package to be able to import it in any script irrespective of the current working directory.
 Because all dependencies of nanotron are already installed in the Dockerfile, no extra libraries will be installed at this point.
 If the installation of the package under development creates artefacts on the shared filesystem (such as binaries from compiled C++/CUDA source code), this results in a race condition when run from multiple nodes.
 Therefore, in this case and also when additional external libraries are to be installed, you should either use venv as shown in previous tutorials, or directly build everything in the Dockerfile.