Skip to content

Commit d0b3f3d

Browse files
committed
Shorten more lines
1 parent 6161900 commit d0b3f3d

File tree

2 files changed

+24
-18
lines changed

2 files changed

+24
-18
lines changed

_episodes/11-dask-configuration.md

Lines changed: 23 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -112,11 +112,14 @@ cluster:
112112
This tells ESMValCore to start a new cluster of one worker, that can use 2
113113
gigabytes (GiB) of memory and run computations using 2 threads. For a more
114114
extensive description of the available arguments and their values, see
115-
[``distributed.LocalCluster``](https://distributed.dask.org/en/stable/api.html#distributed.LocalCluster).
115+
[``distributed.LocalCluster``](https://distributed.dask.org/
116+
en/stable/api.html#distributed.LocalCluster).
116117
117-
To see this configuration in action, run we will run a version
118-
of [recipe_easy_ipcc.yml](https://docs.esmvaltool.org/en/latest/recipes/recipe_examples.html) with just two datasets. This recipe takes a few minutes to run, once you have the data available. Download
119-
the recipe [here](../files/recipe_easy_ipcc_short.yml) and run it
118+
To see this configuration in action, run we will run a version of
119+
[recipe_easy_ipcc.yml](https://docs.esmvaltool.org/
120+
en/latest/recipes/recipe_examples.html) with just two datasets.
121+
This recipe takes a few minutes to run, once you have the data available.
122+
Download the recipe [here](../files/recipe_easy_ipcc_short.yml) and run it
120123
with the command:
121124
122125
~~~bash
@@ -188,8 +191,9 @@ asked to do.
188191
>> threads_per_worker: 2
189192
>> memory_limit: 4GiB
190193
>>```
191-
>> and run the recipe again with the command ``esmvaltool run recipe_easy_ipcc_short.yml``. The time it took to run the recipe is printed
192-
>> to the screen.
194+
>> and run the recipe again with the command
195+
>> ``esmvaltool run recipe_easy_ipcc_short.yml``.
196+
>> The time it took to run the recipe is printed to the screen.
193197
>>
194198
> {: .solution}
195199
{: .challenge}
@@ -229,18 +233,15 @@ if __name__ == '__main__': # Remove this line when running from a Jupyter noteb
229233
memory_limit='4GiB',
230234
)
231235
cluster.adapt(minimum=0, maximum=2)
232-
233236
# Print connection information
234237
print(f"Connect to the Dask Dashboard by opening {cluster.dashboard_link} in a browser.")
235238
print("Add the following text to ~/.esmvaltool/dask.yml to connect to the cluster:" )
236239
print("client:")
237240
print(f' address: "{cluster.scheduler_address}"')
238-
239241
# When running this as a Python script, the next two lines keep the cluster
240242
# running for an hour.
241243
hour = 3600 # seconds
242244
sleep(1 * hour)
243-
244245
# Stop the cluster when you are done with it.
245246
cluster.close()
246247
```
@@ -338,10 +339,10 @@ cluster:
338339
In this example we use the popular SLURM scheduduler, but other schedulers are also supported, see [this list](https://jobqueue.dask.org/en/latest/api.html).
339340

340341
In the above example, ESMValCore will start 64 Dask workers
341-
(with 128 / 64 = 2 threads each) and for that it will need to launch a single SLURM
342-
batch job on the ``compute`` partition. If you would set ``n_workers`` to e.g.
343-
256, it would launch 4 SLURM batch jobs which would each start 64 workers for a
344-
total of 4 x 64 = 256 workers. In the above configuration, each worker is
342+
(with 128 / 64 = 2 threads each) and for that it will need to launch a single
343+
SLURM batch job on the ``compute`` partition. If you would set ``n_workers`` to
344+
e.g. 256, it would launch 4 SLURM batch jobs which would each start 64 workers
345+
for a total of 4 x 64 = 256 workers. In the above configuration, each worker is
345346
allowed to use 240 GiB per job / 64 workers per job = ~4 GiB per worker.
346347

347348
It is important to read the documentation about your HPC system and answer questions such as
@@ -362,14 +363,19 @@ in order to find the optimal configuration for your situation.
362363
> Answer the questions above and create an ``~/.esmvaltool/dask.yml`` file that
363364
> matches your situation. To benefit from using an HPC system, you will probably
364365
> need to run a larger recipe than the example we have used so far. You could
365-
> try the full version of that recipe (``esmvaltool run examples/recipe_easy_ipcc.yml``) or use your own recipe. To understand performance, you may want
366-
> to experiment with different configurations.
366+
> try the full version of that recipe (
367+
> ``esmvaltool run examples/recipe_easy_ipcc.yml``) or use your own recipe.
368+
> To understand how the different settings affect performance, you may want to
369+
> experiment with different configurations.
367370
>
368371
>> ## Solution
369372
>>
370373
>> The best configuration depends on the HPC system that you are using.
371-
>> Discuss your answer with the instructor and the class if possible. If you are
372-
>> taking this course by yourself, you can have a look at the [Dask configuration examples in the ESMValCore documentation](https://docs.esmvaltool.org/projects/ESMValCore/en/latest/quickstart/configure.html#dask-distributed-configuration).
374+
>> Discuss your answer with the instructor and the class if possible.
375+
>> If you are taking this course by yourself, you can have a look at the
376+
>> [Dask configuration examples in the ESMValCore documentation](
377+
>> https://docs.esmvaltool.org/projects/ESMValCore/en/latest/quickstart/
378+
>> configure.html#dask-distributed-configuration).
373379
>>
374380
> {: .solution}
375381
{: .challenge}

bin/lesson_check.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -556,7 +556,7 @@ def __init__(self, args, filename, metadata, metadata_len, text, lines, doc):
556556
(re.compile(r'README\.md'), CheckNonJekyll),
557557
(re.compile(r'index\.md'), CheckIndex),
558558
(re.compile(r'reference\.md'), CheckReference),
559-
(re.compile(os.path.join('_episodes', '*\.md')), CheckEpisode),
559+
(re.compile(os.path.join('_episodes', r'*\.md')), CheckEpisode),
560560
(re.compile(r'.*\.md'), CheckGeneric)
561561
]
562562

0 commit comments

Comments
 (0)