-
Couldn't load subscription status.
- Fork 43
Closed
Description
This issue keeps track of the performance improvements implemented for recipe_extremes_wind_3h.yml
reported in #1915 (comment) as part of the ESiWACE3 service project.
Settings
To run the recipe, the following settings are used:
~/.esmvaltool/config-user.yml
max_parallel_tasks: 1~/.esmvaltool/dask.yml
cluster:
type: dask_jobqueue.SLURMCluster
queue: compute
account: bd0854
cores: 128
memory: 256GiB
processes: 32
interface: ib0
local_directory: /scratch/b/b381141/dask-tmp
n_workers: 32
walltime: '8:00:00'Profiling
A smaller version of the recipe was used for profiling runs. It uses only the wind_fut variable.
Conda environment used for profiling: environment.yml.
The profiles attached below were created with py-spy using the command
py-spy record \
--idle \
--rate 10 \
--subprocesses \
--format speedscope \
esmvaltool run recipe_extremes_wind_3h.ymland can be viewed with https://www.speedscope.app:
- Initial run profile_recipe_extremes_wind_3h.json (40 minutes)
- Faster Faster trivial equality checks for coordinates and arrays SciTools/iris#5691 profile
... mainiris and ESMValCore branch on 2024-09-12 runs including Loadesmvalcore.dataset.Datasetobjects in parallel using Dask #2517 and Save all files in a task at the same time to avoid recomputing intermediate results #2522: single node (8 minutes), 2 nodes (5:20 minutes), 4 nodes (3:20 minutes)
Metadata
Metadata
Assignees
Labels
No labels