Improve JUWELS jobqueue config for safe operation on diskless nodes

[kathoef]

For example, the memory request is very "optimistic". Here, rather choose the reported "available" value:

$ srun --account=??? --time=00:05:00 --partition=??? free -h
srun: job 2730378 queued and waiting for resources
srun: job 2730378 has been allocated resources
              total        used        free      shared  buff/cache   available
Mem:           92Gi       5.9Gi        83Gi       2.7Gi       3.1Gi        83Gi
Swap:            0B          0B          0B

Also, with the current way of importing the Dask jobqueue configuration there is no way of e.g. injecting Dask distributed options via configuration files, e.g. disabling worker disk spilling. This would be important for a smooth operation of Dask jobqueue workers on the diskless nodes, though.

[kathoef]

And/or set local_directory to $SCRATCH.

[kathoef]

Also useful,

$ srun --account=??? --time=00:05:00 --partition=??? top -b -n 1 | head -n 11
srun: job 2730534 queued and waiting for resources
srun: job 2730534 has been allocated resources
top - 14:36:34 up 1 day, 23:26,  0 users,  load average: 0.15, 0.13, 0.80
Tasks: 901 total,   1 running, 900 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.3 us,  0.4 sy,  0.0 ni, 99.2 id,  0.0 wa,  0.0 hi,  0.1 si,  0.0 st
MiB Mem :  94806.0 total,  85295.9 free,   6110.7 used,   3399.4 buff/cache
MiB Swap:      0.0 total,      0.0 free,      0.0 used.  85050.6 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
12709 hoeflic+  20   0   14328   5900   1704 S  47.1   0.0   0:00.11 psilogger
 2644 root       0 -20   50.8g   2.7g 329492 S   5.9   3.0 170:25.20 mmfsd
12718 hoeflic+  20   0  273044   5136   3664 R   5.9   0.0   0:00.02 top
    1 root      20   0  242376  11384   8656 S   0.0   0.0   0:12.29 systemd