optionally withhold some cores on each node for system tasks

[garlick]

Problem: in some situations, applications run faster when some number of cores are reserved for system services. In other situations, applications perform best when all the cores are available to the application.

Apparently LSF on sierra will by default only allocate 40 of 44 cores per node to a job. There is an option users can add to their job request to indicate that 42 or 44 be allocated.

[garlick]

Hmm, alternatively, maybe a simpler solution not involving fluxion would be add a way to configure a batch/alloc instance to set aside N cores per node from the R assigned to it when it bootstraps, thus fluxion would be unaware of those reserved cores.

[grondo]

Interesting. The brokers could perhaps then bind themselves to the reserved cores to avoid impacting job tasks run in the instance. If this were supported in the broker config, then it could conceivably be passed to the instance via something like --conf=reserved-cores=...

[garlick]

Moving to flux-core.

[wihobbs]

Possibly related to #4372? (although may be outdated discussion)

[garlick]

Good catch @wihobbs!

[ryanday36]

Adam Bertsch was just asking about this. I was wondering if the things that we were discussing for rediscovering GPU resources (#6418) would be useful for rediscovering the system task cores when users wanted them, but that discussion in #4372 makes it sound like probably not.

[grondo]

It would really depend on how the feature was implemented. In retrospect, enabling a low-noise config on a per-instance basis might be easiest, this could result in binding the broker+modules to a certain set of cores (configurable?) and remove those from the R shared with the scheduler. Admittedly, this is an off the cuff statement and may not handle all the requirements properly (e.g. it doesn't handle the case of binding other system processes to the system reserved cores)

[grondo]

Possibly related #5240

[grondo]

See comment in #5240 for a prototype (found here)

[wihobbs]

@eleon has a few thoughts here shared today with @trws and me. Tagging him so he can share more.

[eleon]

Thanks for the pointer, @wihobbs

The issue I see is that on El Cap systems, Flux is giving users cores that are "reserved" for system tasks (more details below).

My suggestion for working with systems that provide noise mitigation via core specialization is as follows:

1. If users ask for a full node (-x), then Flux provides all of the cores, including system cores.
2. If users ask for a number of cores (-c), then Flux provides the requested number of cores, but draws them from a pool that excludes the system cores.

Right now, when a user requests, say, 8 cores, one of the cores is a system core. This causes an issue for mpibind, because mpibind will only schedule work on user cores (by default). This means that the job will only run on 7 cores, potentially oversubscribing the resources:

leon@tuolumne1071:~$ flux run -N1 -n8 -c1 -o mpibind=off -o cpu-affinity=per-task sh -c 'taskset -pc $$' 
pid 75901's current affinity list: 89,185
pid 75900's current affinity list: 88,184
pid 75902's current affinity list: 90,186
pid 75903's current affinity list: 91,187
pid 75904's current affinity list: 92,188
pid 75905's current affinity list: 93,189
pid 75907's current affinity list: 95,191
pid 75906's current affinity list: 94,190

leon@tuolumne1071:~$ flux run -N1 -n8 -c1 sh -c 'taskset -pc $$' 
pid 75913's current affinity list: 89
pid 75914's current affinity list: 185
pid 75915's current affinity list: 90,186
pid 75916's current affinity list: 91,187
pid 75917's current affinity list: 92,188
pid 75918's current affinity list: 93,189
pid 75919's current affinity list: 94,190
pid 75920's current affinity list: 95,191

[grondo]

This is unfortunately expected behavior because Flux is configured with all available cores. So Flux has no way to know which cores are reserved for the system. The only thing configured to know which cores are system cores is mpibind, which has no way to easily communicate this to Flux.

A better long term solution would be to configure Flux to know which cores are reserved. This could be done now by changing the system config to include only the cores not reserved for the system. E.g. instead of cores = "0-95", use cores = "IDSET OF USER CORES". The drawback here is that a user could not request 96 cores at the system level. However, since the system instance is configured for exclusive node scheduling, perhaps this is not necessarily a showstopper?

The other, perhaps better long term solution would be to have a different way to configure Flux with the reserved cores separately from the main resource config. There was a kind of experimental PR here: #6547, but at this point that's just an idea.

@trws had some more concrete ideas which he can share here.

[grondo]

Posted in a slack discussion, but I'm reposting here in case this workaround is useful while we determine the best path forward.

You can launch a subinstance of Flux which has the reserved cores removed from R by ensuring the mpibind plugin is active for the subinstance job, then using resource.rediscover=true so that the resource config is rediscovered using hwloc. Since mpibind has set the CPU affinity mask before topology discovery occurs, the new Flux instance will only be configured with the available cores.

The best way to do this now is to use a helper script, here start.sh

#!/bin/sh
cat <<EOF >${FLUX_JOB_TMPDIR}/conf.toml
resource.rediscover = true
EOF
flux start -c ${FLUX_JOB_TMPDIR} "$@"

Then launch this script instead of using flux alloc or flux run flux start:

$ flux resource list
     STATE PROPERTIES NNODES NCORES NGPUS NODELIST
      free pall,pdeb+      1     96     4 tuolumne1006
 allocated                 0      0     0 
      down                 0      0     0 
$ flux run -o pty.interactive -n1 -c96 ./start.sh
$ flux resource list
     STATE NNODES NCORES NGPUS NODELIST
      free      1     84     4 tuolumne1006
 allocated      0      0     0 
      down      0      0     0 

Jobs run in this subinstance should only use the user cores not the system cores.