Skip to content

Workload Assignment

Jump to bottom
Andrew Grant edited this page Nov 1, 2023 · 11 revisions

When a Client requests a workload from the Server, the Server will filter out any tests which the Client is not qualified to complete. From there it will filter out to only include the highest priority of those qualified-workloads. Lastly, from there, a workload will be selected based on the distribution of other works on other tests, to maintain a balanced distribution of workers.

Determining Qualified Workloads

  1. Remove any workloads which contain a dev or base engine that is not supported by the Client. This can occur if the Client does not have a sufficiently new compiler to build the engine; does not have the requested CPU instruction sets; does not have an Operating System that is compatible with the engine; does not have a Fine-Grained Access Token for a private engine.

  2. Remove any workloads with unmet Syzygy requirements. A workload can request up to 7-man Syzygy tablebases for both Adjudication and for WDL during play. Tests which specifically request certain sized Syzygy tables are generally uncommon, and should be used carefully to account for different hardware configurations.

  3. Remove tests that require more threads than we have connected. For a test where both engines have the same number of threads, this means removing any test where our machine does not have the capacity to run at least single concurrent game. For smp-odds tests, we will artificially cut the number of threads of the Client in half if they are clearly dipping into Hyperthreads, to make that determination. An an example, imagine a 4-core/8-thread CPU. It is capable of playing a test that is 4 threads vs 4 threads with concurrency=2, and also a test that is 8 threads vs 8 threads with concurrency=1. It is also capable of playing a test that 4 threads vs 1 threads with concurrency=1, as we will refuse to use Hyperthreads for smp-odds. Lastly, the worker would refuse a test that is 8 threads vs 1 threads. This is because one engine would be using hyperthreads, but the other would be getting full cores.

  4. Remove tests where we would exceed the Thread Limit or Worker Limit by joining the test. These limits are optionally imposed on tests, to control the number of workers and threads that are allowed to be on that at one time. Generally this is not used.

Selecting from Qualified Workloads

  1. We will compute the distribution of workers, relative to the throughput of each test. This is done by diving the total number of threads assigned to each workload by the throughput of the workload. We call this the ratio. We compute the fair-ratio, which would be the ratio if all workers had been distributed evenly with respect to their thread count, and each tests' throughput.

  2. If the worker's most recent workload is in the list of qualified ones, then it will be repeated, so long as no workload is getting less than 75% of the fair-ratio. Otherwise, the worker will be assigned the workload with the lowest ratio. If multiple workloads share the same lowest ratio, then one will be selected at random.

Concurrency Settings

The server provides three critical values in the workload JSON response. These are cutechess-count, concurrency-per, and games-per-cutechess. These values are a function of the number of threads and sockets, as well as the nature of the test. They are explained below.

  1. cutechess-count indicates the number of cutechess copies that should be running at one time. For a typical workload, where each engine is playing with one thread, cutechess-count will be equal to the number of sockets on the worker, as provided via --nsockets or -N when starting the Client. If the workload is an SPSA tune, using the MULTIPLE method of distributing SPSA-points, then cutechess-count will be the maximum number of concurrent games divided by two. Finally, if the previous condition is not true, and the workload uses more than 1 thread for either engine, then cutechess-count will be set to 1.

Clone this wiki locally