Parsl-Vine: Serial Task Latency, Round 2

[dthain]

@colinthomas-z80 fixed some corner cases in the Parsl-TaskVine executor in #3432 that improved the sequential-chained task latency from about 1 task/s to about 6 tasks/s. However, we should be able to do better than this: @gpauloski observed similar performance in #3892

While this isn't the ideal use case for TaskVine (it would be better to submit many tasks at once) it is a useful benchmark that stresses the critical path through the executor module. Let's figure out what the bottleneck is, and how we can do better.

[benclifford]

Using parsl-perf, the parsl "generate a load of tasks" performance tool, gives me these results. This is all in the parsl github repo so you should be able to run these commands yourself and generate your own equivalent or contradictory results.

It's submitting a bunch of tasks, not sequential tasks - so this issue may be mistitled.

Baseline (parsl with in-process execution - no fancy execution tech at all):

$ parsl-perf --config parsl/tests/configs/local_threads.py --time 30
...
Tasks per second: 2001.290

Using parsl's HighPerformanceExecutor:

$ parsl-perf --config parsl/tests/configs/htex_local.py --time 30
Tasks per second: 1038.789

Using Work Queue + coprocesses:

This one slows down quite dramatically as you increase the target time (aka increase the number of tasks at once) so I think there's some linear-access-cost data structure somewhere...

$ parsl-perf --config parsl/tests/configs/workqueue_ex.py --time 30
Tasks per second: 340.689

So all of the above are in a pretty decent 300-2000 tasks per second range. Probably we don't need to quibble too much about the exact numbers there because even things like changing Python logging configuration can change things drastically at this task rate.

Using Task Vine (in regular mode, but performance is only slightly better with serverless mode enabled):

$ parsl-perf --config parsl/tests/configs/taskvine_ex.py --time 30
...
==== Iteration 2 ====
Will run 50 tasks to target 30.0 seconds runtime
Submitting tasks / invoking apps
All 50 tasks submitted ... waiting for completion
Submission took 0.010 seconds = 4813.073 tasks/second
Runtime: actual 25.673s vs target 30.0s
Tasks per second: 1.948

With different target durations (and so different numbers of tasks submitted) this is pretty consistent around 2 tasks/second - I tried with a 10 minute target runtime and the resulting 1043 tasks execute/complete at 1.7 tasks per second.

That ~2, much lower than the 300-2000 tasks/second of the other executors, is what concerns me.

I have not dug into what this looks like any deeper than this - just came prominently to my attention when talking to @gpauloski about what become #3892. It's probably worth me spending an hour or two on this from a parsl perspective though.

[dthain]

Yikes, something is way off there.
Does that last command generate TV manager/worker logs that we can see?

[benclifford]

attached is the parsl rundir generated by $ parsl-perf --config parsl/tests/configs/taskvine_ex.py --time 30

using master branch cctools
4b43490

$ vine_worker --version
vine_worker version 8.0.0 DEVELOPMENT (released 2024-07-31 17:16:38 +0000)
	Built by benc@parsl-dev-3-12-3510 on 2024-07-31 17:16:38 +0000
	System: Linux parsl-dev-3-12-3510 6.5.0-44-generic #44~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Tue Jun 18 14:36:16 UTC 2 x86_64 GNU/Linux
	Configuration: --prefix=/home/benc/parsl/src/cctools/I

task vine logs and other parsl stuff is in there.

cc-issue-3894.zip

[dthain]

Ok, so here is what I can see from a quick look at the manager log:

2024/07/31 17:24:34.67 vine_manager[240872] vine: parsl-dev-3-12-3510 (127.0.0.1:59416) busy on 'python3 exec_parsl_function.py map function argument result'
2024/07/31 17:24:34.67 vine_manager[240872] vine: Task 9 state change: READY (1) to RUNNING (2)
2024/07/31 17:24:34.67 vine_manager[240872] vine: rx from parsl-dev-3-12-3510 (127.0.0.1:59416): info tasks_running 1
2024/07/31 17:24:35.19 vine_manager[240872] vine: rx from parsl-dev-3-12-3510 (127.0.0.1:59416): cache-update file-rnd-pdfqcnirzsqkkoq 1 0 8 33188 510444 1722446674673169 X

• Most of the manager-worker interactions are very fast. All the task/file interactions are streamed in less than .01s, which is good and expected.

• The worker is reporting four cores available, and could run four tasks at once. However, each task is also consuming four cores -- this would be the conservative default if no other description was provided. I think you want the benchmark to describe each task as consuming one core and a small amount of memory, so that you get parity with the other executors. This alone should give you a 4x increase in throughput.

• At the worker, the time to actually execute each task (from "busy on python3" to "cache-update") is pretty consistently 0.25-0.5s through the run. That seems awfully high, and it seems likely the task itself has some kind of crazy startup overhead even for Python. (?)

@colinthomas-z80 I'll let you take it from here, can you set up the python benchmark and see what we get?

[benclifford]

telling tasks to use a single core does speed things up by 2.5x-ish

$ parsl-perf --config parsl/tests/configs/taskvine_ex.py --time 30 --resources '{"cores":1, "memory":1, "disk":1}'
...
Tasks per second: 4.989

To avoid Python startup costs, I guess I should be pushing on serverless mode, in the same way that we prefer coprocess mode with WQ... that worked earlier today (with slightly improved performance but still around the 2 tasks/second rate) when I was using a build from April but now seems to be failing with a cctools build from today. I'll debug that a little bit and write more here.

Serverless mode isn't tested in our CI - although I would like it to be.

[benclifford]

ok, looked like I needed to make clean harder...

Here's a config for serverless mode: (not in our git repo, that's why I'm pasting it here)

$ cat parsl/tests/configs/taskvine_serverless.py 
from parsl.config import Config
from parsl.data_provider.file_noop import NoOpFileStaging
from parsl.data_provider.ftp import FTPInTaskStaging
from parsl.data_provider.http import HTTPInTaskStaging
from parsl.executors.taskvine import TaskVineExecutor, TaskVineManagerConfig


def fresh_config():
    return Config(executors=[TaskVineExecutor(manager_config=TaskVineManagerConfig(port=9000, shared_fs=True),
                                              worker_launch_method='factory', function_exec_mode='serverless')])

and parsl-perf runs like this:

$ parsl-perf --config parsl/tests/configs/taskvine_serverless.py --time 30 --resources '{"cores":1, "memory":1, "disk":1}'
...
==== Iteration 2 ====
Will run 66 tasks to target 30.0 seconds runtime
...
Tasks per second: 2.430

[dthain]

Yep something stinks there, thank you for pointing it out.
Will let Colin dig into what's going on...

[colinthomas-z80]

I get proportionally the same result on a 16 core machine.

parsl-perf --config parsl/tests/configs/taskvine_ex.py --time 30 --resources '{"cores":1, "memory":1, "disk":1}

Runtime: actual 31.904s vs target 30.0s
Tasks per second: 15.860
Cleaning up DFK
The end

Continuing some tests. I don't think we are seeing the same blocking wait cycle as before.

[colinthomas-z80]

So far there seems to be two components to the issue:

The first of which is one bug currently being worked on in #3889, when running n core tasks on a local n core worker the requested disk allocation will exceed remaining disk and cause the task to be forsaken. This happens periodically while running the performance benchmark. This is likely the cause of the variability in throughput since its nature is unpredictable, but if it were to never occur the throughput would still be limited to around 4 tasks per second.

The main throughput issue seems to reside somewhere on the worker in the overhead of the exec_parsl_function.py script. The running time of each task recorded at the worker is around .24 seconds, which takes just about all of the time needed to have a limit of 4 tasks per second. I will study the script and perhaps wrestle with a python gprof equivalent. @tphung3 might have some ideas as well.