This needs cctools 7.13.0 (because thats the first release that has hoisting_modules) but I noticed that Parsl CI is still on 7.8.0 so I think there isn't any test coverage for this change.

I tried this out on my laptop with serverless mode turned on: a pytest run reduces from 116 seconds to 20 seconds. So it would be good for timing considerations to switch our CI taskvine test over to this mode, perhaps.

and

parsl-perf --time 20 --config parsl/tests/configs/taskvine_serverless.py

goes from
6 tasks/sec
to
140 tasks/sec

Importing parsl.serialize will (in usual Python) import parsl first which will in turn import a very large part of the parsl package tree - because parsl/__init__.py imports a lot for re-export - to get a "batteries included" style of import parsl.

So I propose that importing plain parsl rather than parsl.serialize should result in the same performance change - I tried that out on my laptop and the results appear to have about the same distribution (by eyeball).

I'm a little unclear what happens for user code that would benefit from this kind of pre-load for other modules.

@benclifford thoughts on getting this merged? Is import parsl your recommended change?

Is import parsl your recommended change?

@dthain yes

Closing in favor of #3902