How to control execution order of plugins (e.g. SchedulerPlugin, WorkerPlugin)

[fangzhouxie]

Hi all,

I have a use case where the execution order of plugins matters. More specifically, I want distributed.diagnostics.plugin.PipInstall to be executed first. Does anyone know what's the best way to achieve this?

Currently I start/setup plugins manually in a wrapper plugin in order to control execution order but this is not ideal and does not work for all cases. So I'm wondering if there's a proper way to control execution order of plugins.

Thanks in advance.

[hendrikmakait]

@fangzhouxie: Generally, there is currently no good way to guarantee ordering between plugins.

I'd be interested to hear more about your use case. What exactly are your ordering requirements here?

[fangzhouxie]

@hendrikmakait Thanks for your reply.

tldr; in my use case, I just need to make sure distributed.diagnostics.plugin.PipInstall runs before any other plugins.

In my use case, I use distributed.diagnostics.plugin.PipInstall to upgrade an existing python package, let's say package_A. So I need to make sure that this PipInstall plugin runs first. Otherwise, other plugins may import modules in older version of package_A before it's upgraded (unlikely but possible), which causes issues because the underlying process will keep using implementations in the older version of package_A in this case.

I tried setting restart_workers=True in PipInstall but it didn't help. I also tried importlib.reload but it's not ideal because I have to reload specific modules (e.g. importlib.reload(package_A.path.to.module)) and re-import them.

[hendrikmakait]

That makes sense to me. Assuming that you have a static cluster where no new workers join, you could force ordering by installing plugins one after another:

• Wait for all workers to start and register with the scheduler
• Register PipInstall plugin and wait for registration to complete (using a synchronous client, it should block until it's installed on all running workers/nannies)
• Register other plugins

Note that this likely won't work for any worker that joins late. The behavior on restarting workers might also be different.

[fangzhouxie]

That solution for a static cluster makes sense to me. Unfortunately we use a dynamic cluster where new workers may join at any time.

Still, many thanks to your time and responses.

[hendrikmakait]

FWIW, I've added an issue for this, feel free to chime in: #8486

[fangzhouxie]

Awesome! Thank you!

[fangzhouxie]

Closed this discussion as an issue has been raised in #8486