Is it safe to use clustermq with doFuture ?

[statquant]

Hello,
I am trying to move to future but I have the following constraints:

• I'd like to keep the foreach::%dorpar% construct
• I'd like to keep using clustermq
  It's important for me to keep clustermq because all the communication between the slurm workers and the master is done on the network as opposed to file like in future.batchtools.
  I think I red in the past that it could be dangerous to use register methods like clustermq::register_dopar_cmq, and because it seems that future.clustermq is not mature yet my questions are:

• is it safe to use doFuture::registerDoFuture in conjunction from clustermq::register_dopar_cmq ?
• if it is is there a best practice ?

[HenrikBengtsson]

... it seems that future.clustermq is not mature ...

Correct, future.clustermq (https://github.com/HenrikBengtsson/future.clustermq) is/was just prototype - I haven't worked on it since (mostly for lack of time and priorities).

• is it safe to use doFuture::registerDoFuture in conjunction from clustermq::register_dopar_cmq?

I'm not sure that I understand your question. Can you please clarify? Please note that when you use doFuture::registerDoFuture(), then you tell %dopar% to use doFuture, which in turn will use whatever future backend is registered via future::plan(). When you use clustermq::register_dopar_cmq(), you tell %dopar% to use clustermq. So, they are using completely independent backends.

[statquant]

Hello, it is not clear because I'm fishing a bit sorry.
I clearly remember that somewhere in your docs you advertise that when using doFuture, it is advised to not use other "base" registering methods like doMC::register***. In the past I've been confronted in a lot of issues using parallelism in R (child processes not killed master hanging when collecting) and it's difficult to diagnose).
I routinely use local parallelism and slurm parallelism alternatively and I saw in your warning a potential explanation of my issues. Given those are 2 different packages I "thought" they could clash in some way.

Long story short I get that you do not expect issues from switching from doFuture to clustermq in the same code and do not have particular advice/warning to give ?