Look up the purrr function on the worker, rather than sending it over#296
Look up the purrr function on the worker, rather than sending it over#296DavisVaughan merged 2 commits intomainfrom
Conversation
So `expr_seed_setup` and `expr_seed_update` still live close together
| ...furrr_purrr_fn <- get( | ||
| !!purrr_fn_name, | ||
| envir = asNamespace("purrr"), | ||
| mode = "function", | ||
| inherits = FALSE | ||
| ) |
There was a problem hiding this comment.
Here's where we now splice in purrr_fn_name, which is just "map" or "map_dbl". Then we let that be looked up on the worker itself.
We're building an expression to send to the worker here, so after the splicing this part of the expression should just look like:
...furrr_purrr_fn <- get(
"map",
envir = asNamespace("purrr"),
mode = "function",
inherits = FALSE
)and the worker gets that
|
Yes, this update makes sense to me. Inconsistent setups between main R session and workers is a problem to all parallel frameworks, including futureverse. Right now, we're crossing our fingers and hope it'll work, and if it doesn't, hopefully there's a run-time error catching it. The worst case scenario is that the error is silent and we get incorrect results. What the future ecosystem could do is to report on package version discrepancies, e.g. produce a FutureWarning, or even an FutureError if there are major version differences. This is to be explored. This also fall under the plan to introduce resource specifications, i.e. package version requirements are in one way similar to OS and GPU requirements. The bottom line is that, furrr should not have to worry about this, but rely on the future core to handle this. |
Closes #253
@HenrikBengtsson does this make sense to you?
Here's the setup:
The API hasn't changed between the two versions, but the internals did.
We used to capture the local version of
purrr::mapand ship that over to the worker. The captured version ofpurrr::map()had internals that looked forpurrr:::map_(), which is an internal function that exists in purrr 1.0.0 but not 0.3.5, so it could fail. That resulted in #253 and https://stackoverflow.com/questions/75660223/r-furure-map-across-multiple-servers.We now ship the name of the purrr function over to the worker, but look up the actual function from the purrr namespace while on the worker itself. That should prevent this issue (though I don't have a very easy way to test this particular issue).