ability to connect to multsession stdin/stdout, akin to dtach

[r2evans]

The default behavior of parallel nodes is good, I'm not questioning the headless default of the child processes.

Is there an ability to setup the stdin/stdout of the R processes so that they could be connected to if need be? In unix-land, the dtach utility is similar to tmux/screen in that it allows disconnection from a terminal (often/typically over ssh) and reconnection. The dtach intent is much simpler: it does not aim to address multi-plexing, it just handles the terminal connection.

I'm considering how best to handle long-running background R processes. The use of multisession works, I can start a long-running process in the background and it works fine. I could even use callr::r_bg for this purpose, tbh. However, in both cases, if somebody happens to either interrupt/pause the child process (occasionally I have an embedded browser() for many reasons), it would be really great to be able to interact with it live, optionally continuing if the issue is resolved. While it could be argued that a long-running headless process should either run fine or quit, never pausing ... I don't disagree, but let's sideline that for the moment, since I think there are times when it may be acceptable to rely on momentary user intervention.

For instance, if I start a process using

proc <- processx::process$new("R", c("--interactive"), stdin="|", stdout="|", stderr="2>&1")

then I can write commands to it, read the output, etc. Granted, now I need to parse the output (such as removing the "\n> " and reacting if I see a trailing "\n+ " "continue" indication), and that's overhead I'd like to avoid :-). One technique is using dtach instead, with

proc <- processx::process$new("dtach", c("-c", "~/dtach_session_1", "R", "--interactive"), pty=TRUE)

Using emacs/ess, I can then attach to that using

• in emacs, M-x shell
• in the shell, dtach -a ~/dtach_session_1
• in emacs, M-x ess-remote and R for the dialect

At this point, I have full visibility and control over the session. (Other than C-c C-k to kill the shell process, I haven't found how to use dtach's native escape to disconnect, a minor inconvenience.)

I recognize this may not be a common use-case for others. Is there any room in multisession-implementation to allow this kind of functionality, either natively or indirectly by calling dtach instead of R?

[HenrikBengtsson]

Since you mention that you can get something going with processx, I wonder if it would be worth looking into supporting this in future.callr, which uses callr that uses processx internally. To add support for it in multisession, we'd need to figure out if it can be achieved using regular parallel/parallelly PSOCK clusters. But it sounds like it would be easier to get a proof-of-concept going with the callr/processx path. Is that something you could look into?

[r2evans]

I can get some of it to work using stock callr:

fun <- function() {
  options(browserNLdisabled = TRUE)
  print(R.version)
  browser()
  1
}
r2 <- callr::r_bg(
  fun, arch="/opt/homebrew/opt/dtach/bin/dtach",
  # libpath = c("/Users/r2/Library/R/4.3","/Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library"),
  cmdargs=c("-c", "/Users/r2/dtach_session_0.sock", "/Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/bin/R", "--no-save", "--interactive"),
  stdout=NULL, stderr=NULL, stdin=NULL, pty=TRUE)
r2$read_output()
# [1] "\033[H\033[J"
r2$is_alive()
# [1] TRUE
r2$read_output()
# [1] "\r\nR version 4.4.2 (2024-10-31) -- \"Pile of Leaves\"\r\nCopyright (C) 2024 The R Foundation for Statistical Computing\r\nPlatform: aarch64-apple-darwin20\r\n\r\nR is free software and comes with ABSOLUTELY NO WARRANTY.\r\nYou are welcome to redistribute it under certain conditions.\r\nType 'license()' or 'licence()' for distribution details.\r\n\r\n  Natural language support but running in an English locale\r\n\r\nR is a collaborative project with many contributors.\r\nType 'contributors()' for more information and\r\n'citation()' on how to cite R or R packages in publications.\r\n\r\nType 'demo()' for some demos, 'help()' for on-line help, or\r\n'help.start()' for an HTML browser interface to help.\r\nType 'q()' to quit R.\r\n\r\n"
r2$write_input("pi\n")
r2$read_output()
# [1] "> [1] 3.141593\r\n> "
r2$write_input("q('no')\n")
r2$read_output()
# [1] "\033[999H\r\n[EOF - dtach terminating]\r\n"
r2$read_output()
# [1] "\033[?25h"
Sys.sleep(1)
r2$is_alive()
# [1] FALSE

Unfortunately ...

• the use of --interactive preempts the use of -f FILE for the script file to execute, so fun() is never called; this is preventing r_bg()'s functionality; and
• omitting --interactive means that browser() and friends don't actually support interaction.

I think what I'm aiming to do requires --interactive, so in trying to find ways to use it and a script file,

• --interactive -f < FILE replaces stdin with the file and therefore doesn't support /dev/stdin-like input
• cat FILE /dev/stdin | R --interactive -f - doesn't work, because R requires a filename argument
• I tried bash's special R --interactive -f <(cat FILE /dev/stdin), but I don't think it's meant to do that (and is not portable)

So while the notion of using dtach ... R ... works, callr's requirement to pass a function (for good reasons) via -f FILE is going to be an issue for me.

We can't use -e "source(FILE)" as a hack, since that is explicitly documented as

-e EXPR               Execute 'EXPR' and exit

Can you think of a way to combine the -f FILE functionality in a way that preserves stdin?

Otherwise, we need to bypass some of r_bg's functionality to grab the temp script file and inject source(FILE)\n into the process's stdin ... I don't know offhand that I'd want to undermine r_bg that much ...

(To be fair, my test/demo using processx::process$new(..) suffers from the exact same issues, requiring the commands to be injected via stdin instead of -e EXPR or -f FILE.)

[HenrikBengtsson]

Otherwise, we need to bypass some of r_bg's functionality to grab the temp script file and inject source(FILE)\n into the process's stdin ... I don't know offhand that I'd want to undermine r_bg that much ...

I see - almost there, but not all the way. I agree, other than hacking together a proof-of-concept, it sounds like we'd need callr to be updated to support this use case. I guess it could be worth reaching out and ask for this independently of Futureverse; dtach + callr::r_bg() would be a neat feature.

[r2evans]

Part of me things this is an issue with .../bin/R, and changes to callr would be a workaround for that not happening.

But yes, I think having callr::r_bg and friends supporting pushing code via stdin instead of -e/-f would help mitigate that pain point.

Be honest with me, do you see dtach+callr::r_bg() being a selling point for anybody outside of "me"? To be fair, while dtach still (mostly, #5, #10) works and is still well distributed, it appears unmaintained.

[HenrikBengtsson]

An alternative route is to implement a new type of PSOCK cluster in parallelly that would spin up local workers using processx::process$new() instead of a using system() that is currently used by parallel and parallelly. The latter launches workers as show below:

> cl <- parallelly::makeClusterPSOCK(1L, dryrun = TRUE)
----------------------------------------------------------------------
Manually, start worker #1 on local machine 'localhost' with:

  '/home/henrik/shared/software/CBI/_ubuntu22_04/R-4.4.2-gcc11/lib/R/bin/Rscript' --default-packages=datasets,utils,grDevices,graphics,stats,methods -e 'options(socketOptions="no-delay")' -e 'workRSOCK<-tryCatch(parallel:::.workRSOCK,error=function(e)parallel:::.slaveRSOCK);workRSOCK()' MASTER=localhost PORT=11718 OUT=/dev/null TIMEOUT=2592000 XDR=FALSE SETUPTIMEOUT=120 SETUPSTRATEGY=sequential

This is mostly set up by parallelly::makeNodePSOCK(). That function is very general and handle lots of cases, e.g. local and remote, etc. A pure processx localhost node, would be much less code, but the gist would be to launch parallel::workRSOCK() via a processx process while setting those environment variables (MASTER, PORT, ...).

[r2evans]

Frankly, I've never been a fan of system, and system2 is no better. I'm sure you're going to some pains to work around their issues (namely, shell-quoting), but I really prefer processx::process$new over them for a few reasons. Perhaps the biggest issue would be adding an Import: to parallelly, not sure if you're happy/willing to do that.

[HenrikBengtsson]

The plan with parallelly was always to stick as close to parallel as possible, mostly with some hope that some of the tweaks and improvements could be merged into parallel. Only so much of that ever happened - I think I got one or two "bug fixes" added upstream. I eventually got quite good at shell quoting (just make sure to use shQuote() everywhere, and it works also when nested). But, yes, if I'd started from scratch, I probably would have done it differently.

Perhaps the biggest issue would be adding an Import: to parallelly, not sure if you're happy/willing to do that.

That's a "problem" that can be punted on - the first goal is to get something that works. To avoid a hard dependency on parallelly, one could start off by relying on Suggests: with some useful error message if processx is not installed. If it turns out it's a great success, it might be that it ends up in a standalone parallelly.processx package, and the thin wrapper future.processx package that you'd use for futures, e.g. plan(future.processx::processx_multisession). Or it could all start out in a future.processx package, and if it turns out there are valuable parts that are independent of the Futureverse, that could be extract into it's own package (that's how parallelly started out, if you remember). Anyway, all this can be decided on later.