A workaround for clusters? #2
Description
future.p2p looks intriguing. Communicating through storage seems like it simplifies the interprocess communication. (I had considered that very same storage-based todo/done pattern in the earliest days of crew, along with Redis, before ultimately choosing mirai.) And since shared storage usually available in some form, the idea of using it to delegate computation to collaborators is promising.
At my workplace, we chafe at the networking security restrictions on our cluster. In particular, reverse tunneling is not allowed, which means we cannot spawn SGE jobs from code running on our laptops. I wonder if {future.p2p} can alleviate pain in these types of situations. We use mounted network drives all the time as a means of sharing storage, and unless there is a specific prohibition against pico.sh, it should be allowed as an intermediary. Combined with nested futures, I wonder if this can give the Futureverse its own equivalent of the clustermq SSH connector. I am curious if this is a pattern you are considering.