Allocating and side-effect reads

[progrium]

The one bit of feedback I've gotten regarding the design of Wanix and its patterns has been about what Plan 9 calls "the clone dance". Some people (including myself!) don't like read operations having side-effects, which is how allocation works. You read a file either called new or under a directory called new, it allocates it and returns the ID.

When I first implemented this, I noticed sometimes resources would unexpectedly be allocated. At the time, allocation was done when the file was open for reading, which depending how the FS is implemented can happen when just doing a stat. Many programs (including the shell) will do this causing one or more allocations. I changed it so allocation doesn't happen until there is an explicit read on the open file, which resolved it enough to prevent stats causing allocation.

As I mentioned, this is loosely modeled after how Plan 9 does it. And actually parts of Linux use this same pattern (like /dev/net/tun). But both Plan 9 and Linux do it a little differently: when you close the file, the resource is cleaned up. In other words:

You keep the file descriptor open for the duration of your use of the resource. Meaning you don't just keep the resulting ID, you keep the FD as well. Two stateful things, more complexity. Plus, what if using the resource spans multiple processes? An ID is just data. Whereas a FD is scoped to the current process.

A lot of Plan 9 isn't documented. I think they ended up using their own primitives to solve problems as they came up, establishing patterns that became institutional knowledge. I would think a side effect read would bother them too. Perhaps it was an early established pattern that looking back they would maybe do differently. Talking to a Plan 9 person, they pointed out that at least when the file is closed, the resource is cleaned up. So if some program unintentionally allocated by opening the file, at least the resource (which would only be allocated, not necessarily initialized) wouldn't be around very long and it would likely be gone by the time the program finished.

As much as that makes sense to me, it does make it quite a bit harder to pull off in Plan 9's shell, let alone common shells of today. Not even sure you can do this in some of them (probably):

<>[5]/net/tcp/clone {
    n=`{read <[0=5]}
    local=/net/tcp/$n
    echo $local
    rc
}

So I simplified it. I'm not trying to exactly re-create Plan 9 or make a strictly compatible system. I'm just defaulting to their choices to get into a similar design headspace. I'm open to straying from Plan 9 when it makes sense. I decided to keep the basic idea, but make it as simple as creating a REST resource, which is more widely understood and familiar these days.

I'd like to get some feedback. I have thoughts on alternatives, but I don't want to propose anything if this isn't a big deal. Let me know if you have thoughts here, or at the very least, please answer this poll...

[jayfresh]

So I'm somewhat biased by the REST point of view, but leaning into that for a second, if you require writing something to /new to get the ID, does that make anything easier? Potentially there's the opportunity to combine allocating the resource and passing the first payload to it

[progrium]

Yes, there are a handful of "easy" solutions like this. I guess I'll also mention the idea that's been suggested of rolling allocation into the ctl file. But are they "simple"? Either of these does mean you now need to do both a write and a read, which is fine, just harder to do from the shell. It would reduce the impact of "I can just use basic shell commands interactively on the shell to do things" because cat is just read, echo > is just write.

And yes, there are abstractions usually on top anyway, like the shell helpers. I just think the more we can flatten abstractions and keep it simple and accessible at all levels, the better. The helpers obfuscate aspects of the design that I would rather people know about than not.

[xplshn]

It doesn't bother me much that you stray from the Plan 9 way of doing things, but I'd always prefer the Plan 9 way given that I use (actually, I dual-boot, because vmx is just too slow to virtualize an OS to then run Chromium) 9front and AliceLinux

[chiefnoah]

It's worth pointing out that a read pretty much always has side effects, whether it's incrementing an offset in the file description (or whatever Plan 9's equivalent is), clearing a buffer (such as from a TCP socket), or triggering an interrupt in some hardware (basically everything else). Just because you're used to those side-effects and don't think about them doesn't mean they aren't there. To that end, it actually makes the most sense for the allocation to be tied to the lifetime of the file descriptor, because it will be automatically cleaned up when the process exits and is a first-class citizen in rfork (whether child procs inherit the FD is configurable). FDs are the core abstraction for this type of problem. The way you've fixed makes the most sense to me and seems the closest to "correct".

I'm not an active plan 9 user, so I cannot test this, but I suspect that ip(3)'s /net/tcp/clone will allocate and return the ID for a new connection on subsequent reads and deallocate all of them when the fd is closeed. You almost certainly want to maintain this dynamic.

[progrium]

Yea, good point on all reads having a side-effect. I'm not a side-effect purist. I'm mostly concerned about unintended allocations and user expectations.

[chiefnoah]

I'd be surprised if there were very many applications that do a "splat" read of arbitrary files (ie. ls only calls stat, not read). I think it would be artificially crippling of 1/4 of your file API to maintain side-effectless reads, especially considering it's the only way to atomically get a value: the read and write APIs are simplex after all. Put another way, any resource that needs the process that requests a value to be guaranteed to be the process that gets that value must use read to do so, a write then read isn't atomic and any process that shares the resource could race (it should be possible to limit this to processes that share FDs, but in the Plan 9 model that's a larger surface area than would be ideal).

This isn't a HTTP GET request, it's a low level system API that most Plan 9 curious people will have no trouble reasoning about.