spfs server does not handle long term connection

[jrray]

Consider the following arrangement of services:

[ client (spk) ] <--> [ spfs server local ] <-- WAN --> [ spfs server remote ] <--> [ fs repo ]

Starting up these services, where spfs server local is configured with "origin" as a grpc repo talking to spfs server remote, the whole stack works (performance is terrible but that is another story) and spk env is able to resolve and enter an environment.

After waiting some amount of time, perhaps as little as 20 minutes but at most 24 hours, spfs server local will have lost its connection to spfs server remote and any attempt to communicate with it from that point on results in an indefinite (or maybe ~15 minutes) hang or whatever timeout spfs server local is using when it configures the grpc repo.

Using ctrl-c on either of the spfs server instances will hang on INFO shutting down gRPC server... for around that same ~15 minutes, except spfs server local doesn't hang if it is configured with a timeout.

The point of trying this setup is for spfs server local to use a FallbackRepository and end up "caching" any payloads accessed through it locally so reading those payloads in the future does not have to traverse the WAN. This is why, for example, the client doesn't talk directly to spfs server remote.

Unfortunately this configuration has turned out to be unreliable and not usable for production use.