Fix Shape.Supervisor not restarting

[robacourt]

Current behaviour

If a child of the Shape.Supervisor crashes more than 3 times in 5 seconds the Shape.Supervisor shuts down and doesn't restart because it is transient. The WAL will then start to build up. If the connection restarts the Connection.Manager with then report that the Shape.Supervisor is :already_present.

This happened in production for AutoArc when the ShapeLogCollector crashed 4 times in under a second due to trying to call a missing Materializer.

Suggested behaviour

If a child of the Shape.Supervisor crashes more than 3 times in 5 seconds the Shape.Supervisor should be restarted but with all shape data wiped. It is important that the shape data is wiped in this situation as otherwise the ShapeLogCollector will go on trying to processes the txn/shape combinations it crashes on, going into an infinite crash loop. Ideally the Shape.Supervisor should be permanent not transient.

Please note that the Connection.Manager should not match on :already_present. :already_present indicates a serious issue has already happened (the shape subsystem has gone down and has not been restarted) and there is no point in handling this gracefully.

[alco]

What if the crashing of the child process is caused by a particular runtime state that the system has gotten into and not the incoming transactions? It feels like purging all shapes is a "hands in the air" approach to trying to fix the problem where we would normally just restart the whole stack.

And if the stack ends up hitting the same crashing behaviour after a restart, this needs to be detected at a higher level that can then shut down the stack for good.

Could you link in the specifics of the issue that caused a child (which child) of the Shapes.Supervisor to crash?

[robacourt]

Could you link in the specifics of the issue that caused a child (which child) of the Shapes.Supervisor to crash?

This PR has the details #3628

What if the crashing of the child process is caused by a particular runtime state that the system has gotten into and not the incoming transactions? It feels like purging all shapes is a "hands in the air" approach to trying to fix the problem where we would normally just restart the whole stack.

And if the stack ends up hitting the same crashing behaviour after a restart, this needs to be detected at a higher level that can then shut down the stack for good.

Good point, feel free to restart more of the system, but don't just restart the stack, the shapes must be wiped as well to prevent an eternal crash loop.

[robacourt]

I think part of the solution is to make the Shapes.Supervisor permanent, that way it will restart but if that restart fails it'll work its way up the tree. Wiping shapes could be achieved by having an explicit stop on storage that's called on a healthy shutdown but if it's started without a stop it wipes all shapes.