Force pipeline termination from pipeline manager

[ryzhyk]

The runner handles the /shutdown command by first calling controller.stop() and then sending a termination message to the actix dispatch loop. Both operations can be slow or even get stuck in some cases. We will need a way to enforce eventual termination from the pipeline manager by killing the process/pod after a timeout.

@lalithsuresh

[lalithsuresh]

@ryzhyk I get the operations being slow, but do we know when/why they get stuck? Because that would be a bug.

[ryzhyk]

• controller.stop() waits for any in-progress iteration of the circuit to complete. Which should normally be fast, but can take a while if there's a perf issue in the circuit or we're processing a very large input batch. Another possible reason is buggy connectors that refuse to shutdown instantly or at all.
• actix is not supposed to take long, but I think I've seen it take a while to shutdown. Not sure what's up with that.