Set a good maximim bound for the input queue (and etcd queue) #2442
Description
In #2430 we found that if we set a bound on the input queue, we address a couple of problems related to the hydra-node getting stuck when lots of transactions are submitted.
While setting a bound has "solved" that problem, a few mysteries remain:
- Why was the logging queue size blocking input processing?
- Can we fix it so there isn't a connection between the input queue size and the logging queue size?
- What happened to the transactions that got lost? (We think they exceeded their ttl and hence were invalid.)
Todo
- Resolve mysteries
- Add test to check that the bound is set correctly (i.e. something that fails with too large a bound, and succeeds with the right bound.)
- Check how large the etcd queue should be as well (that also got changed in the PR)