Redis subscriptions not released when Timeout is raised

[lauri865]

It seems as if connection_pool fails to release redis subscriptions. Once Timeout is raised, it every subsequent try will result in Timeout as well.

Steps to reproduce:

1. Set redis pool size to something low (like 2 to 5)
2. Open a dashboard and spam refresh
3. Refresh and you won't see any widgets populated

[lauri865]

The problem was that disconnection was not detected.

The solution:

• Adding a heartbeat job
• The trick is: the heartbeat needs its dedicated Redis connection

[gottfrois]

Hey there!

Can you elaborate a bit more about what is a heartbeat job in your case?

[lauri865]

Sure, here it is: http://pastebin.com/hgE1bECD

5s is for development purposes, otherwise it can be longer as well.

[lauri865]

All rufus jobs should be set to non-overlapping as well, because the 3s timeout limit allows requests to accumulate, thus using even more redis connections from the pool.

[gottfrois]

Interesting, so is there something we can do in that repo?

[lauri865]

We can. I've since opening the issue changed the logic on my end completely.

Instead of having a subscription for each /events stream, there is now a single subscription pushing to all SSE streams. Seems like a more reasonable approach to me. Or do you any counter arguments?