websocket connection becomes idle after subscribing to multiple accounts

[davidbrota]

Problem

Hi. I'm subscribing to multiple accounts using the "accountSubscribe" RPC method. Those addresses belong to Serum eventqueues, whose data field takes 1 MB. I'm also using a heartbeat to prevent idle timeouts in the connection.

Problem is that after successfully subscribing to 50 different addresses and receiving a result response for each one of them, the server, at some point, stops sending accountNotification responses and replying to the pings. The connection just stays open and idle (the server never closes it).

This scenario happens on both endpoints of the Mainnet Beta. We're also making sure that we aren't hitting the rate limits described there.

When subscribing to Serum asks and bids addresses the connection works perfectly and we don't experience this issue at all.

Any ideas on what might be going on there?

[polachok]

I think we might have the same problem with programSubscribe.
I made a python script to reproduce (sorry for code quality, reduced from a larger program):

from websocket import create_connection
import json
import hashlib

programs = ['9xQeWvG816bUx9EPjHmaT23yvVM2ZWbrrpZb9PusVFin']
accounts = []

def subscribe(sock, id, key, is_program):
    msg = None
    if not is_program:
        msg = {
                "jsonrpc": "2.0",
                "id": id,
                "method": "accountSubscribe",
                "params": [ key, { "commitment": "recent", "encoding": "base64" } ],
        }
    else:
        msg = {
                "jsonrpc": "2.0",
                "id": id,
                "method": "programSubscribe",
                "params": [ key, { "commitment": "recent", "encoding": "base64" } ],
        }
    sock.send(json.dumps(msg))

#ws = create_connection("ws://$MYINTERNALVALIDATOR/")
ws = create_connection("ws://api.mainnet-beta.solana.com/")
for key in accounts:
    id = key
    subscribe(ws, id, key, False)
for key in programs:
    id = key
    subscribe(ws, id, key, True)

print("Sent")
print("Receiving...")

total_length = 0
while True:
    msg =  ws.recv()
    length = len(msg)
    total_length += length
    obj = json.loads(msg)
    # response
    if 'result' in obj:
        print(obj)

    # notification
    if 'method' in obj:
        print("received %s" % obj["method"])
    print("total bytes received: %s (%sMb)" % (total_length, total_length / 1024 / 1024))

After some time it just stops receiving new messages, and if we use our internal validator (not mainnet), we see
Encountered an error: WebSocket at Capacity: Reached the limit of the output buffer for the connection. in validator stdout. We tried setting --rpc-pubsub-max-out-buffer-capacity to higher values, up to 1Gb. It seems like it allows it to work for more time, but doesn't really solve the problem.

It's also reproducible on mainnet (stops after just a few seconds).

[davidbrota]

Thanks @polachok, that was very helpful!

I was able to find the default settings, which seems to be the root cause of our issue. With a low number of subscriptions, the connection works fine.

[polachok]

I think the root cause is in ws-rs