Broadcast subscription and heartbeat lost randomly without exceptions/error raised

[SmaugPool]

Describe the bug

After a random numbers of hours (or days), subscription to broadcast get lost and heartbeat doesn't get replies anymore without any subscription error raised. Once this happens, the channel doesn't receive broadcast events anymore.

2025-11-04 10:34:26,436 - INFO - Attempting to connect to WebSocket at wss://xxx.supabase.co/realtime/v1/websocket?apikey=sb_secret_xxx
2025-11-04 10:34:26,538 - INFO - WebSocket connection established successfully
2025-11-04 10:34:26,571 - INFO - send: {"event":"heartbeat","payload":{},"topic":"phoenix","ref":null,"join_ref":null}
2025-11-04 10:34:26,576 - INFO - receive: '{"ref":null,"event":"phx_reply","payload":{"status":"ok","response":{}},"topic":"phoenix"}'
2025-11-04 10:34:26,576 - INFO - parsed message as event=<ChannelEvents.reply: 'phx_reply'> topic='phoenix' payload=SuccessReplyMessage(status='ok', response=ReplyPostgresChanges(postgres_changes=None)) ref=None
2025-11-04 10:34:26,740 - INFO - send: {"event":"phx_join","payload":{"config":{"broadcast":null,"presence":null,"private":true,"postgres_changes":[]},"access_token":"sb_secret_xxx"},"topic":"r>
2025-11-04 10:34:26,756 - INFO - receive: '{"ref":"1","event":"phx_reply","payload":{"status":"ok","response":{"postgres_changes":[]}},"topic":"realtime:test"}'
2025-11-04 10:34:26,757 - INFO - parsed message as event=<ChannelEvents.reply: 'phx_reply'> topic='realtime:test' payload=SuccessReplyMessage(status='ok', response=ReplyPostgresChanges(postgres_changes=[])) ref='1'
2025-11-04 10:34:26,757 - INFO - realtime:test : event=<ChannelEvents.reply: 'phx_reply'> topic='realtime:test' payload=SuccessReplyMessage(status='ok', response=ReplyPostgresChanges(postgres_changes=[])) ref='1'    
2025-11-04 10:34:26,757 - WARNING - subscription status=SUBSCRIBED err=None
2025-11-04 10:34:26,757 - INFO - receive: '{"ref":null,"event":"presence_state","payload":{},"topic":"realtime:test"}'
2025-11-04 10:34:26,757 - INFO - parsed message as event=<ChannelEvents.presence_state: 'presence_state'> topic='realtime:test' payload={} ref=None
2025-11-04 10:34:26,757 - INFO - realtime:test : event=<ChannelEvents.presence_state: 'presence_state'> topic='realtime:test' payload={} ref=None    
2025-11-04 10:34:36,742 - WARNING - subscription status=TIMED_OUT err=None
2025-11-04 10:34:51,573 - INFO - send: {"event":"heartbeat","payload":{},"topic":"phoenix","ref":null,"join_ref":null}
2025-11-04 10:34:51,578 - INFO - receive: '{"ref":null,"event":"phx_reply","payload":{"status":"ok","response":{}},"topic":"phoenix"}'
2025-11-04 10:34:51,578 - INFO - parsed message as event=<ChannelEvents.reply: 'phx_reply'> topic='phoenix' payload=SuccessReplyMessage(status='ok', response=ReplyPostgresChanges(postgres_changes=None)) ref=None
2025-11-04 10:35:16,575 - INFO - send: {"event":"heartbeat","payload":{},"topic":"phoenix","ref":null,"join_ref":null}
2025-11-04 10:35:16,580 - INFO - receive: '{"ref":null,"event":"phx_reply","payload":{"status":"ok","response":{}},"topic":"phoenix"}'
2025-11-04 10:35:16,581 - INFO - parsed message as event=<ChannelEvents.reply: 'phx_reply'> topic='phoenix' payload=SuccessReplyMessage(status='ok', response=ReplyPostgresChanges(postgres_changes=None)) ref=None
2025-11-04 10:35:41,577 - INFO - send: {"event":"heartbeat","payload":{},"topic":"phoenix","ref":null,"join_ref":null}
2025-11-04 10:35:41,583 - INFO - receive: '{"ref":null,"event":"phx_reply","payload":{"status":"ok","response":{}},"topic":"phoenix"}'
2025-11-04 10:35:41,583 - INFO - parsed message as event=<ChannelEvents.reply: 'phx_reply'> topic='phoenix' payload=SuccessReplyMessage(status='ok', response=ReplyPostgresChanges(postgres_changes=None)) ref=None
2025-11-04 10:36:06,579 - INFO - send: {"event":"heartbeat","payload":{},"topic":"phoenix","ref":null,"join_ref":null}
2025-11-04 10:36:06,590 - INFO - receive: '{"ref":null,"event":"phx_reply","payload":{"status":"ok","response":{}},"topic":"phoenix"}'
2025-11-04 10:36:06,590 - INFO - parsed message as event=<ChannelEvents.reply: 'phx_reply'> topic='phoenix' payload=SuccessReplyMessage(status='ok', response=ReplyPostgresChanges(postgres_changes=None)) ref=None
2025-11-04 10:36:31,582 - INFO - send: {"event":"heartbeat","payload":{},"topic":"phoenix","ref":null,"join_ref":null}
2025-11-04 10:36:31,589 - INFO - receive: '{"ref":null,"event":"phx_reply","payload":{"status":"ok","response":{}},"topic":"phoenix"}'
2025-11-04 10:36:31,590 - INFO - parsed message as event=<ChannelEvents.reply: 'phx_reply'> topic='phoenix' payload=SuccessReplyMessage(status='ok', response=ReplyPostgresChanges(postgres_changes=None)) ref=None
...
2025-11-04 13:51:07,387 - INFO - send: {"event":"heartbeat","payload":{},"topic":"phoenix","ref":null,"join_ref":null}
2025-11-04 13:51:07,413 - INFO - receive: '{"ref":null,"event":"phx_reply","payload":{"status":"ok","response":{}},"topic":"phoenix"}'
2025-11-04 13:51:07,413 - INFO - parsed message as event=<ChannelEvents.reply: 'phx_reply'> topic='phoenix' payload=SuccessReplyMessage(status='ok', response=ReplyPostgresChanges(postgres_changes=None)) ref=None
2025-11-04 13:51:32,389 - INFO - send: {"event":"heartbeat","payload":{},"topic":"phoenix","ref":null,"join_ref":null}
2025-11-04 13:51:32,398 - INFO - receive: '{"ref":null,"event":"phx_reply","payload":{"status":"ok","response":{}},"topic":"phoenix"}'
2025-11-04 13:51:32,398 - INFO - parsed message as event=<ChannelEvents.reply: 'phx_reply'> topic='phoenix' payload=SuccessReplyMessage(status='ok', response=ReplyPostgresChanges(postgres_changes=None)) ref=None
2025-11-04 13:51:57,391 - INFO - send: {"event":"heartbeat","payload":{},"topic":"phoenix","ref":null,"join_ref":null}
2025-11-04 13:52:22,392 - INFO - send: {"event":"heartbeat","payload":{},"topic":"phoenix","ref":null,"join_ref":null}
2025-11-04 13:52:47,394 - INFO - send: {"event":"heartbeat","payload":{},"topic":"phoenix","ref":null,"join_ref":null}
2025-11-04 13:53:12,395 - INFO - send: {"event":"heartbeat","payload":{},"topic":"phoenix","ref":null,"join_ref":null}
2025-11-04 13:53:37,397 - INFO - send: {"event":"heartbeat","payload":{},"topic":"phoenix","ref":null,"join_ref":null}
2025-11-04 13:54:02,398 - INFO - send: {"event":"heartbeat","payload":{},"topic":"phoenix","ref":null,"join_ref":null}
2025-11-04 13:54:27,400 - INFO - send: {"event":"heartbeat","payload":{},"topic":"phoenix","ref":null,"join_ref":null}
2025-11-04 13:54:52,401 - INFO - send: {"event":"heartbeat","payload":{},"topic":"phoenix","ref":null,"join_ref":null}
2025-11-04 13:55:17,403 - INFO - send: {"event":"heartbeat","payload":{},"topic":"phoenix","ref":null,"join_ref":null}
2025-11-04 13:55:42,404 - INFO - send: {"event":"heartbeat","payload":{},"topic":"phoenix","ref":null,"join_ref":null}
2025-11-04 13:56:07,406 - INFO - send: {"event":"heartbeat","payload":{},"topic":"phoenix","ref":null,"join_ref":null}
2025-11-04 13:56:32,407 - INFO - send: {"event":"heartbeat","payload":{},"topic":"phoenix","ref":null,"join_ref":null}
2025-11-04 13:56:57,407 - INFO - send: {"event":"heartbeat","payload":{},"topic":"phoenix","ref":null,"join_ref":null}
2025-11-04 13:57:22,409 - INFO - send: {"event":"heartbeat","payload":{},"topic":"phoenix","ref":null,"join_ref":null}
2025-11-04 13:57:47,410 - INFO - send: {"event":"heartbeat","payload":{},"topic":"phoenix","ref":null,"join_ref":null}
2025-11-04 13:58:12,411 - INFO - send: {"event":"heartbeat","payload":{},"topic":"phoenix","ref":null,"join_ref":null}
2025-11-04 13:58:37,413 - INFO - send: {"event":"heartbeat","payload":{},"topic":"phoenix","ref":null,"join_ref":null}
2025-11-04 13:59:02,416 - INFO - send: {"event":"heartbeat","payload":{},"topic":"phoenix","ref":null,"join_ref":null}
2025-11-04 13:59:27,418 - INFO - send: {"event":"heartbeat","payload":{},"topic":"phoenix","ref":null,"join_ref":null}
2025-11-04 13:59:52,421 - INFO - send: {"event":"heartbeat","payload":{},"topic":"phoenix","ref":null,"join_ref":null}
2025-11-04 14:00:17,424 - INFO - send: {"event":"heartbeat","payload":{},"topic":"phoenix","ref":null,"join_ref":null}
2025-11-04 14:00:42,426 - INFO - send: {"event":"heartbeat","payload":{},"topic":"phoenix","ref":null,"join_ref":null}
2025-11-04 14:01:07,428 - INFO - send: {"event":"heartbeat","payload":{},"topic":"phoenix","ref":null,"join_ref":null}
2025-11-04 14:01:32,430 - INFO - send: {"event":"heartbeat","payload":{},"topic":"phoenix","ref":null,"join_ref":null}
2025-11-04 14:01:57,432 - INFO - send: {"event":"heartbeat","payload":{},"topic":"phoenix","ref":null,"join_ref":null}
2025-11-04 14:02:22,433 - INFO - send: {"event":"heartbeat","payload":{},"topic":"phoenix","ref":null,"join_ref":null}
2025-11-04 14:02:47,435 - INFO - send: {"event":"heartbeat","payload":{},"topic":"phoenix","ref":null,"join_ref":null}
2025-11-04 14:03:12,437 - INFO - send: {"event":"heartbeat","payload":{},"topic":"phoenix","ref":null,"join_ref":null}
2025-11-04 14:03:37,438 - INFO - send: {"event":"heartbeat","payload":{},"topic":"phoenix","ref":null,"join_ref":null}
2025-11-04 14:04:02,440 - INFO - send: {"event":"heartbeat","payload":{},"topic":"phoenix","ref":null,"join_ref":null}
2025-11-04 14:04:27,442 - INFO - send: {"event":"heartbeat","payload":{},"topic":"phoenix","ref":null,"join_ref":null}
2025-11-04 14:04:52,444 - INFO - send: {"event":"heartbeat","payload":{},"topic":"phoenix","ref":null,"join_ref":null}
2025-11-04 14:05:17,445 - INFO - send: {"event":"heartbeat","payload":{},"topic":"phoenix","ref":null,"join_ref":null}
...

The heartbeart code does not seem to check that a reply is received:

supabase-py/src/realtime/src/realtime/_async/client.py

Line 237 in 60f07a8

async def _heartbeat(self) -> None:

And there is no heartbeat callback to allow users to detect the issue:

supabase-py/src/realtime/src/realtime/_async/channel.py

Line 553 in 60f07a8

elif isinstance(message, HeartbeatMessage): # do nothing

Reproduction

#!/usr/bin/env python3    
    
import os, asyncio, logging    
from supabase.client import AsyncClient    
    
logging.basicConfig(level=logging.INFO, format="%(asctime)s - %(levelname)s - %(message)s")    
    
async def main():    
    def on_subscribe(status, err):    
        logging.warning(f"subscription status={status} err={err}")    
    
    def on_message(msg):    
        logging.info(msg)    
    
    client = AsyncClient(os.environ.get("SUPABASE_URL"), os.environ.get("SUPABASE_KEY"))    
    channel = client.channel("test", {"config": {"private": True}})    
    await channel.on_broadcast("UPDATE", on_message).subscribe(on_subscribe)    
    while True:    
        await asyncio.sleep(1)     
             
asyncio.run(main())

Steps to reproduce

Let the reproduction snippet above run until the issue happens.
It can take minutes, hours or days.

Library affected

realtime

Library version

supabase 2.22.4

Python version

python 3.12.3

[SmaugPool]

By adding a channel state log:

    while True:    
        await asyncio.sleep(15)    
        logging.info(channel.state)  

I notice that the channel first get closed after an auto-reconnect, not sure why yet:

2025-11-05 00:33:43,309 - INFO - send: {"event":"heartbeat","payload":{},"topic":"phoenix","ref":null,"join_ref":null}
2025-11-05 00:33:43,324 - INFO - receive: '{"ref":null,"event":"phx_reply","payload":{"status":"ok","response":{}},"topic":"phoenix"}'
2025-11-05 00:33:43,324 - INFO - parsed message as event=<ChannelEvents.reply: 'phx_reply'> topic='phoenix' payload=SuccessReplyMessage(status='ok', response=ReplyPostgresChanges(postgres_changes=None)) ref=None
2025-11-05 00:33:48,961 - INFO - ChannelStates.JOINED
2025-11-05 00:34:02,443 - ERROR - WebSocket connection closed with code: 1006, reason: 
2025-11-05 00:34:02,443 - INFO - Initiating auto-reconnect sequence...
2025-11-05 00:34:02,443 - INFO - Attempting to connect to WebSocket at wss://xxx.supabase.co/realtime/v1/websocket?apikey=sb_secret_xxx
2025-11-05 00:34:02,655 - INFO - WebSocket connection established successfully
2025-11-05 00:34:02,656 - INFO - Rejoining channel after reconnection: realtime:test
2025-11-05 00:34:02,656 - INFO - send: {"event":"phx_leave","payload":{},"topic":"realtime:test","ref":"2","join_ref":null}
2025-11-05 00:34:02,656 - INFO - send: {"event":"phx_join","payload":{"config":{"broadcast":null,"presence":{"key":"","enabled":false},"private":true,"postgres_changes":[]},"access_token":"sb_secret_xxx"},"topic":"realtime:test","ref":"3","join_ref":null}
2025-11-05 00:34:02,656 - INFO - send: {"event":"heartbeat","payload":{},"topic":"phoenix","ref":null,"join_ref":null}
2025-11-05 00:34:02,670 - INFO - receive: '{"ref":"2","event":"phx_reply","payload":{"status":"ok","response":{}},"topic":"realtime:test"}'
2025-11-05 00:34:02,670 - INFO - parsed message as event=<ChannelEvents.reply: 'phx_reply'> topic='realtime:test' payload=SuccessReplyMessage(status='ok', response=ReplyPostgresChanges(postgres_changes=None)) ref='2'
2025-11-05 00:34:02,670 - INFO - realtime:test : event=<ChannelEvents.reply: 'phx_reply'> topic='realtime:test' payload=SuccessReplyMessage(status='ok', response=ReplyPostgresChanges(postgres_changes=None)) ref='2'
2025-11-05 00:34:02,670 - INFO - channel realtime:test leave
2025-11-05 00:34:02,670 - INFO - channel realtime:test closed
2025-11-05 00:34:02,774 - INFO - receive: '{"ref":"3","event":"phx_reply","payload":{"status":"ok","response":{"postgres_changes":[]}},"topic":"realtime:test"}'
2025-11-05 00:34:02,774 - INFO - parsed message as event=<ChannelEvents.reply: 'phx_reply'> topic='realtime:test' payload=SuccessReplyMessage(status='ok', response=ReplyPostgresChanges(postgres_changes=[])) ref='3'
2025-11-05 00:34:02,774 - INFO - receive: '{"ref":null,"event":"phx_reply","payload":{"status":"ok","response":{}},"topic":"phoenix"}'
2025-11-05 00:34:02,774 - INFO - parsed message as event=<ChannelEvents.reply: 'phx_reply'> topic='phoenix' payload=SuccessReplyMessage(status='ok', response=ReplyPostgresChanges(postgres_changes=None)) ref=None
2025-11-05 00:34:03,964 - INFO - ChannelStates.CLOSED
2025-11-05 00:34:12,666 - WARNING - subscription status=RealtimeSubscribeStates.TIMED_OUT err=None
2025-11-05 00:34:18,970 - INFO - ChannelStates.CLOSED
2025-11-05 00:34:27,662 - INFO - send: {"event":"heartbeat","payload":{},"topic":"phoenix","ref":null,"join_ref":null}
2025-11-05 00:34:27,683 - INFO - receive: '{"ref":null,"event":"phx_reply","payload":{"status":"ok","response":{}},"topic":"phoenix"}'
2025-11-05 00:34:27,683 - INFO - parsed message as event=<ChannelEvents.reply: 'phx_reply'> topic='phoenix' payload=SuccessReplyMessage(status='ok', response=ReplyPostgresChanges(postgres_changes=None)) ref=None
2025-11-05 00:34:33,977 - INFO - ChannelStates.CLOSED

Then a few hours later the heartbeat doesn't get replies anymore:

2025-11-05 07:03:40,227 - INFO - send: {"event":"heartbeat","payload":{},"topic":"phoenix","ref":null,"join_ref":null}
2025-11-05 07:03:40,243 - INFO - receive: '{"ref":null,"event":"phx_reply","payload":{"status":"ok","response":{}},"topic":"phoenix"}'
2025-11-05 07:03:40,243 - INFO - parsed message as event=<ChannelEvents.reply: 'phx_reply'> topic='phoenix' payload=SuccessReplyMessage(status='ok', response=ReplyPostgresChanges(postgres_changes=None)) ref=None
2025-11-05 07:03:45,234 - INFO - ChannelStates.CLOSED
2025-11-05 07:04:00,250 - INFO - ChannelStates.CLOSED
2025-11-05 07:04:05,230 - INFO - send: {"event":"heartbeat","payload":{},"topic":"phoenix","ref":null,"join_ref":null}
2025-11-05 07:04:05,245 - INFO - receive: '{"ref":null,"event":"phx_reply","payload":{"status":"ok","response":{}},"topic":"phoenix"}'
2025-11-05 07:04:05,245 - INFO - parsed message as event=<ChannelEvents.reply: 'phx_reply'> topic='phoenix' payload=SuccessReplyMessage(status='ok', response=ReplyPostgresChanges(postgres_changes=None)) ref=None
2025-11-05 07:04:15,260 - INFO - ChannelStates.CLOSED
2025-11-05 07:04:30,241 - INFO - send: {"event":"heartbeat","payload":{},"topic":"phoenix","ref":null,"join_ref":null}
2025-11-05 07:04:30,261 - INFO - ChannelStates.CLOSED
2025-11-05 07:04:45,276 - INFO - ChannelStates.CLOSED
2025-11-05 07:04:55,249 - INFO - send: {"event":"heartbeat","payload":{},"topic":"phoenix","ref":null,"join_ref":null}
2025-11-05 07:05:00,279 - INFO - ChannelStates.CLOSED
2025-11-05 07:05:15,291 - INFO - ChannelStates.CLOSED
2025-11-05 07:05:20,251 - INFO - send: {"event":"heartbeat","payload":{},"topic":"phoenix","ref":null,"join_ref":null}
2025-11-05 07:05:30,302 - INFO - ChannelStates.CLOSED
2025-11-05 07:05:45,264 - INFO - send: {"event":"heartbeat","payload":{},"topic":"phoenix","ref":null,"join_ref":null}
2025-11-05 07:05:45,302 - INFO - ChannelStates.CLOSED
2025-11-05 07:06:00,318 - INFO - ChannelStates.CLOSED
2025-11-05 07:06:10,274 - INFO - send: {"event":"heartbeat","payload":{},"topic":"phoenix","ref":null,"join_ref":null}

[o-santi]

Thank you for raising the issue. I've flagged this for help with the realtime team, for this may have something to do with the server side, but in the meantime, have you assured that this isn't a connectivity issue?

The realtime client will change its state to CLOSED if it isn't able to send a message through the socket, or if the socket is closed, so my first instinct is to think about this.

[SmaugPool]

have you assured that this isn't a connectivity issue

I use it on several servers in datacenters all over the world. It happens for all of them at some point. It might be normal sporadic world routing issues, even if its seems weird at this frequency, but I think auto-reconnect should reconnect anyway in this case, shouldn't it ? From my tests, it doesn't reconnect without some code handling it. It tries to reconnect, and the websocket reconnection works as seen in the logs, but the channel get closed anyway and does not reopen automatically.

Note that during auto-reconnect, there is again the unexpected TIMED_OUT #1297 issue, so this might be related.

[o-santi]

I don't think the time out issue was related, unless something inside your code is registering a timeout callback that could cause this (the reproduction doesn't). Instead, I'm more keen to thinking that this might be due to the program slowly eating up RAM until something goes wrong, specially given you mentioned that issue only happens after a couple hours/days.

Do you have any information on that?

[SmaugPool]

I don't think it's a memory issue. Here is a test process with the issue

USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
supa      897281  0.0  0.6 221156 53888 ?        Ssl  Nov07   0:16 python3

I don't think it's related to file descriptor exhaustion or like either:

COMMAND    PID  FD      TYPE             DEVICE SIZE/OFF     NODE NAME
python3 897281 cwd       DIR                8,1     4096        2 /
python3 897281 rtd       DIR                8,1     4096        2 /
python3 897281 txt       REG                8,1  5937768   129445 /usr/bin/python3.10
python3 897281 mem       REG                8,1   642864     5139 /var/lib/.../venv/lib/python3.10/site-packages/propcache/_helpers_c.cpython-310-x86_64-linux-gnu.so
python3 897281 mem       REG                8,1  1031960     9671 /var/lib/.../venv/lib/python3.10/site-packages/yarl/_quoting_c.cpython-310-x86_64-linux-gnu.so
python3 897281 mem       REG                8,1   788880     5400 /var/lib/.../venv/lib/python3.10/site-packages/multidict/_multidict.cpython-310-x86_64-linux-gnu.so
python3 897281 mem       REG                8,1   339896     8715 /var/lib/.../venv/lib/python3.10/site-packages/_cffi_backend.cpython-310-x86_64-linux-gnu.so
python3 897281 mem       REG                8,1 12923664     9020 /var/lib/.../venv/lib/python3.10/site-packages/cryptography/hazmat/bindings/_rust.abi3.so
python3 897281 mem       REG                8,1    44624   147850 /usr/lib/python3.10/lib-dynload/_zoneinfo.cpython-310-x86_64-linux-gnu.so
python3 897281 mem       REG                8,1   187200   138959 /usr/lib/x86_64-linux-gnu/libmpdec.so.2.5.1
python3 897281 mem       REG                8,1   180632   147832 /usr/lib/python3.10/lib-dynload/_decimal.cpython-310-x86_64-linux-gnu.so
python3 897281 mem       REG                8,1   125488   138908 /usr/lib/x86_64-linux-gnu/libgcc_s.so.1
python3 897281 mem       REG                8,1  4768792     8570 /var/lib/.../venv/lib/python3.10/site-packages/pydantic_core/_pydantic_core.cpython-310-x86_64-linux-gnu.so
python3 897281 mem       REG                8,1   170456   138945 /usr/lib/x86_64-linux-gnu/liblzma.so.5.2.5
python3 897281 mem       REG                8,1    63728   151398 /usr/lib/python3.10/lib-dynload/_hashlib.cpython-310-x86_64-linux-gnu.so
python3 897281 mem       REG                8,1  4455728   128780 /usr/lib/x86_64-linux-gnu/libcrypto.so.3
python3 897281 mem       REG                8,1   667864   128875 /usr/lib/x86_64-linux-gnu/libssl.so.3
python3 897281 mem       REG                8,1   225296   151403 /usr/lib/python3.10/lib-dynload/_ssl.cpython-310-x86_64-linux-gnu.so
python3 897281 mem       REG                8,1    23664   147839 /usr/lib/python3.10/lib-dynload/_queue.cpython-310-x86_64-linux-gnu.so
python3 897281 mem       REG                8,1    21448   133633 /usr/lib/x86_64-linux-gnu/libpthread.so.0
python3 897281 mem       REG                8,1    14664   133635 /usr/lib/x86_64-linux-gnu/librt.so.1
python3 897281 mem       REG                8,1    30936   128610 /usr/lib/x86_64-linux-gnu/libuuid.so.1.3.0
python3 897281 mem       REG                8,1    33736     8156 /var/lib/.../venv/lib/python3.10/site-packages/websockets/speedups.cpython-310-x86_64-linux-gnu.so
python3 897281 mem       REG                8,1    45240   147835 /usr/lib/python3.10/lib-dynload/_lzma.cpython-310-x86_64-linux-gnu.so
python3 897281 mem       REG                8,1    74848   138836 /usr/lib/x86_64-linux-gnu/libbz2.so.1.0.4
python3 897281 mem       REG                8,1    14688   147847 /usr/lib/python3.10/lib-dynload/_uuid.cpython-310-x86_64-linux-gnu.so
python3 897281 mem       REG                8,1    49160   147833 /usr/lib/python3.10/lib-dynload/_json.cpython-310-x86_64-linux-gnu.so
python3 897281 mem       REG                8,1    32120   147818 /usr/lib/python3.10/lib-dynload/_bz2.cpython-310-x86_64-linux-gnu.so
python3 897281 mem       REG                8,1    78624   147817 /usr/lib/python3.10/lib-dynload/_asyncio.cpython-310-x86_64-linux-gnu.so
python3 897281 mem       REG                8,1    14536   147825 /usr/lib/python3.10/lib-dynload/_contextvars.cpython-310-x86_64-linux-gnu.so
python3 897281 mem       REG                8,1    27002   148451 /usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache
python3 897281 mem       REG                8,1  3048928   131014 /usr/lib/locale/locale-archive
python3 897281 mem       REG                8,1  2220400   133621 /usr/lib/x86_64-linux-gnu/libc.so.6
python3 897281 mem       REG                8,1   108936   139094 /usr/lib/x86_64-linux-gnu/libz.so.1.2.11
python3 897281 mem       REG                8,1   194872   128188 /usr/lib/x86_64-linux-gnu/libexpat.so.1.8.7
python3 897281 mem       REG                8,1   940560   133624 /usr/lib/x86_64-linux-gnu/libm.so.6
python3 897281 mem       REG                8,1    14672   151400 /usr/lib/python3.10/lib-dynload/_opcode.cpython-310-x86_64-linux-gnu.so
python3 897281 mem       REG                8,1   240936   128310 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
python3 897281   0r      CHR                1,3      0t0        5 /dev/null
python3 897281   1u     unix 0xffff938f74596ec0      0t0 91734280 type=STREAM
python3 897281   2u     unix 0xffff938f74596ec0      0t0 91734280 type=STREAM
python3 897281   3u  a_inode               0,14        0    13497 [eventpoll]
python3 897281   4u     unix 0xffff93904708c000      0t0 91825001 type=STREAM
python3 897281   5u     unix 0xffff93904708ea80      0t0 91825002 type=STREAM
python3 897281   6u     IPv4           91802764      0t0      TCP static.xx.xx.xx.xx.clients.your-server.de:33534->xx.xx.xx.xx:https (CLOSE_WAIT)
python3 897281   7u     IPv4           91734292      0t0      TCP static.xx.xx.xx.xx.clients.your-server.de:36810->xx.xx.xx.xx:https (CLOSE_WAIT)
python3 897281   8u     IPv4           91825536      0t0      TCP static.xx.xx.xx.xx.clients.your-server.de:35232->xx.xx.xx.xx:https (CLOSE_WAIT)
python3 897281   9u     IPv4           91802771      0t0      TCP static.xx.xx.xx.xx.clients.your-server.de:33562->xx.xx.xx.xx:https (CLOSE_WAIT)
python3 897281  11u     IPv4           91825006      0t0      TCP static.xx.xx.xx.xx.clients.your-server.de:35246->xx.xx.xx.xx:https (CLOSE_WAIT)

What's weird is that in the current case, the channel is not even closed, here are the previous logs:

2025-11-07 17:52:01,142 - INFO - subscription status=SUBSCRIBED err=None    
2025-11-07 17:52:11,126 - WARNING - subscription status=TIMED_OUT err=None    
2025-11-07 17:52:25,970 - INFO - send: {"event":"heartbeat","payload":{},"topic":"phoenix","ref":null,"join_ref":null}    
2025-11-07 17:52:25,974 - INFO - receive: '{"ref":null,"event":"phx_reply","payload":{"status":"ok","response":{}},"topic":"phoenix"}'    
2025-11-07 17:52:25,975 - INFO - parsed message as event=<ChannelEvents.reply: 'phx_reply'> topic='phoenix' payload=SuccessReplyMessage(status='ok', response=ReplyPostgresChanges(postgres_changes=None)) ref=None 
...
2025-11-07 17:52:25,970 - INFO - send: {"event":"heartbeat","payload":{},"topic":"phoenix","ref":null,"join_ref":null}
2025-11-07 17:52:25,974 - INFO - receive: '{"ref":null,"event":"phx_reply","payload":{"status":"ok","response":{}},"topic":"phoenix"}'
2025-11-07 17:52:25,975 - INFO - parsed message as event=<ChannelEvents.reply: 'phx_reply'> topic='phoenix' payload=SuccessReplyMessage(status='ok', response=ReplyPostgresChanges(postgres_changes=None)) ref=None
2025-11-07 17:52:50,972 - INFO - send: {"event":"heartbeat","payload":{},"topic":"phoenix","ref":null,"join_ref":null}
2025-11-07 17:52:50,984 - INFO - receive: '{"ref":null,"event":"phx_reply","payload":{"status":"ok","response":{}},"topic":"phoenix"}'
2025-11-07 17:52:50,984 - INFO - parsed message as event=<ChannelEvents.reply: 'phx_reply'> topic='phoenix' payload=SuccessReplyMessage(status='ok', response=ReplyPostgresChanges(postgres_changes=None)) ref=None
2025-11-07 17:53:15,974 - INFO - send: {"event":"heartbeat","payload":{},"topic":"phoenix","ref":null,"join_ref":null}
...
2025-11-07 19:27:51,361 - INFO - parsed message as event=<ChannelEvents.reply: 'phx_reply'> topic='phoenix' payload=SuccessReplyMessage(status='ok', response=ReplyPostgresChanges(postgres_changes=None)) ref=None
2025-11-07 19:28:16,352 - INFO - send: {"event":"heartbeat","payload":{},"topic":"phoenix","ref":null,"join_ref":null}
2025-11-07 19:28:41,353 - INFO - send: {"event":"heartbeat","payload":{},"topic":"phoenix","ref":null,"join_ref":null}
2025-11-07 19:29:06,354 - INFO - send: {"event":"heartbeat","payload":{},"topic":"phoenix","ref":null,"join_ref":null}
2025-11-07 19:29:31,355 - INFO - send: {"event":"heartbeat","payload":{},"topic":"phoenix","ref":null,"join_ref":null}
2025-11-07 19:29:56,357 - INFO - send: {"event":"heartbeat","payload":{},"topic":"phoenix","ref":null,"join_ref":null}
2025-11-07 19:30:21,358 - INFO - send: {"event":"heartbeat","payload":{},"topic":"phoenix","ref":null,"join_ref":null}
...

And I have the following loop:

    while True:
        if channel.is_closed or channel.is_errored:
            logging.warning("channel closed or errored")
            return
        await asyncio.sleep(1)

I will update to last version to avoid the TIMED_OUT issue and run the test again.

[SmaugPool]

For now, I'm not able to reproduce with version 2.24.0. It has been running on 4 servers since 2 days ago without the issue. I will continue monitoring.

[SmaugPool]

It happened again with supabase 2.24.0, this time after 3 days (sometimes it just takes hours, sometimes days).

The heartbeat stopped without any error raised despite the channel state still being ChannelStates.JOINED.

More critically, the channel is in JOINED state but is not actually receiving any messages.

Here is the server with the issue when the heartbeat stops:

2025-11-11 14:53:28,550 - INFO - send: {"event":"heartbeat","payload":{},"topic":"phoenix","ref":null,"join_ref":null}    
2025-11-11 14:53:28,550 - INFO - channel: joined    
2025-11-11 14:53:28,567 - INFO - receive: '{"ref":null,"event":"phx_reply","payload":{"status":"ok","response":{}},"topic":"phoenix"}'    
2025-11-11 14:53:28,568 - INFO - parsed message as event=<ChannelEvents.reply: 'phx_reply'> topic='phoenix' payload=SuccessReplyMessage(status='ok', response=ReplyPostgresChanges(postgres_changes=None)) ref=None    
2025-11-11 14:53:53,556 - INFO - send: {"event":"heartbeat","payload":{},"topic":"phoenix","ref":null,"join_ref":null}    
2025-11-11 14:53:53,556 - INFO - channel: joined    
2025-11-11 14:53:53,573 - INFO - receive: '{"ref":null,"event":"phx_reply","payload":{"status":"ok","response":{}},"topic":"phoenix"}'    
2025-11-11 14:53:53,573 - INFO - parsed message as event=<ChannelEvents.reply: 'phx_reply'> topic='phoenix' payload=SuccessReplyMessage(status='ok', response=ReplyPostgresChanges(postgres_changes=None)) ref=None    
2025-11-11 14:54:18,567 - INFO - send: {"event":"heartbeat","payload":{},"topic":"phoenix","ref":null,"join_ref":null}    
2025-11-11 14:54:18,567 - INFO - channel: joined    
2025-11-11 14:54:43,592 - INFO - send: {"event":"heartbeat","payload":{},"topic":"phoenix","ref":null,"join_ref":null}    
2025-11-11 14:54:43,593 - INFO - channel: joined    
2025-11-11 14:55:08,618 - INFO - send: {"event":"heartbeat","payload":{},"topic":"phoenix","ref":null,"join_ref":null}    
2025-11-11 14:55:08,619 - INFO - channel: joined    
2025-11-11 14:55:33,644 - INFO - send: {"event":"heartbeat","payload":{},"topic":"phoenix","ref":null,"join_ref":null}    
2025-11-11 14:55:33,645 - INFO - channel: joined    
2025-11-11 14:55:58,651 - INFO - send: {"event":"heartbeat","payload":{},"topic":"phoenix","ref":null,"join_ref":null}    
2025-11-11 14:55:58,652 - INFO - channel: joined    
2025-11-11 14:56:23,677 - INFO - send: {"event":"heartbeat","payload":{},"topic":"phoenix","ref":null,"join_ref":null}    
2025-11-11 14:56:23,677 - INFO - channel: joined    
2025-11-11 14:56:48,691 - INFO - send: {"event":"heartbeat","payload":{},"topic":"phoenix","ref":null,"join_ref":null}    
2025-11-11 14:56:48,691 - INFO - channel: joined    
2025-11-11 14:57:13,715 - INFO - send: {"event":"heartbeat","payload":{},"topic":"phoenix","ref":null,"join_ref":null}    
2025-11-11 14:57:13,715 - INFO - channel: joined  
...

With nothing happening at the time of the broadcast:

2025-11-12 08:46:01,993 - INFO - send: {"event":"heartbeat","payload":{},"topic":"phoenix","ref":null,"join_ref":null}
2025-11-12 08:46:01,993 - INFO - channel: joined
2025-11-12 08:46:27,019 - INFO - send: {"event":"heartbeat","payload":{},"topic":"phoenix","ref":null,"join_ref":null}
2025-11-12 08:46:27,019 - INFO - channel: joined
2025-11-12 08:46:52,045 - INFO - send: {"event":"heartbeat","payload":{},"topic":"phoenix","ref":null,"join_ref":null}
2025-11-12 08:46:52,045 - INFO - channel: joined
2025-11-12 08:47:17,071 - INFO - send: {"event":"heartbeat","payload":{},"topic":"phoenix","ref":null,"join_ref":null}
2025-11-12 08:47:17,071 - INFO - channel: joined

Compared to another server that still receives broadcast messages:

2025-11-12 08:46:25,674 - INFO - send: {"event":"heartbeat","payload":{},"topic":"phoenix","ref":null,"join_ref":null}
2025-11-12 08:46:25,675 - INFO - channel: joined
2025-11-12 08:46:25,697 - INFO - receive: '{"ref":null,"event":"phx_reply","payload":{"status":"ok","response":{}},"topic":"phoenix"}'
2025-11-12 08:46:25,698 - INFO - parsed message as event=<ChannelEvents.reply: 'phx_reply'> topic='phoenix' payload=SuccessReplyMessage(status='ok', response=ReplyPostgresChanges(postgres_changes=None)) ref=None
2025-11-12 08:46:43,395 - INFO - receive: '{"ref":null,"event":"broadcast","payload":{"event":"INSERT","meta":{"id":"...                                                                                                          
2025-11-12 08:46:43,395 - INFO - parsed message as event=<ChannelEvents.broadcast: 'broadcast'> topic='realtime:gateways' payload={'event': 'INSERT', 'payload': {'id': ...

When looking at connections with netstat or ss, the one in failure does not have any established connection anymore.

The test uses the following main loop:

async def main(args):
    client = AsyncClient(url, key)
    await client.realtime.connect()

    def on_subscribe(status, err):
        if status == RealtimeSubscribeStates.SUBSCRIBED:
            logging.info(f"subscription status={status} err={err}")
            channel.on_broadcast("INSERT", on_message)
            channel.on_broadcast("UPDATE", on_message)
            channel.on_broadcast("DELETE", on_message)
        else:
            logging.warning(f"subscription status={status} err={err}")

    def on_message(msg):
        logging.info(f"{msg['event']} {msg['payload']['table']}")

    await channel.subscribe(on_subscribe)

    while True:
        logging.info(f"channel: {channel.state}")
        if channel.is_closed or channel.is_errored:
            logging.warning("channel closed or errored")
            return
        await asyncio.sleep(25)

while True:
    asyncio.run(main(args))

So:

1. Closed sockets are not always detected nor handled
2. Channel state can incorrectly be JOINED despite the low level connection being closed
3. Heartbeat reply (or lack of) is ignored and not used to detect and handle connection issues

[SmaugPool]

With another server, the websocket connection got closed and auto-reconnect failed, but subscribe callback is not called and the channel stays in state JOINED too without any exception raised:

2025-11-12 10:40:28,570 - INFO - send: {"event":"heartbeat","payload":{},"topic":"phoenix","ref":null,"join_ref":null}                                                       
2025-11-12 10:40:28,570 - INFO - channel: joined                                                       
2025-11-12 10:40:48,602 - ERROR - WebSocket connection closed with code: 1006, reason:                                                       
2025-11-12 10:40:48,602 - INFO - Initiating auto-reconnect sequence...                                                       
2025-11-12 10:40:48,602 - INFO - Attempting to connect to WebSocket at wss://xxx.supabase.co/realtime/v1/websocket?apikey=sb_secret_xxx                                                       
2025-11-12 10:40:53,576 - INFO - channel: joined                                                       
2025-11-12 10:40:58,608 - ERROR - Connection attempt failed: timed out during opening handshake                                                       
2025-11-12 10:40:58,608 - INFO - Retry 1/5: Next attempt in 1.00s (backoff=1.0s)                                                       
2025-11-12 10:41:09,620 - ERROR - Connection attempt failed: timed out during opening handshake                                                       
2025-11-12 10:41:09,620 - INFO - Retry 2/5: Next attempt in 4.00s (backoff=2.0s)                                                       
2025-11-12 10:41:18,582 - INFO - channel: joined                                                       
2025-11-12 10:41:23,631 - ERROR - Connection attempt failed: timed out during opening handshake                                                       
2025-11-12 10:41:23,631 - INFO - Retry 3/5: Next attempt in 16.00s (backoff=4.0s)                                                       
2025-11-12 10:41:43,587 - INFO - channel: joined                                                       
2025-11-12 10:41:49,639 - ERROR - Connection attempt failed: timed out during opening handshake                                                       
2025-11-12 10:41:49,639 - INFO - Retry 4/5: Next attempt in 64.00s (backoff=8.0s)                                                       
2025-11-12 10:42:08,607 - INFO - channel: joined                                                       
2025-11-12 10:42:33,630 - INFO - channel: joined                                                       
2025-11-12 10:42:58,636 - INFO - channel: joined                                                       
2025-11-12 10:43:03,649 - ERROR - Connection attempt failed: timed out during opening handshake                                                       
2025-11-12 10:43:03,650 - ERROR - Connection failed permanently after 5 attempts. Error: timed out during opening handshake                                                       
2025-11-12 10:43:23,657 - INFO - channel: joined                                                       
2025-11-12 10:43:48,682 - INFO - channel: joined                                                       
2025-11-12 10:44:13,708 - INFO - channel: joined                                                       
2025-11-12 10:44:38,733 - INFO - channel: joined                                                       
2025-11-12 10:45:03,755 - INFO - channel: joined                                                       
2025-11-12 10:45:28,770 - INFO - channel: joined                                                       
2025-11-12 10:45:53,780 - INFO - channel: joined                                                       
2025-11-12 10:46:18,806 - INFO - channel: joined  

[SmaugPool]

I reproduced it with more debug, I changed the main loop to:

    while True:
        logging.info(
            f"channel={channel.state} ws={channel.socket._ws_connection.state} close_code={channel.socket._ws_connection.close_code} close_reason={channel.socket._ws_connection.close_reason}"
        )
        if channel.is_closed or channel.is_errored:
            logging.warning("channel closed or errored")
            return
        await asyncio.sleep(25)

We can see that the websocket get closed silently with the channel state staying inconsistently to JOINED state:

2025-11-14 03:44:37,362 - INFO - send: {"event":"heartbeat","payload":{},"topic":"phoenix","ref":null,"join_ref":null}    
2025-11-14 03:44:37,363 - INFO - channel=joined ws=1 close_code=None close_reason=None    
2025-11-14 03:44:37,383 - INFO - receive: '{"ref":null,"event":"phx_reply","payload":{"status":"ok","response":{}},"topic":"phoenix"}'    
2025-11-14 03:44:37,383 - INFO - parsed message as event=<ChannelEvents.reply: 'phx_reply'> topic='phoenix' payload=SuccessReplyMessage(status='ok', response=ReplyPostgresC>    
2025-11-14 03:45:02,373 - INFO - send: {"event":"heartbeat","payload":{},"topic":"phoenix","ref":null,"join_ref":null}    
2025-11-14 03:45:02,373 - INFO - channel=joined ws=3 close_code=1001 close_reason=    
2025-11-14 03:45:27,399 - INFO - send: {"event":"heartbeat","payload":{},"topic":"phoenix","ref":null,"join_ref":null}    
2025-11-14 03:45:27,399 - INFO - channel=joined ws=3 close_code=1001 close_reason=    
2025-11-14 03:45:52,425 - INFO - send: {"event":"heartbeat","payload":{},"topic":"phoenix","ref":null,"join_ref":null}    
2025-11-14 03:45:52,425 - INFO - channel=joined ws=3 close_code=1001 close_reason=   

The _ws_connection.state goes from state OPEN=1 to state CLOSED=3 with close_code 1001 without subscription callback called or error raised:
https://websockets.readthedocs.io/en/stable/reference/sansio/common.html#websockets.protocol.State

Usually, disconnections are detected and lead to auto-reconnect, for example:

2025-11-13 17:26:38,434 - ERROR - WebSocket connection closed with code: 1006, reason:
2025-11-13 17:26:38,435 - INFO - Initiating auto-reconnect sequence...

But not in this case, which is the issue.