investigate listen routine after reconnect behavior

[switchupcb]

Problem

Session listen routine shutdown at a random point after reconnecting, which causes the bot to go offline.

Information

{"level":"info","time":"2025-02-20T22:38:53.563828089Z","session":"3857a104cdfd6f67916ac62cd8fe5394","message":"reconnecting session due to Opcode 7 Reconnect"}
{"level":"info","time":"2025-02-20T22:38:54.625509618Z","session":"3857a104cdfd6f67916ac62cd8fe5394","client":"948649712649662524","message":"connecting session"}
{"level":"info","time":"2025-02-20T22:38:54.714353538Z","session":"3857a104cdfd6f67916ac62cd8fe5394","message":"received Resumed event"}
{"level":"info","time":"2025-02-20T23:17:38.520428786Z","session":"3857a104cdfd6f67916ac62cd8fe5394","message":"closed listen routine"}

This log is not copied and pasted as-is.

Here is the behavior after the listen routine shuts down.

{"level":"info","time":"2025-02-20T23:18:06.012564258Z","session":"3857a104cdfd6f67916ac62cd8fe5394","message":"queued heartbeat"}
{"level":"info","time":"2025-02-20T23:18:47.262763072Z","session":"3857a104cdfd6f67916ac62cd8fe5394","message":"queued heartbeat"}
{"level":"info","time":"2025-02-20T23:19:28.513469149Z","session":"3857a104cdfd6f67916ac62cd8fe5394","message":"queued heartbeat"}

This behavior shows the pulse attempting to queue heartbeats, but cannot hear anything back since there is no listen routine. So, the bot goes offline on Discord as the bot doesn't beat.

The bot should also shutdown when a routine returns an error, which also cancels the the context. So, context isn't closed but listen returns nil.

What could cause a listen routine to go down?

Here are some things.

New refactor involves context changes.
-- context: https://github.com/switchupcb/disgo/blob/v10/wrapper/session_routine_manager.go#L180
-- context: https://github.com/switchupcb/disgo/blob/v10/wrapper/session_routine_manager_actions.go#L83

Reproducing the issue before any changes are made.

[switchupcb]

There is an issue with s.Context function with a new listener routine and how the routine handles errors.

How does this issue happen?

Here is the error that returned after a reconnect.

{"level":"info","time":"2025-02-21T06:35:16.230873728Z","session":"95030dd9e9eb319ac84b05371f9b60fb","message":"received Ready event"}
{"level":"info","error":"failed to get reader: received close frame: status = StatusNormalClosure and reason = \"\"","message":"listen routine error"}

This error is returned from the following code.

if err = socket.Read(s.Context, s.Conn, payload); err != nil {
    break
}

You can trace this function to find the code which returns this error.

conn.Reader(ctx)

...

func (c *Conn) reader(ctx context.Context) (_ MessageType, _ io.Reader, err error) {
...

h, err := c.readLoop(ctx)

...

func (c *Conn) readLoop(ctx context.Context) (header, error) {

...

func (c *Conn) handleControl(ctx context.Context, h header) (err error) {
    ...
    err = fmt.Errorf("received close frame: %w", ce)

This error is referenced in the websocket library issue tracker: coder/websocket#403.

Yup, use https://pkg.go.dev/nhooyr.io/websocket#CloseStatus to check that it's a normal closure and if so, ignore the error.

The solution to handle this error is referenced in the websocket library issue tracker: https://github.com/coder/websocket/.issues/403#issuecomment-1720136980

• This code was removed during refactor Session #78, but must be added back to validateGatewayCloseError to indicate a "graceful" disconnect.
• This code can also be added to the listen routine so this error cannot be received by the manager errgroup, which increases the accuracy of reporting problematic errors experienced during disconnection.

However, this error is not the root cause of the issue because there should not be a closed connection immediately after receiving a ready event.

So, this issue occurs when the Ready event is received, which returns from the initial function in connect at https://github.com/switchupcb/disgo/blob/v10/wrapper/session_routine_manager_actions.go#L133. The listen routine is spawned and immediately receives a closure indicating "the connection is already closed" after receiving a ready event.

So, why is the connection is closed after receiving the ready event?

s.Context manipulation or s.Conn manipulation or Discord is closing the connection.

• s.Context manipulation could be due to fatcontexts or error occurring in connection which prompts a disconnection which passes through a currently undefined state equivalent to SessionStateConnectingWebsocket (but while reconnecting). So, disconnect is called with a normal closure and doesn't error, which means the manager is locked while waiting for other sessions to lock or some variation thereof.
• s.Conn manipulation would be due to Discord closing connection.

Reproducing the issue again before any changes are made.

[switchupcb]

So, why is the connection is closed after receiving the ready event?

There are multiple issues.

1. A diagram must be created to solve the problem with state code handling logic, which was not implemented to 100% in hindsight causing mutex deadlocks.
2. s.Conn close causes an error in the listen routine, so the listen routine returns an error upon close which cancels the manager's context.
3. Canceled manager context is either not recovered due to 1 or other issues.

Here are the steps to resolve this issue.

• Reproduce issue once more with production-level logging implemented.
• Create state diagram for session_manager.go.
• Use diagram to handle all unproblematic errors (e.g., listen routine) correctly.
• Use diagram to create separate s.Context from manager routine and managed routines.