Listener randomly turns non-responsive after some time

[audunsolemdal]

Created issue here for more context #3100
was asked to create support ticket

Describe the bug

The listener works fine to start with and jobs are assigned as usual. After some time (days) it seems that the listener freezes. It seems random when this occurs. This results in requested jobs not being assigned.

Once the listener pod is deleted/recreated, jobs are instantly assigned as expected. Without restarting the listener pod it seems that the listener pod stops working without any output indicating that it failed. I do not see any blocked network traffic, so I do not believe that to be the issue.

Describe the expected behavior

Business as usual

Controller pod logs

Logs are completely empty in the relevant period

Runner pod logs

Listener logs:
--- several previous successful refreshing_client runs on an hourly basis before this
2023-11-22T18:35:24Z   INFO    refreshing_client    message queue token is expired during GetNextMessage, refreshing...
2023-11-22T18:35:24Z    INFO    refreshing token        {"githubConfigUrl": "https://github.com/myorg"}
2023-11-22T18:35:24Z    INFO    getting access token for GitHub App auth        {"accessTokenURL": "https://api.github.com/app/installations/myappid/access_tokens"}
2023-11-23T08:35:29Z    INFO    getting runner registration token       {"registrationTokenURL": "https://api.github.com/orgs/myorg/actions/runners/registration-token"}
2023-11-23T08:35:30Z    INFO    getting Actions tenant URL and JWT      {"registrationURL": "https://api.github.com/actions/runner-registration"}
-- No further output until pod is restarted