tests(k8s): handle port not being open when proxying into k8s (#1414)

The best fix is to make the SocketProxy more robust so that it doesn't crash when a connection attempt fails. By catching the expected BrokenPipeError, the proxy can simply discard the failed connection and continue listening for the next attempt from the Wait.until loop. This turns your test from a "hope it works" scenario into a reliable polling check.

Author: jiridanek

tests/containers/socket_proxy.py

@@ -90,11 +90,19 @@ def listen_and_serve_until_canceled(self):
                 # ISSUE-922: socket.accept() blocks, so if cancel() did not come very fast, we'd loop over and block
                 if self.server_socket in readable:
                     client_socket, addr = self.server_socket.accept()
-                    logging.info(f"Accepted connection from {addr[0]}:{addr[1]}")
-                    # handle client synchronously, which means that there can be at most one at a time
-                    self._handle_client(client_socket)
+                    logging.info(f"Proxy accepted connection from {addr[0]}:{addr[1]}")
+                    try:
+                        # handle client synchronously, which means that there can be at most one at a time
+                        self._handle_client(client_socket)
+                    except (BrokenPipeError, ConnectionResetError) as e:
+                        # BrokenPipeError happens when the proxy connects to the pod, but the service inside is not yet listening.
+                        # The client (Wait.until) will retry.
+                        logging.info(f"Proxy connection to remote failed, likely due to service not being ready: {e}")
+                        # The client_socket is closed by the `with` statement in `_handle_client`.
+                        # We continue the loop to accept the next connection attempt.
+                        continue
         except Exception as e:
-            logging.exception("Proxying failed to listen", exc_info=e)
+            logging.exception("Proxying failed with an unhandled exception", exc_info=e)
             raise
         finally:
             self.server_socket.close()