Question / feature request: using truesocket recording mode for an in-process HTTP server

[jayaddison]

Hi again @mindflayer - thanks for your responses in sphinx-doc/sphinx#11324.

To recap (and for future readers): the improvement requested in that issue is to allow Sphinx to use session-based (pooled) HTTP requests instead of creating one TCP connection per request when it checks documentation hyperlinks. While investigating how to add tests for an implementation of that, I discovered mocket.

It turned out that mocket wasn't applicable for the implementation I attempted, because I wanted to retain both the test HTTP server and test HTTP client. In that plan, measurement of actual traffic from client to server was required, instead of what mocket generally provides: mocking of the server's responses (useful -- and in fact preferable -- in many other situations).

With that explained, the question / feature request is:

Would it be possible for mocket's recording mode to support a socket server that is in the same Python process as the socket client?

If helpful: I can provide example code that I've attempted this with, and the error output(s) that I've encountered. But I think I should ask the feasibility question first.

[mindflayer]

Yes, please share a snippet of code and I'll have a look at it.

[jayaddison]

Thank you. Here is a sample that recreates the situation:

# Sample code derived from: https://docs.python.org/3/library/http.server.html#http.server.SimpleHTTPRequestHandler and sphinx: https://github.com/sphinx-doc/sphinx.git
from contextlib import contextmanager
from http.server import SimpleHTTPRequestHandler, ThreadingHTTPServer
import mocket
from threading import Thread
from tempfile import mkdtemp
from urllib.request import urlopen


class RedirectHandler(SimpleHTTPRequestHandler):
    protocol_version = "HTTP/1.1"

    def do_GET(self):
        if self.path == "/origin":
            self.send_response(302, "Found")
            self.send_header("Location", "http://127.0.0.1:8000/destination")
            self.end_headers()

        if self.path == "/destination":
            self.send_response(200, "OK")
            self.end_headers()


class RedirectServer(Thread):
    def __init__(self):
        super().__init__()
        self.server = ThreadingHTTPServer(("127.0.0.1", 8000), RedirectHandler)

    def run(self):
        self.server.serve_forever(poll_interval=0.001)

    def close(self):
        self.server.shutdown()


@contextmanager
def redirect_server():
    server = RedirectServer()
    server.start()
    try:
        yield server
    finally:
        server.close()


with mocket.Mocketizer(truesocket_recording_dir=mkdtemp()):
    with redirect_server():
        urlopen("http://127.0.0.1:8000/origin")

[mindflayer]

The first problem I see is that your are spawning your server while already inside mocket, while on the contrary the server should bootstrap before we mock the socket module.

After I fixed that, I got a:

  File ".../python-mocket/mocket/mocket.py", line 366, in true_sendall
    self.true_socket.sendall(data, *args, **kwargs)
BrokenPipeError: [Errno 32] Broken pipe

and that's what I am trying to investigate now.

[mindflayer]

My bad, that was a rookie mistake. What I get instead is that the client gets stuck on:

  File ".../python-mocket/mocket/mocket.py", line 375, in true_sendall
    recv = self.true_socket.recv(self._buflen)

[jayaddison]

I've never felt like I completely understand socket-based programming (to the level of intuitively knowing how to write socket-based communication code), but what I remember is that although each socket can operate like a peer, generally a socket will self-categorize as either a 'server' or 'client' type of socket during initialization: the former will always bind to an address and port to listen on.

(I also forget why bind and listen are separate calls, but that's beside the point)

My guess here is that we only want mocket to perform custom logic for sockets that are categorized as 'client'. And to do that effectively, we need to identify the type of the socket before any ambiguous calls happen (such as send, recv -- basically, calls that are valid for both types of socket).

[mindflayer]

My guess here is that we only want mocket to perform custom logic for sockets that are categorized as 'client'.

I don't think there is a procedural way to distinguish between a client and a server socket, apart from when it starts to use its primitives. Also, servers use both server and client sockets.
Quoting from the same page:

The first thing to note, is that the web browser’s “client” socket
and the web server’s “client” socket are identical beasts.

See the examples from https://docs.python.org/3/howto/sockets.html#creating-a-socket

[jayaddison]

I don't think there is a procedural way to distinguish between a client and a server socket, apart from when it starts to use its primitives. Also, servers use both server and client sockets.

@mindflayer is that enough evidence to prove this feature request invalid? (I think that it's ok - maybe good - if it is)

[jayaddison]

My bad, that was a rookie mistake. What I get instead is that the client gets stuck on:

  File ".../python-mocket/mocket/mocket.py", line 375, in true_sendall
    recv = self.true_socket.recv(self._buflen)

After attempting an implementation here (essentially: a lot of if conditions to check whether the socket.socket-inherited object is in 'server mode', and to call the super-method if so), I've caught up to this change, and see the same behaviour here.

I've also added some debug print statements to check the progress of socket creation and startup.

If I read the code and behaviour correctly: the mocket library code is single-threaded, and the server-side socket (from the sample code above) is initialized and begins listening. The process gets stuck, though, because after the client issues a request (in recording mode -- with no existing response data that it can echo from), the client thread is going to wait forever, and the server process won't get any CPU-and-I/O time to receive the request and respond.

[mindflayer]

I'll try to spend a bit more time on it. No reason for closing it as invalid.