`ReverseProxyPlugin` doesn't return a response to the client

[eaftan]

Check FAQs
Please check Frequently Asked Questions
before opening a bug report.

Describe the bug

The included example ReverseProxyPlugin doesn't seem to work. It doesn't return any response to the client.

To Reproduce
Steps to reproduce the behavior:

1. In one terminal, run the following command: proxy --enable-reverse-proxy --plugins proxy.plugin.ReverseProxyPlugin
2. In another terminal, run this command to send a request: curl -v localhost:8899/get
3. The curl command never gets a response; it reports "Empty reply from server"

Here are the outputs I see on my machine when I do this.

Proxy

% proxy --enable-reverse-proxy --plugins proxy.plugin.ReverseProxyPlugin
2024-07-23 14:02:31,090 - pid:2514 [I] plugins.load:89 - Loaded plugin proxy.http.proxy.HttpProxyPlugin
2024-07-23 14:02:31,091 - pid:2514 [I] plugins.load:89 - Loaded plugin proxy.http.server.HttpWebServerPlugin
2024-07-23 14:02:31,094 - pid:2514 [I] plugins.load:89 - Loaded plugin proxy.http.server.reverse.ReverseProxy
2024-07-23 14:02:31,094 - pid:2514 [I] plugins.load:89 - Loaded plugin proxy.plugin.ReverseProxyPlugin
2024-07-23 14:02:43,471 - pid:2519 [I] reverse.on_access_log:168 - 127.0.0.1:65505 - GET /get - curl/8.6.0 -> None - 10198.24ms

Curl

% curl -v localhost:8899/get
* Host localhost:8899 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:8899...
* connect to ::1 port 8899 from ::1 port 65504 failed: Connection refused
*   Trying 127.0.0.1:8899...
* Connected to localhost (127.0.0.1) port 8899
> GET /get HTTP/1.1
> Host: localhost:8899
> User-Agent: curl/8.6.0
> Accept: */*
> 
* Empty reply from server
* Closing connection
curl: (52) Empty reply from server

Expected behavior
I expect the output that httpbin.org/get returns:

{
  "args": {}, 
  "headers": {
    "Accept": "*/*", 
    "Host": "httpbin.org", 
    "User-Agent": "curl/8.6.0", 
    "X-Amzn-Trace-Id": "Root=1-66a01aff-7ebaa35e304be47c40746c5b"
  }, 
  "origin": "24.7.24.238", 
  "url": "http://httpbin.org/get"
}

Version information
I see this behavior on two different machines.

• OS: MacOS 14.5 and Ubuntu Linux 20.04.1
• Browser: N/A
• Device: M1 Macbook Pro, an Azure VM for Ubuntu
• proxy.py Version: 2.4.5.dev9+ga7077cf

Additional context
Add any other context about the problem here.

Screenshots
If applicable, add screenshots to help explain your problem.

[abhinavsingh]

@eaftan Thank you very much for reporting this. In last release, I pushed a bug while adding support for dynamic routes. I went to trace back the changes and looks like I even modified tests to make it work thinking change was just a side effect of changing httpbin to httpbingo 🤦‍♂️

[abhinavsingh]

Looks into the tests a bit further, looks like there is an issue with httpbingo (just like httpbin) when using from GitHub workflows. I'll investigate into tests later, but for now reverse proxy with static routes must work fine. I'll make a new release early coming week.

[eaftan]

Thank you! 🙏

[creaexo]

Hello, @abhinavsingh!
I have the same problem as described above.

To Reproduce

Steps to reproduce the behavior:

Install in a new virtual environment proxy.py==2.4.7
In one terminal, run the following command: .venv3/bin/proxy --enable-reverse-proxy --plugins proxy.plugin.ReverseProxyPlugin
In another terminal, run this command to send a request: curl -v localhost:8899/get
The curl command never gets a response; it reports "Empty reply from server"

Proxy (I've shortened the error tracing a bit)

.venv3/bin/proxy --enable-reverse-proxy --plugins proxy.plugin.ReverseProxyPlugin
2024-09-18 19:36:35,182 - pid:115447 [I] plugins.load:89 - Loaded plugin proxy.http.proxy.HttpProxyPlugin
2024-09-18 19:36:35,182 - pid:115447 [I] plugins.load:89 - Loaded plugin proxy.http.server.HttpWebServerPlugin
2024-09-18 19:36:35,182 - pid:115447 [I] plugins.load:89 - Loaded plugin proxy.http.server.reverse.ReverseProxy
2024-09-18 19:36:35,182 - pid:115447 [I] plugins.load:89 - Loaded plugin proxy.plugin.ReverseProxyPlugin
2024-09-18 19:36:41,471 - pid:115454 [W] handler.handle_readables:240 - Exception when receiving from client 
"....................................................LONG ERROR TRACE................................................................."
    self.upstream.wrap(
  File "/home/path_to_my_project/proxer/.venv3/lib/python3.11/site-packages/proxy/core/connection/server.py", line 55, in wrap
    ctx = ssl.create_default_context(ssl.Purpose.SERVER_AUTH, cafile=ca_file)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/ssl.py", line 770, in create_default_context
    context.load_verify_locations(cafile, capath, cadata)
FileNotFoundError: [Errno 2] No such file or directory
2024-09-18 19:36:41,472 - pid:115454 [I] reverse.on_access_log:169 - 127.0.0.1:38500 - GET /get - curl/7.81.0 -> None - 371.14ms

Curl

curl -v localhost:8899/get
*   Trying 127.0.0.1:8899...
* Connected to localhost (127.0.0.1) port 8899 (#0)
> GET /get HTTP/1.1
> Host: localhost:8899
> User-Agent: curl/7.81.0
> Accept: */*
> 
* Empty reply from server
* Closing connection 0
curl: (52) Empty reply from server

Version information

OS: Ubuntu 22.04.4 LTS
Browser: N/A
proxy.py Version: 2.4.5, 2.4.7

[abhinavsingh]

@creaexo Looks like Ubuntu doesn't have necessary CA certificates installed. Try:

pip install certifi

If above alone does not work, also try:

sudo apt-get install ca-certificates

Please let me know.

[creaexo]

Thank you for your reply! After the above recommendations, a successful response began to come alternating with an unsuccessful one. Why do you think not every answer is successful?

Proxy

.venv3/bin/proxy --enable-reverse-proxy --plugins proxy.plugin.ReverseProxyPlugin
2024-09-19 11:52:31,136 - pid:120607 [I] plugins.load:89 - Loaded plugin proxy.http.proxy.HttpProxyPlugin
2024-09-19 11:52:31,136 - pid:120607 [I] plugins.load:89 - Loaded plugin proxy.http.server.HttpWebServerPlugin
2024-09-19 11:52:31,136 - pid:120607 [I] plugins.load:89 - Loaded plugin proxy.http.server.reverse.ReverseProxy
2024-09-19 11:52:31,136 - pid:120607 [I] plugins.load:89 - Loaded plugin proxy.plugin.ReverseProxyPlugin
2024-09-19 11:52:34,166 - pid:120614 [I] reverse.on_access_log:169 - 127.0.0.1:38808 - GET /get - curl/7.81.0 -> None - 74.46ms
2024-09-19 11:52:38,290 - pid:120614 [I] tcp_upstream.read_from_descriptors:87 - Upstream SSLWantReadError, will retry
2024-09-19 11:52:38,550 - pid:120614 [I] reverse.on_access_log:169 - 127.0.0.1:38820 - GET /get - curl/7.81.0 -> None - 408.62ms

Two curl requests are executed, one after the other

curl -v localhost:8899/get

*   Trying 127.0.0.1:8899...
* Connected to localhost (127.0.0.1) port 8899 (#0)
> GET /get HTTP/1.1
> Host: localhost:8899
> User-Agent: curl/7.81.0
> Accept: */*
> 
* Empty reply from server
* Closing connection 0
curl: (52) Empty reply from server


curl -v localhost:8899/get

*   Trying 127.0.0.1:8899...
* Connected to localhost (127.0.0.1) port 8899 (#0)
> GET /get HTTP/1.1
> Host: localhost:8899
> User-Agent: curl/7.81.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< access-control-allow-credentials: true
< access-control-allow-origin: *
< content-type: application/json; charset=utf-8
< date: Thu, 19 Sep 2024 06:52:38 GMT
< content-length: 555
< server: Fly/a5e9a2096 (2024-09-17)
< via: 1.1 fly.io
< fly-request-id: 01J84GVS02PZ98XB43E9335WJ5-waw
< 
{
  "args": {},
  "headers": {
    "Accept": [
      "*/*"
    ],
    "Host": [
      "localhost:8899"
    ],
    "User-Agent": [
      "curl/7.81.0"
    ],
    "Via": [
      "1.1 fly.io"
    ],
    "X-Forwarded-For": [
      "my_ip, my_ip"
    ],
    "X-Forwarded-Port": [
      "443"
    ],
    "X-Forwarded-Proto": [
      "https"
    ],
    "X-Forwarded-Ssl": [
      "on"
    ],
    "X-Request-Start": [
      "t=1726728758274304"
    ]
  },
  "method": "GET",
  "origin": "my_ip",
  "url": "https://localhost:8899/get"
}
* Connection #0 to host localhost left intact

[creaexo]

@abhinavsingh, also I have problem with standard ReverseProxyPlugin.handle_route, which must redirect http://localhost:8899/get/1 to http://httpbingo.org/get?id=1.

Proxy

.venv3/bin/proxy --enable-reverse-proxy --plugins proxy.plugin.ReverseProxyPlugin
2024-09-19 12:20:01,701 - pid:122607 [I] plugins.load:89 - Loaded plugin proxy.http.proxy.HttpProxyPlugin
2024-09-19 12:20:01,702 - pid:122607 [I] plugins.load:89 - Loaded plugin proxy.http.server.HttpWebServerPlugin
2024-09-19 12:20:01,702 - pid:122607 [I] plugins.load:89 - Loaded plugin proxy.http.server.reverse.ReverseProxy
2024-09-19 12:20:01,702 - pid:122607 [I] plugins.load:89 - Loaded plugin proxy.plugin.ReverseProxyPlugin
2024-09-19 12:20:17,308 - pid:122611 [I] reverse.on_access_log:169 - 127.0.0.1:33674 - GET /get?id=1 - curl/7.81.0 -> http://httpbingo.org/get?id=1 - 79.78ms
2024-09-19 12:20:25,383 - pid:122618 [I] reverse.on_access_log:169 - 127.0.0.1:58530 - GET /get?id=1 - curl/7.81.0 -> http://httpbingo.org/get?id=1 - 81.17ms
2024-09-19 12:20:27,641 - pid:122619 [I] reverse.on_access_log:169 - 127.0.0.1:58542 - GET /get?id=1 - curl/7.81.0 -> http://httpbingo.org/get?id=1 - 74.19ms

Thre curl requests

curl -v http://127.0.0.1:8899/get/1                                     
*   Trying 127.0.0.1:8899...
* Connected to 127.0.0.1 (127.0.0.1) port 8899 (#0)
> GET /get/1 HTTP/1.1
> Host: 127.0.0.1:8899
> User-Agent: curl/7.81.0
> Accept: */*
> 
* Empty reply from server
* Closing connection 0
curl: (52) Empty reply from server

curl -v http://127.0.0.1:8899/get/1
*   Trying 127.0.0.1:8899...
* Connected to 127.0.0.1 (127.0.0.1) port 8899 (#0)
> GET /get/1 HTTP/1.1
> Host: 127.0.0.1:8899
> User-Agent: curl/7.81.0
> Accept: */*
> 
* Empty reply from server
* Closing connection 0
curl: (52) Empty reply from server

curl -v http://127.0.0.1:8899/get/1
*   Trying 127.0.0.1:8899...
* Connected to 127.0.0.1 (127.0.0.1) port 8899 (#0)
> GET /get/1 HTTP/1.1
> Host: 127.0.0.1:8899
> User-Agent: curl/7.81.0
> Accept: */*
> 
* Empty reply from server
* Closing connection 0
curl: (52) Empty reply from server

[abhinavsingh]

Unsure what the issue might be at your end, but I am unable to reproduce the empty reply from server on my Mac laptop. I tried about 10-15 hits and got response for all the requests.

I see in your logs you encountered Upstream SSLWantReadError, will retry, are you seeing it repeatedly every now and then?

[creaexo]

Yes, I regularly see the message Upstream SSLWantReadError, will retry, when using ReverseProxyPlugin

[creaexo]

After an unsuccessful attempt to implement a redirect scenario with http://127.0.0.1:8899/get/1, on https://httpbingo.org/get?id=1 in the browser, I decided to add prints to the synchronous methods proxy/plugin/reverse_proxy.ReverseProxy.
When I try run http://localhost:8899/get in browser and take expected result, returns this methods names:

write_to_descriptors
read_from_descriptors
handle_upstream_data
write_to_descriptors
read_from_descriptors
handle_upstream_data
write_to_descriptors
read_from_descriptors
write_to_descriptors
read_from_descriptors
on_client_connection_close
on_access_log

When I try run http://localhost:8899/get/1 in browser and always take unexpected result (empty response), returns this methods names:

routes
handle_request
on_client_connection_close
on_access_log

Also I haven't any exceptions.

Maybe this will be a clue...