HTTP Server with Dual Interfaces

[DaneSlattery]

I have an ESP32s3 running Ethernet via EspSpiEth() and a WiFi Access Point.

When I first connect my laptop to the AP over WiFI, I can connect to the HTTP server and make requests.

When I then connect my laptop to an ethernet switch, I can no longer make any requests to the HTTP server, and I get responses like:
Error: read ECONNRESET

When I then re-connect my laptop to the AP over WiFi, all subsequent requests return
Error: read ECONNRESET.

I discovered this while trying to get the EspMdns() service to respond to the host name.local on all network interfaces (WIFI_STA, WIFI_AP, ETH_CL).

[ivmarkov]

I don't think this problem has anything to do with edge-http? Why are you opening it here?

[DaneSlattery]

I am not sure where the issue lies, but it is specific to the HTTP server response.

Perhaps the issue lies elsewhere.

I have run the following experiment to try and explain the issue:

The wifi interface get's ip 192.168.35.106. The Ethernet interface gets ip 192.168.35.104.

I will target the wifi IP and the ethernet IP using
`curl --location 'http://192.168.35.<wifi/eth>:80/api/relay'
I will also ping the device in each configuration.

1. Only WiFi connected, ethernet unplugged:

wifi:

curl --location 'http://192.168.35.106:80/api/relay'
"Open"%               
          
ping 192.168.35.106
PING 192.168.35.106 (192.168.35.106) 56(84) bytes of data.
64 bytes from 192.168.35.106: icmp_seq=1 ttl=64 time=33.5 ms
64 bytes from 192.168.35.106: icmp_seq=2 ttl=64 time=56.0 ms

ethernet:

curl --location 'http://192.168.35.104:80/api/relay'
curl: (7) Failed to connect to 192.168.35.104 port 80 after 14299 ms: Couldn't connect to server

ping 192.168.35.104
PING 192.168.35.104 (192.168.35.104) 56(84) bytes of data.
From 192.168.35.109 icmp_seq=1 Destination Host Unreachable
From 192.168.35.109 icmp_seq=2 Destination Host Unreachable

2. WiFi connected, Ethernet connected.

wifi:

curl --location 'http://192.168.35.106:80/api/relay'
"Open"%               
          
ping 192.168.35.106
PING 192.168.35.106 (192.168.35.106) 56(84) bytes of data.
64 bytes from 192.168.35.106: icmp_seq=1 ttl=64 time=207 ms
64 bytes from 192.168.35.106: icmp_seq=2 ttl=64 time=229 ms

ethernet:

curl --location 'http://192.168.35.104:80/api/relay'
"Open"%

ping 192.168.35.104
PING 192.168.35.104 (192.168.35.104) 56(84) bytes of data.
64 bytes from 192.168.35.104: icmp_seq=1 ttl=64 time=5.45 ms
64 bytes from 192.168.35.104: icmp_seq=2 ttl=64 time=5.84 ms

3. WiFi disconnected, Ethernet connected.

wifi:

curl --location 'http://192.168.35.106:80/api/relay'
curl: (7) Failed to connect to 192.168.35.106 port 80 after 12292 ms: Couldn't connect to server
          
ping 192.168.35.106
PING 192.168.35.106 (192.168.35.106) 56(84) bytes of data.
From 192.168.35.109 icmp_seq=6 Destination Host Unreachable
From 192.168.35.109 icmp_seq=7 Destination Host Unreachable

ethernet:

curl --location 'http://192.168.35.104:80/api/relay'
curl: (56) Recv failure: Connection reset by peer

ping 192.168.35.104
PING 192.168.35.104 (192.168.35.104) 56(84) bytes of data.
64 bytes from 192.168.35.104: icmp_seq=1 ttl=64 time=33.7 ms
64 bytes from 192.168.35.104: icmp_seq=2 ttl=64 time=5.16 ms

At this point, all requests will show Connection reset by peer.

I raised the issue here because I have another service on the same device on port 502 (modbus TCP). That service is run on an async-io tcp listener:
let listener = Async::<TcpListener>::bind("0.0.0.0:502")?;
and that never seems to fail.

The HTTP server on the other hand runs on

let addr = "0.0.0.0:80";
    info!("Running HTTP web server on {addr}");
    let acceptor = edge_nal_std::Stack::new()
        .bind(addr.parse().unwrap())
        .await
        .map_err(edge_http::io::Error::Io)?;

[DaneSlattery]

I am suspicious of the connection reset by peer part here. Underneath edge_nal_std::Stack::new is async_io::Async::<TcpListener>, so I'm not exactly sure where the difference lies.

TLDR is that my 502 service seems bulletproof across different network connectivity states,
while the http://:80 service seems less bulletproof, and I have no idea why.

[DaneSlattery]

Another curiousity is that this is far from consistent behaviour, which is why I previously asked about time out logic in the API (thank you again). It seems that eventually, the clog clears. The last clue I can give is from the logs:

W (2024-10-17 15:33:07.745) edge_http::io::server: Handler task 0: Error when handling request: Handler(Http(Io(Os { code: 104, kind: ConnectionReset, message: "Connection reset by peer" })))

W (2024-10-17 15:33:07.747) edge_http::io::server: Handler task 0: Error when closing the socket: Os { code: 128, kind: NotConnected, message: "Socket is not connected" }

W (2024-10-17 15:33:07.771) edge_http::io::server: Handler task 0: Error when handling request: Handler(Http(Io(Os { code: 104, kind: ConnectionReset, message: "Connection reset by peer" })))

W (2024-10-17 15:33:07.773) edge_http::io::server: Handler task 0: Error when closing the socket: Os { code: 128, kind: NotConnected, message: "Socket is not connected" }

W (2024-10-17 15:33:07.789) edge_http::io::server: Handler task 0: Error when handling request: Handler(Http(Io(Os { code: 104, kind: ConnectionReset, message: "Connection reset by peer" })))

W (2024-10-17 15:33:07.792) edge_http::io::server: Handler task 0: Error when closing the socket: Os { code: 128, kind: NotConnected, message: "Socket is not connected" }

W (2024-10-17 15:33:07.809) edge_http::io::server: Handler task 0: Error when handling request: Handler(Http(Io(Os { code: 104, kind: ConnectionReset, message: "Connection reset by peer" })))

W (2024-10-17 15:33:07.811) edge_http::io::server: Handler task 0: Error when closing the socket: Os { code: 128, kind: NotConnected, message: "Socket is not connected" }

[ivmarkov]

For these experiments, are you running edge-http from latest master, or an older version?

The versions before latest master did have a timeout of 6s at the connection start which could've caused the connection reset, while the new version doesn't, in that it is now your responsibility to put timeouts in the server, as documented (by wrapping the acceptor and/or by wrapping the complete request-response or handler).

I.e. did you run cargo update before your experiments?

[DaneSlattery]

I will run a cargo update, but I am using master and confirm.

I see. I think I misunderstood the timeouts, I thought they were still built in.