Adapters being flooded with half-closed connections, causing adapter instability

[aaronjwood]

This may be related to #99 and #81. I'm seeing that connections are not being reused when HA reaches out to the adapter.

192.168.1.199 is one of my adapters and 192.168.1.4 is my HA instance. Ignore the entries related to DNS, I have my firewall setup to enforce everything to go to my own DNS server so clients cannot break out of what is given to them via DHCP. Of course Fujitsu does not honor what the local DHCP server pushes out 😠

There are dozens and dozens and dozens more states on this page than what the screenshot shows. What's interesting (and annoying) is that the adapter doesn't seem to support connection reuse:

curl -v 192.168.1.199 192.168.1.199 > /dev/null 
* Established connection to 192.168.1.199 (192.168.1.199 port 80) from 192.168.1.63 port 64507
...
* Connection #0 to host 192.168.1.199:80 left intact
* Connection 0 seems to be dead
* shutting down connection #0
* Hostname 192.168.1.199 was found in DNS cache
*   Trying 192.168.1.199:80...
* Established connection to 192.168.1.199 (192.168.1.199 port 80) from 192.168.1.63 port 64508
...

The amount of ever-accumulating (but old ones eventually aging out) half-closed connections (FIN_WAIT_2) seems to cause instability with my adapters over time. I'm guessing since they are cheap little things they choke on handling even a small-ish amount of connections (the TCP/IP stack on the adapter itself should have these states) that pile up over time. I don't really know why all of these connections are in a half-closed state. Is it due to something the underlying library is doing?

I have an idea for working around the dumb adapters not supporting connection reuse. I threw this together to show a working example:

import socket
import  time

SOURCE_IP = '0.0.0.0'
SOURCE_PORT = 20091
DESTINATION_HOST = '192.168.1.199'
DESTINATION_PORT = 80

while True:
    try:
        with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
            s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
            s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEPORT, 1)
            s.bind((SOURCE_IP, SOURCE_PORT))
            s.connect((DESTINATION_HOST, DESTINATION_PORT))
            print(f"Connected from source port {SOURCE_PORT} to {DESTINATION_HOST}:{DESTINATION_PORT}")
            s.sendall("GET / HTTP/1.1\r\nHost: 192.168.1.199\r\nConnection: keep-alive\r\n\r\n".encode()) # This gives back a 200
            # We can't keep using the socket since the adapter forcefully closes the connection at this point.
            # Start over with a new connection via our loop but use the same source port that we did before.
    except Exception as e:
        print(f"Error binding to port or connecting: {e}")
    time.sleep(1)

Now if you run this you'll see that there are no more lingering half-closed connections, and the mess of accumulating half-closed connections are gone:

[ohshazbot]

Interesting discovery, will need to look into this harder, but I'm not surprised to hear this. We could be misusing aiohttp sessions that could be the root of the fin_wait_2 connections, will look at that harder, may need to adjust how sessions get constructed or may just be about trimming timeout parameters.

Just so I have a better understanding of what I'm looking at, these firewall diagnostics are from your network between HA & the fujitsu device or are these firewall metrics on your home assistant box itself?

[aaronjwood]

Just so I have a better understanding of what I'm looking at, these firewall diagnostics are from your network between HA & the fujitsu device or are these firewall metrics on your home assistant box itself?

Yup, the former. This is on my router which == opnsense.

[ohshazbot]

Alright, then that does mean at minimum that tcp connection settings could be tweaked to reduce the lifespan of these as well. I will hopefully start fiddling with this integration next week so I'll take a deeper dive then.

[aaronjwood]

Maybe the part of using the same source port isn't needed based on your findings. My thinking there was that the adapters were doing something stupid and not closing out the connections correctly, which would mean states would keep piling up as we go through various ephemeral ports.

[aaronjwood]

I found something that may be useful to you: if you create a ClientSession and pass in (or use the default) of version=aiohttp.HttpVersion11 against an HTTP 1.0 server it reproduces the FIN_WAIT_2 problem. Looks like the session we take and reuse from HA is using this default. I didn't see anywhere that the version is read from the response and the session is adjusted appropriately. In fact I don't think you can adjust this part of a session on-the-fly.

One thing that fixes this is to close the session when you're done with the call, but it defeats the point of using the session as the connection pooling can't be used, etc. The other fix is to have a session using HTTP 1.0 which allows reuse without presenting the problem, you just wouldn't want to use that session for anything else but crappy HTTP 1.0 stuff :)

This issue coupled with not being able to configure the session for the right HTTP version at runtime seems to be problematic, I don't fully understand why this wouldn't be hidden from you and automatically handled under the hood. I did not dig very much but it seems that the aiohttp tries to do some kind of connection juggling when it shouldn't (e.g. HTTP 1.0 closing connections after each response)

BTW, if I wasn't explicit about this in my OP the adapters are actually using HTTP 1.0. Just making sure that detail didn't slip :)