Excessive Load on Proxy Leads to Instability

[JohnJ9ml]

This is the start of a research project, and one I didn't expect. I'll have to spend more time on this over the past few days, I'm going to report the challenge right now while it's fresh.

With the new releases of pypowerwall and PD, I got everything current and then started the process of slowly migrating my own calls over to the proxy that was running in the docker images supporting the dashboard.

I currently have an RPi5/8gb RAM supporting the PD installation. That's all that's running on it.
I currently have an RPi4/8gb RAM supporting my own probe of the PW3 data, with the proxy running and a Java program that talks to the proxy via the /pw/* endpoints I added, and then pushes the data into a database.
Both boxes have a USB Wifi with antenna tied in with the inverter's Wifi.
Both boxes are in the garage, hardwire connected to my local LAN, and less than four feet away from the inverter (which is on the outside wall, wood-sided house (no stucco and chicken wire)).

When each of these boxes hangs out in this fashion and supports only its own load, things seem to go fine for the most part. I've noted very rare issues on the pypowerwall box, but they're generally the timeout variety talking to the PW3, and they seem to always recover by the next call or two.

With the release of the new code, I started inching my way over to have my Java program talk to the proxy running on the PD box. I did not shut down the proxy on the RPi4 box, but there were no active calls going to it. I watched the load on the PD box, and it stayed very low: always in the .5 to 1.0 user load range, and the proxy python process would sometimes get up to 25% when watched in top, but nothing higher, and often in the 5% range.

If I left this configuration in place, however, after 30 minutes or so the PD proxy would somehow buckle under the load, lose contact with the inverter, and then every call starts resulting in an Error 113 - Host is unreachable. It seems as though it wedges the Wifi network.

I tried this three times, and eventually this was always the case.

Left separately with two proxies talking to the inverter, things seem okay.

My plan at this point is to not lose my own data, so I'm going to leave the RPi4 in place as it is now. At the same time, I'm going to take another computer running the Java program "read-only" and let it hit the PD proxy as my program usually does, but do what I can to capture the docker logs and see if I can catch anything revealing at the time the proxy seems to fail. If that isn't conclusive, I'll probably come back for suggestions as to how to run the proxy in debug mode within the docker image and see if there is more to learn there.

I've been assuming all along that the HTTP server inside the proxy is multi-threaded, and the load wouldn't seem excessive for it. I've watched the proxy running in debug mode while I was developing and felt it was adequately managing its own cached data. It's as though there is some possible combination of the dashboard and my own program making requests, and maybe there is just the right combination that causes a problem in the proxy core? Causes a problem on the wifi? Causes a problem someplace I haven't yet imagined?

That's the point of the research project! I just didn't expect there was going to be one.

[jasonacox]

Hi @JohnJ9ml

If I left this configuration in place, however, after 30 minutes or so the PD proxy would somehow buckle under the load, lose contact with the inverter, and then every call starts resulting in an Error 113 - Host is unreachable. It seems as though it wedges the Wifi network.

Powerwall DDOS?

I'm wondering if you triggered a DDOS mechanism on the Powerwall. When it is in this condition (113 errors on the PD proxy box) are you still able to connect to the Powerwall from the other box? In any case, here are some Proxy tweaks you could make to test for optimal performance:

# These are environmental settings that the proxy looks for - set them in pypowerwall.env for PD or via docker run

# Increase the cache TTL -  default 5s - this will reduce load on the WiFi and Powerwall
PW_CACHE_EXPIRE=15

# Adjust how long to wait for Powerwall to respond - default 10s 
# This will reduce the number of retries if WiFi and Powerwall are having issues and slow responding.
PW_TIMEOUT=20

# Increase or decrease number of persistent concurrent connections - default 15, use 0 to disable
PW_POOL_MAXSIZE=20

WiFi Issues

Are you seeing excessive retries or errors on the WiFi link? I would check the interface:

iwconfig wlan0

This could point to hardware limitations or signal congestion. You can check which channel it is using and change it to see if it performs better.

# See what channel it is using (freq)
iw dev wlan0 info

# Change the channel
iw dev wlan0 set channel 1

Bug in pypowerwall?

I haven't seen any issues but I'm still running the older firmware. I have an RPi4 that is using its built-in WiFi to talk to the PW access point. I haven't tried load testing it but will see if I can do that. As mentioned above, caching or other tweaks may help if the issue is too much chatter on the WiFi channel. But I would also suggest trying debug mode to see if that helps.

# Turn on Debug mode
PW_DEBUG=yes

Naturally, if you or anyone else has ideas on how to improve the proxy, please let me know (or better yet, submit a PR).

[JohnJ9ml]

Thank you for the additional sleuthing tips. I may be able to get to this today.

Yes, I knew later on I'd left one fact out: while the PD system was falling over like this, the other Pi continued to work just fine connected to the inverter via Wifi, even with a normal load on it.

I am also back behind on the firmware - still at 25.2.7. I was pressing to get this all in place before I get upgraded.

I'll report back when I have more information and more logging. Debugging is my preferred course at this point, particularly because the two Pi solution, while seemingly being a lot to eat, otherwise continues to work okay. In this two pic configuration, I went through the past 20 hours without a single failure or dropped request.