Looking for ideas how to extract data to Home Assistant

[damclarkd]

Jason, 1st off - Great tool! I have used this for over year since getting solar and 2 PW's. Running on rpi4 with zero issues. Getting all the data I want on Grafana dashboards.
Recently moved over to Home Assistant running on a separate VM on a NUC. HA has a Tesla Powerwall integration that I am testing but having some issues maintaining a connection. (Not at all as stable as Powerwall-Dashboard). What I am wondering is if all the info is in Powerwall-Dashboard with pypowerwall, is there a way to pull that data out and get it into HA somehow?
Again, just looking for suggestions on a path to pursue.

Thanks and cheers!

[jasonacox]

Hi @damclarkd ! Thanks for the great feedback!

On the HA Tesla Powerwall integration, I think I may know why you see some issues connecting to the Powerwall. The reason I wrote the pyPowerwall API was because I couldn't find a good library that didn't have connection issues or that didn't cause the Powerwall Gateway to crash after a while. Upon examination, most of the other libraries flood the Powerwall Gateway API with a mountain of requests. The web server in the Gateway isn't that robust, especially if you are hitting it with many different monitoring tools. I decided to write a library that would help with that. The pyPowerwall library caches all API calls for 10s (default setting) and reuses the same session token until it expires. That made a huge impact on stability.

The HA plugin for Tesla Powerwall appears to be using the tesla_powerwall python library but doesn't provide for any caching. I suspect pyPowerwall could be used instead. It would offer more stability as well as offer up additional data to graph via the device vitals payload that as far as I know, tesla_powerwall doesn't yet pull (it is a protobuf encoded payload). I will take a look to see if we can fork the integration and use pyPowerwall instead. If you are already running the pyPowerwall Proxy container, it could be as simple as pointing the integration to use that. But, I would welcome any help if anyone has time. :)

[damclarkd]

So since posting here I have made several changes to my lab setup that has currently solved the HA integration problem. For the last 24 hours the Tesla PW integration has been rock solid.
Steps taken to reduce load on PW calls:
1.) I have your Powerwall Dashboard running on docker in a VM. As stated already I was able to move all my historical data off the install on a RPi into this new VM install.
2.) I removed another system on a RPi that was polling the PW and uploading to PVoutput.
3.) I removed a testing lab RPi from PW usage.

Because I find your PW Dashboard to be so stable, that is the only local process polling the PW's now.

After all those changes, the HA PW integration has stopped dropping offline.
I now use Node Red in HA to gather and push the data I need up to PVoutput. Because that data is already in HA, there is no extra load on calls to the PW.
Because not everything I want in HA is in the Tesla PW Integration, I am using REST GET commands in Node Red to pypowerwall to get the extra data I am looking for. Again, no extra load on calls to PW.
While all this is new, it appears that with your PW Dashboard/pypowerwall installed locally and the HA Integration being the only other load, all seems happy.
Time will tell. Thanks for your tools and help.
Cheers

[jasonacox]

That's great news @damclarkd - some follow up questions if you don't mind:

• What are you using to push the data into PVoutput? For those not using HA, I'm wondering if I can make this easy since the same data is in InfluxDB, perhaps with an included script/cronjob.
• In yoursetup, it seems like you have two processes polling PW - the Dashboard and the HA Tesla PW Integration - is that correct? If I were to submit an update (PR or new integration) to the HA integration to allow it to use the pypowerwall proxy and include Powerwall+ and other metrics (e.g. string data or other elements you are using Node Red to gather), would that be interesting?
• Can you share some screenshots or code details on your HA setup, specifically the Node Red added data?

Thanks for the feedback and help!

[damclarkd]

@jasonacox In answer to your bullet point:

1. I am currently using Node Red in HA to push to PVoutput. I was using https://github.com/VirtualWolf/powerwall-to-pvoutput-uploader previously. Obviously that was adding load on the calls to PW. I found and adapted someones flow for uploading to PVoutput to fit my needs. PVoutput does have pretty good documentation for their API at https://pvoutput.org/help/api_specification.html. I am a donating member so have access to 5 minute uploads and extended data. I would think that this PVOutput option as part of PW Dashboard/pypowerwall would be well received. I can send a JSON of the flow that you can load into Node Red if that would help.
2. Yes, very interesting. In fact I have started to move away from the entities in the HA PW Integration because I am getting everything I need (plus more) just using GET calls to my pypowerwall install. (Why have more calls to the PW two just get duplicate data?) With the integration we are limited to only what they expose. I am using GET calls in Node Red for those pypowerwall data points I want but you can also use the REST sensor configuration in HA to pull and populate entities.
3. Sure. Do you want those posted here?

[jasonacox]

Thanks @damclarkd - this is super helpful! Yes, please feel free to copy/paste any images (Github convert images pastes to uploads) or code that would be helpful for the community here. I'll work on integrating more of it into the Dashboard stack. My TODO thoughts:

1. Add an easy option/automation to upload 5-minute or daily stats to PVOutput. Please send the JSON of the flow you use in Node Red so others can use that too. I created a tools directory placeholder in the repo that we can use to list these tools and any helpful files or scripts.
2. Easier and more comprehensive data integration with HA - I think your approach is genius and suspect others could quickly do the same (Node Red or REST config path). I suspect the popularity of HA is due to the ease of adding an integration with a simple click. I'm going to work on a near in-place pypowerwall library version of the current integration (with added data point) as well as an option to pull the data via the pypowerwall proxy. Can you share what data set that you are pulling via NodeRed right now?

[damclarkd]

@jasonacox
Node Red GET's from PyPowerwall currently. (Very much in flux as I convert over from HA Tesla Powerwall entities.)

[damclarkd]

Hello @jasonacox
Just wanted to update you that the Node Red solution using GET's to pyPowewall mentioned above, is working so well that I have abandoned using the HA Tesla Powerwall integration all together. I get everything I want, plus more, and thus don't have the extra calls to the PW from the HA Integration. Still think your idea of using pyPowerwall in the HA Integration would be a great solution long term. But for now I am very happy without the HA Integration.
Cheers!

[Dave-Jones-1963]

Please can you show how you are doing this, the basic http://localhost:8675/CSV page I am easily able to collect the data but for example from the http://localhost:8675/vitals page I am being frustrated! Maybe because I have only just started trying to use JSON......

[jasonacox]

If you need CSV formats, try http://localhost:8675/csv/v2?headers and get:

Grid,Home,Solar,Battery,BatteryLevel,GridStatus,Reserve
1568.00,1558.00,10.00,0.00,23.60,1,20

The vitals payload is n JSON format. You can seer all the APIs here: https://github.com/jasonacox/pypowerwall/blob/main/proxy/API.md

[Dave-Jones-1963]

Yes got that - and my multiscrape for the .csv works a treat

• name: Powerwall-scrape1
  resource: http://localhost:8675/csv
  scan_interval: 60
  sensor:
  • unique_id: py_grid_power
    name: Grid power
    select: "body"
    value_template: '{{ value.split(",")[0] }}'
    state_class: measurement
    device_class: energy
    unit_of_measurement: W
  • unique_id : py_home_power
    name: Home power
    value_template: '{{ value.split(",")[1] }}'
    state_class: measurement
    device_class: energy
    unit_of_measurement: W
  • unique_id: py_solar_power
    name: Solar Power
    value_template: '{{ value.split(",")[2] }}'
    state_class: measurement
    device_class: energy
    unit_of_measurement: W
  • unique_id: py_battery_power
    name: Battery power
    value_template: '{{ value.split(",")[3] }}'
    state_class: measurement
    device_class: energy
    unit_of_measurement: W
  • unique_id: py_battery_level
    name: Battery level
    value_template: '{{ value.split(",")[4] }}'
    state_class: measurement
    unit_of_measurement: percentage
    device_class: energy

I am having difficulty with the JSON which just doesn't work - I have probably got it completely wrong!

• name: Powerwall-scrape2
  resource: http://localhost:8675/vitals
  scan_interval: 60
  sensor:
  • unique_id: py_PVAC_Fout
    name: PVAC Fout
    value_template: '{{ value_json.PVAC--(redacted).PVAC_Fout }}'
    state_class: measurement
    device_class: energy
    unit_of_measurement: Mhz
  • unique_id: py_PVAC_Pout
    name: PVAC Pout
    value_template: '{{ value_json.PVAC--(redacted).PVAC_Pout }}'
    state_class: measurement
    device_class: energy
    unit_of_measurement: VAC
  • unique_id: py_PVAC_Vout
    name: PVAC Vout
    value_template: '{{ value_json.PVAC--(redacted).PVAC_Vout | round(2) }}'
    state_class: measurement
    device_class: energy
    unit_of_measurement: W

[jasonacox]

ChatGPT can help you write the software if you provide it a sample of the JSON.

[Dave-Jones-1963]

Sorted without ChatGPT - value_template: '{{ value_json ["PVAC--(redacted)"]["PVAC_Vout"] | round(2) }} is the format for the HA json in multiscrape and now I am happy - just need to workout what is battery in and battery out as two separate entities as opposed to the +/- that the vitals page gives ("PVAC_Pout": -520,). I believe that "METER_Y_CTA_InstRealPower": 965, is showing me the instantaneous solar power generaltion and "METER_X_CTA_InstReactivePower": -437, is the power I am using in the house. Please correct me if I am wrong.

[chinswain]

Did you ever get anything setup attempting to connect Home Assistant via pyPowerwall proxy @jasonacox? Seems like the perfect solution with all of the current issues around the Tesla API and offical HA integration not being able to use the PW3 API.

[jasonacox]

Interesting. What options are there for HA right now? I don't mind trying to help get a plugin to work, but don't want to duplicate if there is another project we could contribute to. Any HA gurus able to help?

[TristanBrotherton]

Nothing that works with Powerwall 3....

[jasonacox]

Could we get pypowerwall to work with HA? I scanned the https://developers.home-assistant.io/docs/add-ons page and it looks possible. Has anyone built one?

[pavandave]

Hey, I wanted to chime in. I tried to update the not-working python code with pypowerwall library but because urllib3 is being used and HA doesn't like that, it doesn't work properly.

Thread and my last comment - home-assistant/core#115513 (comment)

[jasonacox]

Thanks @pavandave - I don't think this is urllib3 per se. This seems more likely an issue with HA wanting the pypowerwall code to be asyncio. I want to do that, but it is a non-trivial effort. Any volunteers to help? :)