Pros/Cons of a Different Tech Stack

[s-crypt]

Hey y'all,
I would just like to throw the idea out there of brainstorming a different tech stack for this project. Not for any reason other than "why not." For those who have experience in related software, what would you think the pros and cons from a technical standpoint of using different software to build this project?

Changes I was thinking:

• Docker -> Podman
• InfluxDB -> PostgreSQL/Timescale Table
• Telegraf -> Fluentd
• Weather411 -> Eliminate. Use a Fluentd API call to table in Postgres/Timescale table instead
• PyPowerwall -> Eliminate? Fluentd
  • Use the great work done on PyPowerwall by Jason and others to convert into Fluentd calls and transformations
  • ... if possible. I don't fully understand the scope of fluentd and PyPowerwall

Why those?

1. Podman usually has less overhead (correct me if I am wrong)
   a. Added benefit of non-root containers
2. I am not a huge fan of InfluxData as a company.
   a. Their consistent breaking changes with their DB (in what feels like most versions) and their move towards closed-source puts a sour taste in my mouth.
3. The current DB is slightly older as there have been some recent advances with speed, scalability, and efficiency.
   a. I know that PSQL/Timescale takes up more disk space, but I think that is a good tradeoff as it uses less memory and CPU. I am running mine on a Raspberry Pi 3B+ so the lighter the better.
4. I like SQL far more than Influx's (thankfully now dead) attempt at their own query language, and Timescale/PostgreSQL is very compliant and standardized.
5. I also prefer relational over NoSQL or tags.
6. I am a big fan and supporter of standardization

What do y'all think? Am I spewing incoherent and technologically illiterate garbage? Do you have different suggestions or thoughts?

This is not a feature request, complaint, or anything of the sort. I love this project and it works perfectly. I just do not have the experience in time series databases and log ingestion and would be interested to learn other's thoughts and perspectives.

TL;DR:

I love this project and it works great the way it is. I think it would be interesting to discuss the pros and cons of using different underlying tech 'cause I don't know a lot.

[youzer-name]

TLDR: I wouldn't be upset to see InfluxDB go, but I'm not sure the other changes would be worth it.

• For Docker vs. Podman - the main advantage touted for podman usually is that it's rootless. Is that a real concern for this project? There are several contributors who would have to go through the learning curve of switching over. Wold the end products be significantly different?
• I have no love for InfluxDB. (Ask me about how I deleted my entire dataset because I typed "where date" instead of "where time"). I even copy some of the data over to MySQL so I can work with relational tables and standard SQL capabilities (like extracting the day of week or month value from a datetime field). My cost/savings calculations all come from MySQL because InfluxQL simply can't do the required queries. However the one thing Influx does that MySQL can't easily replicate is the grouping and fill functions. This is essential for a lot of the graphs being displayed. From a quick search, it looks like TimescaleDB does the same using 'time_bucket' in the query while using a more SQL-like syntax.
• There would have to be a migration tool to move existing data from InfluxDB to TimescaleDB. People have years of data already in InfluxDB.
• You list Telegraf, Weather411 and pyPowerwall on the changes, but don't mention them in the 'why those', so I'm not clear on what the advantages would be.

[s-crypt]

Hey! Thanks for the reply.

Just to expand on my initial thoughts based on your perspective and questions:

1. From what I understand, podman is nearly directly compatible with docker (to the point where you just have to replace the docker command with podman). With the absence of a daemon with podman, this should mean lower overhead.

I deleted my entire dataset because I typed "where date" instead of "where time"

• Yikes 😳. Hopefully you had a backup

2. Based on your quick search, and your expeience in using MySQL and InfluxDB with this dataset, are you saying that Timescale fulfills both of your needs with the "buckets" for grouping and the SQL calculations?

3. Of course migration would be difficult, but I think it would be not too difficult thanks to tools that are already made

4. I mentioned Telegraf, Weather411, and pyPowerwall to be changed to Fluentd because it could serve as a unified logging ingest and processing platform, and removing two containers for an even lighter weight program. With a centralized ingest and manipulation program, pushes and changes can be pushed out a little easier by just changing a config file rather than a whole new docker image. (I am probably blowing hot air here, because I have not looked into the depths of Weather411 and pyPowerwall)

• Fluentd is a vendor and database agnostic tool similar to telegraf with a huge community and tons of plugins
  

I do want to reiterate that I am not requesting or suggesting these changes, just want to hear different perspectives and experiences.

[youzer-name]

Hey! Thanks for the reply.

Just to expand on my initial thoughts based on your perspective and questions:

1. From what I understand, podman is nearly directly compatible with docker (to the point where you just have to replace the docker command with podman). With the absence of a daemon with podman, this should mean lower overhead.

I deleted my entire dataset because I typed "where date" instead of "where time"

• Yikes 😳. Hopefully you had a backup

Sure. Did I ever test that backup? Of course not. Did the backup work? Of course not...

Luckily the history tool had recently been developed so I was able to pull most of the data from the cloud.

2. Based on your quick search, and your expeience in using MySQL and InfluxDB with this dataset, are you saying that Timescale fulfills both of your needs with the "buckets" for grouping and the SQL calculations?

I think so. When I have some time I'm going to try to set up a TimescaleDB container and move some data over to it to see if I can run my TOU queries in it.

[youzer-name]

I think so. When I have some time I'm going to try to set up a TimescaleDB container and move some data over to it to see if I can run my TOU queries in it.

It's not looking good for TimescaleDB. They have two docker images... the '-ha' or high availability version which is based on Ubuntu, and the regular version based on Alpine. The ha version won't run on ARM64, so I can't install it on my Raspberry Pi, and the regular version doesn't include the TimescaleDB toolkit, which is needed to do integral function. Integrals are absolutely essential for calculating kwh from kw values.

The Alpine package manager doesn't recognize the Timescale toolkit as a valid package. It looks like there might be a way to build a custom docker container with the toolkit on ARM64, but I'm not sure if that's the case.

[jasonacox]

I love this discussion. I'm all for experiments and "what-ifs" to learn or to expand functionality of the project for the community to meet our mission. Some general thoughts on how we consider changes and features for the Dashboard project:

• Easy - The goal has always been to make this stack easy to spin up and run for enthusiast of all levels. We have spent a lot of time trying to ensure the setup.sh and upgrade.sh scripts are easy to use and run on a variety of platforms. We target Linux, Raspberry Pi, MacOS, Windows, and where possible, various NAS devices that include shell + container support. Our community is a good cross section of users on those platforms so I think about it a lot when I review changes. A decision to change any element of the platform needs to include "easy" and ensuring that by testing on all of those.
• Beautiful - The goal of a dashboard is to make metrics visually intuitive and aesthetically appealing. I know this can sound subjective, but we have been very hesitant to upgrade or change elements that would be a step back in terms of the look of the dashboard elements. Grafana (and plugins) have made aesthetic changes over time that reduce the visible data or look less beautiful, cohesive or intuitive. Often those are temporary dips as they switch to new technology underneath. In any case, we don't upgrade just to upgrade. It needs to add to the look or function of the dashboard.
• Reliable - The goal is to minimize ongoing support of running the Dashboard by the community. We want to minimize time answering troubleshooting issues due to brittle, complex or unreliable components. Several contributors and collaborators in this community, including me, have spent many hours helping others navigate through configuration or data issues, especially early on. The stack we choose needs to be easy to setup, diagnose an maintain by enthusiast of all levels of capability.
• Extendable - Not everyone will want to extend the platform and will be happy with the base installation, which to be fair, includes a lot. But there are some who want to add other integrations. Over time we have added the ability for power users to add customization while staying within the stack that allows them to use upgrade.sh to add new features or stay current on bug fixes. This has included many helpful tools like the tesla-history import tool that @mcbirse built, that many find essential now. I suspect we can do even more in this area.

The different tech stack discussions is healthy. Some of what you are discussing may help those goals. If the investment of time results in better value for the community, I'm all for exploring.