Old version reported of prometheus node exporter

[andonovski]

I just installed this add-on. The issue I have is that Prometheus node exporter reports version 1.7.0. I see that it should be 1.8.1, based what is in Dockerfile.
What could be wrong?

P.S. just for info, latest node exporter is 1.8.2. It was released in June this year.

[loganmarchione]

If you navigate to http://your_home_assistant_ip_address:9100/metrics and search for 1.7.0 and 1.8.1, which do you see?

[andonovski]

Hi, Thanks for quick reply. I do get version 1.7.0. It's the node_exporter_build_info metric. This is why I am asking. I followwd yoir steps to add your repo and install it, nothing special about it.

…

On Sat, Nov 9, 2024, 15:04 Logan Marchione ***@***.***> wrote: If you navigate to http://your_home_assistant_ip_address:9100/metrics and search for 1.7.0 and 1.8.1, which do you see? — Reply to this email directly, view it on GitHub <#62 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ASPGAGTI6UPK4S6LNJGD2BDZ7YI6XAVCNFSM6AAAAABRO7KGOOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINRWGIZTAMBRGY> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[loganmarchione]

Hmm, it has to be something with your setup. Do you have node exporter running on the host machine? Did you install a specific version of the package? Did you have an older version of this add-on installed at one point? Did you check for updates in Home Assistant?

It should look like this:

# TYPE node_exporter_build_info gauge
node_exporter_build_info{branch="HEAD",goarch="amd64",goos="linux",goversion="go1.22.3",revision="400c3979931613db930ea035f39ce7b377cdbb5b",tags="unknown",version="1.8.1"} 1

[andonovski]

Just crossed my mind, totally forgot about this. I installed Prometheus add-on, I guess this is probably the issue.
The annoying part is that the default Prometheus add-on doesn't expose all the OS metrics as it should.

[loganmarchione]

Ah ok, that might be it. Yes unfortunately their add-on only exposes Home Assistant related metrics, not OS metrics.

Can you try to change the port on this add-on and run both?

[andonovski]

I am definitely missing something regarding HA.
I tried using different port, I just get connection refused. I tried accessing the new port with IP and FQDN of the HA appliance. No, luck.
Removed official Prometheus from config.yaml, reconfigure Proemtheus Node Exporter to listen to 9100. Access it using FQDN I get node build info version 1.7.0, but by accessing using IP I get 1.8.2.
Weird behavior, I am missing something...

[andonovski]

To reply to myself, Issue solved it seems. Maybe it will help someone in the future, maybe I missed something in the docs.
Anyway, I tried accessing with FQDN which is actually published on reverse proxy. This isn't working as it should. Accessing using IP or FQDN of the server itself, which is different than the one in reverse proxy, is working properly, good Prometheus version is shown.

[loganmarchione]

Glad you narrowed it down to the reverse proxy. I guess it must be something in the proxy config?

[andonovski]

Yes, maybe something with proxy. I use Linuxserver.io SWAG. It's Nginx proxy. This is the config created from their template for home assintant:
server {
listen 443 ssl;
listen [::]:443 ssl;

server_name ha.domain.com;
include /config/nginx/ssl.conf;
client_max_body_size 0;
location / {
    include /config/nginx/proxy.conf;
    include /config/nginx/resolver.conf;
    set $upstream_app homeassistant.andonovski.net;
    set $upstream_port 8123;
    set $upstream_proto http;
    proxy_pass $upstream_proto://$upstream_app:$upstream_port;

}

location ~ ^/(api|local|media)/ {
    include /config/nginx/proxy.conf;
    include /config/nginx/resolver.conf;
    set $upstream_app homeassistant.domain.com;
    set $upstream_port 8123;
    set $upstream_proto http;
    proxy_pass $upstream_proto://$upstream_app:$upstream_port;
}

}

Of course, domain.com is in internal DNS as well as both FQDNs. The home assistant server name is homeassistant, there is DNS A record in domain.com zone. The ha is cname record in DNZ zone domain.com pointing to reverse proxy.

If i go to ha.domain.com:9100/metrics I get all host metrics with prometheus exporter version 1.7.0.
If I go to homeassistant.domain.com:9100/metrics I get all host metrics with prometheus exporter version 1.8.2
And of course, getting to ha.domain.com/api/prometheus only gives the metrics for the devices configured in home assistant.
Also, in home assistant I didn't configure in Settings->System->Network the option Home assistant URL option. This also might have something to do with this behavior.

But anyway, figured out what was the problem. Hopefully it will not change this behavior and give some other weird issues.
And once again, thanks for all the replies. Maybe you will see something in proxy configuration.

[loganmarchione]

I'm not super-familiar with the SWAG proxy (though I have run Nginx in the past).

I think what's happening is if you visit ha.domain.com:9100/metrics the ha.domain.com part becomes homeassistant.domain.com:8123. You're basically trying to double-port your request like homeassistant.domain.com:8123:9100/metrics. Can you make a different entry just for metrics? Like ha-metrics.domain.com?

Like you, I also access my Prometheus metrics by going directly to hostname.domain.com:9100/metrics so that I don't go through a proxy. This add-on isn't really setup to work through a proxy out-of-the-box, so everyone's setup will be different.

I'll leave this issue open in case anyone else runs into something similar.