feat: prometheus metrics on separate http port

[saksmt]

Use case: pikvm exposed to internet via port forwarding, prometheus metrics scrapped in intranet without need to bother about authentication

[mdevaev]

Prometheus by default is closed by auth. What is use case for this?

[saksmt]

Prometheus by default is closed by auth. What is use case for this?

Total isolation on network level (imagine closed off trusted intranet where only some endpoints are exposed to internet and all of the internal communication is trusted and secure by default) which is more secure and does not require fiddling with authentication in prometheus scrapping (especially this point becomes valid when using OTP)

[saksmt]

Here's an example:

Given: server, pikvm and a network.

Network is walled off and only allows:
0.0.0.0:443 -> server:443 {homelab: git, grafana, ...}
0.0.0.0:22080 -> pikvm:80
0.0.0.0:22443 -> pikvm:443

In this case you'd want maximum security for pikvm, hence you'll enable TOTP. But you still want to collect metrics from pikvm and store them on server for alerting/dashboarding/etc.

Reasonable solution in that case is to disallow prometheus metrics on default port in pikvm, disable authentication for them and expose them lets say on 22091 to be collected by server. That way prometheus metrics from pikvm are unavailable from internet and are still possible to be collected by server since that does not require port forward for 22091

---

Maybe there's a bit more elegant solution to do this in python server itself (I mean binding route to different port/socket/...), but since I don't really know how to do that and as far as I understood all binding to actual ports (as opposed to unix sockets) is done only in nginx, I've done it that way. Which works and is actually tested on my pikvm through copying template and prepending block with needed values (<% ... %>)

[mdevaev]

I think I forgot to answer. It's reasonable and I'll think about what the best solution might be. It is likely that what you are suggesting is exactly what it is.