[feature] Adding metrics for the exporter itself and the HTTP clients

[Packetslave]

Here at $DAYJOB, we have augmented the exporter with system-level metrics such as process_cpu_seconds_total, go_* and gc_* from the standard Go prometheus library. This is useful for monitoring the health of the exporter, setting alerts on CPU usage, etc.

We have also instrumented the HTTP clients used to talk to api.fastly.com and rt.fastly.com using the standard HTTP middleware from the Go prometheus library (e.g.promhttp.InstrumentRoundTripperCounter and promhttp.InstrumentRoundTripperDuration) to expose metrics for calls to the API endpoints such as count by HTTP status code and request latency.

To avoid maintaining our fork, we would love to upstream these changes, but would like to discuss how best to make them fit into the current code.

Option 1: In our current code, to avoid touching the per-service metrics, we create a separate HTTP server on a different "admin" port (6060) and bind the default Prometheus registry to /metrics on it. This was the easiest thing to do with minimal changes to existing code.

Option 2: repurpose the /metrics endpoint and serve system-level metrics there using the default registry, instead of the current service listings page. That could be moved to, e.g. /services. This would be similar to what other exporters do, where /metrics is the standard place for system-level metrics, but would be a breaking change from a user perspective.

Option 3: add a separate HTTP path, such as /system or /metrics2 for system-level metrics.

Thoughts?

[leklund]

I really like the idea of adding system-level metrics!

Ideally we'd prefer to avoid breaking changes so either Option 1 or Option 3 seem like the best path forward. I'm leaning Option 1. It keeps the exporter's system level metrics separate from the service metrics and I'm guessing keeps the code fairly separate as well.

[Packetslave]

Sounds good. I can put a PR together from our internal fork using Option 1.