[Feature] Start all Immich metrics with word "immich_".

[reefland]

The feature

Finding Immich metrics is a chore as none of the metrics contain the word "immich" which makes finding and discovery of metrics annoying. I don't want to have to curl both the endpoints to figure out metric names instead of going to Prometheus Web interface and type "immich" and they all start showing up as you type.

Looks like the initial round of metrics are developer focused. Please consider adding metrics that can be used for a typical dashboards... number of: images, movies, albums, people, objects, faces not associated to person, etc.

Platform

• Server
• Web
• Mobile

[bo0tzz]

I think setting a global prefix is not supported by the library we use: pragmaticivan/nestjs-otel#265

@mertalev thoughts?

[mertalev]

Only the io metrics can easily use a prefix since they're custom. We'd have to monkeypatch things to force the prefix in other cases.

All the metrics have immich as the service name, so that might also help with discovery.

[mertalev]

Looks like the initial round of metrics are developer focused. Please consider adding metrics that can be used for a typical dashboards... number of: images, movies, albums, people, objects, faces not associated to person, etc.

Yep, this was initially aimed as more of a development / debugging tool. But I also want to add these kinds of metrics as well as job metrics.

[mertalev]

I've made an upstream PR to allow prefixing the API metrics here. I'm not sure if we should change the repo metrics now, or wait for that change so they're consistent.

[MMK21Hub]

As a workaround, here's part of my prometheus.yml that shows how to add a prefix to Immich's metrics on the scraping side:

  - job_name: immich_microservices
    static_configs:
      - targets: ["immich-microservices:8081"]
    metric_relabel_configs:
      - source_labels: [ __name__ ]
        target_label: __name__
        regex: '(.*)'
        action: replace
        replacement: immich_${1}

Note that I don't think you need to add the same relabeling config to the immich_server scraper, as its metrics are already prefixed with immich_

[mertalev]

That's a really good find! We should add that to the official config. A small tweak can be to change the regex to this: ^(?!immich).*$. The job metrics do have the immich prefix and others in the future may as well, so this prevents the prefix from being added twice.

[reefland]

When I tried your regex suggestion above regex: '^(?!immich).*$' Prometheus failed config loading with:

error parsing regexp: invalid or unsupported Perl syntax: `(?!`

[mertalev]

Hmm, Go's regex engine apparently doesn't support lookarounds.

[bo0tzz]

I don't have a good setup to test it myself, but would it work to just do this?

  - job_name: immich_microservices
    static_configs:
      - targets: ["immich-microservices:8081"]
    metric_relabel_configs:
      - source_labels: [ __name__ ]
        target_label: __name__
        regex: '(immich_)?(.*)'
        action: replace
        replacement: immich_${2}

[Opa-]

The last regex you shared works well 👍

[bo0tzz]

Stumbling on this thread again a while later, I think the proper approach would be not to have an immich_ prefix on the names, but instead to add an app: immich label (or similar) to each metric instead.