Skip to content

Add Guidance wrt Labelling to Naming and Rules Best Practices #2691

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 7 commits into
base: main
Choose a base branch
from
Open

Add Guidance wrt Labelling to Naming and Rules Best Practices #2691

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
e57a291
Add Important Labels subsection, with job and instance called out
conallob Jul 15, 2025
a40367a
Add a new section about labels
conallob Jul 15, 2025
ccf009a
Fix typo
conallob Aug 14, 2025
0e87e09
Update docs/practices/naming.md
conallob Aug 14, 2025
a79aa9b
Update docs/practices/naming.md
conallob Aug 15, 2025
a0ebfa9
Iterate on the job label description
conallob Aug 17, 2025
7694523
Update docs/practices/naming.md
conallob Oct 7, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions docs/practices/naming.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,17 @@ of unit and type information in the metric name will cause certain series to col

## Labels

* `job`
* The `job` label is a primary key to differentiate metrics from each other.
* If not specified in PromQL expressions, they will match unrelated metrics with the same name. This is especially true in a multi system or multi tenant installation
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure this is really a useful note here, as this applies to all label matching.

Suggested change
* If not specified in PromQL expressions, they will match unrelated metrics with the same name. This is especially true in a multi system or multi tenant installation

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It applies to all labels. But job and instance are two uniform labels found on every metric, including ubiquitous synthetic metrics such as up

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, but it's not related to job, but related to "target labels" and discovery. That is a different thing and related to querying, not creating labels.


WARNING: When using `without`, be careful not to strip out the `job` label accidentally.

Comment on lines +93 to +94
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This warning doesn't make a lot of sense to me. It has a high probability of being quoted as copy-pasta without being understood. Let's just drop it.

Suggested change
WARNING: When using `without`, be careful not to strip out the `job` label accidentally.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't want to encourage copy-pasta, but this is an important point.

If using alerting expressions like up{job=bla} > 0 for 3m , you need to be careful not to accidentally strip the job label. If you do, your alert no longer works as intended.

I'll polish the wording here

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doesn't seem related to naming practices, which is what this guide is about.

* `instance`
* The `instance` label will include the `ip:port` what was scraped, providing a crucial breadcrumb for debugging scrape time issues
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
* The `instance` label will include the `ip:port` what was scraped, providing a crucial breadcrumb for debugging scrape time issues
* The `instance` label by default will include the `ip:port` what was scraped.


### General Labelling Advice

Use labels to differentiate the characteristics of the thing that is being measured:

* `api_http_requests_total` - differentiate request types: `operation="create|update|delete"`
Expand Down
18 changes: 14 additions & 4 deletions docs/practices/rules.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,8 @@ This page documents proper naming conventions and aggregation for recording rule
Keeping the metric name unchanged makes it easy to know what a metric is and
easy to find in the codebase.

IMPORTANT: `job` label acts as a primary key. It is **strongly** recommended that you use it to scope your PromQL expressions to the system you are monitoring.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is misleading. Prometheus doesn't have the concept of "primary key". Not even metric names are a "primary key".

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair, especially since folks used to SQL DBs will jump to the conclusion that it's a SQL DB, which it isn't.

Iterated on the language to avoid creating ambiguity


To keep the operations clean, `_sum` is omitted if there are other operations,
as `sum()`. Associative operations can be merged (for example `min_min` is the
same as `min`).
Expand All @@ -27,6 +29,18 @@ If there is no obvious operation to use, use `sum`. When taking a ratio by
doing division, separate the metrics using `_per_` and call the operation
`ratio`.

## Labels

NOTE: Omitting a label in a PromQL expression is the functional equivalent of specifying `label=*`

* In both recorded rules and alerting expressions, always specify a `job` label to prevent expression mismatches from occuring.
This is especially important in multi-tenant systems where the same metric names may be exported by different jobs or the
same job (e.g `node_exporter) in multiple, distinct deployments

* Always specify a `without` clause with the labels you are aggregating away.
This is to preserve all the other labels such as `job`, which will avoid
conflicts and give you more useful metrics and alerts.

## Aggregation

* When aggregating up ratios, aggregate up the numerator and denominator
Expand All @@ -40,10 +54,6 @@ Instead keep the metric name without the `_count` or `_sum` suffix and replace
the `rate` in the operation with `mean`. This represents the average
observation size over that time period.

* Always specify a `without` clause with the labels you are aggregating away.
This is to preserve all the other labels such as `job`, which will avoid
conflicts and give you more useful metrics and alerts.

## Examples

_Note the indentation style with outdented operators on their own line between
Expand Down