Skip to content

prometheus.scrape OpenMetrics parser fails on blank lines between metric families, despite spec requiring they be ignoredΒ #5638

@sjclement

Description

@sjclement

Component(s)

prometheus.scrape

What's wrong?

When scraping an endpoint that returns application/openmetrics-text with blank lines between metric families, Alloy's OpenMetrics parser treats the blank line as an invalid token and marks the scrape as up=0, dropping all metrics from metric families after the first blank line.

The OpenMetrics specification states that blank lines should be ignored by parsers.

Steps to reproduce

Scrape any endpoint returning the following OpenMetrics response (note the blank line between metric families):

# HELP django_health_check_status Health check status (1 = healthy, 0 = unhealthy)
# TYPE django_health_check_status gauge
django_health_check_status{check="Database(alias='default')"} 1
# HELP django_health_check_response_time_seconds Health check response time in seconds
# TYPE django_health_check_response_time_seconds gauge
django_health_check_response_time_seconds{check="Database(alias='default')"} 0.014698

# HELP django_health_check_overall_status Overall health check status
# TYPE django_health_check_overall_status gauge
django_health_check_overall_status 1
# EOF

Expected behavior

All 3 metric families are parsed successfully, up=1.

Actual behavior

Parsing stops at the blank line. Only the first metric family is scraped. up=0.
The following error is visible in the Alloy UI target debug page under last_error:
expected a valid start token, got "\n" ("INVALID") while parsing: "\n"

Environment

Alloy version: grafana/alloy:latest [v1.13.1]
Deployment: Docker container
Scrape target: django-health-check library (https://github.com/codingjoe/django-health-check), which emits blank lines between metric families in its OpenMetrics renderer

System information

Docker Image Debian 13

Software version

grafana/alloy:latest image ( v1.13.1 )

Configuration

prometheus.scrape "django_health" {
  targets = [{
    __address__ = "django-app:8000",
  }]
  metrics_path    = "/health/"
  params          = { "format" = ["openmetrics"] }
  forward_to      = [prometheus.remote_write.grafana_cloud.receiver]
  scrape_interval = "15s"
  job_name        = "django_health"
}

prometheus.remote_write "grafana_cloud" {
  endpoint {
    url = #######

    basic_auth {
      username = ######
      password = ######
    }
  }
}

Logs


Tip

React with πŸ‘ if this issue is important to you.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions