Help with Prometheus Scrape query

[satellite-no]

Im trying to setup a prometheus scrape with a query parameter but having troubles. I have a successful source setup with the below where I pull back many many metrics and then just drop the ones I dont want. However my preference would be to only pull back the metrics that I want (filter fast and filter early).

sources:
  redis: 
    type: prometheus_scrape
    endpoints:
      - "https://redis_server/metrics"
    scrape_interval_secs: 60
    tls:
      verify_certificate: false

The below is an example of one of the metrics im trying ot capture but am still pulling ALL metrics. Tried both items in list seperately which is based on the documentation from the scrape source.

sources:
  redis: 
    type: prometheus_scrape
    endpoints:
      - "https://redis_server/metrics"
    query:
      "match[]":
        - '{name="bdb_avg_latency"}'
        - '{__name__=~"bdb_avg_latency"}'
    scrape_interval_secs: 60
    tls:
      verify_certificate: false

Any insight would be helpful even if im just interpriting what the query param does.

[pront]

Hi @satellite-no, thanks for sharing this.

filter fast and filter early
👍

Hmm, your prometheus_scrape configuration looks correct to me. I am now wondering if Prometheus does not allow filtering metrics at scrape time when pulling from a /metrics endpoint (but I don't have a local environment setup to validate this).

You can use a filter transform in Vector though:

transforms:
  filter_metrics:
    type: filter
    inputs:
      - redis
    condition: '{{ name == "bdb_avg_latency" }}'

[satellite-no]

Hmm, maybe im just confused with the purpose of the query option. If Prometheus does not allow filtering metrics at scrape time when pulling from a /metrics endpoint I cannot see a purpose for the query parameter to exist or how it was ever tested and implemented.

FYI, Im using the below to remove the stuff i dont want but seems terribley inificient when i want 5% of what is being pulled lol.

type: filter
inputs:
  - prometheus_scrape_redis
condition:
  type: vrl
  source: |
    .name == "bdb_avg_latency" ||
    .name == "node_free_memory" ||
    .name == "node_cpu_idle" ||
    .name == "node_persistent_storage_free" ||
    .name == "node_ephemeral_storage_free" ||
    .name == "node_ingress_bytes" ||
    .name == "node_egress_bytes" ||
    .name == "redis_total_commands_processed" ||
    .name == "redis_used_memory" ||
    .name == "redis_process_cpu_usage_percent" ||
    .name == "redis_connected_clients" ||
    .name == "redis_used_memory" ||
    .name == "redis_db0_keys" ||
    .name == "bdb_crdt_syncer_local_ingress_lag_time" ||
    .name == "bdb_crdt_syncer_status" ||
    .name == "bdb_replicaof_syncer_local_ingress_lag_time" ||
    .name == "bbdb_replicaof_syncer_status" ||
    .name == "bdb_up" ||
    .name == "node_up" ||
    .name == "node_provisional_memory" ||
    .name == "bdb_conns"

[pront]

Right, can you see the produced URL and/or any logs on the Prometheus side? Just wanted to ensure that the final URL is conforming to the Prometheus HTTP API.

[satellite-no]

Well thats the fun part, I dont have any visiblity to these boxes other then the metrics endpoint. I assume (but not sure) that they are using the redis exporter that we are just scrapping thinking its v1 and not v2 which might be an issue(?).

The part im confused about and this is my inexperiance with prometheus is I dont see an way to query specific metrics in the proper prometheus scrape config. Researching online the only way i can see to do this is with the metric_relabel_configs seen below (source)

    metric_relabel_configs:
    - source_labels: [__name__]
      action: keep
      regex: '(node_cpu)'

[pront]

small nit: You can reply on the thread above directly

Unfortunately, I don't have time to setup a dev environment now but maybe someone from the community has experience and can step in here. If you can run some experiments, even without visibility into the boxes, you can observe what is ingested into Vector.

[satellite-no]

Yep, I've done quite a bit of testing thats why I came here basically to understand the purpose of the query parameter. Seems like maybe we arent sure however?

[pront]

Unfortunately, I am not sure. But it's my TODO list to setup a prometheus environment to better assist with such questions.

[squiishyy]

I am also running into the exact same issue as @satellite-no and would love to not have to scrape all metrics from kube-state-metrics to actually utilize a small subset in aggregations later.

Does filtering them out drop them from vectors memory immediately? Worried about resource consumption for large metric generators

[pront]

Does filtering them out drop them from vectors memory immediately?

Generally yes, if you ingest them they will increase the memory usage before they hit the filter transform.

[bobbyhubbard]

For posterity. VRL includes a bunch of nice functions including starts_with and contains. When query failed to work (never got it working with the redis_exporter metrics endpoint), I did as suggested here but I did not want to include a single line for all 100 metrics I wanted to collect. So...

transforms:
  kdb_metrics_filter:
    type: "filter"
    inputs: ["kdb_metrics_source"]
    condition: 
      type: vrl
      ## Exclude the following key prefixes
      source: |
        !starts_with!(.name, "redis_exporter") && 
        !starts_with!(.name, "redis_commands") && 
        !starts_with!(.name, "redis_aof") && 
        !starts_with!(.name, "redis_allocator") && 
        !starts_with!(.name, "go")