Sink one gauge metric to prometheus by remote write, query get multiple

[xxm404]

vector version: vector 0.26.0 (x86_64-unknown-linux-gnu c6b5bc2 2022-12-05)
vector config:

# Change this to use a non-default directory for Vector data storage:
data_dir = "/var/lib/vector"
[sources.vector_logs]
type = "internal_logs"

[sinks.console]
type = "console"
inputs = ["vector_logs"]
encoding.codec = "text"

# Random Syslog-formatted logs
[sources.prometheus_style_metrics]
type = "kafka"
bootstrap_servers = "172.31.16.247:9092,172.31.15.146:9092"
group_id = "vector-script-metric-0"
topics = [ "scriptMetric" ]

[transforms.test_parse]
type = "remap"
inputs = [ "prometheus_style_metrics" ]
source = '''
.metric = parse_json!(.message)
.metric.metric.timestamp = to_timestamp!(parse_json!(.message).metric.timestamp, unit: "milliseconds")
'''

# Parse Syslog logs
# See the Vector Remap Language reference for more info: https://vrl.dev
[transforms.make_metric]
type = "lua"
inputs = [ "test_parse" ]
# inputs = [ "prometheus_style_metrics" ]
version = "2"
[transforms.make_metric.hooks]
process = """
function (event, emit)
  emit(event.log.metric)
end"""

# Print parsed logs to stdout
[sinks.print]
type = "console"
inputs = ["make_metric"]
encoding.codec = "json"
[sinks.prometheus_write]
type = "prometheus_remote_write"
inputs = [ "make_metric" ]
endpoint = "http://host:9091/insert/0/prometheus/api/v1/write"

when i sink one metric, vector console print

{"name":"cpu_time_milliseconds","namespace":"script_metric","tags":{"script_name":"monitor/test.php"},"timestamp":"2023-03-24T08:48:00.383Z","kind":"absolute","gauge":{"value":33.0}}

use grafana query script_metric_cpu_time_milliseconds result

The result is not one data point, but multiple, the expectation is that only one data point appears for the corresponding event.

[jszwedko]

Hi @xxm404 !

Can you verify if the prometheus_remote_write sink there is actually writing more than one payload? I believe if you run with VECTOR_LOG=debug that should be clear. I'm wondering if this is just an artifact of Prometheus/Grafana that shows it as multiple.

[xxm404]

Thanks for the reply, when run with VECTOR_LOG=debug debug log like this

2023-03-26T12:17:25.257994Z DEBUG sink{component_kind="sink" component_id=prometheus_write component_type=prometheus_remote_write component_name=prometheus_write}:request{request_id=0}:http: vector::internal_events::http_client: Sending HTTP request. uri=http://host:9091/insert/0/prometheus/api/v1/write method=POST version=HTTP/1.1 headers={"x-prometheus-remote-write-version": "0.1.0", "content-encoding": "snappy", "content-type": "application/x-protobuf", "user-agent": "Vector/0.26.0 (x86_64-unknown-linux-gnu c6b5bc2 2022-12-05)", "accept-encoding": "identity"} body=[119 bytes]
Sending HTTP request.
2023-03-26T12:17:25.259497Z DEBUG sink{component_kind="sink" component_id=prometheus_write component_type=prometheus_remote_write component_name=prometheus_write}:request{request_id=0}:http: vector::internal_events::http_client: HTTP response. status=204 No Content version=HTTP/1.1 headers={"x-server-hostname": "ip-172-31-9-183", "date": "Sun, 26 Mar 2023 12:17:25 GMT"} body=[empty]
HTTP response.

looks like writing one payload to prometheus, but i still get more than one data point in grafana, I'm confused

[jszwedko]

Gotcha, yeah, maybe this is a Prometheus or Grafana question then. I'd recommend raising it with them.

[xxm404]

Ok I will try to ask Prometheus or Grafana for help.