Solution for vector to vector metrics data pipeline using http

[ee07b415]

Hi Dear Vector community,

I'm recently doing a project plan to use vector for both agent and aggregator role for collecting the metrics data. The vector agent will working in the data plane where our clients generate some metrics data, on our control plane we have another vector aggregator cluster collecting the data received from the data plane then forward to the centralized prometheus server for monitoring.

Here I runs into the trouble for the metrics data, because #17056, I can't use the native_json as the codec for http sink and source, so I use json as the codec. Then the http source in the control plane got the data:

{
	"aggregated_summary":{
		"count":2502,
		"quantiles":[{"quantile":0.0,"value":0.000031142},{"quantile":0.25,"value":0.000038841},{"quantile":0.5,"value":0.000045922},{"quantile":0.75,"value":0.000050285},{"quantile":1.0,"value":0.00008474}],
		"sum":0.114286825},
	"kind":"absolute",
	"name":"go_gc_duration_seconds",
	"path":"/",
	"source_type":"http_server",
	"tags":{"nodename":"ip-",
	"service_name":"nodeExporter",
	"source_type":"nodeExporter"},
	"timestamp":"2023-09-12T19:08:20.427244440Z"
}

But this data is in log format so actually it is:

{"log": "{{above_data}}"}

directly connect this source to prometheus_exporter won't work, because prometheus_exporter sink looking for metric input, so I have to add the process for translate the data, I tried some solution as I saw in the discussions and issues, for example:

  transforms:
    parse_http:
      type: remap
      inputs:
      - http_server
      source: |
        .metric = .
        .metric.timestamp = to_timestamp!(.metric.timestamp)
     to_metric:
      type: lua
      inputs:
      - parse_http
      version: '2'
      hooks:
        function (event, emit)
            event.metric = event.log.metric
            event.log = nil
            emit(event)
        end

This will work for gauge and counter type metrics, but not for the histgram and summary, the reason I find from the source code: https://github.com/vectordotdev/vector/blob/master/lib/vector-core/src/event/lua/metric.rs#L781, vector seems think the histgram and summary type data in lua using different schema from the vector metrics. So if I want to use the lua transform the "metrics-log" to metrics, I need add the steps to transform the data which already in the vector metrics format to lua format, this will downgrade the performance a lot(need 5x resource). I'm wandering before the fix of the native type on the http source/sink, what is the solution we are using for the vector to vector metrics data pipeline using http source and sink.

[jszwedko]

Hey @ee07b415 !

Thanks for filing #17056. We'll try to look at that shortly. In the meanwhile you could try the native codec (protobuf) to see if that suffers the same issues.

You could also consider using the vector source/sink to send data from one Vector to another. This is HTTP/2-based (gRPC).

[ee07b415]

Hi @jszwedko , The native codec also runs into the same panic for metrics. I'm testing with the vector sink but it doesn't support the header/auth feature at this time which I could not launch in the prod.

[jszwedko]

Thanks @ee07b415 . We are looking into that issue currently and so should hopefully be able to resolve it.