meaning of vector_source_lag_time_seconds

[evanzhang87]

We are replacing logstash with Vector, our filebeat send data to vector via logstash source, but it looks not very fast.

vector_source_lag_time_seconds_bucket{component_id="logstash",component_kind="source",component_type="logstash",le="0.015625"} 0
vector_source_lag_time_seconds_bucket{component_id="logstash",component_kind="source",component_type="logstash",le="0.03125"} 0
vector_source_lag_time_seconds_bucket{component_id="logstash",component_kind="source",component_type="logstash",le="0.0625"} 0
vector_source_lag_time_seconds_bucket{component_id="logstash",component_kind="source",component_type="logstash",le="0.125"} 12
vector_source_lag_time_seconds_bucket{component_id="logstash",component_kind="source",component_type="logstash",le="0.25"} 301
vector_source_lag_time_seconds_bucket{component_id="logstash",component_kind="source",component_type="logstash",le="0.5"} 59930
vector_source_lag_time_seconds_bucket{component_id="logstash",component_kind="source",component_type="logstash",le="1"} 623390
vector_source_lag_time_seconds_bucket{component_id="logstash",component_kind="source",component_type="logstash",le="2"} 1254199
vector_source_lag_time_seconds_bucket{component_id="logstash",component_kind="source",component_type="logstash",le="4"} 1624082
vector_source_lag_time_seconds_bucket{component_id="logstash",component_kind="source",component_type="logstash",le="8"} 2325502
vector_source_lag_time_seconds_bucket{component_id="logstash",component_kind="source",component_type="logstash",le="16"} 2971113
vector_source_lag_time_seconds_bucket{component_id="logstash",component_kind="source",component_type="logstash",le="32"} 2976016
vector_source_lag_time_seconds_bucket{component_id="logstash",component_kind="source",component_type="logstash",le="64"} 2979728
vector_source_lag_time_seconds_bucket{component_id="logstash",component_kind="source",component_type="logstash",le="128"} 2979811
vector_source_lag_time_seconds_bucket{component_id="logstash",component_kind="source",component_type="logstash",le="256"} 2979812
vector_source_lag_time_seconds_bucket{component_id="logstash",component_kind="source",component_type="logstash",le="512"} 2979834
vector_source_lag_time_seconds_bucket{component_id="logstash",component_kind="source",component_type="logstash",le="1024"} 2979834
vector_source_lag_time_seconds_bucket{component_id="logstash",component_kind="source",component_type="logstash",le="2048"} 2979834
vector_source_lag_time_seconds_bucket{component_id="logstash",component_kind="source",component_type="logstash",le="4096"} 2979834
vector_source_lag_time_seconds_bucket{component_id="logstash",component_kind="source",component_type="logstash",le="+Inf"} 2979834

The Doc explained: The difference between the timestamp recorded in each event and the time when it was ingested, expressed as fractional seconds.
Data sent by filebeat will have a time key @timestamp, so is the metric comes from here?
Furthermore, according to my metric value, is the logstash source a bottleneck?

[jszwedko]

Data sent by filebeat will have a time key @timestamp, so is the metric comes from here?

That's correct. It should indicate the difference between @timestamp and the time in which the logstash source received the event.

Furthermore, according to my metric value, is the logstash source a bottleneck?

It could be, but not necessarily. If filebeat held onto the event for a while before sending it to Vector it would also manifest as high lag time. In your case it looks like all events were below 512 seconds of lag and the vast majority (80%) were under 8 seconds of lag.