Skip to content

[Metrics] Improve ES Output metrics #1231

@lucabelluccini

Description

@lucabelluccini

The common problem in Logstash performance cases is to determine if there is slowness in Logstash itself, the inputs or if there’s pushback / backpressure from the outputs.
I think it would be valuable to enhance the Elasticsearch output, adding some histograms/stats about:

  • Number of concurrent requests / in flight to Elasticsearch
  • Average/Distribution of RTT of indexing request to Elasticsearch
  • Occurrence of errors (retriable, not retriable), possibly with status codes (429, 503, 502…)
  • Distribution of bulk requests sizes (both in terms of bytes per bulk, in terms of number of documents per bulk)
  • Whenever possible, distribution of different data streams or index names involved per bulk
  • Other useful metrics

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions