es_dead_letter_index

[ltflb-bgdi]

Describe the enhancement:

The new output option es_deadletter_index allows to send events which returned from elastic with an error code to a dedicated deadletter index/datastream. This works very well to analyse issues like mapping errors etc.

Currently it seems that failing messages are forwarded to the deadletter index regardless of the error type.
IMO it would make sense to improve this behavior and not forward messages with certain types of errors.
See the example below.

Describe a specific use case for the enhancement or feature:

Our deadletter index contains events which failed due to read timeouts.

ConnectionTimeout caused by - ReadTimeoutError(HTTPSConnectionPool(host='acme.es.eu-central-1.aws.cloud.es.io', port=443): Read timed out. (read timeout=30))

These errors are caused by cloudfront log congestion during some peak hours or when redriving events from the continuing dealetter que. IMO these events should not be sent to the deadletter queue, since they would be kind lost (unparsed) and cause even more congestion. These cases should be still handled by the continuing queue and finally the continuing dlq.

[zmoog]

@ltflb-bgdi, we release version v1.18.0 that contains one change that should improve this scenario.

Starting from v1.18.0, the ES output will not forward retryable errors to the dead letter index (DLI).

For example, if the Elasticsearch client returns an error that does not contain the HTTP status code (probably a connection error) or the status code is 429, the ES output will skip the DLI.