Failed document handler

[felixbarny]

Instead of dropping documents that have failed ingestion due to an exception during pipeline execution or indexing, it should be possible to store the failed document.

High-level options where to store the failed documents:

1. In a dedicated data stream
2. Within the same data stream, in dedicated failure backing indices
3. Within the same index, only storing the _source and with an indicator that these reflect failed documents

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[lachlann562]

This would be extremely helpful, we recently discovered we were losing a huge number of messages because of inconsistency in data type and we couldn't find any easy way to identify these messages or recover the data associated with them. The abscence of log records is a huge risk when you are reliant on ELK as the "source of truth".

[ruflin]

@lachlann562 Glad to hear this will be useful for you. To also expose the failure in the UI as soon as the feature lands, we are working on a dataset quality page. Some related issue can be found here: elastic/kibana#184572

[flash1293]

@felixbarny can this be closed?

[wolframhaussig]

@felixbarny can this be closed?

Why should this be closed? It would be extremely helpful and reduce our effort to look for the missing data in a long pipe of components (app->openshift->ingest pipeline root->ingest pipeline app->index).