Metrics for failed docs

[flash1293]

Description

In the context of the data quality page / quality tab in streams, we aim to give the user a comprehensive overview over whether their data is stored successfully. Right now there are two data sources for this: the failure store and the _ignored field. However, there is another common category of ingestion-related configuration errors which is very hard to debug currently: permission problems.

In case the shipper is not properly configured to send data to the targeted destination, data is dropped (almost) silently. There are some messages in the local shipper logs, but those are hard to access depending on the setup.

It would be great if we could make the user aware within the UI as part of the quality tab.

The idea is to have a counter or similar to keep track of attempts to send data to a certain data stream:

• In case a doc within a _bulk request is dropped because the user doesn't have write permissions to the targeted index, increase the per-datastream counter and set a per-datastream "last occurred" timestamp in memory locally on the node
• When fetching data stream stats, collect the values for these counters and "last occurred" values and include them in the response
• In the UI, show this information along with the other quality-related information
• This only happens if the whole request isn't rejected wholesale (e.g. because there is no API key provided or the API key is invalid/doesn't exist)
• If a non-existing data stream is targeted, no information is collected as well (only for cases where the targeted data stream exists already)

Open questions:

• What information to capture exactly? Just a counter and "last occurred" timestamp or more than that?