Provide more insight into indexing failures

[martijnvg]

The goal of this issue to do the following improvements in order to improve visibility into indexing failures:

• Start to keep track of indexing failures caused by a predefined exception classification in InternalIndexingStats. For example indexing failures caused by: VersionConflictEngineException, other client errors and internal server errors. These counter stats will then be exposes in IndexingStats and consumed by IndicesMetrics (which would stores indexing stats as apm metrics). (Metrics for indexing failures due to version conflicts #119067)
• Add new debug logging for indexing failures. Add a new IndexingOperationListener implementation that logs indexing failure as debug using a new logger. The postIndex(ShardId, Engine.Index, Exception) method should be overwritten to log exceptions and the postIndex(ShardId, Engine.Index, Engine.IndexResult) method should delegate to it if the index result type is failure. (Add additional debug logging for indexing failure. #117728)

[elasticsearchmachine]

Pinging @elastic/es-storage-engine (Team:StorageEngine)

[elasticsearchmachine]

Pinging @elastic/es-distributed-indexing (Team:Distributed Indexing)

[consulthys]

@martijnvg I see this issue is still open, is there anything else that is being worked on in the context of providing more insights into indexing failures? Thanks

[martijnvg]

This issue is complete. I just didn't get closed. I updated the description to link all sub tasks to pull requests.