Add APM metrics for duration of search phases at the coordinating node

[chrisparrinello]

New metrics created:

• es.search.coordinator.phases.can_match.duration.histogram
• es.search.coordinator.phases.dfs.duration.histogram
• es.search.coordinator.phases.dfs_query.duration.histogram
• es.search.coordinator.phases.expand.duration.histogram
• es.search.coordinator.phases.fetch.duration.histogram
• es.search.coordinator.phases.fetch_lookup_fields.duration.histogram
• es.search.coordinator.phases.rank_feature.duration.histogram

[elasticsearchmachine]

Pinging @elastic/es-search-foundations (Team:Search Foundations)

[elasticsearchmachine]

Hi @chrisparrinello, I've created a changelog YAML for you.

[javanna]

Hey @chrisparrinello , would it make sense to start with one latency measured on the coord node, and next expand to others in following PRs?

Take for instance DFS, or query, or fetch, to start with. We only measure these so far on the shards, but we'd like to see the overall latency for these on the coord nodel so that it is comparable to the took time.

[chrisparrinello]

@javanna Yeah I think I might need to take that approach with this as well based on the CI results which I've run into issues around opening the point-in-time around "reflecting" the metric name from the phase. A more straightforward targeted metric collection is doable but the downside I think is that metric object might need to be passed around a little more to get it down to the SearchPhase implementations. I'll take a stab at the query phase first.