Implement CCS telemetry export as part of _cluster/stats

[smalyshev]

Adds this structure to stats:

"ccs": {
     "_search": {
        "total": 7,
        "success": 7,
        "skipped": 0,
        "took": {
            "max": 36,
            "avg": 20,
            "p90": 33
        },
        "took_mrt_true": {
            "max": 33,
            "avg": 15,
            "p90": 33
        },
        "took_mrt_false": {
            "max": 36,
            "avg": 26,
            "p90": 36
        },
        "remotes_per_search_max": 3,
        "remotes_per_search_avg": 2.0,
        "failure_reasons": { ... },
        "features": { ... },
        "clients": { ... },
        "clusters": { ... }
     }
  }

with telemetry data about CCS searches.

[github-actions]

Documentation preview:

• ✨ Changed pages

[elasticsearchmachine]

Pinging @elastic/es-search-foundations (Team:Search Foundations)

[quux00]

A few changes requested.

Also, this has fewer tests / test changes than I would have guessed. Is ClusterStatsMonitoringDocTests.java and the docs tests the only ones to change for this?

Ideas on how we might generate other tests that also exercise searches - such as an IT test like you did for populating the CCSUsageTelemetry class?

[quux00]

Note: I usually also ask the PM to review the end user docs.
cc: @naj-h

[smalyshev]

Yes, that's a good idea.

[naj-h]

Will the include_remotes option be available to enable these metrics? If so, should we document it?

[smalyshev]

@naj-h Yes on both, in the next pull following this one.

[smalyshev]

Ideas on how we might generate other tests that also exercise searches - such as an IT test like you did for populating the CCSUsageTelemetry class?

We already test the collection part and the JSON rendering part in existing tests, so the new test only verifies the part where it is connected to a specific stats request. I plan to do a bit more roundtrip testing with remote clusters functionality but for this I think most of it is already covered. But if you see parts that are missing we could add more.

[quux00]

LGTM