[Profiling] Ignore events count value

[rockdaboot]

Background

To display statistically sound data, Universal Profiling queries require to fetch a lot of documents (aka profiling events). This can be more than 100k, even if only 20k documents are needed.

For this, the number of documents allowed in a single response is set to 150k. It also requires to increase a cluster-wide setting search.max_buckets to 150k. The alternative, to paginate the response, is/was not an option as it increased the query latency unacceptably.

The new (and still experimental) random_sampler aggregation can not be used as every profiling event document has a weight (aka count).

Problem

On serverless, the cluster-wide setting search.max_buckets can no longer be changed and it defaults to 64k. The option to fetch the data in a paginated way is too slow (up to 15 sequential requests).

Solution

The profiling events were recently switched to have nanosecond precise timestamps.
This made it very unlikely to have events with a count value != 1.
So with ES 9.2, the count value is either dropped or always set to 1 (see also open-telemetry/opentelemetry-collector-contrib#40947).

With profiling event documents having all the same weight, we can leverage the random_sampler aggregation to reduce the number of documents to be fetched to a maximum of 20k. This allows using a paginated response (1 additional request/response roundtrip) without massively increasing the total query latency.

This PR is the first step, using the aggregated doc_count value instead of using aggregated count values. In a second PR, we'll switch to use the random_sampler aggregation in combination with pagination.

[elasticsearchmachine]

Pinging @elastic/obs-ux-infra_services-team (Team:obs-ux-infra_services)