Catch up DLS with recent Lucene changes.

[jpountz]

Lucene introduced the Bits#applyMask API to speed up the evaluation of non-scoring queries (e.g. aggregations-only queries) and is considering making BitSet a sealed class that only allows FixedBitSet and SparseFixedBitSet as sub-classes to control the performance impact of virtual calls.

As a consequence, this change:

• renames CombinedBitSet to CombinedBits,
• makes it implement Bits instead of extending BitSet,
• implements CombinedBits#applyMask,
• refactors how the query and live docs are intersected in ContextIndexSearcher.

Closes #120627

[elasticsearchmachine]

Pinging @elastic/es-security (Team:Security)

[tvernum]

Ping @joegallo, FYI.

[tvernum]

LGTM (I think I understood it all)

[tvernum]

Is there a reason we use a mix of WINDOW_SIZE and bitSet.length() ?

They should be the same, and it looks like the code relies on them being the same, but perhaps you had a deeper for reason mixing them.

[jpountz]

No reason! Let me use WINDOW_SIZE everywhere.

[joegallo]

LGTM

[joegallo]

I updated my previous benchmark for testing DLS to account for these changes. Specifically, #120627 mentions Lucene 10.2 is introducing a new Bits#applyMask API that is important to have good query evaluation performance in the presence of deletes, so I added a step in my benchmark that deletes 5% of the docs at random from the eight metricbeat indices (each index has approximately 1 million docs, and I delete 50,000 from each of them).

Looking at the flamegraphs, the speed up is pretty clear:

The time spent in any .*Bits.applyMask method drops from 5.12% of all profiled cpu time to 1.26%. Everything else stays more or less the same outside of that, though.

In terms of the overall speed up, here's what means for the high level operation of running a search via a DLS role that hits an index for which there are deleted documents:

MAIN:
=====

(cache constrained to 48mb, lots of cache churn)
|                                                Mean Throughput |              dls-search |  1121.89        |  ops/s |
|                                              Median Throughput |              dls-search |  1123.23        |  ops/s |
|                                        50th percentile latency |              dls-search |    98.781       |     ms |
|                                        90th percentile latency |              dls-search |   170.912       |     ms |
|                                        99th percentile latency |              dls-search |   263.398       |     ms |

(unconstrained cache, lots of cache hits)
|                                                Mean Throughput |              dls-search |  2038.01        |  ops/s |
|                                              Median Throughput |              dls-search |  2042.57        |  ops/s |
|                                        50th percentile latency |              dls-search |    55.4865      |     ms |
|                                        90th percentile latency |              dls-search |    87.5875      |     ms |
|                                        99th percentile latency |              dls-search |   132.864       |     ms |

THIS PR:
========

(cache constrained to 48mb, lots of cache churn)
|                                                Mean Throughput |              dls-search |  1152.41        |  ops/s |
|                                              Median Throughput |              dls-search |  1152.34        |  ops/s |
|                                        50th percentile latency |              dls-search |    95.733       |     ms |
|                                        90th percentile latency |              dls-search |   167.011       |     ms |
|                                        99th percentile latency |              dls-search |   260.01        |     ms |

(unconstrained cache, lots of cache hits)
|                                                Mean Throughput |              dls-search |  2180.96        |  ops/s |
|                                              Median Throughput |              dls-search |  2190.19        |  ops/s |
|                                        50th percentile latency |              dls-search |    51.5064      |     ms |
|                                        90th percentile latency |              dls-search |    81.5829      |     ms |
|                                        99th percentile latency |              dls-search |   125.111       |     ms |

If you compare the main versus this-pr case, you'll notice that there's a modest but real improvement (a few percent, in this benchmark) in both throughput and latency for both the cache-constrained and non cache-constrained cases.