WIP: Batch + move parts of reduce during query phase execution to data nodes #118490
Conversation
This is not used as ref-counted, we never increment the count so we can simplify things a little and make it just a releasable.
This redose CanMatchPreFilterSearchPhase to be simpler and without the needless heavy locking. We were locking on every shard result which could introduce a lot of contention when a large number of data nodes (especially when they hold large shard counts) are queried. We probably could even do without the volatile in some cases but either way, this commit does away with any contention on the results since we will never access the same cell in the result array from more than a single thread. There is no need to have a bitset here and deal with the synchronization it entails when we at the same time also set up two arrays of shard_count length and only need a single array to begin with. => share array for failures, min-max and matches, remove all synchronization, further simplify the code.
We only need this rather large map in `run`, lets create it on the fly there to save the rather large redundant field and save on state and complexity in general.
We don't need to use this request, the handler for freeing of scroll requests literally goes to the same transport handler and doesn't come with the list of indices. The original security need for keeping the list of indices around is long gone.
… all-in-one-query-change
…l-in-one-query-change
…l-in-one-query-change
… all-in-one-query-change
|
It looks like this PR modifies one or more |
|
It looks like this PR modifies one or more |
|
It looks like this PR modifies one or more |
|
It looks like this PR modifies one or more |
|
It looks like this PR modifies one or more |
|
It looks like this PR modifies one or more |
|
It looks like this PR modifies one or more |
|
It looks like this PR modifies one or more |
|
It looks like this PR modifies one or more |
|
It looks like this PR modifies one or more |
|
It looks like this PR modifies one or more |
|
It looks like this PR modifies one or more |
|
It looks like this PR modifies one or more |
|
Closing in favor of #121885, with the latest round of cleanups we have a shortcut to finish this it turns out. |
2 participants
WIP: please hold off on reviewing, there's about 500 lines of changes missing in this one yet and this is merely to enable me to run tests on CI a couple times. Should be ready to review within the next 24h though.