Fix testSearchConcurrencyDoesNotCreateMoreTasksThanThreads failure (#116204)

This test was somehow difficult to write in the first place. We had to come up
with a threshold of how many tasks max are going to be created, but that is
not that easy to calculate as it depends on how quickly such tasks can be created
and be executed.

We should have rather used a higher threshold to start with, the important part
is anyways that we create a total of tasks that is no longer dependent on the
number of segments, given there are much less threads available to execute them.

Closes #116048

Author: javanna

muted-tests.yml

@@ -289,9 +289,6 @@ tests:
 - class: org.elasticsearch.upgrades.FullClusterRestartIT
   method: testSnapshotRestore {cluster=OLD}
   issue: https://github.com/elastic/elasticsearch/issues/111777
-- class: org.elasticsearch.search.DefaultSearchContextTests
-  method: testSearchConcurrencyDoesNotCreateMoreTasksThanThreads
-  issue: https://github.com/elastic/elasticsearch/issues/116048
 - class: org.elasticsearch.analysis.common.CommonAnalysisClientYamlTestSuiteIT
   issue: https://github.com/elastic/elasticsearch/issues/116134
 - class: org.elasticsearch.xpack.ml.integration.DatafeedJobsRestIT

server/src/test/java/org/elasticsearch/search/DefaultSearchContextTests.java

@@ -1056,8 +1056,9 @@ public int size() {
         assertThat(executor.getCompletedTaskCount(), greaterThanOrEqualTo((long) numIters * executorPoolSize));
         // while we parallelize we also limit the number of tasks that each searcher submits
         assertThat(executor.getCompletedTaskCount(), lessThan((long) numIters * numSegmentTasks));
-        // *3 is just a wild guess to account for tasks that get executed while we are still submitting
-        assertThat(executor.getCompletedTaskCount(), lessThan((long) numIters * executorPoolSize * 3));
+        // *10 is just a wild guess to account for tasks that get executed while we are still submitting. The important bit is that it is a
+        // much lower factor than the number of segments
+        assertThat(executor.getCompletedTaskCount(), lessThan((long) numIters * executorPoolSize * 10));
     }
 
     private void doTestSearchConcurrency(ThreadPoolExecutor executor, int numIters, int numSegmentTasks, AtomicInteger completedTasks)