Fix SO-Error in SearchWithRandomDisconnectsIT (#122593)

Obvious SO exception possibilitiy if we encounter exceptions back to back in the callback.
Use promise style pattern instead of callback in the individual loops to limit stack-depth
(this should be good enough for a fix, technically you could still run into very deep stacks
if the search completes between the `isDone` check and adding the listener back-to-back
a couple times but practically this should be good enough since all the instant-done situations
are from the search fully failing outright.

closes #116175

Author: original-brownbear

muted-tests.yml

@@ -72,9 +72,6 @@ tests:
 - class: org.elasticsearch.action.search.SearchPhaseControllerTests
   method: testProgressListener
   issue: https://github.com/elastic/elasticsearch/issues/116149
-- class: org.elasticsearch.search.basic.SearchWithRandomDisconnectsIT
-  method: testSearchWithRandomDisconnects
-  issue: https://github.com/elastic/elasticsearch/issues/116175
 - class: org.elasticsearch.xpack.deprecation.DeprecationHttpIT
   method: testDeprecatedSettingsReturnWarnings
   issue: https://github.com/elastic/elasticsearch/issues/108628

server/src/internalClusterTest/java/org/elasticsearch/search/basic/SearchWithRandomDisconnectsIT.java

@@ -14,6 +14,7 @@
 import org.elasticsearch.action.search.SearchResponse;
 import org.elasticsearch.action.support.PlainActionFuture;
 import org.elasticsearch.common.settings.Settings;
+import org.elasticsearch.common.util.concurrent.ListenableFuture;
 import org.elasticsearch.discovery.AbstractDisruptionTestCase;
 import org.elasticsearch.index.IndexModule;
 import org.elasticsearch.index.IndexSettings;
@@ -67,11 +68,15 @@ public void onFailure(Exception e) {
                 }
 
                 private void runMoreSearches() {
-                    if (done.get() == false) {
-                        prepareRandomSearch().execute(this);
-                    } else {
-                        finishFuture.onResponse(null);
+                    while (done.get() == false) {
+                        final ListenableFuture<SearchResponse> f = new ListenableFuture<>();
+                        prepareRandomSearch().execute(f);
+                        if (f.isDone() == false) {
+                            f.addListener(this);
+                            return;
+                        }
                     }
+                    finishFuture.onResponse(null);
                 }
             });
         }