Fix deadlock in SearchPhaseControllerTests cancellation tests

  The cancellation tests could deadlock when threads are delayed by OS
  scheduling. If cancellation triggers before all threads start, late
  threads may hit a code path where batchReduceSize causes the latch
  callback to be deferred to a MergeTask. Under certain timing conditions,
  these callbacks never execute, causing latch.await() to hang indefinitely.

  Ensure latch.countDown() is always called by wrapping consumeResult in
  try-catch. This guarantees test completion regardless of cancellation
  timing or exceptions.

  Fixes opensearch-project#19094

Signed-off-by: Atri Sharma <atri.jiit@gmail.com>

Author: atris

server/src/test/java/org/opensearch/action/search/SearchPhaseControllerTests.java

@@ -1826,7 +1826,13 @@ private static void consumeShardLevelQueryPhaseResultsAsync(int expectedNumResul
                 result.setShardIndex(index);
                 result.size(1);
 
-                consumer.consumeResult(result, latch::countDown);
+                try {
+                    consumer.consumeResult(result, latch::countDown);
+                } catch (Exception e) {
+                    // Ensure latch counts down even on cancellation
+                    latch.countDown();
+                    // Don't rethrow - let the thread complete normally
+                }
             });
             threads[index].start();
         }