Skip to content

Force Connect Test: Fix flaky test caused by unreliable remote cluster stalling #132641

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
pawankartik-elastic merged 3 commits into from
Aug 11, 2025
Merged

Force Connect Test: Fix flaky test caused by unreliable remote cluster stalling #132641

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
ca0f209
Fix flaky test caused by unreliable remote cluster stalling
pawankartik-elastic Aug 11, 2025
a15e556
Merge branch 'main' into pkar/force-connect-test-fix
pawankartik-elastic Aug 11, 2025
21de54a
Fix line ending
pawankartik-elastic Aug 11, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 0 additions & 6 deletions muted-tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -455,9 +455,6 @@ tests:
- class: org.elasticsearch.xpack.esql.ccq.MultiClustersIT
method: testLookupJoinAliasesSkipOld
issue: https://github.com/elastic/elasticsearch/issues/131697
- class: org.elasticsearch.indices.cluster.RemoteSearchForceConnectTimeoutIT
method: testTimeoutSetting
issue: https://github.com/elastic/elasticsearch/issues/131656
- class: org.elasticsearch.packaging.test.DockerTests
method: test151MachineDependentHeapWithSizeOverride
issue: https://github.com/elastic/elasticsearch/issues/123437
Expand All @@ -473,9 +470,6 @@ tests:
- class: org.elasticsearch.index.engine.MergeWithLowDiskSpaceIT
method: testRelocationWhileForceMerging
issue: https://github.com/elastic/elasticsearch/issues/131789
- class: org.elasticsearch.indices.cluster.FieldCapsForceConnectTimeoutIT
method: testTimeoutSetting
issue: https://github.com/elastic/elasticsearch/issues/132179
- class: org.elasticsearch.test.rest.yaml.MDPYamlTestSuiteIT
method: test {yaml=mdp/10_basic/Index using shared data path}
issue: https://github.com/elastic/elasticsearch/issues/132223
Expand Down
18 changes: 10 additions & 8 deletions ...nalClusterTest/java/org/elasticsearch/indices/cluster/FieldCapsForceConnectTimeoutIT.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,21 +73,23 @@ protected Map<String, Boolean> skipUnavailableForRemoteClusters() {

public void testTimeoutSetting() {
var latch = new CountDownLatch(1);
for (String nodeName : cluster(LOCAL_CLUSTER).getNodeNames()) {
MockTransportService mts = (MockTransportService) cluster(LOCAL_CLUSTER).getInstance(TransportService.class, nodeName);

mts.addConnectBehavior(
cluster(LINKED_CLUSTER_1).getInstance(TransportService.class, (String) null),
((transport, discoveryNode, profile, listener) -> {
for (String localNodeName : cluster(LOCAL_CLUSTER).getNodeNames()) {
MockTransportService localMts = (MockTransportService) cluster(LOCAL_CLUSTER).getInstance(
TransportService.class,
localNodeName
);
for (String remoteNodeName : cluster(LINKED_CLUSTER_1).getNodeNames()) {
TransportService remoteTs = cluster(LINKED_CLUSTER_1).getInstance(TransportService.class, remoteNodeName);
localMts.addConnectBehavior(remoteTs, ((transport, discoveryNode, profile, listener) -> {
Copy link
Contributor

@drempapis drempapis Aug 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I understand well, the updated code blocks all remote targets per local node, making the timeout behavior-assertions reliable.

try {
latch.await();
} catch (InterruptedException e) {
throw new AssertionError(e);
}

transport.openConnection(discoveryNode, profile, listener);
})
);
}));
}
}

// Add some dummy data to prove we are communicating fine with the remote.
Expand Down
18 changes: 10 additions & 8 deletions ...ClusterTest/java/org/elasticsearch/indices/cluster/RemoteSearchForceConnectTimeoutIT.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,21 +70,23 @@ protected Map<String, Boolean> skipUnavailableForRemoteClusters() {

public void testTimeoutSetting() {
var latch = new CountDownLatch(1);
for (String nodeName : cluster(LOCAL_CLUSTER).getNodeNames()) {
MockTransportService mts = (MockTransportService) cluster(LOCAL_CLUSTER).getInstance(TransportService.class, nodeName);

mts.addConnectBehavior(
cluster(REMOTE_CLUSTER_1).getInstance(TransportService.class, (String) null),
((transport, discoveryNode, profile, listener) -> {
for (String localNodeName : cluster(LOCAL_CLUSTER).getNodeNames()) {
MockTransportService localMts = (MockTransportService) cluster(LOCAL_CLUSTER).getInstance(
TransportService.class,
localNodeName
);
for (String remoteNodeName : cluster(REMOTE_CLUSTER_1).getNodeNames()) {
TransportService remoteTs = cluster(REMOTE_CLUSTER_1).getInstance(TransportService.class, remoteNodeName);
localMts.addConnectBehavior(remoteTs, ((transport, discoveryNode, profile, listener) -> {
try {
latch.await();
} catch (InterruptedException e) {
throw new AssertionError(e);
}

transport.openConnection(discoveryNode, profile, listener);
})
);
}));
}
}

// Add some dummy data to prove we are communicating fine with the remote.
Expand Down