[CI] DockerTests test151MachineDependentHeapWithSizeOverride failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-periodic-packaging #9262 / rhel-9_packaging-tests-unix
• elasticsearch-periodic-packaging #9262 / almalinux-8_packaging-tests-unix
• elasticsearch-pull-request #82802 / ubuntu-2404_destructivedistrotest.docker_packaging-tests-unix-sample
• elasticsearch-pull-request #82639 / ubuntu-2404_destructivedistrotest.docker_packaging-tests-unix-sample
• elasticsearch-periodic-packaging #9220 / rhel-7_packaging-tests-unix
• elasticsearch-periodic-packaging #9164 / opensuse-leap-15_packaging-tests-unix
• elasticsearch-periodic-packaging #9150 / rhel-8_packaging-tests-unix
• elasticsearch-periodic-packaging #9136 / ubuntu-2204_packaging-tests-unix

Reproduction Line:

null

Applicable branches:
8.17

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

org.apache.http.NoHttpResponseException: localhost:9200 failed to respond

Issue Reasons:

• [8.17] 2 consecutive failures in step ubuntu-2404_destructivedistrotest.docker_packaging-tests-unix-sample
• [8.17] 8 failures in test test151MachineDependentHeapWithSizeOverride (1.2% fail rate in 645 executions)
• [8.17] 2 failures in step ubuntu-2404_destructivedistrotest.docker_packaging-tests-unix-sample (66.7% fail rate in 3 executions)
• [8.17] 5 failures in pipeline elasticsearch-periodic-packaging (31.3% fail rate in 16 executions)
• [8.17] 2 failures in pipeline elasticsearch-pull-request (50.0% fail rate in 4 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

This has been muted on branch main

Mute Reasons:

• [main] 21 failures in test test151MachineDependentHeapWithSizeOverride (2.1% fail rate in 986 executions)
• [main] 11 failures in step ubuntu-2004_destructivedistrotest.docker_packaging-tests-unix-sample (5.7% fail rate in 193 executions)
• [main] 9 failures in step rhel-8_destructivedistrotest.docker_packaging-tests-unix-sample (4.7% fail rate in 193 executions)
• [main] 16 failures in pipeline elasticsearch-pull-request (8.3% fail rate in 193 executions)

Build Scans:

• elasticsearch-periodic-packaging #6320 / opensuse-leap-15_packaging-tests-unix
• elasticsearch-pull-request #59272 / rhel-8_destructivedistrotest.docker_packaging-tests-unix-sample
• elasticsearch-pull-request #59268 / ubuntu-2004_destructivedistrotest.docker_packaging-tests-unix-sample
• elasticsearch-pull-request #59268 / rhel-8_destructivedistrotest.docker_packaging-tests-unix-sample
• elasticsearch-pull-request #59266 / ubuntu-2004_destructivedistrotest.docker_packaging-tests-unix-sample
• elasticsearch-pull-request #59261 / rhel-8_destructivedistrotest.docker_packaging-tests-unix-sample
• elasticsearch-pull-request #59259 / rhel-8_destructivedistrotest.docker_packaging-tests-unix-sample
• elasticsearch-pull-request #59259 / ubuntu-2004_destructivedistrotest.docker_packaging-tests-unix-sample
• elasticsearch-pull-request #59258 / rhel-8_destructivedistrotest.docker_packaging-tests-unix-sample
• elasticsearch-pull-request #59243 / ubuntu-2004_destructivedistrotest.docker_packaging-tests-unix-sample

[elasticsearchmachine]

Pinging @elastic/es-delivery (Team:Delivery)

[elasticsearchmachine]

This has been muted on branch 9.0

Mute Reasons:

• [9.0] 9 failures in test test151MachineDependentHeapWithSizeOverride (4.8% fail rate in 189 executions)
• [9.0] 3 failures in step rocky-8_packaging-tests-unix (75.0% fail rate in 4 executions)
• [9.0] 3 failures in pipeline elasticsearch-periodic-packaging (75.0% fail rate in 4 executions)
• [9.0] 2 failures in pipeline elasticsearch-pull-request (11.1% fail rate in 18 executions)

Build Scans:

• elasticsearch-periodic-packaging #6343 / rocky-8_packaging-tests-unix
• elasticsearch-pull-request #59489 / ubuntu-2004_destructivedistrotest.docker_packaging-tests-unix-sample
• elasticsearch-periodic-packaging #6336 / rocky-8_packaging-tests-unix
• elasticsearch-periodic-packaging #6336 / almalinux-8_packaging-tests-unix
• elasticsearch-pull-request #59343 / rhel-8_destructivedistrotest.docker_packaging-tests-unix-sample
• elasticsearch-periodic-packaging #6329 / rhel-8_packaging-tests-unix
• elasticsearch-periodic-packaging #6329 / almalinux-9_packaging-tests-unix
• elasticsearch-periodic-packaging #6329 / rocky-8_packaging-tests-unix
• elasticsearch-periodic-packaging #6329 / rhel-9_packaging-tests-unix

[slobodanadamovic]

The test fails after 3m 25.409s. The cluster logs show attempts to connect via HTTP instead of HTTPS.

{"@timestamp":"2025-07-14T22:40:29.336Z", "log.level": "WARN", "message":"received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/172.17.0.2:9200, remoteAddress=/172.17.0.1:36258}", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[6e09c04df3b0][transport_worker][T#4]","log.logger":"org.elasticsearch.http.netty4.Netty4HttpServerTransport","elasticsearch.cluster.uuid":"RVjAzbVaSCW3BtQ_cOz0Fg","elasticsearch.node.id":"_FPmSma0T2ekXHYAt6UoQA","elasticsearch.node.name":"6e09c04df3b0","elasticsearch.cluster.name":"docker-cluster"}	

The waitForElasticSearch method in ServerUtils waits arround ~3 minutes. It only tries to get HTTP CA certificate at the beginning of the method. Then loops and calls _cluster/health based on the presence of CA certificate when the method got first called.

This is an issue because the auto-configuration that generates HTTP certificates can take longer (like in this case). I believe the solution here is to move fetching of CA certificate within the retry loop.

Caused by: java.net.SocketException: Connection reset	
at sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:318)	
•••
at org.elasticsearch.packaging.util.ServerUtils.execute(ServerUtils.java:153)	
at org.elasticsearch.packaging.util.ServerUtils.waitForElasticsearch(ServerUtils.java:249)	
at org.elasticsearch.packaging.util.docker.Docker.lambda$waitForElasticsearch$10(Docker.java:611)	
at org.elasticsearch.packaging.util.docker.Docker.withLogging(Docker.java:628)	
at org.elasticsearch.packaging.util.docker.Docker.waitForElasticsearch(Docker.java:611)	
at org.elasticsearch.packaging.util.docker.Docker.waitForElasticsearch(Docker.java:598)	
at org.elasticsearch.packaging.test.DockerTests.machineDependentHeapTest(DockerTests.java:1064)	
at org.elasticsearch.packaging.test.DockerTests.test151MachineDependentHeapWithSizeOverride(DockerTests.java:1035)	
•••