Unmute docker tests, add more logging and increase startup timeout #131203

slobodanadamovic · 2025-07-14T12:29:42Z

Unmuting 33 DockerTests with added additional logging on failures.
Also, bumping docker startup timeout to account for slow security
auto-configuration, and additionally moving CA certificates inside
the retry while loop.

Resolves #129808
Resolves #123437
Resolves #120911
Resolves #120914
Resolves #120910
Resolves #120918
Resolves #120925
Resolves #124990
Resolves #116636
Resolves #125680
Resolves #116147
Resolves #126232
Resolves #126145
Resolves #116866
Resolves #126936
Resolves #127081
Resolves #127158
Resolves #127138
Resolves #128018
Resolves #128031
Resolves #128110
Resolves #128006
Resolves #128174
Resolves #128144
Resolves #128165
Resolves #128075
Resolves #128117
Resolves #128121
Resolves #128115
Resolves #128230
Resolves #128120
Resolves #128996
Resolves #128867

- bump docker startup timeout to account for slow security autoconfiguration Resolves elastic#129808

slobodanadamovic · 2025-07-15T10:23:21Z

qa/packaging/src/test/java/org/elasticsearch/packaging/util/ServerUtils.java

        while (started == false && timeElapsed < waitTime) {
            if (System.currentTimeMillis() - lastRequest > requestInterval) {
+                if (caCert == null) {
+                    caCert = getCaCert(installation);


In case of slow auto-configuration, the HTTP CA certificate may not have been configured when waitForElasticsearch method is called. This would result in received plaintext http traffic on an https channel, closing connection Netty4HttpChannel and test failure after 3 minutes. See this comment for more details: #123437 (comment)

slobodanadamovic · 2025-07-15T10:32:44Z

qa/packaging/src/test/java/org/elasticsearch/packaging/test/PackagingTestCase.java

            failed = true;
+            if (installation != null && installation.distribution.isDocker()) {
+                logger.warn("Test {} failed. Printing logs for failed test...", description.getMethodName());
+                FileUtils.logAllLogs(installation.logs, logger);


Some of the uncaught docker failures did not include any cluster logs, which makes it super hard to troubleshoot.

elasticsearchmachine · 2025-07-15T11:09:54Z

Pinging @elastic/es-security (Team:Security)

elasticsearchmachine · 2025-07-15T13:22:35Z

💔 Backport failed

Status	Branch	Result
❌	9.1	Commit could not be cherrypicked due to conflicts
❌	8.19	Commit could not be cherrypicked due to conflicts
❌	9.0	Commit could not be cherrypicked due to conflicts
❌	8.18	Commit could not be cherrypicked due to conflicts

You can use sqren/backport to manually backport by running backport --upstream elastic/elasticsearch --pr 131203

…lastic#131203) Unmuting 33 DockerTests with added additional logging on failures. Also, bumping docker startup timeout to account for slow security auto-configuration, and additionally moving CA certificates inside the retry while loop. Resolves elastic#129808 Resolves elastic#123437 Resolves elastic#120911 Resolves elastic#120914 Resolves elastic#120910 Resolves elastic#120918 Resolves elastic#120925 Resolves elastic#124990 Resolves elastic#116636 Resolves elastic#125680 Resolves elastic#116147 Resolves elastic#126232 Resolves elastic#126145 Resolves elastic#116866 Resolves elastic#126936 Resolves elastic#127081 Resolves elastic#127158 Resolves elastic#127138 Resolves elastic#128018 Resolves elastic#128031 Resolves elastic#128110 Resolves elastic#128006 Resolves elastic#128174 Resolves elastic#128144 Resolves elastic#128165 Resolves elastic#128075 Resolves elastic#128117 Resolves elastic#128121 Resolves elastic#128115 Resolves elastic#128230 Resolves elastic#128120 Resolves elastic#128996 Resolves elastic#128867 (cherry picked from commit 4806156) # Conflicts: # muted-tests.yml

slobodanadamovic · 2025-07-15T13:46:36Z

💚 All backports created successfully

Status	Branch	Result
✅	9.1
✅	9.0
✅	8.19
✅	8.18

Questions ?

Please refer to the Backport tool documentation

…lastic#131203) Unmuting 33 DockerTests with added additional logging on failures. Also, bumping docker startup timeout to account for slow security auto-configuration, and additionally moving CA certificates inside the retry while loop. Resolves elastic#129808 Resolves elastic#123437 Resolves elastic#120911 Resolves elastic#120914 Resolves elastic#120910 Resolves elastic#120918 Resolves elastic#120925 Resolves elastic#124990 Resolves elastic#116636 Resolves elastic#125680 Resolves elastic#116147 Resolves elastic#126232 Resolves elastic#126145 Resolves elastic#116866 Resolves elastic#126936 Resolves elastic#127081 Resolves elastic#127158 Resolves elastic#127138 Resolves elastic#128018 Resolves elastic#128031 Resolves elastic#128110 Resolves elastic#128006 Resolves elastic#128174 Resolves elastic#128144 Resolves elastic#128165 Resolves elastic#128075 Resolves elastic#128117 Resolves elastic#128121 Resolves elastic#128115 Resolves elastic#128230 Resolves elastic#128120 Resolves elastic#128996 Resolves elastic#128867 (cherry picked from commit 4806156) # Conflicts: # muted-tests.yml

…out (#131203) (#131293) * Unmute docker tests, add more logging and increase startup timeout (#131203) Unmuting 33 DockerTests with added additional logging on failures. Also, bumping docker startup timeout to account for slow security auto-configuration, and additionally moving CA certificates inside the retry while loop. Resolves #129808 Resolves #123437 Resolves #120911 Resolves #120914 Resolves #120910 Resolves #120918 Resolves #120925 Resolves #124990 Resolves #116636 Resolves #125680 Resolves #116147 Resolves #126232 Resolves #126145 Resolves #116866 Resolves #126936 Resolves #127081 Resolves #127158 Resolves #127138 Resolves #128018 Resolves #128031 Resolves #128110 Resolves #128006 Resolves #128174 Resolves #128144 Resolves #128165 Resolves #128075 Resolves #128117 Resolves #128121 Resolves #128115 Resolves #128230 Resolves #128120 Resolves #128996 Resolves #128867 (cherry picked from commit 4806156) # Conflicts: # muted-tests.yml * call dumpDebug

…eout (#131203) (#131294) * Unmute docker tests, add more logging and increase startup timeout (#131203) Unmuting 33 DockerTests with added additional logging on failures. Also, bumping docker startup timeout to account for slow security auto-configuration, and additionally moving CA certificates inside the retry while loop. Resolves #129808 Resolves #123437 Resolves #120911 Resolves #120914 Resolves #120910 Resolves #120918 Resolves #120925 Resolves #124990 Resolves #116636 Resolves #125680 Resolves #116147 Resolves #126232 Resolves #126145 Resolves #116866 Resolves #126936 Resolves #127081 Resolves #127158 Resolves #127138 Resolves #128018 Resolves #128031 Resolves #128110 Resolves #128006 Resolves #128174 Resolves #128144 Resolves #128165 Resolves #128075 Resolves #128117 Resolves #128121 Resolves #128115 Resolves #128230 Resolves #128120 Resolves #128996 Resolves #128867 (cherry picked from commit 4806156) # Conflicts: # muted-tests.yml * call dumpDebug

…eout (#131203) (#131295) * Unmute docker tests, add more logging and increase startup timeout (#131203) Unmuting 33 DockerTests with added additional logging on failures. Also, bumping docker startup timeout to account for slow security auto-configuration, and additionally moving CA certificates inside the retry while loop. Resolves #129808 Resolves #123437 Resolves #120911 Resolves #120914 Resolves #120910 Resolves #120918 Resolves #120925 Resolves #124990 Resolves #116636 Resolves #125680 Resolves #116147 Resolves #126232 Resolves #126145 Resolves #116866 Resolves #126936 Resolves #127081 Resolves #127158 Resolves #127138 Resolves #128018 Resolves #128031 Resolves #128110 Resolves #128006 Resolves #128174 Resolves #128144 Resolves #128165 Resolves #128075 Resolves #128117 Resolves #128121 Resolves #128115 Resolves #128230 Resolves #128120 Resolves #128996 Resolves #128867 (cherry picked from commit 4806156) # Conflicts: # muted-tests.yml * call dumpDebug

This test timed out due to slow security auto-configuration. The elastic#131203 increased timeout from 30s to 45s, which should hopefully be enough. Resolves elastic#131107

…out (#131203) (#131292) * Unmute docker tests, add more logging and increase startup timeout (#131203) Unmuting 33 DockerTests with added additional logging on failures. Also, bumping docker startup timeout to account for slow security auto-configuration, and additionally moving CA certificates inside the retry while loop. Resolves #129808 Resolves #123437 Resolves #120911 Resolves #120914 Resolves #120910 Resolves #120918 Resolves #120925 Resolves #124990 Resolves #116636 Resolves #125680 Resolves #116147 Resolves #126232 Resolves #126145 Resolves #116866 Resolves #126936 Resolves #127081 Resolves #127158 Resolves #127138 Resolves #128018 Resolves #128031 Resolves #128110 Resolves #128006 Resolves #128174 Resolves #128144 Resolves #128165 Resolves #128075 Resolves #128117 Resolves #128121 Resolves #128115 Resolves #128230 Resolves #128120 Resolves #128996 Resolves #128867 (cherry picked from commit 4806156) # Conflicts: # muted-tests.yml * call dump debug logs

brianseeders · 2025-07-16T16:27:35Z

@slobodanadamovic Our periodic packaging jobs have tons of failures now after unmuting all of these tests... See here for an example: https://buildkite.com/elastic/elasticsearch-periodic-packaging/builds/9057

unmute docker tests to get some fresh failures

836b465

- bump docker startup timeout to account for slow security autoconfiguration Resolves elastic#129808

slobodanadamovic self-assigned this Jul 14, 2025

slobodanadamovic added >test Issues or PRs that are addressing/adding tests :Security/Security Security issues without another label Team:Security Meta label for security team test-release Trigger CI checks against release build labels Jul 14, 2025

elasticsearchmachine added the v9.2.0 label Jul 14, 2025

slobodanadamovic changed the title ~~Unmute docker tests to get some fresh failures~~ Unmute docker tests to get some fresh logs Jul 14, 2025

slobodanadamovic removed the test-release Trigger CI checks against release build label Jul 14, 2025

slobodanadamovic and others added 2 commits July 14, 2025 16:43

Merge branch 'main' into docker-tests

2d63199

add more logging to help troubleshooting future failures

0c386d2

slobodanadamovic changed the title ~~Unmute docker tests to get some fresh logs~~ Unmute docker tests, add more logging and increase startup timeout Jul 15, 2025

Merge branch 'main' into docker-tests

765dae7

slobodanadamovic commented Jul 15, 2025

View reviewed changes

slobodanadamovic added auto-backport Automatically create backport pull requests when merged v9.1.1 v8.19.1 v9.0.5 v8.18.5 labels Jul 15, 2025

slobodanadamovic commented Jul 15, 2025

View reviewed changes

slobodanadamovic requested review from a team July 15, 2025 11:09

slobodanadamovic marked this pull request as ready for review July 15, 2025 11:09

richard-dennehy approved these changes Jul 15, 2025

View reviewed changes

slobodanadamovic merged commit 4806156 into elastic:main Jul 15, 2025
39 checks passed

elasticsearchmachine added the backport pending label Jul 15, 2025

This was referenced Jul 15, 2025

[CI] DockerTests test084SymlinkToFileWithInvalidPermissionsIsRejected failing #129124

Closed

[CI] DockerTests test071BindMountCustomPathWithDifferentUID failing #120917

Open

This was referenced Jul 15, 2025

[CI] ServiceAccountIT testAuthenticateShouldNotFallThroughInCaseOfFailure failing #120902

Closed

[CI] DockerTests class failing #120978

Closed

[9.1] Unmute docker tests, add more logging and increase startup timeout (#131203) #131292

Merged

slobodanadamovic mentioned this pull request Jul 15, 2025

[CI] DockerTests test083EnvironmentVariablesUsingFilesHaveCorrectPermissions failing #129935

Closed

slobodanadamovic mentioned this pull request Jul 15, 2025

Unmute DockerTests#test090SecurityCliPackaging #131303

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Unmute docker tests, add more logging and increase startup timeout #131203

Unmute docker tests, add more logging and increase startup timeout #131203

Uh oh!

slobodanadamovic commented Jul 14, 2025 •

edited

Loading

Uh oh!

slobodanadamovic Jul 15, 2025 •

edited

Loading

Uh oh!

slobodanadamovic Jul 15, 2025 •

edited

Loading

Uh oh!

elasticsearchmachine commented Jul 15, 2025

Uh oh!

Uh oh!

elasticsearchmachine commented Jul 15, 2025

Uh oh!

slobodanadamovic commented Jul 15, 2025

Uh oh!

brianseeders commented Jul 16, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Unmute docker tests, add more logging and increase startup timeout #131203

Unmute docker tests, add more logging and increase startup timeout #131203

Uh oh!

Conversation

slobodanadamovic commented Jul 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

slobodanadamovic Jul 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

slobodanadamovic Jul 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

elasticsearchmachine commented Jul 15, 2025

Uh oh!

Uh oh!

elasticsearchmachine commented Jul 15, 2025

💔 Backport failed

Uh oh!

slobodanadamovic commented Jul 15, 2025

💚 All backports created successfully

Questions ?

Uh oh!

brianseeders commented Jul 16, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

slobodanadamovic commented Jul 14, 2025 •

edited

Loading

slobodanadamovic Jul 15, 2025 •

edited

Loading

slobodanadamovic Jul 15, 2025 •

edited

Loading