Skip to content

Conversation

@adamfarley
Copy link
Contributor

@adamfarley adamfarley commented Dec 10, 2025

This excludes any tests that could interrupt a release, and also helps keep our test runs green at the expense of nonblocker test executions. Plus it reduces complexity by reducing the number of exclude files, which makes maintenance easier.

Note 1: All "unreliable" tests excluded in this commit have been identified as not blocking a release (nonblocker) in at least one previous release cycle.

Note 2: The issues related to these infrequently failing tests will be updated to include a title change and an additional tag to indicate they they are infrequent failure.

@sophia-guo - Do you think this PR can go in in parallel to that work, or do you feel we should hold fire until the related issues are updated?

Resolves #6593

@smlambert
Copy link
Contributor

I am looking at the issue links under which these exclusions are being done, and we should not be referring to release triage issues such as #6421

Copy link
Contributor

@smlambert smlambert left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should be referencing actual issues that articulate the intermittent failure (look in aqa-tests, infrastructure, and JBS existing issues that describe the problem, if an issue does not exist that describes the actual type of failure, then we should create one, some testcases may be failing in the same way, and can be grouped under the same issue)

@adamfarley
Copy link
Contributor Author

I am looking at the issue links under which these exclusions are being done, and we should not be referring to release triage issues such as #6421

Fair enough. Will screen them out.

@adamfarley
Copy link
Contributor Author

adamfarley commented Dec 11, 2025

It seems many of the triage issues have expired job links, so I'm going through each of the failing unit tests and identifying the actual problem for each based on more recent test runs. I'll keep an eye out for failures sharing symptoms (with each other and existing issues).

Plan: For each unreliable test:

  • If a test is linked to a triage issue, comment out the line.
  • Uncomment the line after one of these:
    • Finding an issue for the failure
    • Finding a recent failure and raising an issue
  • Remove the line if there are no open issues, and the test has passed 100% of the time in recent runs.

Status

JDK8: Complete.

JDK11: Complete. - https://ci.adoptium.net/job/Grinder/16033

JDK17: Complete. - https://ci.adoptium.net/view/Test_grinder/job/Grinder/16186/

JDK21: Complete. - https://ci.adoptium.net/job/Grinder/16196/

JDK25: Grinders rerunning. - https://ci.adoptium.net/job/Grinder/16283/

adamfarley and others added 4 commits January 30, 2026 16:50
This excludes any tests that could interrupt a release, and also
helps keep our test runs green at the expense of nonblocker
test executions. Plus it reduces complexity by reducing the number
of exclude files, which makes maintenance easier.

Addendum: All "unreliable" tests excluded in this commit have
been identified as not blocking a release (nonblocker) in at least
one previous release cycle.

Signed-off-by: Adam Farley <[email protected]>
@adamfarley adamfarley force-pushed the exclude_unreliable_tests branch from 086c92d to 859ed2e Compare January 30, 2026 16:53
@karianna karianna requested a review from smlambert January 31, 2026 02:14
Signed-off-by: Adam Farley <[email protected]>
as they have not been seen in recent runs.

Signed-off-by: Adam Farley <[email protected]>
Where "processing" involves finding a recent reproduction or open
issue, and attributing the exclusion to that issue (as opposed to a
mass-triage issue that contains few details).

Signed-off-by: Adam Farley <[email protected]>
Note that many could not be reproduced in grinders, nor found to have
failed in pipeline test runs. These were removed, and may need to be
re-added later if they occur again.

Signed-off-by: Adam Farley <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Exclude unreliables in the usual exclude files

3 participants