Report usage of preview API only when the project compliance is at latest #3943

[jarthana]

What it does

Fixes #3943

How to test

Author checklist

• I have thoroughly tested my changes
• The change is following the coding conventions
• I have signed the Eclipse Contributor Agreement (ECA)

[github-actions]

Test Results

    11 files  +  3      11 suites  +3   18m 17s ⏱️ +38s
79 450 tests +125  79 450 ✅ +156  0 💤 ±0  0 ❌  - 31 
79 510 runs  +125  79 510 ✅ +156  0 💤 ±0  0 ❌  - 31 

Results for commit 5a12e44. ± Comparison against base commit 6798c4b.

♻️ This comment has been updated with latest results.

[stephan-herrmann]

@jarthana has this change ever been tested at compliance 16?

In Y-builds I found PreviewFeatureTest.test001 and test002 to fail at compliance 16, see https://download.eclipse.org/eclipse/downloads/drops4/Y20251125-1000/testresults/html/org.eclipse.jdt.core.tests.compiler_ep438Y-unit-linux-x86_64-java26_linux.gtk.x86_64_26.html

Looking at current I-builds, we simply do not execute these two tests because:

• below 16 the entire test class is skipped
• above 16 those particular tests are skipped :)

OTOH, this raises the question, where are compliance levels specified for Y-build tests? Looks like tests are run for all from 1.8 to 26, which is more than we need, but at least it revealed this issue in PreviewFeatureTest.

[stephan-herrmann]

OTOH, this raises the question, where are compliance levels specified for Y-build tests? Looks like tests are run for all from 1.8 to 26, which is more than we need, but at least it revealed this issue in PreviewFeatureTest.

This one I found: test.xml lacks a definition -Dcompliance.jre.26=... -> over to #4658

[jarthana]

@jarthana has this change ever been tested at compliance 16?

In Y-builds I found PreviewFeatureTest.test001 and test002 to fail at compliance 16, see https://download.eclipse.org/eclipse/downloads/drops4/Y20251125-1000/testresults/html/org.eclipse.jdt.core.tests.compiler_ep438Y-unit-linux-x86_64-java26_linux.gtk.x86_64_26.html

Looking at current I-builds, we simply do not execute these two tests because:

* below 16 the entire test class is skipped

* above 16 those particular tests are skipped :)

I don't remember the history now. Sorry, poor memory. But this test has been a pain to maintain since the preview APIs keep changing, getting removed etc. But still should be consistent if --release is effective.

Anyway, thanks for reporting it. I have a PR here for this - #4666

[jarthana]

The cause still remains though. Even now, with selective compliance and some suites or tests only run at particular levels, there could be other tests too.