The `TEST-scylla-3.11.4.0` test gets marked as "PASSED" without running any tests

[vponomaryov]

Argus: enterprise-2024.1/driver-tests/java-driver-matrix-test#36
Scylla: 2024.1.17

TEST-scylla-3.11.4.0 test results:

• Failure (0)
• Error (0)
• Passed (0)
• Skipped (654)
• Disabled (632)

Screenshot from Argus:

[Bouncheck]

Has this previously looked different for matrix tests? I'll try to look into how those numbers are fetched but it seems that the test at least runs something according to console output:

20:41:12  [WARNING] Tests run: 1286, Failures: 0, Errors: 0, Skipped: 654, Time elapsed: 5,073.073 s - in TestSuite
20:41:19  [INFO] 
20:41:19  [INFO] Results:
20:41:19  [INFO] 
20:41:19  [WARNING] Tests run: 993, Failures: 0, Errors: 0, Skipped: 361
20:41:19  [INFO] 
20:41:19  [INFO] ------------------------------------------------------------------------
20:41:19  [INFO] BUILD SUCCESS
20:41:19  [INFO] ------------------------------------------------------------------------
20:41:19  [INFO] Total time:  01:25 h
20:41:19  [INFO] Finished at: 2025-04-14T18:41:18Z
20:41:19  [INFO] ------------------------------------------------------------------------
20:41:19  INFO:root:All tests are passed for '3.11.4.0' version
20:41:19  INFO:root:=== JAVA DRIVER MATRIX RESULTS FOR 3.11.4.0 ===
20:41:19  INFO:root:Creating a new report file in '/jenkins/workspace/enterprise-2024.1/driver-tests/java-driver-matrix-test/scylla-java-driver-matrix/reports/TEST-scylla-3.11.4.0.xml' path
20:41:19  INFO:root:time: 5073.073, tests: 1286, errors: 0, skipped: 654, failures: 0
20:41:19  INFO:root:Removing all run's xml files of '3.11.4.0' version

[Bouncheck]

Looking at 4.x xml test reports (which do not have this issue) this is how successful test looks like:

<testcase name="should_insert_json_using_simple_statement_with_custom_codec" classname="datastax.4.19.0.com.datastax.oss.driver.querybuilder.JsonInsertIT" time="0.088"/>
<testcase name="should_insert_json_using_simple_statement_with_codec_registry" classname="datastax.4.19.0.com.datastax.oss.driver.querybuilder.JsonInsertIT" time="0.088"/>

When i look at 3.x xml report for some reason there is this system out section:

<testcase name="should_fail_when_synchronous_call_on_io_thread" classname="scylla.3.11.4.0.com.datastax.driver.core.AsyncQueryTest" time="0.002">
<system-out/>
</testcase>

This causes it somehow to be classified as "Disabled" in Argus.
Looking at console output we can see that it actually passed

19:17:12  -----------------------------------------------
19:17:12  Starting com.datastax.driver.core.AsyncQueryTest.should_fail_when_synchronous_call_on_io_thread [Test #19]...
19:17:12  SUCCESS: should_fail_when_synchronous_call_on_io_thread
19:17:12  Test   : 00:00:00
19:17:12  Elapsed: 00:00:34

I'm guessing that everything that ended up in "Disabled" tab here is actually a passing test. The numbers do add up.

[vponomaryov]

Has this previously looked different for matrix tests?

I looked at the history of this test and can say following:

• The TEST-3.11.4.0 has always had this problem that tests were marked as disabled
• The TEST-3.11.5.0, if it was run, had the same problem.
• The TEST-3.10.2-scylla-1 didn't have this problem when was used on 2023-11-20 (enterprise-2024.1/driver-tests/java-driver-matrix-test#3)
• The TEST-3.7.1-scylla-2 also was ok from this point of view.

[Bouncheck]

The difference in xml outputs comes from different versions of surefire plugin used.
Current scylla-4.x uses 3.0.0 and scylla-3.x uses 3.0.0-M6.
The change causing this additional section to be printed came in 3.0.0-M5: https://issues.apache.org/jira/browse/SUREFIRE-1744
Unfortunately the option to toggle it comes later in 3.3.1: https://issues.apache.org/jira/browse/SUREFIRE-1934

One solution is to upgrade (OR downgrade) surefire in all affected java driver versions and disable this output.
Another solution to this would be adjusting Argus to handle this xml output correctly.
edit: I think this is the code that handles that:
https://github.com/scylladb/argus/blob/ebaa6ea95b3717c55af9164fc0eb3920c9972fc3/frontend/TestRun/jUnitResults.svelte#L34-L52

[dkropachev]

@Bouncheck is it solved ?

[Bouncheck]

Technically yes, but needs to be confirmed with another run.