Test Fix: Resolve unintended test failures caused by non-deterministic ordering of elements in the serialized XML output

[Jake-WangZhi]

Due to the non-deterministic behavior of serialized XML output, the order of items in the class can vary. This may lead to an AssertionError when comparing an item object with its expected value.

The example below shows a failure under NonDex on line 54

Click on to see more details on the error message when running each test

[INFO] Using auto detected provider org.apache.maven.surefire.junitcore.JUnitCoreProvider
[INFO] 
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running com.fasterxml.jackson.dataformat.xml.ser.TestSerializationWithFilter
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.364 s <<< FAILURE! -- in com.fasterxml.jackson.dataformat.xml.ser.TestSerializationWithFilter
[ERROR] com.fasterxml.jackson.dataformat.xml.ser.TestSerializationWithFilter.testPullRequest616 -- Time elapsed: 0.353 s <<< FAILURE!
junit.framework.ComparisonFailure: expected:<<Item><[b>10</b><c>100</c]></Item>> but was:<<Item><[c>100</c><b>10</b]></Item>>
        at junit.framework.Assert.assertEquals(Assert.java:100)
        at junit.framework.Assert.assertEquals(Assert.java:107)
        at junit.framework.TestCase.assertEquals(TestCase.java:260)
        at com.fasterxml.jackson.dataformat.xml.ser.TestSerializationWithFilter.testPullRequest616(TestSerializationWithFilter.java:54)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at junit.framework.TestCase.runTest(TestCase.java:177)
        at junit.framework.TestCase.runBare(TestCase.java:142)
        at junit.framework.TestResult$1.protect(TestResult.java:122)
        at junit.framework.TestResult.runProtected(TestResult.java:142)
        at junit.framework.TestResult.run(TestResult.java:125)
        at junit.framework.TestCase.run(TestCase.java:130)
        at junit.framework.TestSuite.runTest(TestSuite.java:241)
        at junit.framework.TestSuite.run(TestSuite.java:236)
        at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:90)
        at org.junit.runners.Suite.runChild(Suite.java:128)
        at org.junit.runners.Suite.runChild(Suite.java:27)
        at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
        at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
        at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
        at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
        at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
        at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
        at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
        at org.apache.maven.surefire.junitcore.JUnitCore.run(JUnitCore.java:49)
        at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:120)
        at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeEager(JUnitCoreWrapper.java:95)
        at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:75)
        at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:69)
        at org.apache.maven.surefire.junitcore.JUnitCoreProvider.invoke(JUnitCoreProvider.java:146)
        at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:385)
        at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:162)
        at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:507)
        at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:495)

[INFO] 
[INFO] Results:
[INFO] 
[ERROR] Failures: 
[ERROR]   TestSerializationWithFilter.testPullRequest616:54 expected:<<Item><[b>10</b><c>100</c]></Item>> but was:<<Item><[c>100</c><b>10</b]></Item>>
[INFO] 
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0
[INFO] 

As shown here, the order of the items is non-deterministic.

To reproduce each, run these at the root directory:

mvn  edu.illinois:nondex-maven-plugin:2.1.7:nondex     -Dtest=com.fasterxml.jackson.dataformat.xml.ser.TestSerializationWithFilter#testPullRequest616 -DnondexRuns=10

(Note: The failing test might not be seen if every test happens to have the item objects in the expected order. Try running it several times or increase the the number of runs with -DnondexRuns= to reproduce the issue.)

The log output can be found here for your reference:
mvn-nondex-1730966644.log

To resolve this, we can annotate the fields in the Item class to enforce a specific order.

After applying the fix, the test should now pass with NonDex as expected:

[INFO] No Test Failed with this configuration.
[INFO] *********
[INFO] All tests pass without NonDex shuffling
[INFO] ####################
[INFO] Across all seeds:
[INFO] Test results can be found at: 
[INFO] file:///home/jakew4/jackson-dataformat-xml/.nondex/bRRc1NJsYd2JeHNw3GqrSmQbA610uRmFyU3UpFsCJJQ=/test_results.html
[INFO] [NonDex] The id of this run is: bRRc1NJsYd2JeHNw3GqrSmQbA610uRmFyU3UpFsCJJQ=
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  21.258 s
[INFO] Finished at: 2024-11-07T02:50:04-06:00
[INFO] ------------------------------------------------------------------------

The log file showing the test passing after the fix can be found here:
mvn-nondex-fix-1730968323.log

Please let me know if this approach works for you. If not, I'm happy to discuss alternatives and am willing to spend more time to address the test in the way you'd prefer. Thank you!

[cowtowncoder]

Thank you @Jake-WangZhi !