[CI] SpatialExtentAggregationNoLicenseIT testStExtentAggregationWithShapes failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-periodic-platform-support #7294 / oraclelinux-9_platform-support-unix
• elasticsearch-periodic-platform-support #7294 / ubuntu-2404_platform-support-unix
• elasticsearch-periodic-platform-support #7294 / almalinux-8_platform-support-unix
• elasticsearch-periodic-platform-support #7294 / rocky-8_platform-support-unix
• elasticsearch-periodic-platform-support #7294 / windows-2022_checkpart3_platform-support-windows
• elasticsearch-pull-request #65142 / part-3
• elasticsearch-pull-request #65101 / part-3
• elasticsearch-pull-request #65101 / part-3
• elasticsearch-intake #20883 / part3
• elasticsearch-periodic-platform-support #7287 / ubuntu-2404_platform-support-unix

Reproduction Line:

./gradlew ":x-pack:plugin:esql:internalClusterTest" --tests "org.elasticsearch.xpack.esql.spatial.SpatialExtentAggregationNoLicenseIT.testStExtentAggregationWithShapes" -Dtests.seed=60EDFAE803E8DCE9 -Dtests.locale=bm-Latn-ML -Dtests.timezone=Australia/Lord_Howe -Druntime.java=24

Applicable branches:
main

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

junit.framework.AssertionFailedError: Expected exception VerificationException but no exception was thrown

Issue Reasons:

• [main] 2 consecutive failures in step oraclelinux-9_platform-support-unix
• [main] 2 consecutive failures in step ubuntu-2404_platform-support-unix
• [main] 21 failures in test testStExtentAggregationWithShapes (9.5% fail rate in 222 executions)
• [main] 2 failures in step oraclelinux-9_platform-support-unix (50.0% fail rate in 4 executions)
• [main] 6 failures in step part-3 (5.9% fail rate in 101 executions)
• [main] 2 failures in step ubuntu-2404_platform-support-unix (50.0% fail rate in 4 executions)
• [main] 2 failures in step rocky-9_platform-support-unix (66.7% fail rate in 3 executions)
• [main] 4 failures in pipeline elasticsearch-periodic-platform-support (100.0% fail rate in 4 executions)
• [main] 5 failures in pipeline elasticsearch-pull-request (5.2% fail rate in 97 executions)
• [main] 2 failures in pipeline elasticsearch-periodic (50.0% fail rate in 4 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

Pinging @elastic/es-analytical-engine (Team:Analytics)

[elasticsearchmachine]

This has been muted on branch 9.0

Mute Reasons:

• [9.0] 2 failures in test testStExtentAggregationWithShapes (10.0% fail rate in 20 executions)

Build Scans:

• elasticsearch-periodic-platform-support #7279 / rocky-8_platform-support-unix
• elasticsearch-pull-request #64933 / part-3

[luigidellaquila]

New test #125584, marking as medium risk because it's about license enforcement
cc @craigtaverner

[craigtaverner]

Tested with -Dtests.iters=1000 on both 8.x and main and busy running that on 9.0 branch. The interesting thing is this fails over 20% on 8.x, and 0% on main. There seems to be some underlying issue with the test infrastructure that is fixed on main. The thread stack seems to hint at an assertion deep in the stack related to closed resources. Further investigation is needed to figure out what fix was made on main, and backport that if possible.

[craigtaverner]

OK. On further investigation, I can get this to fail on different branches with different seeds:

seed	8.x	9.0	main
-Dtests.seed=712BF32772B4874C	FAIL	FAIL	PASS
-Dtests.seed=17AC5C5D140EC9BF	PASS	PASS	FAIL

My earlier tests with -Dtests.iters=1000, did pass with 0% failure on main, but re-testing later gave a high failure rate. And sometimes an early failure caused all further iterations to fail. A common theme not reported in the failure is the following assertion failure:

    java.lang.AssertionError: Request breaker not reset to 0 on node: node_s0
    Expected: <0L>
         but: was <240L>
        at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
        at org.junit.Assert.assertThat(Assert.java:964)
        at org.elasticsearch.test.InternalTestCluster.lambda$ensureEstimatedStats$35(InternalTestCluster.java:2472)
        at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:1498)
        at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:1471)
        at org.elasticsearch.test.InternalTestCluster.ensureEstimatedStats(InternalTestCluster.java:2470)
        at org.elasticsearch.test.TestCluster.assertAfterTest(TestCluster.java:181)
        at org.elasticsearch.test.InternalTestCluster.assertAfterTest(InternalTestCluster.java:2520)
        at org.elasticsearch.test.ESIntegTestCase.afterInternal(ESIntegTestCase.java:582)
        at org.elasticsearch.test.ESIntegTestCase.cleanUpCluster(ESIntegTestCase.java:2327)
        at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
        at java.base/java.lang.reflect.Method.invoke(Method.java:565)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1763)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:1004)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
        at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
        at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
        at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
        at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
        at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
        at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
        at java.base/java.lang.Thread.run(Thread.java:1447)

[craigtaverner]

The above is not related to this test itself, but clearly some of the code being activated by this test must have a leak of some kind. This code is testing features that were merged months ago. Since it even fails on 8.x, it is possible this leak exists in even earlier branches. I will test on the version the feature was first committed to.