[CI] DeprecationHttpIT testDeprecatedSettingsReturnWarnings failing

[mosche]

Build scan:
https://gradle-enterprise.elastic.co/s/nlw52dnqiof4a/tests/:x-pack:plugin:deprecation:qa:rest:javaRestTest/org.elasticsearch.xpack.deprecation.DeprecationHttpIT/testDeprecatedSettingsReturnWarnings

Reproduction line:

./gradlew ':x-pack:plugin:deprecation:qa:rest:javaRestTest' --tests "org.elasticsearch.xpack.deprecation.DeprecationHttpIT.testDeprecatedSettingsReturnWarnings" -Dtests.seed=94B983B3041BD3C8 -Dtests.locale=ga -Dtests.timezone=Africa/Algiers -Druntime.java=21

Applicable branches:
main

Reproduces locally?:
No

Failure history:
Failure dashboard for org.elasticsearch.xpack.deprecation.DeprecationHttpIT#testDeprecatedSettingsReturnWarnings

Failure excerpt:

java.lang.AssertionError: 
Expected: a collection with size <2>
     but: collection size was <0>

  at __randomizedtesting.SeedInfo.seed([94B983B3041BD3C8:B992366E42E13572]:0)
  at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
  at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:6)
  at org.elasticsearch.test.ESTestCase.assertThat(ESTestCase.java:2188)
  at org.elasticsearch.xpack.deprecation.DeprecationHttpIT.lambda$testDeprecatedSettingsReturnWarnings$1(DeprecationHttpIT.java:142)
  at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:1294)
  at org.elasticsearch.xpack.deprecation.DeprecationHttpIT.testDeprecatedSettingsReturnWarnings(DeprecationHttpIT.java:139)
  at jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
  at java.lang.reflect.Method.invoke(Method.java:580)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
  at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.junit.rules.RunRules.evaluate(RunRules.java:20)
  at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
  at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at org.junit.rules.RunRules.evaluate(RunRules.java:20)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
  at org.junit.rules.RunRules.evaluate(RunRules.java:20)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
  at java.lang.Thread.run(Thread.java:1583)

[elasticsearchmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[mosche]

I briefly peeked into a recent failure (buildscan), the test timed out waiting 30secs for the deprecation to appear in logs.
To me doesn't look like a test failure, but I'd have to look further into it.

The test logs:

[2024-05-07T10:01:00,440][INFO ][o.e.x.d.DeprecationHttpIT] [testDeprecatedSettingsReturnWarnings] before test
мај 07, 2024 10:01:00 AM org.elasticsearch.client.RestClient logResponse
WARNING: request [PUT http://127.0.0.1:40969/_cluster/settings] returned 2 warnings: [299 Elasticsearch-8.15.0-b0fb722cc5d8f8e60ac49112f229811ce2cdf7cf "[test.setting.deprecated.true2] setting was deprecated in Elasticsearch and will be removed in a future release."],[299 Elasticsearch-8.15.0-b0fb722cc5d8f8e60ac49112f229811ce2cdf7cf "[test.setting.deprecated.true1] setting was deprecated in Elasticsearch and will be removed in a future release."]
[2024-05-07T10:01:00,963][WARN ][o.e.x.d.DeprecationHttpIT] [testDeprecatedSettingsReturnWarnings] []
[2024-05-07T10:01:00,989][WARN ][o.e.x.d.DeprecationHttpIT] [testDeprecatedSettingsReturnWarnings] []
[2024-05-07T10:01:01,031][WARN ][o.e.x.d.DeprecationHttpIT] [testDeprecatedSettingsReturnWarnings] []
[2024-05-07T10:01:01,087][WARN ][o.e.x.d.DeprecationHttpIT] [testDeprecatedSettingsReturnWarnings] []
[2024-05-07T10:01:01,122][WARN ][o.e.x.d.DeprecationHttpIT] [testDeprecatedSettingsReturnWarnings] []
[2024-05-07T10:01:01,192][WARN ][o.e.x.d.DeprecationHttpIT] [testDeprecatedSettingsReturnWarnings] []
[2024-05-07T10:01:01,265][WARN ][o.e.x.d.DeprecationHttpIT] [testDeprecatedSettingsReturnWarnings] []
[2024-05-07T10:01:01,370][WARN ][o.e.x.d.DeprecationHttpIT] [testDeprecatedSettingsReturnWarnings] []
[2024-05-07T10:01:01,535][WARN ][o.e.x.d.DeprecationHttpIT] [testDeprecatedSettingsReturnWarnings] []
[2024-05-07T10:01:01,830][WARN ][o.e.x.d.DeprecationHttpIT] [testDeprecatedSettingsReturnWarnings] []
[2024-05-07T10:01:02,379][WARN ][o.e.x.d.DeprecationHttpIT] [testDeprecatedSettingsReturnWarnings] []
[2024-05-07T10:01:03,451][WARN ][o.e.x.d.DeprecationHttpIT] [testDeprecatedSettingsReturnWarnings] []
[2024-05-07T10:01:05,544][WARN ][o.e.x.d.DeprecationHttpIT] [testDeprecatedSettingsReturnWarnings] []
[2024-05-07T10:01:09,674][WARN ][o.e.x.d.DeprecationHttpIT] [testDeprecatedSettingsReturnWarnings] []
[2024-05-07T10:01:17,906][WARN ][o.e.x.d.DeprecationHttpIT] [testDeprecatedSettingsReturnWarnings] []
[2024-05-07T10:01:34,318][WARN ][o.e.x.d.DeprecationHttpIT] [testDeprecatedSettingsReturnWarnings] []
мај 07, 2024 10:01:34 AM org.elasticsearch.client.RestClient logResponse
WARNING: request [PUT http://[::1]:41399/_cluster/settings] returned 2 warnings: [299 Elasticsearch-8.15.0-b0fb722cc5d8f8e60ac49112f229811ce2cdf7cf "[test.setting.deprecated.true2] setting was deprecated in Elasticsearch and will be removed in a future release."],[299 Elasticsearch-8.15.0-b0fb722cc5d8f8e60ac49112f229811ce2cdf7cf "[test.setting.deprecated.true1] setting was deprecated in Elasticsearch and will be removed in a future release."]
[2024-05-07T10:01:34,466][INFO ][o.e.x.d.DeprecationHttpIT] [testDeprecatedSettingsReturnWarnings] after test
REPRODUCE WITH: ./gradlew ':x-pack:plugin:deprecation:qa:rest:javaRestTest' --tests "org.elasticsearch.xpack.deprecation.DeprecationHttpIT.testDeprecatedSettingsReturnWarnings" -Dtests.seed=446C182627B25D32 -Dtests.locale=sr-ME -Dtests.timezone=US/Mountain -Druntime.java=21

However, looking at the deprecation log you'll see the following:

{"@timestamp":"2024-05-07T16:01:34.325Z", "log.level": "WARN",  "data_stream.dataset":"deprecation.elasticsearch","data_stream.namespace":"default","data_stream.type":"logs","elasticsearch.elastic_product_origin":"","elasticsearch.event.category":"settings","elasticsearch.http.request.x_opaque_id":"testDeprecatedSettingsReturnWarnings","event.code":"test.setting.deprecated.true2","message":"[test.setting.deprecated.true2] setting was deprecated in Elasticsearch and will be removed in a future release." , "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"deprecation.elasticsearch","process.thread.name":"elasticsearch[javaRestTest-0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.deprecation.common.settings.Settings","elasticsearch.cluster.uuid":"dDcqp0Y3Tte6TEqtUezFsw","elasticsearch.node.id":"3E_sV7YrQJW0ySfuirx-0g","elasticsearch.node.name":"javaRestTest-0","elasticsearch.cluster.name":"javaRestTest"}
{"@timestamp":"2024-05-07T16:01:34.326Z", "log.level": "WARN",  "data_stream.dataset":"deprecation.elasticsearch","data_stream.namespace":"default","data_stream.type":"logs","elasticsearch.elastic_product_origin":"","elasticsearch.event.category":"settings","elasticsearch.http.request.x_opaque_id":"testDeprecatedSettingsReturnWarnings","event.code":"test.setting.deprecated.true1","message":"[test.setting.deprecated.true1] setting was deprecated in Elasticsearch and will be removed in a future release." , "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"deprecation.elasticsearch","process.thread.name":"elasticsearch[javaRestTest-0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.deprecation.common.settings.Settings","elasticsearch.cluster.uuid":"dDcqp0Y3Tte6TEqtUezFsw","elasticsearch.node.id":"3E_sV7YrQJW0ySfuirx-0g","elasticsearch.node.name":"javaRestTest-0","elasticsearch.cluster.name":"javaRestTest"}

These logs correspond to the request in the finally block, when resetting/removing the deprecated settings again.
It appears as if the first request was neither logged nor indexed 🤷

[mosche]

Another at https://gradle-enterprise.elastic.co/s/nlw52dnqiof4a/tests/task/:x-pack:plugin:deprecation:qa:rest:javaRestTest/details/org.elasticsearch.xpack.deprecation.DeprecationHttpIT/testDeprecatedSettingsReturnWarnings?top-execution=1

[mosche]

test logs

[2024-05-13T16:43:56,601][INFO ][o.e.x.d.DeprecationHttpIT] [testDeprecatedSettingsReturnWarnings] before test	
Beal 13, 2024 4:43:56 P.M. org.elasticsearch.client.RestClient logResponse	
WARNING: request [PUT http://[::1]:44625/_cluster/settings] returned 2 warnings: [299 Elasticsearch-8.15.0-0b2e5584f70ece2e157d7be47529637c505a7d4c "[test.setting.deprecated.true2] setting was deprecated in Elasticsearch and will be removed in a future release."],[299 Elasticsearch-8.15.0-0b2e5584f70ece2e157d7be47529637c505a7d4c "[test.setting.deprecated.true1] setting was deprecated in Elasticsearch and will be removed in a future release."]	
[2024-05-13T16:43:56,903][WARN ][o.e.x.d.DeprecationHttpIT] [testDeprecatedSettingsReturnWarnings] []	
[2024-05-13T16:43:56,921][WARN ][o.e.x.d.DeprecationHttpIT] [testDeprecatedSettingsReturnWarnings] []	
[2024-05-13T16:43:56,944][WARN ][o.e.x.d.DeprecationHttpIT] [testDeprecatedSettingsReturnWarnings] []	
[2024-05-13T16:43:56,965][WARN ][o.e.x.d.DeprecationHttpIT] [testDeprecatedSettingsReturnWarnings] []	
[2024-05-13T16:43:56,994][WARN ][o.e.x.d.DeprecationHttpIT] [testDeprecatedSettingsReturnWarnings] []	
[2024-05-13T16:43:57,033][WARN ][o.e.x.d.DeprecationHttpIT] [testDeprecatedSettingsReturnWarnings] []	
[2024-05-13T16:43:57,086][WARN ][o.e.x.d.DeprecationHttpIT] [testDeprecatedSettingsReturnWarnings] []	
[2024-05-13T16:43:57,173][WARN ][o.e.x.d.DeprecationHttpIT] [testDeprecatedSettingsReturnWarnings] []	
[2024-05-13T16:43:57,329][WARN ][o.e.x.d.DeprecationHttpIT] [testDeprecatedSettingsReturnWarnings] []	
[2024-05-13T16:43:57,610][WARN ][o.e.x.d.DeprecationHttpIT] [testDeprecatedSettingsReturnWarnings] []	
[2024-05-13T16:43:58,143][WARN ][o.e.x.d.DeprecationHttpIT] [testDeprecatedSettingsReturnWarnings] []	
[2024-05-13T16:43:59,190][WARN ][o.e.x.d.DeprecationHttpIT] [testDeprecatedSettingsReturnWarnings] []	
[2024-05-13T16:44:01,259][WARN ][o.e.x.d.DeprecationHttpIT] [testDeprecatedSettingsReturnWarnings] []	
[2024-05-13T16:44:05,375][WARN ][o.e.x.d.DeprecationHttpIT] [testDeprecatedSettingsReturnWarnings] []	
[2024-05-13T16:44:13,588][WARN ][o.e.x.d.DeprecationHttpIT] [testDeprecatedSettingsReturnWarnings] []	
[2024-05-13T16:44:30,000][WARN ][o.e.x.d.DeprecationHttpIT] [testDeprecatedSettingsReturnWarnings] []	
Beal 13, 2024 4:44:30 P.M. org.elasticsearch.client.RestClient logResponse	
WARNING: request [PUT http://127.0.0.1:43361/_cluster/settings] returned 2 warnings: [299 Elasticsearch-8.15.0-0b2e5584f70ece2e157d7be47529637c505a7d4c "[test.setting.deprecated.true2] setting was deprecated in Elasticsearch and will be removed in a future release."],[299 Elasticsearch-8.15.0-0b2e5584f70ece2e157d7be47529637c505a7d4c "[test.setting.deprecated.true1] setting was deprecated in Elasticsearch and will be removed in a future release."]	
[2024-05-13T16:44:30,105][INFO ][o.e.x.d.DeprecationHttpIT] [testDeprecatedSettingsReturnWarnings] after test	
REPRODUCE WITH: ./gradlew ':x-pack:plugin:deprecation:qa:rest:javaRestTest' --tests "org.elasticsearch.xpack.deprecation.DeprecationHttpIT.testDeprecatedSettingsReturnWarnings" -Dtests.seed=94B983B3041BD3C8 -Dtests.locale=ga -Dtests.timezone=Africa/Algiers -Druntime.java=21	

deprecation logs

{"@timestamp":"2024-05-13T15:43:56.712Z", "log.level": "WARN",  "data_stream.dataset":"deprecation.elasticsearch","data_stream.namespace":"default","data_stream.type":"logs","elasticsearch.elastic_product_origin":"","elasticsearch.event.category":"settings","elasticsearch.http.request.x_opaque_id":"testDeprecationIndexingCacheReset","event.code":"test.setting.deprecated.true2","message":"[test.setting.deprecated.true2] setting was deprecated in Elasticsearch and will be removed in a future release." , "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"deprecation.elasticsearch","process.thread.name":"elasticsearch[javaRestTest-0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.deprecation.common.settings.Settings","elasticsearch.cluster.uuid":"I7qWkaMEQfmweHuYh9_Dqw","elasticsearch.node.id":"JJANCj_nTX-XcfDM9Kk5mw","elasticsearch.node.name":"javaRestTest-0","elasticsearch.cluster.name":"javaRestTest"}
{"@timestamp":"2024-05-13T15:43:56.718Z", "log.level": "WARN",  "data_stream.dataset":"deprecation.elasticsearch","data_stream.namespace":"default","data_stream.type":"logs","elasticsearch.elastic_product_origin":"","elasticsearch.event.category":"settings","elasticsearch.http.request.x_opaque_id":"testDeprecationIndexingCacheReset","event.code":"test.setting.deprecated.true1","message":"[test.setting.deprecated.true1] setting was deprecated in Elasticsearch and will be removed in a future release." , "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"deprecation.elasticsearch","process.thread.name":"elasticsearch[javaRestTest-0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.deprecation.common.settings.Settings","elasticsearch.cluster.uuid":"I7qWkaMEQfmweHuYh9_Dqw","elasticsearch.node.id":"JJANCj_nTX-XcfDM9Kk5mw","elasticsearch.node.name":"javaRestTest-0","elasticsearch.cluster.name":"javaRestTest"}
{"@timestamp":"2024-05-13T15:44:30.010Z", "log.level": "WARN",  "data_stream.dataset":"deprecation.elasticsearch","data_stream.namespace":"default","data_stream.type":"logs","elasticsearch.elastic_product_origin":"","elasticsearch.event.category":"settings","elasticsearch.http.request.x_opaque_id":"testDeprecatedSettingsReturnWarnings","event.code":"test.setting.deprecated.true2","message":"[test.setting.deprecated.true2] setting was deprecated in Elasticsearch and will be removed in a future release." , "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"deprecation.elasticsearch","process.thread.name":"elasticsearch[javaRestTest-0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.deprecation.common.settings.Settings","elasticsearch.cluster.uuid":"I7qWkaMEQfmweHuYh9_Dqw","elasticsearch.node.id":"JJANCj_nTX-XcfDM9Kk5mw","elasticsearch.node.name":"javaRestTest-0","elasticsearch.cluster.name":"javaRestTest"}
{"@timestamp":"2024-05-13T15:44:30.011Z", "log.level": "WARN",  "data_stream.dataset":"deprecation.elasticsearch","data_stream.namespace":"default","data_stream.type":"logs","elasticsearch.elastic_product_origin":"","elasticsearch.event.category":"settings","elasticsearch.http.request.x_opaque_id":"testDeprecatedSettingsReturnWarnings","event.code":"test.setting.deprecated.true1","message":"[test.setting.deprecated.true1] setting was deprecated in Elasticsearch and will be removed in a future release." , "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"deprecation.elasticsearch","process.thread.name":"elasticsearch[javaRestTest-0][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.deprecation.common.settings.Settings","elasticsearch.cluster.uuid":"I7qWkaMEQfmweHuYh9_Dqw","elasticsearch.node.id":"JJANCj_nTX-XcfDM9Kk5mw","elasticsearch.node.name":"javaRestTest-0","elasticsearch.cluster.name":"javaRestTest"}