Disable queryable built-in roles feature for core and datastream YAML tests

[slobodanadamovic]

This PR disables the "queryable built-in roles" feature for the CoreWithSecurityClientYamlTestSuiteIT and DataStreamsClientYamlTestSuiteIT YAML test suites.
The feature was enabled by default in the #120323 PR, which asynchronously creates the .security index after cluster formation and indexes all built-in roles. The asynchronous creation of the .security index introduces non-deterministic behavior in our YAML tests.
Since these test suites are not intended to verify the queryable built-in roles functionality, having the feature enabled introduced flakiness and unnecessary complexity to handle .security in existing tests. These tests would have to exclude the .security index in some way (by adjusting permissions or API calls), and in the end cleanup (delete) the .security index. This simply adds overhead without much gain. The feature is already test covered by XPackRestIT and other integration/REST tests, disabling it here would not compromise test coverage. Instead, it ensures these suites remain deterministic and focused on the behaviors they were designed to verify.

Resolves #121536
Resolves #121513
Resolves #121484
Resolves #121478
Resolves #121290
Resolves #121246
Resolves #121242
Resolves #121238
Resolves #121186
Resolves #121131
Resolves #121130
Resolves #121128
Resolves #121014
Resolves #120965
Resolves #120920
Resolves #120890

[elasticsearchmachine]

Pinging @elastic/es-security (Team:Security)

[jfreden]

Is there a plan to get rid of es.queryable_built_in_roles_enabled=false in these tests in the future? Even if they're not directly related to that feature, the feature being integration tested will run with es.queryable_built_in_roles_enabled=true in production which make me a little hesitant to just turning it off.

[jfreden]

It would be nice if the long term plan is to add it back somehow? Even if there is no relation between the security index creation and those tests today, there might be in the future. If the property is going to stick around for a while it would be nice to rename it to something related to the security index to help with debugging if something goes wrong.

[slobodanadamovic]

Is there a plan to get rid of es.queryable_built_in_roles_enabled=false in these tests in the future? Even if they're not directly related to that feature, the feature being integration tested will run with es.queryable_built_in_roles_enabled=true in production which make me a little hesitant to just turning it off.

No. I was a bit too optimistic that I can make this work for all tests, but the reality is that I need to keep a flag/setting in order to be able to disable it due to the asynchronous nature of .security index creation.
This is only temporary solution which I’m planning to make a permanent by keeping the feature flag in some way (e.g by converting it to a setting or renaming system property to something like xpack.security.queryable_built_in_roles.enabled or xpack.security.auto_built_in_roles_indexing.enabled). This will be then used to disable the queryable built-in role feature in tests (similar to geoip downloader).

I think it's safe to turn it off for these tests. If you have a concrete concern, I think we should address that by adding more test coverage. But I don't think that this would reduce the confidence in feature working as intended. The problem is that the async nature of .security index creation is introducing too much flakiness in YAML tests which don't have nice mechanisms for ensuring consistent test executions. FWIW We have better control over it in integration and rest tests and we leverage them to have higher coverage.

[jfreden]

LGTM!

[elasticsearchmachine]

💔 Backport failed

Status	Branch	Result
❌	9.0	Commit could not be cherrypicked due to conflicts
❌	8.18	Commit could not be cherrypicked due to conflicts
❌	8.x	Commit could not be cherrypicked due to conflicts

You can use sqren/backport to manually backport by running backport --upstream elastic/elasticsearch --pr 121541

[slobodanadamovic]

💚 All backports created successfully

Status	Branch	Result
✅	8.x
✅	9.0
✅	8.18

Questions ?

Please refer to the Backport tool documentation