Skip to content

Conversation

slobodanadamovic
Copy link
Contributor

@slobodanadamovic slobodanadamovic commented Jan 16, 2025

Making the es.queryable_built_in_roles_enabled feature flag enabled by default.

In order to keep the scope of this PR small, I've disabled the feature for some of the tests,
because they are either non-trivial to adjust or the gain is not worthy the effort to do it now.
None of the disabled tests are bugs. See my comments for each disabled test.

I will be adjusting these disabled tests in a follow-up PR and later remove the flag completely.

@slobodanadamovic slobodanadamovic added >non-issue :Security/Authorization Roles, Privileges, DLS/FLS, RBAC/ABAC Team:Security Meta label for security team auto-backport Automatically create backport pull requests when merged v9.0.0 v8.18.0 labels Jan 16, 2025
@slobodanadamovic slobodanadamovic self-assigned this Jan 16, 2025
@slobodanadamovic
Copy link
Contributor Author

💚 All backports created successfully

Status Branch Result
8.x

Questions ?

Please refer to the Backport tool documentation

slobodanadamovic added a commit to slobodanadamovic/elasticsearch that referenced this pull request Jan 27, 2025
Making the `es.queryable_built_in_roles_enabled` feature flag enabled by default.
This feature makes the built-in roles automatically indexed in `.security` index and available
for querying via Query Role API. The consequence of this is that `.security` index is now
created eagerly (if it's not existing) on cluster formation.

In order to keep the scope of this PR small, the feature is disabled for some of the tests,
because they are either non-trivial to adjust or the gain is not worthy the effort to do it now.
The tests will be adjusted in a follow-up PR and later the flag will be removed completely.

Relates to elastic#117581

(cherry picked from commit 52e0f21)

# Conflicts:
#	modules/dot-prefix-validation/build.gradle
#	test/framework/src/main/java/org/elasticsearch/test/InternalTestCluster.java
#	x-pack/plugin/security/src/internalClusterTest/java/org/elasticsearch/xpack/security/authc/esnative/ReservedRealmElasticAutoconfigIntegTests.java
elasticsearchmachine pushed a commit that referenced this pull request Jan 27, 2025
…120886)

* Enable queryable built-in roles feature by default (#120323)

Making the `es.queryable_built_in_roles_enabled` feature flag enabled by default.
This feature makes the built-in roles automatically indexed in `.security` index and available
for querying via Query Role API. The consequence of this is that `.security` index is now
created eagerly (if it's not existing) on cluster formation.

In order to keep the scope of this PR small, the feature is disabled for some of the tests,
because they are either non-trivial to adjust or the gain is not worthy the effort to do it now.
The tests will be adjusted in a follow-up PR and later the flag will be removed completely.

Relates to #117581

(cherry picked from commit 52e0f21)

# Conflicts:
#	modules/dot-prefix-validation/build.gradle
#	test/framework/src/main/java/org/elasticsearch/test/InternalTestCluster.java
#	x-pack/plugin/security/src/internalClusterTest/java/org/elasticsearch/xpack/security/authc/esnative/ReservedRealmElasticAutoconfigIntegTests.java

* Update InternalTestCluster.java

remove line snuck after resolving merge confilcs

* Update build.gradle

fix build.gradle

* Update build.gradle

fix build.gradle by removing invalid task

* remove non-existing timeout parameter on 8.x branch
slobodanadamovic added a commit that referenced this pull request Feb 4, 2025
… tests (#121541)

This PR disables the "queryable built-in roles" feature for the `CoreWithSecurityClientYamlTestSuiteIT` and `DataStreamsClientYamlTestSuiteIT` YAML test suites. 
The feature was enabled by default in the #120323 PR, which asynchronously creates the `.security` index after cluster formation and indexes all built-in roles. The asynchronous creation of the `.security` index introduces non-deterministic behavior in our YAML tests. 
Since these test suites are not intended to verify the queryable built-in roles functionality, having the feature enabled introduced flakiness and unnecessary complexity to handle `.security` in existing tests. These tests would have to exclude the `.security` index in some way (by adjusting permissions or API calls), and in the end cleanup (delete) the `.security` index. This simply adds overhead without much gain. The feature is already test covered by `XPackRestIT` and other integration/REST tests, disabling it here would not compromise test coverage. Instead, it ensures these suites remain deterministic and focused on the behaviors they were designed to verify.

Resolves #121536 
Resolves #121513 
Resolves #121484 
Resolves #121478 
Resolves #121290 
Resolves #121246 
Resolves #121242 
Resolves #121238 
Resolves #121186 
Resolves #121131 
Resolves #121130 
Resolves #121128 
Resolves #121014 
Resolves #120965 
Resolves #120920 
Resolves #120890
slobodanadamovic added a commit to slobodanadamovic/elasticsearch that referenced this pull request Feb 4, 2025
… tests (elastic#121541)

This PR disables the "queryable built-in roles" feature for the `CoreWithSecurityClientYamlTestSuiteIT` and `DataStreamsClientYamlTestSuiteIT` YAML test suites.
The feature was enabled by default in the elastic#120323 PR, which asynchronously creates the `.security` index after cluster formation and indexes all built-in roles. The asynchronous creation of the `.security` index introduces non-deterministic behavior in our YAML tests.
Since these test suites are not intended to verify the queryable built-in roles functionality, having the feature enabled introduced flakiness and unnecessary complexity to handle `.security` in existing tests. These tests would have to exclude the `.security` index in some way (by adjusting permissions or API calls), and in the end cleanup (delete) the `.security` index. This simply adds overhead without much gain. The feature is already test covered by `XPackRestIT` and other integration/REST tests, disabling it here would not compromise test coverage. Instead, it ensures these suites remain deterministic and focused on the behaviors they were designed to verify.

Resolves elastic#121536
Resolves elastic#121513
Resolves elastic#121484
Resolves elastic#121478
Resolves elastic#121290
Resolves elastic#121246
Resolves elastic#121242
Resolves elastic#121238
Resolves elastic#121186
Resolves elastic#121131
Resolves elastic#121130
Resolves elastic#121128
Resolves elastic#121014
Resolves elastic#120965
Resolves elastic#120920
Resolves elastic#120890

(cherry picked from commit d1beb01)

# Conflicts:
#	muted-tests.yml
slobodanadamovic added a commit to slobodanadamovic/elasticsearch that referenced this pull request Feb 4, 2025
… tests (elastic#121541)

This PR disables the "queryable built-in roles" feature for the `CoreWithSecurityClientYamlTestSuiteIT` and `DataStreamsClientYamlTestSuiteIT` YAML test suites.
The feature was enabled by default in the elastic#120323 PR, which asynchronously creates the `.security` index after cluster formation and indexes all built-in roles. The asynchronous creation of the `.security` index introduces non-deterministic behavior in our YAML tests.
Since these test suites are not intended to verify the queryable built-in roles functionality, having the feature enabled introduced flakiness and unnecessary complexity to handle `.security` in existing tests. These tests would have to exclude the `.security` index in some way (by adjusting permissions or API calls), and in the end cleanup (delete) the `.security` index. This simply adds overhead without much gain. The feature is already test covered by `XPackRestIT` and other integration/REST tests, disabling it here would not compromise test coverage. Instead, it ensures these suites remain deterministic and focused on the behaviors they were designed to verify.

Resolves elastic#121536
Resolves elastic#121513
Resolves elastic#121484
Resolves elastic#121478
Resolves elastic#121290
Resolves elastic#121246
Resolves elastic#121242
Resolves elastic#121238
Resolves elastic#121186
Resolves elastic#121131
Resolves elastic#121130
Resolves elastic#121128
Resolves elastic#121014
Resolves elastic#120965
Resolves elastic#120920
Resolves elastic#120890

(cherry picked from commit d1beb01)

# Conflicts:
#	muted-tests.yml
slobodanadamovic added a commit to slobodanadamovic/elasticsearch that referenced this pull request Feb 4, 2025
… tests (elastic#121541)

This PR disables the "queryable built-in roles" feature for the `CoreWithSecurityClientYamlTestSuiteIT` and `DataStreamsClientYamlTestSuiteIT` YAML test suites.
The feature was enabled by default in the elastic#120323 PR, which asynchronously creates the `.security` index after cluster formation and indexes all built-in roles. The asynchronous creation of the `.security` index introduces non-deterministic behavior in our YAML tests.
Since these test suites are not intended to verify the queryable built-in roles functionality, having the feature enabled introduced flakiness and unnecessary complexity to handle `.security` in existing tests. These tests would have to exclude the `.security` index in some way (by adjusting permissions or API calls), and in the end cleanup (delete) the `.security` index. This simply adds overhead without much gain. The feature is already test covered by `XPackRestIT` and other integration/REST tests, disabling it here would not compromise test coverage. Instead, it ensures these suites remain deterministic and focused on the behaviors they were designed to verify.

Resolves elastic#121536
Resolves elastic#121513
Resolves elastic#121484
Resolves elastic#121478
Resolves elastic#121290
Resolves elastic#121246
Resolves elastic#121242
Resolves elastic#121238
Resolves elastic#121186
Resolves elastic#121131
Resolves elastic#121130
Resolves elastic#121128
Resolves elastic#121014
Resolves elastic#120965
Resolves elastic#120920
Resolves elastic#120890

(cherry picked from commit d1beb01)

# Conflicts:
#	muted-tests.yml
elasticsearchmachine pushed a commit that referenced this pull request Feb 4, 2025
… tests (#121541) (#121663)

This PR disables the "queryable built-in roles" feature for the `CoreWithSecurityClientYamlTestSuiteIT` and `DataStreamsClientYamlTestSuiteIT` YAML test suites.
The feature was enabled by default in the #120323 PR, which asynchronously creates the `.security` index after cluster formation and indexes all built-in roles. The asynchronous creation of the `.security` index introduces non-deterministic behavior in our YAML tests.
Since these test suites are not intended to verify the queryable built-in roles functionality, having the feature enabled introduced flakiness and unnecessary complexity to handle `.security` in existing tests. These tests would have to exclude the `.security` index in some way (by adjusting permissions or API calls), and in the end cleanup (delete) the `.security` index. This simply adds overhead without much gain. The feature is already test covered by `XPackRestIT` and other integration/REST tests, disabling it here would not compromise test coverage. Instead, it ensures these suites remain deterministic and focused on the behaviors they were designed to verify.

Resolves #121536
Resolves #121513
Resolves #121484
Resolves #121478
Resolves #121290
Resolves #121246
Resolves #121242
Resolves #121238
Resolves #121186
Resolves #121131
Resolves #121130
Resolves #121128
Resolves #121014
Resolves #120965
Resolves #120920
Resolves #120890

(cherry picked from commit d1beb01)

# Conflicts:
#	muted-tests.yml
elasticsearchmachine pushed a commit that referenced this pull request Feb 4, 2025
… tests (#121541) (#121658)

This PR disables the "queryable built-in roles" feature for the `CoreWithSecurityClientYamlTestSuiteIT` and `DataStreamsClientYamlTestSuiteIT` YAML test suites.
The feature was enabled by default in the #120323 PR, which asynchronously creates the `.security` index after cluster formation and indexes all built-in roles. The asynchronous creation of the `.security` index introduces non-deterministic behavior in our YAML tests.
Since these test suites are not intended to verify the queryable built-in roles functionality, having the feature enabled introduced flakiness and unnecessary complexity to handle `.security` in existing tests. These tests would have to exclude the `.security` index in some way (by adjusting permissions or API calls), and in the end cleanup (delete) the `.security` index. This simply adds overhead without much gain. The feature is already test covered by `XPackRestIT` and other integration/REST tests, disabling it here would not compromise test coverage. Instead, it ensures these suites remain deterministic and focused on the behaviors they were designed to verify.

Resolves #121536
Resolves #121513
Resolves #121484
Resolves #121478
Resolves #121290
Resolves #121246
Resolves #121242
Resolves #121238
Resolves #121186
Resolves #121131
Resolves #121130
Resolves #121128
Resolves #121014
Resolves #120965
Resolves #120920
Resolves #120890

(cherry picked from commit d1beb01)

# Conflicts:
#	muted-tests.yml
elasticsearchmachine pushed a commit that referenced this pull request Feb 4, 2025
… tests (#121541) (#121664)

This PR disables the "queryable built-in roles" feature for the `CoreWithSecurityClientYamlTestSuiteIT` and `DataStreamsClientYamlTestSuiteIT` YAML test suites.
The feature was enabled by default in the #120323 PR, which asynchronously creates the `.security` index after cluster formation and indexes all built-in roles. The asynchronous creation of the `.security` index introduces non-deterministic behavior in our YAML tests.
Since these test suites are not intended to verify the queryable built-in roles functionality, having the feature enabled introduced flakiness and unnecessary complexity to handle `.security` in existing tests. These tests would have to exclude the `.security` index in some way (by adjusting permissions or API calls), and in the end cleanup (delete) the `.security` index. This simply adds overhead without much gain. The feature is already test covered by `XPackRestIT` and other integration/REST tests, disabling it here would not compromise test coverage. Instead, it ensures these suites remain deterministic and focused on the behaviors they were designed to verify.

Resolves #121536
Resolves #121513
Resolves #121484
Resolves #121478
Resolves #121290
Resolves #121246
Resolves #121242
Resolves #121238
Resolves #121186
Resolves #121131
Resolves #121130
Resolves #121128
Resolves #121014
Resolves #120965
Resolves #120920
Resolves #120890

(cherry picked from commit d1beb01)

# Conflicts:
#	muted-tests.yml
fzowl pushed a commit to voyage-ai/elasticsearch that referenced this pull request Feb 4, 2025
… tests (elastic#121541)

This PR disables the "queryable built-in roles" feature for the `CoreWithSecurityClientYamlTestSuiteIT` and `DataStreamsClientYamlTestSuiteIT` YAML test suites. 
The feature was enabled by default in the elastic#120323 PR, which asynchronously creates the `.security` index after cluster formation and indexes all built-in roles. The asynchronous creation of the `.security` index introduces non-deterministic behavior in our YAML tests. 
Since these test suites are not intended to verify the queryable built-in roles functionality, having the feature enabled introduced flakiness and unnecessary complexity to handle `.security` in existing tests. These tests would have to exclude the `.security` index in some way (by adjusting permissions or API calls), and in the end cleanup (delete) the `.security` index. This simply adds overhead without much gain. The feature is already test covered by `XPackRestIT` and other integration/REST tests, disabling it here would not compromise test coverage. Instead, it ensures these suites remain deterministic and focused on the behaviors they were designed to verify.

Resolves elastic#121536 
Resolves elastic#121513 
Resolves elastic#121484 
Resolves elastic#121478 
Resolves elastic#121290 
Resolves elastic#121246 
Resolves elastic#121242 
Resolves elastic#121238 
Resolves elastic#121186 
Resolves elastic#121131 
Resolves elastic#121130 
Resolves elastic#121128 
Resolves elastic#121014 
Resolves elastic#120965 
Resolves elastic#120920 
Resolves elastic#120890
slobodanadamovic added a commit that referenced this pull request Feb 16, 2025
This PR fixes SecuritySingleNodeTestCase and ProfileIntegTests tests.

- The security single node test failures are solved by ensuring every test starts with security index created and available. This is in order to have consistent state for every test. With the changes introduce in the #120323 PR, only the first test would execute with .security index being created async. Subsequent tests would execute without security index creation due to the fact that whole cluster is wiped after each test. This caused a flakiness only for the first test, because there was no mechanism in place to ensure that the .security index is active before test execution.

 - The profile integration tests are solved by introducing an anonymous role which don't have application privileges. The application privileges are resolved from the .security index and assigned to all users, including the es_test_root user which is used during cluster wiping. Due to asynchronous nature of cluster setup and .security index creation, this now causes flakiness. The main problem is that wiping is done asynchronously and uses es_test_root which had assigned anonymous rac_role which depends on .security index being available for search in order to resolve application privileges. The application privilege resolution is done in buildRoleFromDescriptors which currently does not wait for security index availability(can be improved - but still wouldn't fix internal cluster tests). This wasn't a problem before just because we simply return empty results when .security index does not exist. There is some complexity in making internal clusters wait for availability of security shards before the test, so I think this solution is acceptable given that it's not required for this tests to have anonymous role with application privileges.

Resolves #121022
Resolves #121096
Resolves #121101
Resolves #120988
Resolves #121108
Resolves #120983
Resolves #120987
Resolves #121179
Resolves #121183
Resolves #121346
Resolves #121151
Resolves #120985
Resolves #121039
Resolves #121483
Resolves #121116
Resolves #121258
Resolves #121486
slobodanadamovic added a commit to slobodanadamovic/elasticsearch that referenced this pull request Feb 17, 2025
This PR fixes SecuritySingleNodeTestCase and ProfileIntegTests tests.

- The security single node test failures are solved by ensuring every test starts with security index created and available. This is in order to have consistent state for every test. With the changes introduce in the elastic#120323 PR, only the first test would execute with .security index being created async. Subsequent tests would execute without security index creation due to the fact that whole cluster is wiped after each test. This caused a flakiness only for the first test, because there was no mechanism in place to ensure that the .security index is active before test execution.

 - The profile integration tests are solved by introducing an anonymous role which don't have application privileges. The application privileges are resolved from the .security index and assigned to all users, including the es_test_root user which is used during cluster wiping. Due to asynchronous nature of cluster setup and .security index creation, this now causes flakiness. The main problem is that wiping is done asynchronously and uses es_test_root which had assigned anonymous rac_role which depends on .security index being available for search in order to resolve application privileges. The application privilege resolution is done in buildRoleFromDescriptors which currently does not wait for security index availability(can be improved - but still wouldn't fix internal cluster tests). This wasn't a problem before just because we simply return empty results when .security index does not exist. There is some complexity in making internal clusters wait for availability of security shards before the test, so I think this solution is acceptable given that it's not required for this tests to have anonymous role with application privileges.

Resolves elastic#121022
Resolves elastic#121096
Resolves elastic#121101
Resolves elastic#120988
Resolves elastic#121108
Resolves elastic#120983
Resolves elastic#120987
Resolves elastic#121179
Resolves elastic#121183
Resolves elastic#121346
Resolves elastic#121151
Resolves elastic#120985
Resolves elastic#121039
Resolves elastic#121483
Resolves elastic#121116
Resolves elastic#121258
Resolves elastic#121486

(cherry picked from commit 369c641)

# Conflicts:
#	muted-tests.yml
#	x-pack/plugin/security/src/internalClusterTest/java/org/elasticsearch/xpack/security/authc/esnative/ReservedRealmElasticAutoconfigIntegTests.java
slobodanadamovic added a commit to slobodanadamovic/elasticsearch that referenced this pull request Feb 17, 2025
This PR fixes SecuritySingleNodeTestCase and ProfileIntegTests tests.

- The security single node test failures are solved by ensuring every test starts with security index created and available. This is in order to have consistent state for every test. With the changes introduce in the elastic#120323 PR, only the first test would execute with .security index being created async. Subsequent tests would execute without security index creation due to the fact that whole cluster is wiped after each test. This caused a flakiness only for the first test, because there was no mechanism in place to ensure that the .security index is active before test execution.

 - The profile integration tests are solved by introducing an anonymous role which don't have application privileges. The application privileges are resolved from the .security index and assigned to all users, including the es_test_root user which is used during cluster wiping. Due to asynchronous nature of cluster setup and .security index creation, this now causes flakiness. The main problem is that wiping is done asynchronously and uses es_test_root which had assigned anonymous rac_role which depends on .security index being available for search in order to resolve application privileges. The application privilege resolution is done in buildRoleFromDescriptors which currently does not wait for security index availability(can be improved - but still wouldn't fix internal cluster tests). This wasn't a problem before just because we simply return empty results when .security index does not exist. There is some complexity in making internal clusters wait for availability of security shards before the test, so I think this solution is acceptable given that it's not required for this tests to have anonymous role with application privileges.

Resolves elastic#121022
Resolves elastic#121096
Resolves elastic#121101
Resolves elastic#120988
Resolves elastic#121108
Resolves elastic#120983
Resolves elastic#120987
Resolves elastic#121179
Resolves elastic#121183
Resolves elastic#121346
Resolves elastic#121151
Resolves elastic#120985
Resolves elastic#121039
Resolves elastic#121483
Resolves elastic#121116
Resolves elastic#121258
Resolves elastic#121486

(cherry picked from commit 369c641)

# Conflicts:
#	muted-tests.yml
slobodanadamovic added a commit to slobodanadamovic/elasticsearch that referenced this pull request Feb 17, 2025
This PR fixes SecuritySingleNodeTestCase and ProfileIntegTests tests.

- The security single node test failures are solved by ensuring every test starts with security index created and available. This is in order to have consistent state for every test. With the changes introduce in the elastic#120323 PR, only the first test would execute with .security index being created async. Subsequent tests would execute without security index creation due to the fact that whole cluster is wiped after each test. This caused a flakiness only for the first test, because there was no mechanism in place to ensure that the .security index is active before test execution.

 - The profile integration tests are solved by introducing an anonymous role which don't have application privileges. The application privileges are resolved from the .security index and assigned to all users, including the es_test_root user which is used during cluster wiping. Due to asynchronous nature of cluster setup and .security index creation, this now causes flakiness. The main problem is that wiping is done asynchronously and uses es_test_root which had assigned anonymous rac_role which depends on .security index being available for search in order to resolve application privileges. The application privilege resolution is done in buildRoleFromDescriptors which currently does not wait for security index availability(can be improved - but still wouldn't fix internal cluster tests). This wasn't a problem before just because we simply return empty results when .security index does not exist. There is some complexity in making internal clusters wait for availability of security shards before the test, so I think this solution is acceptable given that it's not required for this tests to have anonymous role with application privileges.

Resolves elastic#121022
Resolves elastic#121096
Resolves elastic#121101
Resolves elastic#120988
Resolves elastic#121108
Resolves elastic#120983
Resolves elastic#120987
Resolves elastic#121179
Resolves elastic#121183
Resolves elastic#121346
Resolves elastic#121151
Resolves elastic#120985
Resolves elastic#121039
Resolves elastic#121483
Resolves elastic#121116
Resolves elastic#121258
Resolves elastic#121486

(cherry picked from commit 369c641)

# Conflicts:
#	muted-tests.yml
#	x-pack/plugin/security/src/internalClusterTest/java/org/elasticsearch/xpack/security/authc/esnative/ReservedRealmElasticAutoconfigIntegTests.java
elasticsearchmachine pushed a commit that referenced this pull request Feb 17, 2025
This PR fixes SecuritySingleNodeTestCase and ProfileIntegTests tests.

- The security single node test failures are solved by ensuring every test starts with security index created and available. This is in order to have consistent state for every test. With the changes introduce in the #120323 PR, only the first test would execute with .security index being created async. Subsequent tests would execute without security index creation due to the fact that whole cluster is wiped after each test. This caused a flakiness only for the first test, because there was no mechanism in place to ensure that the .security index is active before test execution.

 - The profile integration tests are solved by introducing an anonymous role which don't have application privileges. The application privileges are resolved from the .security index and assigned to all users, including the es_test_root user which is used during cluster wiping. Due to asynchronous nature of cluster setup and .security index creation, this now causes flakiness. The main problem is that wiping is done asynchronously and uses es_test_root which had assigned anonymous rac_role which depends on .security index being available for search in order to resolve application privileges. The application privilege resolution is done in buildRoleFromDescriptors which currently does not wait for security index availability(can be improved - but still wouldn't fix internal cluster tests). This wasn't a problem before just because we simply return empty results when .security index does not exist. There is some complexity in making internal clusters wait for availability of security shards before the test, so I think this solution is acceptable given that it's not required for this tests to have anonymous role with application privileges.

Resolves #121022
Resolves #121096
Resolves #121101
Resolves #120988
Resolves #121108
Resolves #120983
Resolves #120987
Resolves #121179
Resolves #121183
Resolves #121346
Resolves #121151
Resolves #120985
Resolves #121039
Resolves #121483
Resolves #121116
Resolves #121258
Resolves #121486

(cherry picked from commit 369c641)

# Conflicts:
#	muted-tests.yml
elasticsearchmachine pushed a commit that referenced this pull request Feb 17, 2025
…122732)

* Fix internal cluster and single node security tests (#121466)

This PR fixes SecuritySingleNodeTestCase and ProfileIntegTests tests.

- The security single node test failures are solved by ensuring every test starts with security index created and available. This is in order to have consistent state for every test. With the changes introduce in the #120323 PR, only the first test would execute with .security index being created async. Subsequent tests would execute without security index creation due to the fact that whole cluster is wiped after each test. This caused a flakiness only for the first test, because there was no mechanism in place to ensure that the .security index is active before test execution.

 - The profile integration tests are solved by introducing an anonymous role which don't have application privileges. The application privileges are resolved from the .security index and assigned to all users, including the es_test_root user which is used during cluster wiping. Due to asynchronous nature of cluster setup and .security index creation, this now causes flakiness. The main problem is that wiping is done asynchronously and uses es_test_root which had assigned anonymous rac_role which depends on .security index being available for search in order to resolve application privileges. The application privilege resolution is done in buildRoleFromDescriptors which currently does not wait for security index availability(can be improved - but still wouldn't fix internal cluster tests). This wasn't a problem before just because we simply return empty results when .security index does not exist. There is some complexity in making internal clusters wait for availability of security shards before the test, so I think this solution is acceptable given that it's not required for this tests to have anonymous role with application privileges.

Resolves #121022
Resolves #121096
Resolves #121101
Resolves #120988
Resolves #121108
Resolves #120983
Resolves #120987
Resolves #121179
Resolves #121183
Resolves #121346
Resolves #121151
Resolves #120985
Resolves #121039
Resolves #121483
Resolves #121116
Resolves #121258
Resolves #121486

(cherry picked from commit 369c641)

# Conflicts:
#	muted-tests.yml
#	x-pack/plugin/security/src/internalClusterTest/java/org/elasticsearch/xpack/security/authc/esnative/ReservedRealmElasticAutoconfigIntegTests.java

* fix compilation error
elasticsearchmachine pushed a commit that referenced this pull request Feb 17, 2025
…122734)

* Fix internal cluster and single node security tests (#121466)

This PR fixes SecuritySingleNodeTestCase and ProfileIntegTests tests.

- The security single node test failures are solved by ensuring every test starts with security index created and available. This is in order to have consistent state for every test. With the changes introduce in the #120323 PR, only the first test would execute with .security index being created async. Subsequent tests would execute without security index creation due to the fact that whole cluster is wiped after each test. This caused a flakiness only for the first test, because there was no mechanism in place to ensure that the .security index is active before test execution.

 - The profile integration tests are solved by introducing an anonymous role which don't have application privileges. The application privileges are resolved from the .security index and assigned to all users, including the es_test_root user which is used during cluster wiping. Due to asynchronous nature of cluster setup and .security index creation, this now causes flakiness. The main problem is that wiping is done asynchronously and uses es_test_root which had assigned anonymous rac_role which depends on .security index being available for search in order to resolve application privileges. The application privilege resolution is done in buildRoleFromDescriptors which currently does not wait for security index availability(can be improved - but still wouldn't fix internal cluster tests). This wasn't a problem before just because we simply return empty results when .security index does not exist. There is some complexity in making internal clusters wait for availability of security shards before the test, so I think this solution is acceptable given that it's not required for this tests to have anonymous role with application privileges.

Resolves #121022
Resolves #121096
Resolves #121101
Resolves #120988
Resolves #121108
Resolves #120983
Resolves #120987
Resolves #121179
Resolves #121183
Resolves #121346
Resolves #121151
Resolves #120985
Resolves #121039
Resolves #121483
Resolves #121116
Resolves #121258
Resolves #121486

(cherry picked from commit 369c641)

# Conflicts:
#	muted-tests.yml
#	x-pack/plugin/security/src/internalClusterTest/java/org/elasticsearch/xpack/security/authc/esnative/ReservedRealmElasticAutoconfigIntegTests.java

* fix compilation error
nielsbauman added a commit to nielsbauman/elasticsearch that referenced this pull request Mar 26, 2025
The V7 REST compat tests shouldn't run with queryable built-in roles
enabled.

Relates elastic#120323

Fixes elastic#121501
Fixes elastic#122226
Fixes elastic#122687
Fixes elastic#123081
Fixes elastic#123147
nielsbauman added a commit to nielsbauman/elasticsearch that referenced this pull request Mar 26, 2025
The V7 REST compat tests shouldn't run with queryable built-in roles
enabled.

Relates elastic#120323

Fixes elastic#121501
Fixes elastic#122226
Fixes elastic#122687
Fixes elastic#123081
Fixes elastic#123147
nielsbauman added a commit that referenced this pull request Mar 27, 2025
The V7 REST compat tests shouldn't run with queryable built-in roles
enabled.

Relates #120323

Fixes #121501
Fixes #122226
Fixes #122687
Fixes #123081
Fixes #123147
nielsbauman added a commit to nielsbauman/elasticsearch that referenced this pull request Mar 31, 2025
The V7 REST compat tests shouldn't run with queryable built-in roles
enabled.

Relates elastic#120323

Fixes elastic#121501
Fixes elastic#122226
Fixes elastic#122687
Fixes elastic#123081
Fixes elastic#123147

(cherry picked from commit a3b9b98)

# Conflicts:
#	muted-tests.yml
#	x-pack/plugin/build.gradle
elasticsearchmachine pushed a commit that referenced this pull request Mar 31, 2025
The V7 REST compat tests shouldn't run with queryable built-in roles
enabled.

Relates #120323

Fixes #121501
Fixes #122226
Fixes #122687
Fixes #123081
Fixes #123147

(cherry picked from commit a3b9b98)

# Conflicts:
#	muted-tests.yml
elasticsearchmachine pushed a commit that referenced this pull request Mar 31, 2025
The V7 REST compat tests shouldn't run with queryable built-in roles
enabled.

Relates #120323

Fixes #121501
Fixes #122226
Fixes #122687
Fixes #123081
Fixes #123147

(cherry picked from commit a3b9b98)

# Conflicts:
#	muted-tests.yml
#	x-pack/plugin/build.gradle
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

auto-backport Automatically create backport pull requests when merged >non-issue :Security/Authorization Roles, Privileges, DLS/FLS, RBAC/ABAC serverless-linked Added by automation, don't add manually Team:Security Meta label for security team v8.18.0 v9.0.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants