Skip to content

Granting kibana_system reserved role access to "all" privileges to .adhoc.alerts* and .internal.adhoc.alerts* indices #127321

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 19 commits into from
Jun 3, 2025
Merged

Granting kibana_system reserved role access to "all" privileges to .adhoc.alerts* and .internal.adhoc.alerts* indices #127321

Show file tree
Hide file tree
Changes from 8 commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
d559964
Granting `kibana_system` reserved role access to "all" privileges to …
e40pud Apr 24, 2025
5966787
Update docs/changelog/127321.yaml
e40pud Apr 24, 2025
9a47df2
[CI] Auto commit changes from spotless
Apr 24, 2025
e34b3e7
Merge branch 'main' into security/genai/adhoc-alerts-index
e40pud May 8, 2025
49cf63d
Replace `"all"` with the specific privileges for the `kibana_system` …
e40pud May 8, 2025
5231345
Fix tests
e40pud May 8, 2025
c0df6dd
Fix CI
e40pud May 8, 2025
f543bd9
Merge branch 'main' into security/genai/adhoc-alerts-index
e40pud May 8, 2025
44ce3ba
Merge branch 'main' into security/genai/adhoc-alerts-index
e40pud May 19, 2025
8be39d4
Merge branch 'main' into security/genai/adhoc-alerts-index
e40pud May 20, 2025
8394f72
Merge branch 'main' into security/genai/adhoc-alerts-index
e40pud May 20, 2025
79dd079
Merge branch 'main' into security/genai/adhoc-alerts-index
e40pud May 27, 2025
4e9dda0
Updated privileges
e40pud May 27, 2025
67c182c
Updated privileges
e40pud May 28, 2025
8d2f9c0
Merge branch 'main' into security/genai/adhoc-alerts-index
e40pud May 28, 2025
e5fbf27
Remove redundant code
e40pud May 28, 2025
f017abc
Merge branch 'main' into security/genai/adhoc-alerts-index
e40pud May 29, 2025
d12adc2
Merge branch 'main' into security/genai/adhoc-alerts-index
e40pud May 29, 2025
4775ac6
Merge branch 'main' into security/genai/adhoc-alerts-index
e40pud May 30, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions docs/changelog/127321.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
pr: 127321
summary: Granting `kibana_system` reserved role access to "all" privileges to `.adhoc.alerts*`
and `.internal.adhoc.alerts*` indices
area: Authorization
type: enhancement
issues: []
6 changes: 6 additions & 0 deletions ...org/elasticsearch/xpack/core/security/authz/store/KibanaOwnedReservedRoleDescriptors.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -265,6 +265,12 @@ static RoleDescriptor kibanaSystem(String name) {
RoleDescriptor.IndicesPrivileges.builder().indices(ReservedRolesStore.ALERTS_INDEX_ALIAS).privileges("all").build(),
// "Alerts as data" public index alias used in Security Solution
// Kibana system user uses them to read / write alerts.
RoleDescriptor.IndicesPrivileges.builder()
.indices(ReservedRolesStore.ADHOC_ALERTS_BACKING_INDEX, ReservedRolesStore.ADHOC_ALERTS_INDEX_ALIAS)
.privileges("create_index", "manage", "read", "write")
Copy link

@jeramysoucy jeramysoucy May 8, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@e40pud Thanks for updating this! The last thing I'd like to ask, is about the manage privilege, which inherently includes all of these privileges:

  • manage_data_stream_lifecycle
  • manage_follow_index
  • manage_ilm
  • manage_leader_index
  • monitor
  • maintenance
  • auto_configure

Aiming for the minimum necessary privileges, could we include just a subset of these? If not, could you provide justification to include them? Sorry to have to ask. We like to be as thorough as we can when changing this role.

Copy link
Contributor Author

@e40pud e40pud May 9, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I went through the ES apis that we use to work with this indices and here is the list of all of them:

  1. Simulate an index (API, Usage in kibana)
  2. Simulate index template (API, Usage in kibana)
  3. Create or update an index template (API, Usage in kibana)
  4. Get data streams (API, Usage in kibana)
  5. Create a data stream (API, Usage in kibana)
  6. Get aliases (API, Usage in kibana)
  7. Get index information (API, Usage in kibana)
  8. Create an index (API, Usage in kibana)
  9. Create or update an alias (API, Usage in kibana)
  10. Get index templates (API, Usage in kibana)
  11. Update field mappings (API, Usage in kibana)
  12. Create or update a lifecycle policy (API, Usage in kibana)
  13. Create or update a component template (API, Usage in kibana)
  14. Update index settings (API, Usage in kibana)
  15. Search (API, Usage in kibana)
  16. Bulk index or delete documents (API, Usage in kibana)

Regarding the manage privilege we need it because when we create the index we also specify the alias for it:

esClient.indices.create({
  index: indexPatterns.name,
  aliases: {
    [indexPatterns.alias]: {
      is_write_index: true,
    },
  },
})

and according to the create_index privilege documentation manage privilege is required.

Privilege to create an index or data stream. A create index request may contain aliases to be added to the index once created. In that case the request requires the manage privilege as well, on both the index and the aliases names.

Copy link

@jeramysoucy jeramysoucy May 19, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@e40pud First, sorry for the delay. Second, sorry to ask for another change. I reviewed this with @slobodanadamovic, and while it is true that we do not have specific alias privileges, we are also not limited to using only end-user named privileges.

In this case, we can reduce the scope being granted by the manage privilege by referencing action names directly instead. Here's an example that Slobodan provided.

Does this seem like a better match here? We'd prefer this from a security perspective, since we can hone in on just the action needed without needing to grant any additional access via manage.

Copy link
Contributor Author

@e40pud e40pud May 19, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the response @jeramysoucy. I'm just not quite sure which actions I will need to use in this case. There are multiple APIs that will have access to the index and it's alias (16 mentioned above) and I do not really know what set of privileges will cover all of them. Is there some kind of map of API => minimum required privilege which I can use to create a correct minimum set of privileges?

I'm also not sure why the documentation states the required manage privilege and still we do not really need it? Is this specific to our use case only or the documentation does not accurately describe it?

Copy link

@jeramysoucy jeramysoucy May 21, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm also not sure why the documentation states the required manage privilege and still we do not really need it? Is this specific to our use case only or the documentation does not accurately describe it?

My understanding is that the documentation is intended for end-users. As user, we would only be able to include named privileges in a role. But as a developer, we do not have this limitation.

IIUC, the example linked above will impart the privilege for the necessary alias actions. Regarding the others you listed, I'll need some help from @elastic/es-security (cc @slobodanadamovic)

Here is what I could identify so far...

  • Simulate an index: ?
  • Simulate index template: manage_index_templates cluster privilege?
  • Create or update an index template: manage_index_templates cluster privilege?
  • Get data streams: view_index_metadata
  • Create a data stream: create_index
  • Get aliases: view_index_metadata
  • Get index information: view_index_metadata
  • Create an index: create_index
  • Create or update an alias: TransportIndicesAliasesAction.NAME (not sure if this also grants update)
  • Get index templates: manage_index_templates cluster privilege?
  • Update field mappings: write? Not sure on this one. Many of the privileges that are marked as granting this privilege also note that it is deprecated as of v8.0.
  • Create or update a lifecycle policy: manage_ilm
  • Create or update a component template: ?
  • Update index settings: ?
  • Search: read
  • Bulk index or delete documents: write

Copy link
Contributor

@slobodanadamovic slobodanadamovic May 27, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Basically, the privileges grant certain actions to be executed. We have named privileges (e.g. manage) that have a set of patterns or actions that they grant. In this particular case it’s the rollover action that was denied.
You can grant it to kibana_system role by adding the actual action name indices:admin/rollover to index privileges, or by referencing the action constant RolloverAction.NAME.

Copy link
Contributor Author

@e40pud e40pud May 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I now updated Serverless roles to reflect this changes and I see the next error:

[2025-05-27T17:04:06.579+02:00][ERROR][plugins.ruleRegistry] Error creating data stream .adhoc.alerts-security.attack.discovery.alerts-default - security_exception
Root causes:
security_exception: action [indices:admin/data_stream/create] is unauthorized for service account [elastic/kibana] on indices [.adhoc.alerts-security.attack.discovery.alerts-default], this action is granted by the index privileges [create_index,manage,all]

Should I add CreateDataStreamAction.NAME (indices:admin/data_stream/create) to index privilege?

Copy link
Contributor

@slobodanadamovic slobodanadamovic May 27, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Strange. This should be allowed. You have create_index over .adhoc.alerts* names, which includes .adhoc.alerts-security.attack.discovery.alerts-default and create_index grants indices:admin/data_stream/create. How did you test this? Is it possible that Elasticsearch changes from this PR were not included in your test?

Copy link
Contributor Author

@e40pud e40pud May 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, that was me testing against not update ES. I checked it again with my changes and this error is gone.

Though, now I see a different one:

[2025-05-27T19:39:54.214+02:00][ERROR][plugins.ruleRegistry] ResponseError: {"errors":true,"took":0,"items":[{"create":{"_index":".adhoc.alerts-security.attack.discovery.alerts-default","_id":"0b666812-ac3a-4aeb-ad08-466789e49891","status":403,"failure_store":"not_enabled","error":{"type":"security_exception","reason":"action [indices:admin/refresh/unpromotable] is unauthorized for service account [elastic/kibana] on indices [.ds-.adhoc.alerts-security.attack.discovery.alerts-default-2025.05.27-000001], this action is granted by the index privileges [maintenance,manage,all]"}}}]}
at Object.bulk (rule_data_client.ts:247:27)
at processTicksAndRejections (node:internal/process/task_queues:95:5)
at createAttackDiscoveryAlerts (index.ts:84:20)
at generateAndUpdateAttackDiscoveries (generate_and_update_discoveries.ts:106:7)

We use refresh in the bulk API and I guess this is causing this error.

When I added "manage" privilege to the list the error is gone. @slobodanadamovic what would be an appropriate privilege to add to that list to cover the refresh=true?

Copy link
Contributor

@slobodanadamovic slobodanadamovic May 28, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For allowing refresh, I suggest granting maintenance privilege.

.build(),
// "Alerts as data" public index alias used in Security Solution
// Kibana system user uses them to read / write alerts.
RoleDescriptor.IndicesPrivileges.builder().indices(ReservedRolesStore.PREVIEW_ALERTS_INDEX_ALIAS).privileges("all").build(),
// "Alerts as data" internal backing indices used in Security Solution
// Kibana system user creates these indices; reads / writes to them via the
Expand Down
14 changes: 12 additions & 2 deletions ...e/src/main/java/org/elasticsearch/xpack/core/security/authz/store/ReservedRolesStore.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,10 @@ public class ReservedRolesStore implements BiConsumer<Set<String>, ActionListene
public static final String PREVIEW_ALERTS_BACKING_INDEX = ".internal.preview.alerts*";
public static final String PREVIEW_ALERTS_BACKING_INDEX_REINDEXED = ".reindexed-v8-internal.preview.alerts*";

/** "Attack Discovery" ad-hoc alerts index */
public static final String ADHOC_ALERTS_INDEX_ALIAS = ".adhoc.alerts*";
public static final String ADHOC_ALERTS_BACKING_INDEX = ".internal.adhoc.alerts*";

/** "Security Solutions" only lists index for value lists for detections */
public static final String LISTS_INDEX = ".lists-*";
public static final String LISTS_INDEX_REINDEXED_V8 = ".reindexed-v8-lists-*";
Expand Down Expand Up @@ -782,7 +786,11 @@ private static RoleDescriptor buildViewerRoleDescriptor() {
.build(),
// Alerts-as-data
RoleDescriptor.IndicesPrivileges.builder()
.indices(ReservedRolesStore.ALERTS_INDEX_ALIAS, ReservedRolesStore.PREVIEW_ALERTS_INDEX_ALIAS)
.indices(
ReservedRolesStore.ALERTS_INDEX_ALIAS,
ReservedRolesStore.PREVIEW_ALERTS_INDEX_ALIAS,
ReservedRolesStore.ADHOC_ALERTS_INDEX_ALIAS
)
.privileges("read", "view_index_metadata")
.build(),
// Universal Profiling
Expand Down Expand Up @@ -846,7 +854,9 @@ private static RoleDescriptor buildEditorRoleDescriptor() {
ReservedRolesStore.ALERTS_INDEX_ALIAS,
ReservedRolesStore.PREVIEW_ALERTS_BACKING_INDEX,
ReservedRolesStore.PREVIEW_ALERTS_BACKING_INDEX_REINDEXED,
ReservedRolesStore.PREVIEW_ALERTS_INDEX_ALIAS
ReservedRolesStore.PREVIEW_ALERTS_INDEX_ALIAS,
ReservedRolesStore.ADHOC_ALERTS_BACKING_INDEX,
ReservedRolesStore.ADHOC_ALERTS_INDEX_ALIAS
)
.privileges("read", "view_index_metadata", "write", "maintenance")
.build(),
Expand Down
11 changes: 11 additions & 0 deletions .../test/java/org/elasticsearch/xpack/core/security/authz/store/ReservedRolesStoreTests.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -624,6 +624,17 @@ public void testKibanaSystemRole() {
".slo-observability." + randomAlphaOfLength(randomIntBetween(0, 13))
).forEach(index -> assertAllIndicesAccessAllowed(kibanaRole, index));

Arrays.asList(
ReservedRolesStore.ADHOC_ALERTS_INDEX_ALIAS + randomAlphaOfLength(randomIntBetween(0, 13)),
ReservedRolesStore.ADHOC_ALERTS_BACKING_INDEX + randomAlphaOfLength(randomIntBetween(0, 13))
).forEach(index -> {
final IndexAbstraction indexAbstraction = mockIndexAbstraction(index);
assertThat(kibanaRole.indices().allowedIndicesMatcher(TransportCreateIndexAction.TYPE.name()).test(indexAbstraction), is(true));
assertThat(kibanaRole.indices().allowedIndicesMatcher(TransportSearchAction.TYPE.name()).test(indexAbstraction), is(true));
assertThat(kibanaRole.indices().allowedIndicesMatcher(TransportUpdateAction.TYPE.name()).test(indexAbstraction), is(true));
assertViewIndexMetadata(kibanaRole, index);
});

// read-only index access, including cross cluster
Arrays.asList(".monitoring-" + randomAlphaOfLength(randomIntBetween(0, 13))).forEach((index) -> {
logger.info("index name [{}]", index);
Expand Down
Loading