Add aws.config source indices to kibana_system role permissions #128350

kcreddy · 2025-05-23T07:37:08Z

Adding logs-aws.config-* data stream indices to the kibana_system privileges. This is required for the latest transform to work.

Add new AWS Config datastream. integrations#13830 (comment)

Similar to #124074

elasticsearchmachine · 2025-05-23T07:38:29Z

Pinging @elastic/es-security (Team:Security)

kc13greiner

LGTM

Reasoning:

Although we generally would not grant the system user access to data indexes, an exception is made for logs-* (documented here) as it is a known collision pattern.

kcreddy · 2025-05-23T17:03:17Z

@kc13greiner, is there anything that should be done to automatically create all the backport PRs?
This PR is very similar to #124074 where many backport PRs got created automatically.

cc: @maxcold

kc13greiner · 2025-05-23T17:44:00Z

@kcreddy I am not sure how the es repo labels work. The auto-backport w/ versions seemed like it would work to me 🤔

kcreddy · 2025-05-26T07:29:37Z

@kcreddy I am not sure how the es repo labels work. The auto-backport w/ versions seemed like it would work to me 🤔

@kc13greiner can you please tag someone to assist with this issue, considering the public release is tomorrow for these branches?
How about creating manual backport PRs, does that work?

kcreddy · 2025-05-26T07:33:47Z

@slobodanadamovic, this PR is very similar to #124074 which you approved previously. But the backport PRs aren't getting created here even though auto-backport label is added. Could you please assist?

cc: @maxcold

slobodanadamovic · 2025-05-26T07:44:29Z

@kcreddy The auto-backport label will only take effect if applied before the PR gets merged. Since it was added after, you'll have to backport it manually or use the the backport tool.

kcreddy · 2025-05-26T07:45:20Z

@kcreddy The auto-backport label will only take effect if applied before the PR gets merged. Since it was added after, you'll have to backport it manually or use the the backport tool.

@slobodanadamovic, Actually I added this label right after creating the PR, even before the review: #128350 (comment)

slobodanadamovic · 2025-05-26T07:54:52Z

@kcreddy The auto-backport label will only take effect if applied before the PR gets merged. Since it was added after, you'll have to backport it manually or use the the backport tool.

@slobodanadamovic, Actually I added this label right after creating the PR, even before the review: #128350 (comment)

I missed that. Sorry!

Not sure why it did not work. The @elastic/es-delivery team might help diagnosing this.

In any case, I think you'll have to use the backport tool locally or backport the PR manually. There is no way to trigger backporting after the PR got merge. Given that this happens from time to time it would be nice to have the ability to manully trigger backport, e.g. via @elasticmachine backport.

kcreddy · 2025-05-26T07:56:50Z

In any case, I think you'll have to use the backport tool locally or backport the PR manually. There is no way to trigger backporting after the PR got merge

Thanks for the quick help ❤️ I will check the backport tool and create PRs.

Given that this happens from time to time it would be nice to have the ability to manully trigger backport, e.g. via @elasticmachine backport.

Strongly agree with this 💯

kcreddy · 2025-05-26T08:50:56Z

💔 Some backports could not be created

Status	Branch	Result
✅	9.0
✅	8.19
✅	8.18
❌	8.17	Could not create pull request:

Manual backport

To create the backport manually run:

backport --pr 128350

Questions ?

Please refer to the Backport tool documentation

…tic#128350) Adding `logs-aws.config-*` data stream indices to the `kibana_system` privileges. This is required for the latest transform to work. Related: - elastic/integrations#13830 (comment) (cherry picked from commit 169527f)

kcreddy · 2025-05-26T09:04:33Z

In any case, I think you'll have to use the backport tool locally or backport the PR manually. There is no way to trigger backporting after the PR got merge

Thanks for the quick help ❤️ I will check the backport tool and create PRs.

Given that this happens from time to time it would be nice to have the ability to manully trigger backport, e.g. via @elasticmachine backport.

Strongly agree with this 💯

@slobodanadamovic, I created backport PRs using the backport tool.
But the PRs #128446 and #128443 got created with next patch version than what we would like to target.
We want: 8.18.2 , the PR created has 8.18.3 label.
We want: 9.0.2 , the PR created has 9.0.3 label.

Also there is an error for 8.17.x which isn't quite intuitive.

…tic#128350) Adding `logs-aws.config-*` data stream indices to the `kibana_system` privileges. This is required for the latest transform to work. Related: - elastic/integrations#13830 (comment) (cherry picked from commit 169527f)

kcreddy · 2025-05-26T12:16:13Z

💚 All backports created successfully

Status	Branch	Result
✅	8.17

Questions ?

Please refer to the Backport tool documentation

) (#128443) Adding `logs-aws.config-*` data stream indices to the `kibana_system` privileges. This is required for the latest transform to work. Related: - elastic/integrations#13830 (comment) (cherry picked from commit 169527f)

) (#128446) Adding `logs-aws.config-*` data stream indices to the `kibana_system` privileges. This is required for the latest transform to work. Related: - elastic/integrations#13830 (comment) (cherry picked from commit 169527f)

) (#128444) Adding `logs-aws.config-*` data stream indices to the `kibana_system` privileges. This is required for the latest transform to work. Related: - elastic/integrations#13830 (comment) (cherry picked from commit 169527f)

) (#128460) Adding `logs-aws.config-*` data stream indices to the `kibana_system` privileges. This is required for the latest transform to work. Related: - elastic/integrations#13830 (comment) (cherry picked from commit 169527f)

…R) workflow (#15230) aws: Add transforms to Config and Inspector data streams for extended protections (CDR) workflow. - Add latest transform to Config and Inspector data streams to help with Cloud Native Vulnerability Management (CNVM)[1] and Cloud Security Posture Management (CSPM)[2] workflows. - Add ILM policy to AWS Config as it does full sync every interval. - Update minimum kibana version to "^8.19.0 || ^9.1.0" to ensure necessary permissions for transform[3]. - Re-add 3.14.2 changelog entry as it is overwritten in VPC Flow PR[4]. - Skip system tests for securityhub* data streams to avoid fleet health degradation due to empty template values by httpjson. This is fixed in 8.19.4 and 9.1.4 by beats#45810[5] and beats#46332[6]. This skip can be removed when the stack version is upgraded to ones containing the fix. [1] https://www.elastic.co/guide/en/security/current/vuln-management-overview.html [2] https://www.elastic.co/docs/solutions/security/cloud/cloud-security-posture-management [3] elastic/elasticsearch#128350 [4] #15077 [5] elastic/beats#45810 [6] elastic/beats#46332

Add aws.config source indices to kibana_system role permissions

ca8088f

kcreddy requested a review from a team as a code owner May 23, 2025 07:37

kcreddy self-assigned this May 23, 2025

elasticsearchmachine added the v9.1.0 label May 23, 2025

kcreddy added >non-issue v8.17.7 v8.18.2 v8.19.0 v9.0.2 labels May 23, 2025

kcreddy requested a review from maxcold May 23, 2025 07:42

kcreddy mentioned this pull request May 23, 2025

Add new AWS Config datastream. elastic/integrations#13830

Merged

5 tasks

maxcold approved these changes May 23, 2025

View reviewed changes

kc13greiner approved these changes May 23, 2025

View reviewed changes

kcreddy merged commit 169527f into elastic:main May 23, 2025
18 checks passed

kcreddy added auto-backport Automatically create backport pull requests when merged and removed auto-backport Automatically create backport pull requests when merged labels May 26, 2025

kcreddy deleted the csp-aws-config branch May 26, 2025 07:27

kcreddy restored the csp-aws-config branch May 26, 2025 07:27

kcreddy mentioned this pull request May 26, 2025

[8.17] Add aws.config source indices to kibana_system role permissions (#128350) #128460

Merged

kcreddy mentioned this pull request May 30, 2025

Add rapid7_insightvm.asset_vulnerability source indices to kibana_system role permissions #128661

Merged

kcreddy mentioned this pull request Aug 5, 2025

Add {m365_defender,microsoft_defender_endpoint}.vulnerability indices to kibana_system role permissions #132445

Merged

Uh oh!

Add aws.config source indices to kibana_system role permissions #128350

Add aws.config source indices to kibana_system role permissions #128350

Uh oh!

Conversation

kcreddy commented May 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticsearchmachine commented May 23, 2025

Uh oh!

kc13greiner left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kcreddy commented May 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kc13greiner commented May 23, 2025

Uh oh!

kcreddy commented May 26, 2025

Uh oh!

kcreddy commented May 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

slobodanadamovic commented May 26, 2025

Uh oh!

kcreddy commented May 26, 2025

Uh oh!

slobodanadamovic commented May 26, 2025

Uh oh!

kcreddy commented May 26, 2025

Uh oh!

kcreddy commented May 26, 2025 • edited by slobodanadamovic Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💔 Some backports could not be created

Manual backport

Questions ?

Uh oh!

kcreddy commented May 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kcreddy commented May 26, 2025

💚 All backports created successfully

Questions ?

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

kcreddy commented May 23, 2025 •

edited

Loading

kcreddy commented May 23, 2025 •

edited

Loading

kcreddy commented May 26, 2025 •

edited

Loading

kcreddy commented May 26, 2025 •

edited by slobodanadamovic

Loading

kcreddy commented May 26, 2025 •

edited

Loading