Skip to content

[Fleet] add privileges to kibana_system to read integrations data #132501

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 7 commits into from
Closed

[Fleet] add privileges to kibana_system to read integrations data #132501

wants to merge 7 commits into from

Conversation

@juliaElastic
Copy link
Contributor

  • Have you signed the contributor license agreement? yes
  • Have you followed the contributor guidelines? yes
  • If submitting code, have you built your formula locally prior to submission with gradle check? yes
  • If submitting code, is your pull request against main? Unless there is a good reason otherwise, we prefer pull requests against main and will backport as needed. yes
  • If submitting code, have you checked that your submission is for an OS and architecture that we support? yes
  • If you are submitting this code for a class then read our policy for that. yes

Relates elastic/kibana#230369
Relates https://github.com/elastic/ingest-dev/issues/5685

kibana_system needs permission to read integrations data streams logs-*,metrics-*,traces-* to support the auto install content packages feature.
Fleet checks ingested data_stream.dataset values to see if any content packages match that can be auto installed.

Context: #132400 (review)
Readd the change to discuss with kibana-security.
PR ready after the revert is merged: #132499

juliaElastic and others added 2 commits August 6, 2025 14:48
@elasticsearchmachine elasticsearchmachine added external-contributor Pull request authored by a developer outside the Elasticsearch team v9.2.0 labels Aug 6, 2025
@juliaElastic juliaElastic added >non-issue :Core/Infra/Plugins Plugin API and infrastructure Team:Fleet and removed external-contributor Pull request authored by a developer outside the Elasticsearch team v9.2.0 labels Aug 6, 2025
@juliaElastic juliaElastic mentioned this pull request Aug 6, 2025
Merged
@juliaElastic juliaElastic added the Team:Core/Infra Meta label for core/infra team label Aug 6, 2025
@juliaElastic juliaElastic marked this pull request as ready for review August 18, 2025 07:47
@juliaElastic juliaElastic requested a review from a team as a code owner August 18, 2025 07:47
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-core-infra (Team:Core/Infra)

@SiddharthMantri
Copy link
Contributor

Hi @juliaElastic - thanks for reverting and reopening this. We are on On-Week, this week - so expect a slightly slower review. But i'll raise this with the team anyway and let you know the outcome.

@juliaElastic juliaElastic added the cloud-deploy Publish cloud docker image for Cloud-First-Testing label Aug 20, 2025
@juliaElastic juliaElastic mentioned this pull request Aug 26, 2025
Merged
10 tasks
SiddharthMantri
SiddharthMantri requested changes Aug 27, 2025
View reviewed changes
Copy link
Contributor

@SiddharthMantri SiddharthMantri left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for reverting the original PR. After discussing internally with @elastic/kibana-security - we can't allow read privileges to Kibana system user on indices that contain Customer data. Please let me know if the alternative solution works.

...va/org/elasticsearch/xpack/core/security/authz/store/KibanaOwnedReservedRoleDescriptors.java
)
.build(),
// Read datasets for auto install content packages feature in Fleet
RoleDescriptor.IndicesPrivileges.builder().indices("logs-*", "metrics-*", "traces-*").privileges("read").build(),
Copy link
Contributor

@SiddharthMantri SiddharthMantri Aug 27, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @juliaElastic - on the related Kibana PR here: https://github.com/elastic/kibana/pull/230369/files#diff-53e6fb38de62c0f181daadc9c2b0301b019268fc469e00d56eeca3a1a523aca1R299-R302 Are you requesting read privileges on logs- and metrics- for reading the dataStream dataset name?

If so, you only require view_index_metadata privileges: https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-get-data-stream.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm we should be able to derive the dataset from the data stream name metrics-<dataset>-<namespace>.

Just to check, is there a way to only read the data_stream.dataset field in the data stream docs?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure tbh. Probably a question for ES team but from my initial understanding of the Datastreams API, you can't derive specific fields like dataset as that would require read privileges again.

Copy link
Contributor Author

@juliaElastic juliaElastic Aug 27, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checked locally and it should be good to use the data streams API, kibana_system already has the privileges:

elasticsearch/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/authz/store/KibanaOwnedReservedRoleDescriptors.java

Line 164 in 0d0832d

RoleDescriptor.IndicesPrivileges.builder().indices("*").privileges("view_index_metadata", "monitor").build(),

I'll change the implementation in kibana and this pr can be closed. Thanks!

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Amazing, thank you for your patience and the changes.

juliaElastic added a commit to elastic/kibana that referenced this pull request Aug 27, 2025
@juliaElastic
[Fleet] enable feature flag enableAutoInstallContentPackages (#232668)
9933241 
## Summary

Closes elastic/ingest-dev#5872

Blocked by elastic/elasticsearch#132501

## Release note

Enable feature to auto install content packages where data is ingested
for the matching datasets defined in the packages' discovery fields.


### Checklist

Check the PR satisfies following conditions. 

Reviewers should verify this PR satisfies this list as well.

- [ ] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [ ] This was checked for breaking HTTP API changes, and any breaking
changes have been approved by the breaking-change committee. The
`release_note:breaking` label should be applied in these situations.
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [ ] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
- [ ] Review the [backport
guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing)
and apply applicable `backport:*` labels.

### Identify risks

Does this PR introduce any risks? For example, consider risks like hard
to test bugs, performance regression, potential of data loss.

Describe the risk, its severity, and mitigation for each identified
risk. Invite stakeholders and evaluate how to proceed before merging.

- [ ] [See some risk
examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx)
- [ ] ...
kowalczyk-krzysztof pushed a commit to kowalczyk-krzysztof/kibana that referenced this pull request Aug 30, 2025
@juliaElastic @kowalczyk-krzysztof
[Fleet] enable feature flag enableAutoInstallContentPackages (elast…
89f6369 
…ic#232668)

## Summary

Closes elastic/ingest-dev#5872

Blocked by elastic/elasticsearch#132501

## Release note

Enable feature to auto install content packages where data is ingested
for the matching datasets defined in the packages' discovery fields.


### Checklist

Check the PR satisfies following conditions. 

Reviewers should verify this PR satisfies this list as well.

- [ ] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [ ] This was checked for breaking HTTP API changes, and any breaking
changes have been approved by the breaking-change committee. The
`release_note:breaking` label should be applied in these situations.
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [ ] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
- [ ] Review the [backport
guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing)
and apply applicable `backport:*` labels.

### Identify risks

Does this PR introduce any risks? For example, consider risks like hard
to test bugs, performance regression, potential of data loss.

Describe the risk, its severity, and mitigation for each identified
risk. Invite stakeholders and evaluate how to proceed before merging.

- [ ] [See some risk
examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx)
- [ ] ...
qn895 pushed a commit to qn895/kibana that referenced this pull request Sep 2, 2025
@juliaElastic @qn895
[Fleet] enable feature flag enableAutoInstallContentPackages (elast…
ce43d9e 
…ic#232668)

## Summary

Closes elastic/ingest-dev#5872

Blocked by elastic/elasticsearch#132501

## Release note

Enable feature to auto install content packages where data is ingested
for the matching datasets defined in the packages' discovery fields.


### Checklist

Check the PR satisfies following conditions. 

Reviewers should verify this PR satisfies this list as well.

- [ ] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md)
- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [ ] This was checked for breaking HTTP API changes, and any breaking
changes have been approved by the breaking-change committee. The
`release_note:breaking` label should be applied in these situations.
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [ ] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
- [ ] Review the [backport
guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing)
and apply applicable `backport:*` labels.

### Identify risks

Does this PR introduce any risks? For example, consider risks like hard
to test bugs, performance regression, potential of data loss.

Describe the risk, its severity, and mitigation for each identified
risk. Invite stakeholders and evaluate how to proceed before merging.

- [ ] [See some risk
examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx)
- [ ] ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@SiddharthMantri SiddharthMantri SiddharthMantri requested changes

Assignees

No one assigned

Labels

cloud-deploy Publish cloud docker image for Cloud-First-Testing :Core/Infra/Plugins Plugin API and infrastructure >non-issue Team:Core/Infra Meta label for core/infra team Team:Fleet v9.2.0

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@juliaElastic @elasticsearchmachine @SiddharthMantri