[Fleet] added privileges for `fleet-synced-integrations` #121753

juliaElastic · 2025-02-05T11:36:57Z

Related to elastic/kibana#206237
Added kibana_system privileges on fleet-synced-integrations
Reason: we are building a feature where this index has to be replicated with CCR to remote clusters, so it can't be a system index.

Have you signed the contributor license agreement? yes
Have you followed the contributor guidelines? yes
If submitting code, have you built your formula locally prior to submission with gradle check? yes
If submitting code, is your pull request against main? Unless there is a good reason otherwise, we prefer pull requests against main and will backport as needed. yes
If submitting code, have you checked that your submission is for an OS and architecture that we support? yes
If you are submitting this code for a class then read our policy for that.

elasticsearchmachine · 2025-02-05T11:38:47Z

Pinging @elastic/es-core-infra (Team:Core/Infra)

kc13greiner · 2025-02-05T14:14:40Z

...va/org/elasticsearch/xpack/core/security/authz/store/KibanaOwnedReservedRoleDescriptors.java

                RoleDescriptor.IndicesPrivileges.builder().indices("logs-fleet_server*").privileges("read", "delete_index").build(),
+                // Fleet creates and writes this index for sync integrations feature
+                RoleDescriptor.IndicesPrivileges.builder()
+                    .indices("fleet-synced-integrations")


Heya @juliaElastic! Is there a reason to move away from the usual .fleet-* system indices? Making it a "data index" could have collisions with customer indices.

It is because we are building a feature where this index has to be replicated with CCR to remote clusters, so it can't be a system index.

Sorry for the delay!

To proceed with this change would you please update the PR description with a Reason section stating your requirement above.

And would you also be able to update this document to include this new index in the Avoid index pattern collisions section: https://www.elastic.co/guide/en/elasticsearch/reference/current/index-templates.html

Updated the PR description.

Do you know where to update the documentation, or is there an elasticsearch docs team to reach out to?

Thanks for updating the description!

The docs are located in this (elasticsearch) repo: docs/reference/indices/index-templates.asciidoc

Thanks, updated here: https://github.com/elastic/elasticsearch/pull/121753/files#diff-a3c27c5107ae1f4a617a6f0006307d2b2d4092f844a1e256e7019ccfc5bc21f8

It seems there is a documentation freeze: #121753 (comment)
This feature is for 9.1 the earliest, what should we do?

Let's remove the docs from this PR and get these privileges merged to unblock the work.

Then create an issue to add the docs after the freeze is lifted in a few weeks.

sorry for the confusion 😵‍💫

thanks, reverted
added as a task here elastic/kibana#206237

juliaElastic · 2025-02-07T09:13:54Z

the test failures don't seem to be related to the change, are they flaky?

java.lang.AssertionError: All incoming requests on node [node_s4] should have finished. Expected 0 bytes for requests in-flight but got 298 bytes; pending tasks [[]] |  
-- | --
Caused by: org.elasticsearch.index.mapper.DocumentParsingException: [1:13] failed to parse: data stream timestamp field [@timestamp] is missing |  
-- | --
  | at org.elasticsearch.index.mapper.DocumentParser.wrapInDocumentParsingException(DocumentParser.java:264) |  
  | at org.elasticsearch.index.mapper.DocumentParser.internalParseDocument(DocumentParser.java:161) |  
  | at org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:98) |  
  | at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:128) |  
  | at org.elasticsearch.index.shard.IndexShard.prepareIndex(IndexShard.java:1048) |  
  | at org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:989) |  
  | at org.elasticsearch.index.shard.IndexShard.applyTranslogOperation(IndexShard.java:2031) |  
  | at org.elasticsearch.index.shard.IndexShard.applyTranslogOperation(IndexShard.java:2018) |  
  | at org.elasticsearch.indices.recovery.RecoveryTarget.lambda$indexTranslogOperations$4(RecoveryTarget.java:454) |  
  | ... 15 more |  
  | Caused by: java.lang.IllegalArgumentException: data stream timestamp field [@timestamp] is missing |  
  | at org.elasticsearch.index.mapper.DataStreamTimestampFieldMapper.extractTimestampValue(DataStreamTimestampFieldMapper.java:210) |  
  | at org.elasticsearch.index.mapper.DataStreamTimestampFieldMapper.postParse(DataStreamTimestampFieldMapper.java:223) |  
  | at org.elasticsearch.index.mapper.DocumentParser.internalParseDocument(DocumentParser.java:158) |  
  | ... 22 more

kc13greiner · 2025-02-07T13:22:58Z

the test failures don't seem to be related to the change, are they flaky?

java.lang.AssertionError: All incoming requests on node [node_s4] should have finished. Expected 0 bytes for requests in-flight but got 298 bytes; pending tasks [[]] |  
-- | --
Caused by: org.elasticsearch.index.mapper.DocumentParsingException: [1:13] failed to parse: data stream timestamp field [@timestamp] is missing |  
-- | --
  | at org.elasticsearch.index.mapper.DocumentParser.wrapInDocumentParsingException(DocumentParser.java:264) |  
  | at org.elasticsearch.index.mapper.DocumentParser.internalParseDocument(DocumentParser.java:161) |  
  | at org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:98) |  
  | at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:128) |  
  | at org.elasticsearch.index.shard.IndexShard.prepareIndex(IndexShard.java:1048) |  
  | at org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:989) |  
  | at org.elasticsearch.index.shard.IndexShard.applyTranslogOperation(IndexShard.java:2031) |  
  | at org.elasticsearch.index.shard.IndexShard.applyTranslogOperation(IndexShard.java:2018) |  
  | at org.elasticsearch.indices.recovery.RecoveryTarget.lambda$indexTranslogOperations$4(RecoveryTarget.java:454) |  
  | ... 15 more |  
  | Caused by: java.lang.IllegalArgumentException: data stream timestamp field [@timestamp] is missing |  
  | at org.elasticsearch.index.mapper.DataStreamTimestampFieldMapper.extractTimestampValue(DataStreamTimestampFieldMapper.java:210) |  
  | at org.elasticsearch.index.mapper.DataStreamTimestampFieldMapper.postParse(DataStreamTimestampFieldMapper.java:223) |  
  | at org.elasticsearch.index.mapper.DocumentParser.internalParseDocument(DocumentParser.java:158) |  
  | ... 22 more

@elastic/es-security can you help us out?

github-actions · 2025-02-07T14:03:17Z

It looks like this PR modifies one or more .asciidoc files. These files are being migrated to Markdown, and any changes merged now will be lost. See the migration guide for details.

This reverts commit b73bae8.

kc13greiner

LGTM!

Thanks for the info

## Summary Closes #206237 Create `fleet-synced-integrations` index in Fleet setup, added async task that populates the index with a doc that includes remote ES output data and installed integrations data. ES change to add `kibana_system` privileges: elastic/elasticsearch#121753 To test locally: - run elasticsearch from source to apply the privilege changes, so that `kibana_system` can create the index. ``` yarn es source -E xpack.security.authc.api_key.enabled=true -E xpack.security.authc.token.enabled=true --source-path=/Users/juliabardi/elasticsearch -E path.data=/tmp/es-data -E xpack.ml.enabled=false ``` - enable the feature flag in `kibana.dev.yml`: `xpack.fleet.enableExperimental: ['enableSyncIntegrationsOnRemote']` - add a remote ES output with sync enabled - install some integrations - wait until Fleet setup and the task runs - verify that the index is created and contains a doc with the expected data ``` GET fleet-synced-integrations/_search "hits": { "total": { "value": 1, "relation": "eq" }, "max_score": 1, "hits": [ { "_index": "fleet-synced-integrations", "_id": "fleet-synced-integrations", "_score": 1, "_source": { "remote_es_hosts": [ { "hosts": [ "http://remote1:80" ], "name": "remote1", "sync_integrations": true } ], "integrations": [ { "package_version": "1.64.1", "updated_at": "2025-02-05T11:03:02.226Z", "package_name": "system" } ] } } ] ``` ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios --------- Co-authored-by: kibanamachine <[email protected]>

added privileges for fleet-synced-integrations

76a013e

juliaElastic self-assigned this Feb 5, 2025

juliaElastic requested a review from a team as a code owner February 5, 2025 11:36

elasticsearchmachine added v9.1.0 needs:triage Requires assignment of a team area label external-contributor Pull request authored by a developer outside the Elasticsearch team labels Feb 5, 2025

juliaElastic added >non-issue :Core/Infra/Plugins Plugin API and infrastructure Team:Fleet Team:Core/Infra Meta label for core/infra team labels Feb 5, 2025

elasticsearchmachine removed the needs:triage Requires assignment of a team area label label Feb 5, 2025

Merge branch 'main' into fleet-synced-integrations

0bf64c5

kc13greiner self-requested a review February 5, 2025 14:11

kc13greiner reviewed Feb 5, 2025

View reviewed changes

juliaElastic mentioned this pull request Feb 6, 2025

[Fleet] add index and task for fleet-synced-integrations elastic/kibana#209762

Merged

1 task

Merge branch 'main' into fleet-synced-integrations

a2484cc

Update index-templates.asciidoc

b73bae8

Revert "Update index-templates.asciidoc"

67f8d40

This reverts commit b73bae8.

kc13greiner approved these changes Feb 7, 2025

View reviewed changes

juliaElastic merged commit a11085c into elastic:main Feb 10, 2025
17 checks passed

This was referenced Feb 20, 2025

[Fleet] Update index-templates.asciidoc #123000

Closed

[Fleet] added privileges for fleet-synced-integrations* #123493

Merged

[Fleet] added privileges for fleet-synced-integrations #121753

[Fleet] added privileges for fleet-synced-integrations #121753

Uh oh!

Conversation

juliaElastic commented Feb 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticsearchmachine commented Feb 5, 2025

Uh oh!

kc13greiner Feb 5, 2025

Choose a reason for hiding this comment

Uh oh!

juliaElastic Feb 5, 2025

Choose a reason for hiding this comment

Uh oh!

kc13greiner Feb 6, 2025

Choose a reason for hiding this comment

Uh oh!

juliaElastic Feb 7, 2025

Choose a reason for hiding this comment

Uh oh!

kc13greiner Feb 7, 2025

Choose a reason for hiding this comment

Uh oh!

juliaElastic Feb 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kc13greiner Feb 7, 2025

Choose a reason for hiding this comment

Uh oh!

kc13greiner Feb 7, 2025

Choose a reason for hiding this comment

Uh oh!

juliaElastic Feb 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

juliaElastic commented Feb 7, 2025

Uh oh!

kc13greiner commented Feb 7, 2025

Uh oh!

github-actions bot commented Feb 7, 2025

Uh oh!

kc13greiner left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

[Fleet] added privileges for `fleet-synced-integrations` #121753

[Fleet] added privileges for `fleet-synced-integrations` #121753

juliaElastic commented Feb 5, 2025 •

edited

Loading

juliaElastic Feb 7, 2025 •

edited

Loading

juliaElastic Feb 7, 2025 •

edited

Loading