[Change Proposal] Ability to add privileges for all data streams of a specific type

[felixbarny]

Currently, the data stream permissions are specific to the dataset that an integration defines.
For example, when adding a custom log integration, you'll have to specify the data_stream.dataset, for example foo, and an API key will be generated with the permissions to send data to logs-foo-default.

This is limiting for integrations that define a data stream that routes events to other data streams. See:

• Ecs logs integration integrations#2972

Example: a log line like this gets sent to the logs-ecs_router-default data stream:

{
  "message": "{\"@timestamp\":\"2022-04-01T12:09:12.375Z\", \"log.level\": \"INFO\", \"message\":\"With event.dataset\", \"data_stream.dataset\": \"foo\"}"
}

The default ingest pipeline for logs-ecs_router-default parses the JSON within the message and uses the data_stream.dataset from the log message to route the message to the logs-foo-default data stream (by overriding the _index field).

The issue is that this will lead to a security exception as the API key used to ingest the data only has permissions to ingest data to logs-ecs_router-default.

This is also an issue for the Azure springcloudlogs integration: All logs are always sent to the springcloudlogs data stream, even if the logs are from different application and, thus, should ideally be routed to their own data streams. Other examples are CloudWatch, k8s logs, PCF logs, and httpjson.

This relates to the discussions about input-only packages but is an independent and decoupled task.

What I'm proposing is to add a flag to the package spec that behaves similar to dataset_is_prefix

package-spec/versions/1/data_stream/manifest.spec.yml

Lines 129 to 132 in d017330

	dataset_is_prefix:
	description: if true, the index pattern in the ES template will contain the dataset as a prefix only
	type: boolean
	default: false

But instead of just adding .* to the index permissions, the flag will allow access to all data streams of a given type, such as logs-*-*.

@ruflin @mtojek @joshdover

[ruflin]

An alternative approach is using default_permissions: true. There is a default set of permissions that gives access to logs-*-*, metrics-*-*, traces-*-*. It gives broader permissions then needed but is a catch all solution for all input packages that might also contain metrics or traces. One of the basic assumptions I make here is that if someone deploys this, the Elastic Agent is deployed in a trusted environment.

[felixbarny]

An alternative approach is using default_permissions: true. There is a default set of permissions that gives access to logs-*-*, metrics-*-*, traces-*-*.

Is this an existing flag? Where can I learn more about it? Ideally, a logging-related input integration would only have access to logs-*-*.

[ruflin]

Is this an existing flag?

No, I just made it up :-) Alternative it could default_permissions: ["logs", "metrics"] to make it an array which by default is empty.

[joshdover]

Thanks for highlighting this use case. We have a few discussions this week around ingest node routing (eg. elastic/elasticsearch#63798) and I think I can provide a more helpful response here after we've had those.

[felixbarny]

@joshdover Did you have a chance to think about this again? This issue is a blocker for full event routing and also the lightweight event routing required for an ECS logging integration (elastic/integrations#2972).

[ruflin]

There is currently work ongoing here around input packages: #325 I expect the packages that need this feature are normally input packages as for integrations packages, the data streams are mostly predefined. Now I'm torn if there should be a config param in the package spec itself for this or if Fleet should so magic around it. I'm leaning towards package a package spec flag as it would allow Fleet to know in advance that a certain package requires much more permissions and also warn the user about it.

[joshdover]

The default ingest pipeline for logs-ecs_router-default parses the JSON within the message and uses the data_stream.dataset from the log message to route the message to the logs-foo-default data stream (by overriding the _index field).

The issue is that this will lead to a security exception as the API key used to ingest the data only has permissions to ingest data to logs-ecs_router-default.

This is also an issue for the Azure springcloudlogs integration: All logs are always sent to the springcloudlogs data stream, even if the logs are from different application and, thus, should ideally be routed to their own data streams. Other examples are CloudWatch, k8s logs, PCF logs, and httpjson.

I'm wondering if instead, this should actually be solved with an index / data stream setting rather than with granting Agents explicit access to every data stream that an ingest pipeline may route to. This would allow us to further decouple the data collection from the ingest processing.

Otherwise, every time a routing ingest pipeline adds or removes a new destination data stream, we'll need to update the API keys for every agent sending data to the routing data stream. Generating API keys has non-trivial performance overhead and when we're talking about 100k+ agents, rolling out an API key change could take quite a bit of time and put a large load on the cluster. Due to this delay, we'd need to also change the order of operations to update all the Agent API keys first, then update the routing ingest pipeline. Otherwise, data could be dropped.

I'm imagining a setting like index.pipeline_routing_allowlist: ["logs-*"] that could be specified on the data stream's index template which would modify the security behavior during the ingest pipeline processing to avoid the security exception.

[felixbarny]

Otherwise, every time a routing ingest pipeline adds or removes a new destination data stream, we'll need to update the API keys

I don't think updates are required when the API keys have permissions for logs-* as that's all it would ever need. Note that it wouldn't be possible to make the list of index names more restrictive so that it only contains patterns that are used by installed integrations, such as logs-ngingx* if the Nginx integration is installed. That's because integrations, such as the ECS logging integration and possibly the Azure springcloudlogs integration, can route to any data dataset, depending on the values in the log message.

I'm imagining a setting like index.pipeline_routing_allowlist: ["logs-*"] that could be specified on the data stream's index template which would modify the security behavior during the ingest pipeline processing to avoid the security exception.

I think that could work, too. However, it would conflate index-level settings with API Key permissions which makes it harder to reason about what you can do with a given API key. Not sure if the ES team would be happy with that.

Also, I don't see why that would be preferable over an integration being able to request permissions for logs-*. Having access to logs-router-default which has a setting like index.pipeline_routing_allowlist: ["logs-*"] would be synonymous to having a permission for logs-*.

[joshdover]

I don't think updates are required when the API keys have permissions for logs-* as that's all it would ever need.

Got it, if we don't expect to ever have finer-grained privileges for these types of packages, I guess going with a purely package-based approach is fine then. Probably an option default_permissions would work fine, though I'd maybe want to make the name of the option a bit more explicit. I also don't think it should live next to dataset_as_prefix since that changes more than just the permissions and this is a permissions-only feature.

What if we put this next to data_stream.elasticsearch.privileges.indices? I'm thinking a name like data_stream.elasticsearch.privileges.allow_routing which would grant create privileges on logs-*-*, metrics-*-*, traces-*-*, synthetics-*-*

Felix are you comfortable opening a PR for the package-spec change? Once we have agreement on the spec change we can create the implementation tasks.

[felixbarny]

Here's the PR: #327