[Fleet] Support for document-based routing via ingest pipelines

[joshdover]

We are moving forward with a solution for "document-based routing" based a new ingest pipeline processor, called the reroute processor. Fleet will be responsible for managing routing rules defined by packages and the end user, and updating ingest pipelines to include the new data_stream_router processor to apply the rules.

Links

• [Feature] Document based ingest routing elasticsearch#63798
• Add reroute processor elasticsearch#76511
• Invoke default pipeline of new index elasticsearch#85931
• Add ability to skip the rest of the processors in a pipeline elasticsearch#85932
• Architecture discussion

Overview diagram

Integration-defined routing rules

Integrations will be able to define routing rules about how data from other integrations or data streams should be routed to their own data stream. For example, the nginx package may define a routing rule for the logs-kubernetes.container_logs data stream to route logs to logs-nginx.access whenever container.image.name == "nginx". Similarly, when the kubernetes package is installed and the nginx was also previously installed, we'll need to ensure the logs-kubernetes.router-{version} ingest pipeline includes a reroute processor for each routing rule defined on the nginx integration.

To support this, we'll need to add a concept of routing rules to the package spec and add support for them in Fleet.

# Integration-defined routing rules
- [ ] https://github.com/elastic/package-spec/issues/514
- [ ] https://github.com/elastic/kibana/issues/155910
- [ ] https://github.com/elastic/kibana/issues/157422
- [ ] https://github.com/elastic/ingest-docs/issues/276

Supporting tasks

We'll need to do a few things in support of these changes as well, namely around API key permissions.

# Supporting tasks
- [ ] https://github.com/elastic/kibana/issues/134971
- [ ] https://github.com/elastic/package-spec/issues/315

cc @ruflin @felixbarny

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[ruflin]

How will we manage the dependency web between packages during installation?

We are thinking it of a dependency because it is the same ingest pipeline for the first implementation. But ideally, there would be a routing API and integrations would just add their bits to it, no necessarily creating a dependency. What if we build this API in Fleet for now that manages the rules and creates the pipeline out of it? What does it mean for the package spec? I'm hoping we can stay away mostly from having dependencies.

[joshdover]

Yeah to be clear, I didn't mean that packages would declare dependencies on one another. But there will be things to consider to ensure that every time a package is installed, all the appropriate pipelines are updated.

[joshdover]

I synced on this today with Observability and discussed what the next steps should be. We came to the conclusion that we should work on a design document that includes the following

Package spec for routing rules

We need to allow packages to define routing rules for how data should be routed from other integration data streams to the integration defining the rule. For example, the Nginx integration should be able to define a rule for the Docker integration on how to detect a document that contains and nginx log and route it to the nginx.access data stream.

While the underlying implementation of the routing rule will be part of the integration's ingest pipeline, we want Fleet/EPM to have control over the order of when routing decisions happen. For these 2 reasons, routing rules should not be defined as part of an integration data stream's regular ingest pipeline. Instead they need to be defined in a separate file in the package.

We should also not abstract away too much of the underlying reroute processor's options. Integrations should be able to use the full power of that processor's capabilities, including using painless to define the rule's condition.

Ingest pipeline design

We need to specify the order in which the following things will be executed in the pipeline:

• The integration-defined processing logic
• The integration-defined routing rules
• User-defined processing logic
• User-defined routing rules
• Default rerouting that should happen for all data streams (maybe)
  • This is essentially re-routing a document to the right data stream if the data_stream.dataset in the document doesn't match the current data stream.
  • This is to handle bugs in shippers and may not be strictly necessary.

We need to strike a balance between allow flexibility and preventing user customization from conflicting with processing and routing defined by integrations.

This design needs to be tested against real use cases (to be provided by Observability).

Part of this should also include the naming convention used for the different pipelines and sub-pipelines (eg @routing).

Fleet API for adding user-defined routing rules

Users need to be able to add their own routing rules. We may want to offer an API in Fleet/EPM to make manging this simpler. Later a UI may be built on top of this.

The design of this API should include how ingest pipelines are updated to apply the new rules or remove ones that are deleted

Internal tracking of rules and ingest pipeline updating procedure

We need to define how Fleet will internally track rules defined in packages and by end users, and how those rules will be installed into ingest pipelines with 0 disruption to ingestion.

This needs to include how package installation will work when new rules are added, how package upgrades will work, and how package uninstallation will work.

[hop-dev]

@joshdover I'd be interested in what we see the user experience being here.

As a user with an nginx docker container in kubernetes, I might be tempted to first look for the nginx integration for example, but the standard flow would then prompt me for the log paths etc which is unnecessary.

or if I go to install the kubernetes integration first, how will I know to then install the nginx integration to capture my logs?

It's almost like we need a dedicated kubernetes onboarding which guides the user to select the kinds of containers they will be deploying and installs the matching integrations first then the kubernetes integration, that way the user starts re-routing data straight away.

As a user I would also want to see if my data is successfully being re-routed, we could consider adding a field when we re-route the data to track the origin of the data and we could then aggregate on it to give the user a summary somewhere of data that is being re-routed.

[felixbarny]

Good points @hop-dev. I'd hope we can come up with a solution that doesn't require the user to manually install integrations. Instead, we'd either pre-install or auto-install the integration when data flows to a corresponding data stream. IIRC, Josh did a spacetime on this.

[hop-dev]

Ah yes I was involved in a spacetime project in this area: https://github.com/elastic/observability-dev/issues/2100

Slightly different to this use case, we used annotations on the containers themselves to prompt the installation of the integration. then configured the integration to consume the data directly

[joshdover]

Actually @hop-dev did the auto-install spacetime :) I did a related one to suggest packages to install based on signals in the common data. Both could be potential options here.

I suspect this is important as well, and ideally we don't build a solution that only works for Kubernetes sources.

I wonder if we could do some 80/20 solution where we pre-install just the ES assets for integrations that define routing rules for another integration. We can then watch for data showing up in those data streams and install the dashboards/other Kibana objects at that time.

[felixbarny]

pre-install just the ES assets for integrations that define routing rules for another integration. We can then watch for data showing up in those data streams and install the dashboards/other Kibana objects at that time.

This makes a lot of sense to me

[weltenwort]

I wonder if we could do some 80/20 solution where we pre-install just the ES assets for integrations that define routing rules for another integration.

Wouldn't it be rather to pre-install the ES assets for integrations that documents are being re-routed to? How else would Kibana know which integration to install?

For @hop-dev's user journey question that would imply that the routing configuration is tied to the lifecycle of the source integration (such as k8s).

[joshdover]

Wouldn't it be rather to pre-install the ES assets for integrations that documents are being re-routed to? How else would Kibana know which integration to install?

I think we're saying the same thing? So if a user installs the k8s integration, Fleet would install the ES assets for all (or most popular) integrations that have routing rules specified for k8s container logs. I imagine that the registry APIs would need to expose a list of other integrations that have routing rules for a particular integration.

For @hop-dev's user journey question that would imply that the routing configuration is tied to the lifecycle of the source integration (such as k8s).

Good point, this would be one of the caveats. We could probably come up with ways to refresh this periodically, but it's a little less than ideal.

I do suspect that users want more control over this - and we need a way to show them more integrations that may have routing rules that are relevant for their use case. This is where I think the design around auto-suggestion of integrations to install may be helpful, where routing only happens once the user has decided to install an integration to enhance the data that is extracted.

Relevant spacetime: https://github.com/elastic/observability-dev/issues/2132

[joshdover]

Synced with @kpollich about this today.

User flow

We discussed the general concept and UX flow from Discover:

• We expect users to be able to use filters in Discover to define a set of data and then choose to create a new dataset/datastream for the documents that match the filter.
• Discover will need a Fleet API to call when the user chooses to create this new dataset. At a high level, this API should allow Discover to:
  • Create a new index template & ingest pipeline for the new dataset (if the target datatset does not exist)
  • Persist the routing rule in a Kibana Saved Object
  • Update the source dataset's ingest pipeline with the reroute processor to route logs to the new dataset
• On package upgrades of the base package, the "forked" datastream(s) for the new dataset should have it's base index template upgraded.
• You should be able to route to datasets that already exist

Example

• You start in Discover, choose logs-*, choose kubernetes.container_logs data set
• Filter for logs on container.image.name: foo
• You need to parse a specific field from the log message that is only in logs from the foo image
• Write a new grok processor to extract the field from message
• Save the new processor in a new dataset as logs-foo-default
• Discover calls Fleet API to:
  • Create a new logs-foo-* index template and ingest pipeline
  • Add a reroute processor to the logs-kubernetes.container_logs ingest pipeline to point to route logs to logs-foo

Questions & some potential answers

• How do we describe the routing rule in the Fleet API? Painless vs. high-level option like KQL
  • Supporting raw painless would allow us to put the Discover filter to painless conversion be the responsibility of the client of the Fleet API.
  • Using a high-level option like KQL would bring that responsibility to the Fleet side, and potentially limit the power of the routing rule API.
• What is the base index template?
  • Based on the source dataset + @Custom
• Which saved object are routing rules saved to?
  • We agreed that rules should be stored on the target dataset/package/package policy, not the source or "sink"
  • We want to use a common model for package-defined and user-defined routing rules. Making user-defined rules defined on "custom" packages would be one way to achieve this.
  • We may also consider short-term options like saving customizations in a particular component template as the source of truth for now. We expect "custom packages" to touch many pieces of the code base.
• Should the ingest pipeline be reusued from the base datastream?
  • Yes, logs should always be parsed the same regardless of if they're routed through the "sink" data stream or directly to the specialized data stream.
  • No, not all transformations are idempotent and we also don't want to pay the processing penalty twice.
• What should the APIs be for adding new routing rules and customizating datasets? We probably need at least:
  • CRUD for routing rules
  • CRUD for datasets
• The requirements for "structure all logs" will also require similar APIs for (out of scope of this issue):
  • CRUD for dataset mappings
  • CRUD for dataset pipeline processors

[joshdover]

@grabowskit @ruflin can you confirm our understanding of the UX and the APIs we're discussing?