[Change Proposal] Support migration to equivalent inputs

[jsoriano]

There are cases when one input is directly replaced by other, their settings are mostly the same and they collect the same data, but they have different implementations. Some examples of that would be the migration from log to filestream, or from httpjson to cel. For example see https://github.com/elastic/integrations/pull/14787/files.

The problem with these migrations is that even when they are equivalent, they require user interaction. When upgrading the policy it needs to be configured again, and the old data is not available for users. This happens because Fleet uses the input type to decide the identifiers in the policies.

Support automatic migration of equivalent inputs.

A maybe naive implementation in Fleet could assume that when an input disappears in a policy and a new one appears, then the new one is replacing the other and the variables could be reused. A less naive implementation would require explicit declarations than an input is replacing an existing one. For example with something like this:

streams:
  - input: cel
    migrateFrom: httpjson

If Fleet is upgrading a policy with these settings, finds that the cel input is not configured, but a httpjson configuration exists, it copies the variables from one to the other.

[narph]

@kpollich , @nimarezainia we continue to face this issue with the:
elastic/integrations#11338
elastic/integrations#14759
https://github.com/elastic/enhancements/issues/25659

and a few others, where the migration from httpjson to cel input would solve performance and limitation issues but that has a serious user experience impact when upgrading (no state persistence , data stream would be disabled and users will have to re-enter credentials)

[nimarezainia]

@narph thanks for bringing this to our attention. I'll speak to @kpollich and see how we could proceed.

[cmacknz]

We have been looking a little bit at the approach we are going to take for log -> filestream is keep the log name but have the implementation redirect to Filestream inside of Filebeat.

Specifically, we are going to have a type of proxy input that knows how to translate log input configuration into filestream input configuration and start filestream. In the end the only part of the log input we'll be left with is the configuration options. See here (private issue).

This avoids all of the messiness around ID preservation. If we were to tackle that, we are moving towards changing the data model of the state on disk vs doing anything in the integration configuration, Elastic Agent, or Fleet to deal with this. elastic/beats#46939. This is not really explored in depth yet.

[MichelLosier]

To add a bit more related context to this story, we had a similar issue at the variable level:

• Support var removals on integration upgrade kibana#238542

Where input level variables were moved to a global scope of the package. The fix in that PR removes the stale variable definitions in the policy so they don't invisibly override the new variable definitions, but it also means on upgrade the user will need to re-fill in the values for those variables since they are treated as new variable definitions.

In that fix I wanted to avoid fleet making the assumption of variable migrations if one is removed, but one exists of same name and type in different scope. I callout there we should consider package spec support for defining variable scope migrations.

[kpollich]

We spoke about this as a team and I think the technical approach that @jsoriano has proposed is generally sound. We need to approach this as two separate streams of work per @cmacknz comment above:

1. The Fleet UX for automatically "spreading" the previous input's fields onto the new one
2. The Agent behavior for preserving state when the input type is changed

@narph, from your perspective is improving the Fleet UX worthwhile without the state persistence work on the agent side? We can implement the migrateFrom package spec field that @jsoriano has proposed above and have Fleet intelligently enable the new input, disable the old one, and spread the fields from the previous input onto the newly-enabled one. This is fairly approachable, but I worry that smoothing the UX here without the agent preserving input state before/after the integration upgrade will result in a situation where we make it very easy to experience data loss.

[narph]

@kpollich , we have integrations that do not require state persistence so I believe it's a step in the right direction. That said, I share your concerns, we continue to encounter issues due to httpjson input limitation and although migrating the data stream to the cel input is the best solution the state persistence is still a blocker for some of the integrations.

I've recently noticed in https://github.com/elastic/ingest-dev/issues/4932#issuecomment-3383258903 that a log/filestream specific solution was proposed where we were hoping for a more generalized solution that could apply to other inputs.
We have been discussing with @cmacknz in https://github.com/elastic/security-team/issues/7819 and @efd6 created a proposal for for state persistence but we were told that it will be picked up by the elastic agent team, @cmacknz is this something that the team is focusing on or plans to in the near future?

[kpollich]

I'm pulling this into our next iteration for dedicated tech definition. I think we should treat this issue as the hub for the ecosystem + Kibana changes and continue using https://github.com/elastic/ingest-dev/issues/4932 to track the agent side state persistence work.

Since integrations will be required to opt into the migration behavior, I'm hoping that integration authors can make the call on whether their integration has risks associated with changing input types on the fly. So, if it's safe, the authors can add the migrateFrom configuration and Fleet will provide the more pleasant migration experience. If it's not deemed safe, the authors should hold off on adding that config. The Fleet team won't be able to verify integration-specific behavior here, and will focus on building the UX as specified here.

[nimarezainia]

I've recently noticed in elastic/ingest-dev#4932 (comment) that a log/filestream specific solution was proposed where we were hoping for a more generalized solution that could apply to other inputs.

@narph we are focusing on the log to filestream transition issues at the moment (as described in this RFC, but you are right that this needs to be generalized solution. @cmacknz I don't think we have any efforts on our side to do this at the moment - correct?

[efd6]

I was looking at the RFC this morning and it struck me that if the input interface were given a way to hand over to another input (very much akin to how http.RoundTripper chaining can be done, or io.Writer, or (golang.org/x/text).Transformer, … ) then solving the log to filestream fix would be the general fix. At the moment, the RFC is very much a single location fix.

[cmacknz]

Yes agree, the fix for log to filestream here needs to be generalized.

[cmacknz]

Let's discuss the details of how to do that in https://github.com/elastic/ingest-dev/issues/6318