Avoid changing tsid creation strategy for an index

[felixbarny]

This avoids that the tsid and routing is calculated differently on the same index. Doing so would have the following consequences:

• De-duplication would not work, for example, when a client re-transmits a batch as the id (which is based on the tsid) will be different.
• When replaying a translog operation, the re-computed tsid and id will differ from the id stored in the translog. This would lead to a failure of the index operation.

This could be seen as a breaking change as operations on backing indices that were allowed before are not allowed after this change. However, this only applies to data streams as index.dimensions will never be set on plain time_series indices. For data streams, you first change the template, then optionally the backing indices, and roll over if changing the backing indices doesn't work.

[kkrik-es]

I kinda like this. Let's try to be as restrictive as possible.

[felixbarny]

I like it better, too but didn't make the change earlier due to potential impact on backwards compatibility. Are you worried about that at all?

[kkrik-es]

Not really.. If anything, I'd think it's a best practice to avoid such changes (label => dimension) outside rollovers. More so, we should only have dimensions and metrics at this point, no labels. Maybe worth documenting better.

[felixbarny]

I'd think it's a best practice to avoid such changes (label => dimension) outside rollovers.

Today, it's already prohibited to change a non-dimension field to a dimension field. However, there are two problematic scenarios where replaying the translog or re-submitting a batch can lead to duplicates or rejections:

1. Adding a new dimension field to a non-dynamic mapping where existing fields already in the translog would be interpreted as dimensions instead of being ignored.
2. Adding a dynamic template defining a dimension to an existing TSDS backing index changes the index.dimensions-based tsid strategy to the index.routing_path-based strategy - addressed in this PR.

(See #135402 (comment) for more details about these scenarios)

It seems that the only way'd feel comfortable with adding the index.dimensions-based strategy is by making the change proposed in this PR: Disallow adding a dynamic template defining a dimension directly to a backing index and require a rollover, which transitions between the strategies in an orderly way.

While we may still need to discuss backwards compatibility implications before removing the feature flag for the new index.dimensions-based tsid creation strategy, I'd suggest to go ahead with this PR.

[elasticsearchmachine]

Pinging @elastic/es-storage-engine (Team:StorageEngine)

[kkrik-es]

I agree, let's proceed with this for v.9.2 and discuss bwc separately. To play it safe, it may be worth adding an index setting to opt out of the dimension routing logic, so that affected users have a way to restore old behavior. @martijnvg wdyt?

[felixbarny]

To play it safe, it may be worth adding an index setting to opt out of the dimension routing logic, so that affected users have a way to restore old behavior.

That would be possible already by explicitly setting the index.routing_path configuration option on a data stream. Were you think an additional boolean index setting? Or maybe a cluster-level setting?

[kkrik-es]

Yeah let's add a separate boolean index setting; index.routing_path gets auto-populated today, users may want to maintain that behavior but opt-out of the new mode. Plus, it may help if we hit a bug in production, to quickly disable it.

[felixbarny]

Plus, it may help if we hit a bug in production, to quickly disable it.

That's the reason I was thinking about making this a cluster-level setting rather than per data stream.

[kkrik-es]

We mostly use index settings for this purpose, to have fine-grained control. It's not expected to be used, and it's not that hard to update all TSDS in a cluster if need be.

[felixbarny]

I'm good either way but wanted to note one thing

it's not that hard to update all TSDS in a cluster if need be

This would change it for all existing data streams. But new data streams would then still use the index.dimensions-based strategy, unless they explicitly opt-out. This is where a cluster-level setting may come in handy.