Template validation is slow

[flash1293]

Component template calculation can be really slow within Elasticsearch because it creates a temporary index to check whether everything is compatible.

This is what happens when a component template is changed that is used in >300 index templates, which can easily happen when using wired streams.

This is caused by a PUT _component_template/logs.child5@stream.layer call:

│ info [o.e.c.m.MetadataIndexTemplateService] [localhost] updating component template [[logs.child5@stream.layer](mailto:logs.child5@stream.layer)]
│ info [o.e.c.s.MasterService] [localhost] took [2.1m/128172ms] to compute cluster state update for [create-component-template [[logs.child5@stream.layer](mailto:logs.child5@stream.layer)], cause [api][org.elasticsearch.cluster.metadata.MetadataIndexTemplateService$3@760a9545]], which exceeds the warn threshold of [10s]

Streams creates one index template per stream, with a stack of component templates made up of its own one and all the parents - so logs.a.b.c.d has a component template stack of:

• logs@stream.layer
• logs.a@stream.layer
• logs.a.b@stream.layer
• logs.a.b.c@stream.layer
• logs.a.b.c.d@stream.layer

For a large hierarchy, there will be lots of index templates using logs.a - if changed, all of them need to be "remerged", which apparently takes a long time. Looks like this could be optimized.

This is not actually failing, but it takes a long time, so the request from the user to Kibana times out in the meantime and a timeout error is reported, even though Kibana server keeps working in the background until everything is done (it's also keeping the lock up for this time)

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[nielsbauman]

Thanks for reporting this, @flash1293!

I recently merged #135457 and #135644. The latter only went in with v9.3.0, but the former was released in v9.2.0 and should have the biggest impact in the scenario you described (the latter mainly concerns no-op updates, which isn't what you're talking about, it seems). Did you already upgrade to v9.2.x? If so, did you also experience the issue there? And finally, is there any chance you're experiencing this on an internal cluster which we could have access to, to do some precise profiling etc.?

[flash1293]

Thanks @nielsbauman , my investigation is from August, so it's definitely a bit outdated here. I can run the scenario again and report back.

the latter mainly concerns no-op updates, which isn't what you're talking about, it seems

That's right, the thing I tested with is adding a field to the mapping.

[flash1293]

This completely fixed the case I was running into 🎉 I tested around a bit with changes hitting 400 index templates at once and it worked just fine (from 2mins to 500ms). I'll close this for now, will open another issue if I run into an edge case that makes problems.