Semantic text mapper restriction specific to system indices

[Supplementing]

Elasticsearch Version

9.2.0

Installed Plugins

No response

Java Version

bundled

OS Version

Darwin Kernel Version 24.6.0

Problem Description

While working on #132506, we encountered an issue where leaving off inference_id and model_settings of a field with type semantic_text would not properly apply the default model as expected on a system index. This causes issues when trying to index items to the new system index as the ESClient sends over a request with the default model, which doesnt match the system descriptor, and causes indexing to fail.

Please see the thread starting here for additional context.

cc: @spong @Mikep86

Steps to Reproduce

1. Inside of the x-pack/plugin/core/template-resources/src/main/resources/fleet-integration-knowledge.json file, remove the hardcoded inference_id and model_settings.
2. Spin up a local instance of kibana and run es from source with this change
3. Try to ingest data into the new index (see [Fleet] Knowledge base integration support kibana#230107 for more info here)
4. Observe that ES logs an error about the index descriptor not matching. It seems the default inference model isnt being added to the field as intended by ES.

Logs (if relevant)

 Cannot update mappings in [.integration_knowledge-7]: system indices can only use mappings from their descriptors, but the mappings in the request [{"_doc":{"dynamic":"false","_meta":{"managed_index_mappings_version":1,"description":"Integration package knowledge base content storage","version":"9.2.0"},"properties":{"content":{"type":"semantic_text","inference_id":".elser-2-elasticsearch","model_settings":{"service":"elasticsearch","task_type":"sparse_embedding"}}}}}] did not match those in the descriptor(s)

[elasticsearchmachine]

Pinging @elastic/search-eng (Team:SearchOrg)

[elasticsearchmachine]

Pinging @elastic/search-relevance (Team:Search - Relevance)

[jimczi]

The first question to ask is: why use a semantic text field in a system index?
The second is: why would a knowledge index be treated as a system index?
This bug is easy to fix, but maybe we should go the other way and disable semantic text for system indices across the board.

[Supplementing]

The first question to ask is: why use a semantic text field in a system index?

The knowledge base docs content field is to be consumed by the O11y and Security AI assistants, so it seems that using semantic_text would be preferred there over normal text. My understanding is we would want the assistants to search based on meaning, rather than just keywords.

The second is: why would a knowledge index be treated as a system index?

Maybe @spong can chime in here with more context if needed, but based on the design doc, it was agreed upon that a system index would make the most sense as its an index to be written to/read by integrations and read by the AI assistants. At this time, there is no direct need to expose it to the end-user directly.

[jimczi]

The tricky part with the system index right now is that it’s meant to be managed internally by Elasticsearch (at least that's how it is setup). So if you change the inference ID or the mappings, you’ll run into the same issue since the index isn’t built for outside management.

Also worth noting: system indices run on a special thread pool that only has a single thread. With that limitation, I’m not sure the KB index in its current form is really set up for success.

[Supplementing]

The tricky part with the system index right now is that it’s meant to be managed internally by Elasticsearch (at least that's how it is setup). So if you change the inference ID or the mappings, you’ll run into the same issue since the index isn’t built for outside management.

We dont intend to update either one at this time, things are fairly set-in-stone from a structure standpoint. If we could just remove the inference_id and let ES default as it should, then we wouldnt need to use it at all in this case.

Also worth noting: system indices run on a special thread pool that only has a single thread. With that limitation, I’m not sure the KB index in its current form is really set up for success.

Hmmm, good to know! Do we have some sort of threshold were we expect it to fall over that I could compare against expected load here? Luckily, this will be a very gradual rollout so we dont expect to just index everything immediately. Each package will need to be updated by the maintainer to have docs relevant to the package, which is going to take some time. Even then, I assume some will have very little or no content while others may have multiple docs. I dont have an exact expectation, but Id like to maybe run some tests on my side with some dummy docs, potentially hundreds of them of varying size, and see how things perform (or dont perform).

[jimczi]

but Id like to maybe run some tests on my side with some dummy docs, potentially hundreds of them of varying size, and see how things perform (or dont perform).

My main worry is how that plays out with the other system indices. They are supposed to be very important for the system to function so we don't want kb indices to use all the resources of the thread pool. One simple solution here would be to setup those indices on the normal thread pools. They should be able to use the entire resources there like any other indices.

We dont intend to update either one at this time, things are fairly set-in-stone from a structure standpoint. If we could just remove the inference_id and let ES default as it should, then we wouldnt need to use it at all in this case.

Ok that's helpful thanks, we should be able to support that.

[Supplementing]

My main worry is how that plays out with the other system indices. They are supposed to be very important for the system to function so we don't want kb indices to use all the resources of the thread pool. One simple solution here would be to setup those indices on the normal thread pools. They should be able to use the entire resources there like any other indices.

Yeah, that makes sense. We certainly need to be aware of that. Let me know how to proceed here, but I think allotting more resources is smart so we dont have to worry about resource hogging from the KB indices.