Specification fixes for 9.2

[pquentin]

As part of the 9.2 feature freeze, I'd like your help to get the following changes in the specification. They're not all 9.2 specific.

Search Relevance

New features

• Add Sparse Vector Index Options Settings to Semantic Text Field elasticsearch#131058
  • Adds sparse_vector index options to semantic_text index specification #5386
• Add support for extended search usage telemetry elasticsearch#135306
  • Add extended search usage to spec #5389
• Refactor semantic text field to align with text field behaviour  elasticsearch#119183
• Bonus: Add a new random rerank retriever elasticsearch#111851 (added in 8.16)

More specific failures

• Allow chunking_settings: null and chunking_settings: {} (added in Support configurable chunking in semantic_text fields elasticsearch#121041). Not sure why they are accepted though to be honest. We could also skip the relevant tests.
  • Support explicit null chunking_settings in semantic_text field mappings #5408
• ChunkRescorer: "Nested" chunk_rescorer, missing chunk_size
• Allow text_embeddings withtout a model id (not currently allowed Add support for knn vector queries on semantic_text fields elasticsearch#119011
  • Support inferred model ID for knn queries over semantic text fields #5390

Elasticsearch Search Relevance

• Release DiskBBQ(bbq_disk) index type for dense_vector fields elasticsearch#135299
  • Adding new bbq_disk index type #5433
• KNN Query visit_percentage elasticsearch#133753

[markjhoy]

I've added the update for the sparse vector index options in semantic_text: #5386

[kderusso]

I have opened the following PRs:

• Add extended search usage to spec #5389 for the search usage telemetry
• Support inferred model ID for knn queries over semantic text fields #5390 for knn queries

@pquentin I could use some advice for how to deal with the empty chunking settings though - we do want the ability to null out custom chunking settings, because this will then go back to model defaults. I don't know the best way to represent those use cases in the spec though.

[pquentin]

@kderusso Oh, using null to go back to defaults is fine! See Avoid null and empty string to mark a value as missing
. Do we also need {} to achieve that though?

[kderusso]

@kderusso Oh, using null to go back to defaults is fine! See Avoid null and empty string to mark a value as missing
. Do we also need {} to achieve that though?

It's technically supported by the API w.r.t. how the serialization works and is the equivalent of resetting it to defaults. I don't think it's needed but not harmful and at this point updating Elasticsearch to remove it would be a breaking change.

ETA: Returning null means it was explicitly set by the user so we shouldn't omit it? Am I misunderstanding?

[pquentin]

Regarding null, the key sentence in my link was "It is fine to accept null in the request if they have special semantics such as unsetting a value." It is indeed fine here. I believe we need to use chunking_settings?: ChunkingSettings | null in SemanticTextProperty.

Regarding {}, the reason it is unfortunate is that it means we need to make strategy and max_chunk_size optional in ChunkingSettings, which:

• is a breaking change for Go
• allows sending only strategy or only max_chunk_size

What I can do instead is split the "We do not set custom chunking settings for null or empty specified chunking settings" YAML test and ignore the chunking_settings: {} case for the purposes of validating the clients. How does that sound?

[kderusso]

What I can do instead is split the "We do not set custom chunking settings for null or empty specified chunking settings" YAML test and ignore the chunking_settings: {} case for the purposes of validating the clients. How does that sound?

@pquentin Sounds good to me, I'll get a PR for the null up. Thanks!

[kderusso]

PR: #5408

[pquentin]

Thanks! I just have to double-check the backports with the rest of the team. I just opened elastic/elasticsearch#136023.

[liranabn]

Hi @pquentin . Can this be closed? Seems like all tasks were completed?

[pquentin]

@liranabn We made great progress, but still have a few issues left! Thanks for asking. It's misleading because I'm linking to Elasticsearch PRs that were merged, but a few tasks are not done yet. Specifically:

• ChunkRescorer: "Nested" chunk_rescorer which fails because ChunkRescorer only supports size and chunking_settings, while this test also includes chunk_rescorer.
• ChunkRescorer: One YAML test uses chunk_size while we only expect size: Reranking based on rescore_chunks on a semantic_text field specifying chunk size
• A stretch goal is that random_reranker isn't supported. Happy to handle this separately since it was added in 8.16 and thus not specific to 9.2.

[liranabn]

@kderusso , will you be able to address the open items?