Update semantic text to use BFLOAT16 by default

[Mikep86]

Updates the semantic_text field to use the BFLOAT16 element type by default for all inference services that use FLOAT.

This change minorly impacts knn query scoring due to the reduced precision of BFLOAT16. This won't matter for 99% of users, but since this may be a problem for some, this PR also adds a way to override the element type used so that these users can continue using FLOAT. An element_type parameter has been added to the dense vector index options:

PUT semantic-embeddings
{
  "mappings": {
    "properties": {
      "content": {
        "type": "semantic_text",
        "inference_id": "my-inference-endpoint",
        "index_options": {
          "dense_vector": {
            "element_type": "float",
            "type": "bbq_disk"
          }
        }
      }
    }
  }
}

Since element_type is not part of standard dense vector index options (and it doesn't make sense to add it there, since element_type is a top-level param for dense_vector), this param was added by extending dense vector index options only for semantic_text. This creates some new scenarios for us to handle:

• Allowing the user to set only the element_type without setting type
• Requiring type only when params other than element_type are set
• Throwing an error when the element_type set in index_options is incompatible with the model element type
• Serializing element_type when include_defaults is true and we defaulted to BFLOAT16

This PR is still in draft because tests are incomplete and there's a potentially unhandled edge case with partial defaults. Sharing now for early feedback before I invest in implementing all the tests.

[elasticsearchmachine]

Hi @Mikep86, I've created a changelog YAML for you.

[Mikep86]

@elasticmachine update branch

[Mikep86]

Some tests are failing because switching to BFLOAT16 is breaking what used to be tied scores. I stabilized those tests in a separate PR: #144251

[benwtrent]

Allowing the user to set only the element_type without setting type

Yep, of course we should do this.

Requiring type only when params other than element_type are set

I don't think we should require type at all? Why are we requiring it? It should be whatever default is for semantic text. If params are set that do not make sense, then of course, we should throw (e.g. setting max_conn with DiskBBQ or cluster size with HNSW).

Throwing an error when the element_type set in index_options is incompatible with the model element type

Yep!

Serializing element_type when include_defaults is true and we defaulted to BFLOAT16

yes, agreed.

I think separating the index options as defined with semantic_text is OK (e.g. element_type is settable on dense_vector. The only other option I could think of is:

{
  "type": "semantic_text",
  "index_options" {
    "dense_vector": {
      "element_type": <TYPE>,
      "index_options": {...} // dense_vector index options like index type & their settings
    }
  }
}

But this might break even more backwards compatibility things.

Regardless, we already need to do a "translation step" from dense_vector to the mapping which is dense_vector: index_options: {...}

[benwtrent]

please keep all refactoring that isn't strictly necessary in a separate pr.

[Mikep86]

@benwtrent

I don't think we should require type at all? Why are we requiring it? It should be whatever default is for semantic text. If params are set that do not make sense, then of course, we should throw (e.g. setting max_conn with DiskBBQ or cluster size with HNSW).

We aren't requiring type, unless some param other than element_type is set. Take an index option like this, for example:

        "index_options": {
          "dense_vector": {
            "element_type": "float",
            "foo": "bar"
          }
        }

The param foo could be associated with some type, we don't know until we pass it to dense_vector for parsing. And even then, it depends on the type that dense_vector defaults to. To simplify this case, I propose that if any param other than element_type is set, type is required so that the index options are a complete definition of the settings to use.

[benwtrent]

The param foo could be associated with some type, we don't know until we pass it to dense_vector for parsing. And even then, it depends on the type that dense_vector defaults to. To simplify this case, I propose that if any param other than element_type is set, type is required so that the index options are a complete definition of the settings to use.

I think this is ok for now. Better to be strict here than not. I do think we COULD be lenient and just assume that type is whatever the default is and if that parameter doesn't work, then we throw requesting type to be changed. But, keeping it strict for now is fine.