Make bbq_hnsw the default index option for dense-vector fields with more than 384 dimensions

[pmpailis]

After the recent benchmarking deemed it sensible to do so, this PR changes the default index options for float vector fields with >= 384 dimensions to bbq_hnsw. Otherwise, we keep the int8_hnsw that was set previously in #106836

[github-actions]

🔍 Preview links for changed docs:

• docs/reference/elasticsearch/mapping-reference/dense-vector.md

🔔 The preview site may take up to 3 minutes to finish building. These links will become live once it completes.

[elasticsearchmachine]

Pinging @elastic/es-docs (Team:Docs)

[elasticsearchmachine]

Pinging @elastic/es-search-relevance (Team:Search Relevance)

[elasticsearchmachine]

Pinging @elastic/es-search-foundations (Team:Search Foundations)

[pmpailis]

Setting it for review, but there are some test cases failing for search.vectors/70_dense_vector_telemetry ; looking into it.

[benwtrent]

We need some tests around default values and ensuring that dynamic dims still have the default of bbq hnsw set.

[benwtrent]

I have a sneaky suspicion that if dims are not set, but we then index a vector of 384 dims, and set the dims to 384, the index type will not be set to bbq_hnsw.

Can you write a test to confirm/deny this?

[pmpailis]

Yeap, your suspicion was right :) Updating to use the builder's dimension setter when reading the float array., so that we can have access to that when setting up default value.

Added tests in DynamicMappingIT and DynamicMappingTests.

[benwtrent]

@pmpailis there is still a bug.

PUT vectors
{
	"mappings": {
		"properties": {
			"vector": {
				"type": "dense_vector"
			}
		}
	}
}

POST vectors/_doc/1
{
	"vector": [0.1, 0.2, 0.3, 0.4, 0.5,..., 0.385]
}

GET vectors

See that the mapped type is not bbq_hnsw.

Its possible to create a dense_vector mapping with a null dims. Then later, when the doc is parsed, we update the mapping to apply that dim update.

I think in that same update, we should adjust the default index type to ensure its applied.

[benwtrent]

I guess the tricky part is knowing if index_options were set by the user (specifically to the defaults, or whatever), or not set at all.

[pmpailis]

oh, that's a nice catch. I've tried restoring the default value to null if dims are not present, and resetting this once we have a document in place. So now:

• if a user provides dims, then we take them into account from the beginning to setup a default value
• if a user does not provide any dimensions, then the mappings do not initialize any information for the index_options either, and we instead delay this decision for when we know dims as well

For example:

PUT vectors
{
	"mappings": {
		"properties": {
			"vector": {
				"type": "dense_vector"
			}
		}
	}
}

GET _mappings

->

{
    "vectors": {
        "mappings": {
            "properties": {
                "vector": {
                    "type": "dense_vector",
                    "index": true,
                    "similarity": "cosine"
                }
            }
        }
    }
}

PUT vectors/_doc/1
{
  "vector": [1.2, 3.4, ..., 8.9] 
}

GET _mappings

->

{
    "vectors": {
        "mappings": {
            "properties": {
                "vector": {
                    "type": "dense_vector",
                    "dims": 1512,
                    "index": true,
                    "similarity": "cosine",
                    "index_options": {
                        "type": "bbq_hnsw",
                        "m": 16,
                        "ef_construction": 100,
                        "rescore_vector": {
                            "oversample": 3
                        }
                    }
                }
            }
        }
    }
}

Not overly familiar with mappings and how they're updated, so i might be missing something (or if this considered breaking change). Does the above make sense?
Added a take on this in ea7c8a2 and a test for the above scenario in DynamicMappingIT#testBBQDynamicMappingWhenFirstIngestingDoc

[pmpailis]

(there is some code duplication that we could potentially avoid if we are to go with this)

[pmpailis]

@benwtrent updated this so if dims is not provided, we postpone deciding which IndexOptions to create until we actually know the dimensions (in DenseVectorFieldMapper#parse)

[pmpailis]

buildkite test this please

[jimczi]

LGTM

[pmpailis]

Thanks for the reviews @benwtrent , @jimczi and @tteofili !

@benwtrent I'll proceed with merging this PR, we can address any leftover comments you may have in a follow up :)