Skip to content

MB-62985: Forcing cosine similarity for backing indexes within bq#2282

Open
Likith101 wants to merge 4 commits intomasterfrom
BQMain
Open

MB-62985: Forcing cosine similarity for backing indexes within bq#2282
Likith101 wants to merge 4 commits intomasterfrom
BQMain

Conversation

@Likith101
Copy link
Member

@Likith101 Likith101 commented Jan 31, 2026

  • Forcing cosine similarity for binary indexes during indexing and query times
  • Added relevant test cases

@coveralls
Copy link

coveralls commented Jan 31, 2026

Coverage Status

coverage: 52.676% (+0.01%) from 52.666%
when pulling 19f9a4d on BQMain
into 9d07cde on master.

@Likith101 Likith101 changed the title Toy: Forcing Cosine for binary indexes MB-62985: Forcing cosine similarity for binary optimized indexes Feb 2, 2026
@Likith101 Likith101 changed the title MB-62985: Forcing cosine similarity for binary optimized indexes MB-62985: Forcing cosine similarity for backing indexes within bq Feb 2, 2026
@abhinavdangeti abhinavdangeti added this to the v2.6.0 milestone Feb 12, 2026
options |= index.SkipFreqNorm

if vectorIndexOptimizedFor == index.IndexOptimizedWithBivfFlat {
similarity = index.CosineSimilarity
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add commentary here, that this is for the backing index - as as default to hamming distance with Bivf.

vectorIndexOptimizedFor = index.DefaultIndexOptimization
}
// BIVF-Flat index needs cosine similarity for correct scoring
// regardless of the similarity metric specified in the mapping.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Update to indicate that this is for the backing flat (ditto above)

vectorIndexOptimizedFor = index.DefaultIndexOptimization
}
// BIVF-Flat index needs cosine similarity for correct scoring
// regardless of the similarity metric specified in the mapping.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto

effectiveOptimizedFor = index.DefaultIndexOptimization
}
if effectiveOptimizedFor == index.IndexOptimizedWithBivfFlat && field.Dims%8 != 0 {
return fmt.Errorf("field: '%s', vector dimensions for bivf-flat must be a multiple"+
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Update this to the same message - use a const for here and below. Maybe use - "incompatible vector dimensionality for BQ".

// BIVF-Flat index optimization requires vector dimensions to be a multiple of 8
if effectiveOptimizedFor == index.IndexOptimizedWithBivfFlat && field.Dims%8 != 0 {
return fmt.Errorf("field: '%s', invalid vector dimension: %d for optimization with BIVF-Flat,"+
" dimension should be a multiple of 8", effectiveFieldName, field.Dims)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See comment above.

return nil, fmt.Errorf("k must be greater than 0 and vector must be non-empty")
}
if fieldMapping.VectorIndexOptimizedFor == index.IndexOptimizedWithBivfFlat {
similarityMetric = index.CosineSimilarity
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Related to comment earlier.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Development

Successfully merging this pull request may close these issues.

4 participants