[BUG] File index dimension setting not consistently applied (1536 vs 1024)

[hannesrudolph]

Type

Bug

Problem / Value

When a user sets an embedding dimension (for example, 1536), the system still produces vectors of a different size (for example, 1024) while the vector database creates a collection with the configured size. This mismatch causes a vector dimension error and blocks indexing and search.

Context

This affects anyone configuring a non-default embedding dimension or switching models with different default dimensions. The problem shows up during index creation and embedding, resulting in a dimension mismatch error reported by the vector database.

Reproduction

1. Steps: Set the "Embedding Dimension" to 1536 in Code Index settings and start indexing
2. Expected result: Both the collection and embeddings use the same dimension (no errors)
3. Actual result: Collection is created at 1536 but embeddings are generated at 1024, resulting in a dimension mismatch error
4. Variations tried: Setting dimension to 1024 works as expected

Constraints/Preferences

• Respect the user-configured dimension as the single source of truth across the entire indexing flow
• If the dimension or model changes, safely handle index reinitialization (or prompt to clear/recreate) before embedding
• Avoid silent fallbacks; provide clear, actionable guidance if a change is required
• Keep error messages user-friendly and specific to the mismatch

[roomote]

I'm exploring a potential fix for the embedding dimension mismatch issue and will open a PR to share progress.

[roomote]

Opened a PR with my attempt at addressing this issue: #8106

The fix prioritizes user-configured embedding dimensions over model defaults, ensuring that when you set a dimension like 1536, it will be respected even if the model has a different default dimension. Feedback welcome!