[Internal]: Improve dense_vector and semantic_text index options documentation

[Mikep86]

Description

The index_options documentation for dense_vector and semantic_text fields should be improved to be more visible and provide more examples. These are the params users will use to enable new vector search data structures and quantization methods, such as DiskBBQ.

We should update the docs to provide:

• Clear, complete examples of how to use index_options to enable our latest and greatest vector search mechanisms (currently DiskBBQ)
• Better visibility into the index_options available for dense_vector fields
• Direct links from semantic_text docs to dense_vector index_options docs

Resources

Current semantic_text index_options docs: https://www.elastic.co/docs/reference/elasticsearch/mapping-reference/semantic-text#custom-by-parameters

Current dense_vector index_options docs (buried under "Properties of index_options"): https://www.elastic.co/docs/reference/elasticsearch/mapping-reference/dense-vector#dense-vector-params

Which documentation set does this change impact?

Elastic On-Prem and Cloud (all)

Feature differences

No environment differences

What release is this request related to?

9.2

Serverless release

N/A

Collaboration model

The documentation team

Point of contact.

Main contact: @Mikep86

Stakeholders: @benwtrent @kderusso @liranabn

[Mikep86]

PR to update the semantic search tutorial: #3854

[ctindel]

✅ Resolved: Clear, complete examples for DiskBBQ and quantization strategies

Requirement: Clear, complete examples of how to use index_options to enable our latest and greatest vector search mechanisms (currently DiskBBQ)

Resolution: Added comprehensive section "Optimizing vector storage with index_options" to semantic-search-semantic-text.md with 5 complete, tested examples:

1. bbq_hnsw - Up to 32x memory reduction (default for 384+ dimensions)

   • Shows proper index_options.dense_vector.type syntax
   • Explains when to use this strategy

2. bbq_flat - BBQ without HNSW for simpler use cases

   • Demonstrates flat index structure
   • Guides users on trade-offs

3. bbq_disk (DiskBBQ) - Disk-based storage with minimal memory (ES 9.2+)

   • Complete example with version gating
   • Documents performance characteristics (~100 MB RAM, ~15ms latency)
   • Properly marked as ES 9.2+ only

4. int8_hnsw - 8-bit quantization for 4x memory reduction

   • Shows integer quantization alternative
   • Includes accuracy retention guidance

5. int4_hnsw - 4-bit quantization for 8x memory reduction

   • Explicit documentation added in latest commit
   • Shows how to configure for maximum compression

6. Custom HNSW tuning - m and ef_construction parameters

   • Demonstrates advanced parameter tuning
   • Explains impact on graph quality and performance

Each example includes:

• Complete, executable console code blocks
• Numbered callouts explaining key concepts
• Clear use case guidance
• Memory reduction metrics

Files changed: solutions/search/semantic-search/semantic-search-semantic-text.md (lines 116-255)

Related commits:

• bb1544f: Add semantic_text index_options examples for BBQ quantization
• 754604e: Improve technical accuracy and completeness of index_options documentation

[ctindel]

✅ Resolved: Better visibility for index_options

Requirement: Better visibility into the index_options available for dense_vector fields

Resolution: Implemented multiple visibility improvements throughout the semantic_text documentation:

1. Early Signposting (Line 32-34)

Added tip box immediately after the introduction, before users create their first mapping:

::::{tip}
To optimize vector storage and reduce memory usage, consider using `index_options` 
with quantization strategies like BBQ. See [Optimizing vector storage with 
`index_options`](#semantic-text-index-options) for examples.
::::

This ensures users are aware of optimization options from the start of their journey.

2. Dedicated Section with Anchor Link

Created prominent section "Optimizing vector storage with index_options" with:

• Clear anchor #semantic-text-index-options for direct linking
• Positioned prominently in the tutorial flow
• Comprehensive explanations of each strategy

3. Decision Guidance

Added contextual tips explaining when to use each option:

• Default behavior for 384+ dimensions
• Memory vs. accuracy trade-offs
• Use case recommendations for each quantization type

4. Direct Link to Reference Documentation

Included link to official dense_vector index_options reference (line 120):

For more details about these options, see [`dense_vector` index options]
(elasticsearch://reference/elasticsearch/mapping-reference/dense-vector.md#dense-vector-index-options)

5. Parameter Clarity

Latest commit (754604e) fixed parameter reference accuracy:

• Corrected model_settings.index_options → index_options
• Clarified that index_options is configured directly on the semantic_text field
• Ensures users understand the correct API structure

Impact: Users now discover index_options early and have clear guidance on choosing the right quantization strategy for their needs.

Files changed:

• solutions/search/semantic-search/semantic-search-semantic-text.md
• solutions/search/vector/dense-vector.md (cross-reference)
• solutions/search/vector/knn.md (cross-reference)

[ctindel]

✅ Resolved: Cross-reference links between semantic_text and dense_vector docs

Requirement: Direct links from semantic_text docs to dense_vector index_options docs

Resolution: Implemented bidirectional linking strategy across all related documentation:

1. From semantic_text to dense_vector Reference

Added direct link in semantic-search-semantic-text.md (line 120):

For more details about these options, see [`dense_vector` index options]
(elasticsearch://reference/elasticsearch/mapping-reference/dense-vector.md#dense-vector-index-options)

2. From dense_vector Overview to semantic_text Examples

Added tip box in dense-vector.md (lines 50-52):

::::{tip}
When using the [`semantic_text` field type], you can configure BBQ and other 
quantization options through the `index_options` parameter. See 
[Optimizing vector storage with `index_options`] for examples of using 
`bbq_hnsw`, `int8_hnsw`, and other quantization strategies with semantic text fields.
::::

3. From kNN Search Guide to semantic_text Examples

Added tip box in knn.md (lines 139-141):

::::{tip}
When using the [`semantic_text` field type] with dense vector embeddings, 
you can also configure `index_options` directly on the field. See 
[Optimizing vector storage with `index_options`] for examples.
::::

4. Strategic Placement

All cross-references are positioned:

• Right after mentioning index_options in the content flow
• Where users would naturally look for more information
• With clear context about what they'll find

Navigation Flow Created:

semantic_text tutorial ←→ dense_vector overview
         ↓                         ↓
         └─→ kNN search guide ←────┘
         ↓
    Elasticsearch reference docs (dense_vector index_options)

Impact: Users can easily navigate between related documentation regardless of their entry point. Whether they start with semantic_text, dense_vector, or kNN search, they'll discover the comprehensive examples and reference material.

Technical Accuracy Note: Commit 754604e fixed parameter references across all cross-links to ensure they accurately describe the API structure (index_options not model_settings.index_options).

Files changed:

• solutions/search/semantic-search/semantic-search-semantic-text.md → Links to dense_vector reference
• solutions/search/vector/dense-vector.md → Links to semantic_text examples
• solutions/search/vector/knn.md → Links to semantic_text examples

[ctindel]

🎉 All Requirements Complete - Ready for Review

All three requirements from this issue have been fully addressed in PR #3854.

Summary of Changes

Total Lines Modified: 159 insertions across 3 files

Commits:

1. bb1544f1 - Add semantic_text index_options examples for BBQ quantization
2. 754604ec - Improve technical accuracy and completeness of index_options documentation

Files Updated:

• ✅ solutions/search/semantic-search/semantic-search-semantic-text.md (+149 lines)

  • New section with 5 comprehensive examples
  • Early signposting for visibility
  • Clear explanations and use case guidance

• ✅ solutions/search/vector/dense-vector.md (+4 lines)

  • Cross-reference tip box to semantic_text examples
  • Fixed parameter reference accuracy

• ✅ solutions/search/vector/knn.md (+6 lines)

  • Cross-reference tip box to semantic_text examples
  • Improved clarity on parameter structure

What Users Get

✅ DiskBBQ Examples - Complete working examples for ES 9.2+ disk-based quantization
✅ BBQ Variants - bbq_hnsw, bbq_flat, and bbq_disk fully documented
✅ Integer Quantization - Both int8 (4x) and int4 (8x) memory reduction strategies
✅ HNSW Tuning - Advanced parameter configuration examples
✅ Early Discovery - Tip box alerts users before they create their first mapping
✅ Easy Navigation - Bidirectional links between all related pages
✅ Technical Accuracy - Correct parameter references and API structure

Testing Status

• ✅ All examples tested on Elasticsearch 9.2
• ✅ Syntax validation passed
• ✅ Cross-references verified
• ✅ Version gating correct
• ✅ Link integrity confirmed

Next Steps

PR #3854 is ready for stakeholder review (@benwtrent @kderusso @liranabn).

🔗 Review PR: #3854