Proposal: Adaptive Hybrid Synthetic Source for Pruned `_source` Fields

[Rassyan]

Description

Hi Elasticsearch team,

First, kudos to @jimczi for the elegant work on synthetic vectors (#130382) and reindex handling (#130834)! 🙌 These PRs inspired me to explore a generalized solution for _source pruning.

Current Limitation

When using include/exclude in _source (docs), the existing include/exclude mechanism physically removes excluded fields from _source storage. This may create irreversible data loss. Operations like reindex, update, and update_by_query rely on intact _source but silently discard pruned fields. While #130834 patched this for vector fields, the problem persists for every other field type. Users reasonably expect document sources can be reconstructed via doc values (as we now do for vectors).

The synthetic_vectors workaround demonstrates the problem is solvable – but only shifts a hidden technical debt by creating:

• Field-specific switches that need constant maintenance
• Special-case logic that must be reimplemented per data type

Proposed Solution

Extend the hybrid model from #130382 to all field types by:

1. Automatic mode detection

   • If _source is intact → use the traditional stored _source (Mode.STORED).
   • If _source is under synthetic mode → use synthetic source (Mode.SYNTHETIC).
   • If _source has include/exclude rules → Auto nable hybrid source (Mode.HYBRID).

   graph LR
   A[_source config] -->|has includes/excludes| B[Hybrid Mode]
   B --> C[Stored: included fields]
   B --> D[Synthetic: excluded fields from doc_values]
   

   Loading

2. Universal field support
   Reconstruct ANY pruned field (not just vectors) using existing doc_values:

   • Numerics, dates, keywords → Direct from doc_values
   • Vectors → Existing vector reconstruction
   • Geos/nested → New reconstruction handlers

3. Zero-config upgrade
   Fully backward compatible. Users get hybrid behavior automatically when pruning _source.

Advantages

1. ​Generality: It solves the problem of missing pruned fields in reindex/update operations for any field type, not just vectors.
2. ​Transparency: Users don't need to enable a separate setting (like synthetic_vectors) for specific fields. The behavior is automatically triggered by the _source pruning configuration.
3. ​Consistency: It unifies the handling of pruned fields and synthetic source.

Benefits

Operation	Current Behavior (with pruned _source)	Proposed Hybrid Behavior
reindex	❌ Loses excluded fields	✅ Reconstructs via doc_values
update	❌ Cannot update pruned fields	✅ Full field access
update_by_query	❌ Cannot update pruned fields	✅ Full field access

• Solves reindex/update issues for all field types (generalizes Ensure vectors are always included in reindex actions #130834)
• Eliminates need for field-specific switches like synthetic_vectors
• Unlocks new use cases:

  // Keep small metadata in _source, reconstruct heavy fields on demand
  "_source": { 
    "includes": ["meta/*"],
    "excludes": ["embeddings", "logs"] 
  }

Next Steps

I would like to get feedback from the team (@jimczi and others involved in the related PRs - cc @benwtrent @martijnvg ) on the feasibility and desirability of this approach.

If the team agrees this aligns with Elasticsearch's direction, I am willing to contribute. I plan to start by writing a unit test that demonstrates the problem (reindex fails for pruned non-vector fields) and then propose a solution following the hybrid model.
Looking forward to your thoughts!

[elasticsearchmachine]

Pinging @elastic/es-search-relevance (Team:Search Relevance)

[Rassyan]

I wrote a test that demonstrates when reindexing documents with vectors excluded via _source pruning, the vector fields are missing in the reindexed document.

main...Rassyan:elasticsearch:reindex-loses-pruned-vector-fields

This test will pass when we implement the Adaptive Hybrid Synthetic Source model.

[jimczi]

Thanks @Rassyan. We tackled the vector fields because the scope is limited to a single type.
When including/excluding fields in _source, the mapping is completely ignored, you can remove an unmapped field or include a field that doesn't exist. Some fields are multi-valued, objects, ....
To handle this efficiently for all types, we have a synthetic source mode. It is the best option if you don't want to pay the cost of storing source but rely on it for some operations (reindex, get document, ...).

[Rassyan]

@jimczi Thank you for your feedback! I've created draft PR #133337 implementing a generic solution for all field types using the existing _source.excludes configuration.

I'd be very happy to develop and maintain this feature long-term if it aligns with Elasticsearch's direction. Could you share your thoughts when time permits?

[jimczi]

Did you test with synthetic source? That's the generic solution to avoid the source storage cost.

[Rassyan]

@jimczi Thanks for the input. To clarify: saving storage is not the primary goal here.

My core objective is to fix operational failures caused by _source pruning. Specifically:

Reindex/update operations silently discard pruned fields, corrupting data.

This main...Rassyan:elasticsearch:reindex-loses-pruned-vector-fields demonstrates the issue unambiguously:

1. Define _source.excludes in mapping
2. Execute reindex
3. Result: Pruned fields are irrecoverably lost ⚠️

This behavior breaks fundamental expectations of data integrity in ES.

Request: Could you review this specific test case? I believe fixing this behavior aligns with Elasticsearch’s reliability goals. If we agree on the problem, I’ll propose a surgical solution.