(feat) Add dynamic templates for customizing dynamic field mappings#2253
Open
ajroetker wants to merge 6 commits intoblevesearch:masterfrom
Open
(feat) Add dynamic templates for customizing dynamic field mappings#2253ajroetker wants to merge 6 commits intoblevesearch:masterfrom
ajroetker wants to merge 6 commits intoblevesearch:masterfrom
Conversation
This implements Elasticsearch-style dynamic templates that allow users to
define rules for how dynamically detected fields are mapped. Templates can
match based on:
- Field name patterns (using glob matching with * and **)
- Full path patterns (supporting ** for multi-level matching)
- Detected data type (string, number, boolean, date)
- Exclusion patterns (unmatch, path_unmatch)
Key features:
- Templates are defined at the DocumentMapping level via DynamicTemplates field
- Templates are inherited from parent mappings but can be overridden
- First matching template wins (evaluated in order)
- Templates support all FieldMapping options (analyzer, store, index, etc.)
- Global dynamic settings (StoreDynamic, IndexDynamic, DocValuesDynamic) are
applied as defaults when templates don't explicitly set them
Example usage:
```go
mapping := NewIndexMapping()
mapping.DefaultMapping.AddDynamicTemplate(
NewDynamicTemplate("keyword_fields").
MatchField("*_keyword").
MatchType("string").
WithMapping(&FieldMapping{
Type: "text",
Analyzer: "keyword",
}),
)
```
Uses github.com/bmatcuk/doublestar/v4 for glob pattern matching with ** support.
…ic-index-templates
Contributor
Author
--- FAIL: TestSparseMutationCheckpointing (19.55s)
Error: rollback_test.go:621: expected 3 protected snapshots, got 1
Error: rollback_test.go:570: remove C:\Users\RUNNER~1\AppData\Local\Temp/bleve-scorch-test-TestSparseMutationCheckpointing\000000000001.zap: The process cannot access the file because it is being used by another process.
FAIL
FAIL github.com/blevesearch/bleve/v2/index/scorch 118.848sSeems like a transient test failure? |
ajroetker
changed the title
…ic-index-templates
Contributor
Author
|
@abhinavdangeti does this one have any potential to make it in the |
Member
|
Hey @ajroetker - thanks for this contribution again. I believe it's a useful feature - let me add it to v2.6.0 for now. However, after the team evaluates all the work we're currently tracking - we may reorganize some into separate releases as needed. |
Contributor
Author
|
I'll work on adding some usage examples to a feature doc as well then! Thanks @abhinavdangeti |
Document the dynamic templates functionality that allows fine-grained control over how dynamically detected fields are mapped during indexing, including pattern matching, type filtering, and template inheritance.
Resolved conflicts in go.mod and go.sum by accepting upstream versions
Required for glob pattern matching in dynamic index template field mappings.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This implements Elasticsearch-style dynamic templates that allow users to define rules for how dynamically detected fields are mapped. This allows indexes where schema not be know ahead of time to dynamically use the full power of bleve mappings, for instance wrappers could define rules for creating geo fields based on pathing or field name.
Templates can match based on:
Notes:
Example usage:
Uses github.com/bmatcuk/doublestar/v4 for glob pattern matching with ** support.