MB-57888: Support for downtime mitigation upon updates to index mapping

[Likith101]

• Added new apis for index update
• Added logic to determine whether index mappings can be updated and what specifically changed between mappings
• Added logic to store and retrieve said information within bolt
• Added checks to prevent deleted data from being referenced within queries till the actual data on the segment is removed during the merge process

Related:

• MB-57888: Index Update bleve_index_api#64
• MB-57888: Index Update (#47) scorch_segment_api#61
• MB-57888: Index Update zapx#318

[CascadingRadium]

hi @Likith101 can you please rebase over master and fix conflicts

[CascadingRadium]

first pass will continue

[CascadingRadium]

second pass

[CascadingRadium]

theres one case which is not implemented.
consider the following scenario

1. OriginalMapping

   "analysis": {
    "analyzers": {
     "custAna": {
      "token_filters": [
       "custFilt"
      ],
      "type": "custom",
      "tokenizer": "unicode"
     }
    },
    "token_filters": {
     "custFilt": {
      "min": 3,
      "max": 255,
      "type": "length"
     }
    },
   "types": {
    "abc": {
     "enabled": true,
     "dynamic": true,
     "default_analyzer": "custAna"
    }
   }
   

2. UpdatedMapping

   "analysis": {
    "analyzers": {
     "custAna": {
      "token_filters": [
       "custFilt"
      ],
      "type": "custom",
      "tokenizer": "unicode"
     }
    },
    "token_filters": {
     "custFilt": {
      "min": 10,
      "max": 255,
      "type": "length"
     }
    },
   "types": {
    "abc": {
     "enabled": true,
     "dynamic": true,
     "default_analyzer": "custAna"
    }
   }

Here assume the custAna analyzer is used in a type_mapping. Now, although the custom analyzer has not changed and the type_mapping is the same, the custom analysis component used within the custom analyzer has changed. This check is not there and it can wrongly flag the update as successful. This is true for all custom analysis component - tokenFilter, tokenMap, tokenizer,

[abhinavdangeti]

Looking mostly ok to me @Likith101 - I've a few requests for some more commentary and the positioning the mapping_key and we should be good here.

[abhinavdangeti]

@Likith101 Would you add a new docs/index_update.md with a title "Ability to reduce downtime during index mapping updates" (in the same format as the other docs there) indicating the status quo and capturing how to leverage this feature for our community users.

I'll leave it be your decision if you want to make that a separate commit, but it'll need to be available before we produce v2.5.0.

[abhinavdangeti]

@Likith101 It seems this is introducing multiple unit test failures that need your attention - please make it a practice to run go test ./... locally with and without the vectors tag before proposing each patch.

[abhinavdangeti]

A vector unit test requires attention ..

--- FAIL: TestIndexUpdateVector (0.07s)
panic: runtime error: invalid memory address or nil pointer dereference [recovered, repanicked]
[signal SIGSEGV: segmentation violation code=0x2 addr=0x30 pc=0x100381494]

goroutine 14502 [running]:
testing.tRunner.func1.2({0x1006bc4c0, 0x100b1ce80})
	/opt/homebrew/Cellar/go/1.25.0/libexec/src/testing/testing.go:1872 +0x190
testing.tRunner.func1()
	/opt/homebrew/Cellar/go/1.25.0/libexec/src/testing/testing.go:1875 +0x31c
panic({0x1006bc4c0?, 0x100b1ce80?})
	/opt/homebrew/Cellar/go/1.25.0/libexec/src/runtime/panic.go:783 +0x120
github.com/blevesearch/bleve/v2/index/scorch.(*IndexSnapshotVectorReader).Next(0x1400039ad00, 0x1?)
	/Users/abhinav.dangeti/Documents/go/src/github.com/blevesearch/bleve/index/scorch/snapshot_index_vr.go:86 +0x74
github.com/blevesearch/bleve/v2/search/searcher.(*KNNSearcher).Next(0x140007488c0, 0x14001b8cb40)
	/Users/abhinav.dangeti/Documents/go/src/github.com/blevesearch/bleve/search/searcher/search_knn.go:118 +0x94
github.com/blevesearch/bleve/v2/search/searcher.(*DisjunctionSliceSearcher).initSearchers(0x140000d66c0, 0x14001b8cb40)
	/Users/abhinav.dangeti/Documents/go/src/github.com/blevesearch/bleve/search/searcher/search_disjunction_slice.go:151 +0x9c
github.com/blevesearch/bleve/v2/search/searcher.(*DisjunctionSliceSearcher).Next(0x140000d66c0, 0x14001b8cb40)
	/Users/abhinav.dangeti/Documents/go/src/github.com/blevesearch/bleve/search/searcher/search_disjunction_slice.go:215 +0x30
github.com/blevesearch/bleve/v2/search/collector.(*KNNCollector).Collect(0x14000647a00, {0x10078e588, 0x100bde840}, {0x100793100, 0x140000d66c0}, {0x100794d68, 0x1400012e4d0})
	/Users/abhinav.dangeti/Documents/go/src/github.com/blevesearch/bleve/search/collector/knn.go:181 +0x264
github.com/blevesearch/bleve/v2.(*indexImpl).runKnnCollector(0x140001d2930, {0x10078e588, 0x100bde840}, 0x14000432f00, {0x100794d68, 0x1400012e4d0}, 0x0)
	/Users/abhinav.dangeti/Documents/go/src/github.com/blevesearch/bleve/search_knn.go:412 +0x354
github.com/blevesearch/bleve/v2.(*indexImpl).SearchInContext(0x140001d2930, {0x10078e588, 0x100bde840}, 0x14000432f00)
	/Users/abhinav.dangeti/Documents/go/src/github.com/blevesearch/bleve/index_impl.go:700 +0x6e0
github.com/blevesearch/bleve/v2.(*indexImpl).Search(0x0?, 0x48?)
	/Users/abhinav.dangeti/Documents/go/src/github.com/blevesearch/bleve/index_impl.go:480 +0x30
github.com/blevesearch/bleve/v2.TestIndexUpdateVector(0x14000910e00)
	/Users/abhinav.dangeti/Documents/go/src/github.com/blevesearch/bleve/search_knn_test.go:1904 +0x1c40
testing.tRunner(0x14000910e00, 0x1007843a8)
	/opt/homebrew/Cellar/go/1.25.0/libexec/src/testing/testing.go:1934 +0xc8
created by testing.(*T).Run in goroutine 1
	/opt/homebrew/Cellar/go/1.25.0/libexec/src/testing/testing.go:1997 +0x364
FAIL	github.com/blevesearch/bleve/v2	48.307s

[abhinavdangeti]

Looking good, let me make one more pass early next week.

[abhinavdangeti]

@Likith101 one check required for scoring_model as well allowing any change between states - `"", "tf-idf", "bm25" to be changeable without index rebuild. Please weigh in on whether that change is required here or in cbft.