fix: Fix inverted index load failure caused by sliced files by sunby · Pull Request #48526 · milvus-io/milvus

sunby · 2026-03-25T09:35:07Z

issue: #48294
In CompactIndexDatas, slice meta may contain files that are not in index_datas. So compact will fail. Now we specify key when loading inverted index to avoid it.

In CompactIndexDatas, slice meta may contain files that are not in index_datas. So compact will fail. Now we specify key when loading inverted index to avoid it. Signed-off-by: sunby <sunbingyi1992@gmail.com>

sre-ci-robot · 2026-03-25T09:35:24Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: sunby
To complete the pull request process, please assign xiaofan-luan after the PR has been reviewed.
You can assign the PR to them by writing /assign @xiaofan-luan in a comment when ready.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

internal/core/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

sre-ci-robot · 2026-03-25T09:36:14Z

[ci-v2-notice]
Notice: New ci-v2 system is enabled for this PR.

To rerun ci-v2 checks, comment with:

/ci-rerun-code-check // for ci-v2/code-check
/ci-rerun-build // for ci-v2/build
/ci-rerun-build-all // for ci-v2/build-all (multi-arch builds)
/ci-rerun-buildenv // for ci-v2/build-env (build milvus-env builder images)
/ci-rerun-ut-integration // for ci-v2/ut-integration, will rerun ci-v2/build
/ci-rerun-ut-go // for ci-v2/ut-go, will rerun ci-v2/build
/ci-rerun-ut-cpp // for ci-v2/ut-cpp
/ci-rerun-ut // for all ci-v2/ut-integration, ci-v2/ut-go, ci-v2/ut-cpp, will rerun ci-v2/build
/ci-rerun-e2e-arm // for ci-v2/e2e-arm
/ci-rerun-e2e-default // for ci-v2/e2e-default
/ci-rerun-ciloop // for ci-v2/ciloop (build + unit tests in one pipeline)
/ci-rerun-gosdk // for ci-v2/go-sdk (Go SDK E2E tests, AMD)
/ci-rerun-gosdk-arm // for ci-v2/go-sdk-arm (Go SDK E2E tests, ARM)

If you have any questions or requests, please contact @zhikunyao.

codecov · 2026-03-25T11:28:43Z

Codecov Report

❌ Patch coverage is 90.65421% with 10 lines in your changes missing coverage. Please review.
✅ Project coverage is 77.81%. Comparing base (7d109c3) to head (7de67d8).
⚠️ Report is 13 commits behind head on master.

Files with missing lines	Patch %	Lines
internal/core/src/index/JsonInvertedIndex.cpp	0.00%	6 Missing ⚠️
internal/core/src/index/InvertedIndexTantivy.cpp	40.00%	3 Missing ⚠️
internal/core/src/index/Utils.cpp	96.15%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           master   #48526       +/-   ##
===========================================
- Coverage   83.98%   77.81%    -6.18%     
===========================================
  Files         627     2100     +1473     
  Lines      103896   349126   +245230     
===========================================
+ Hits        87259   271670   +184411     
- Misses      16637    69177    +52540     
- Partials        0     8279     +8279

Components	Coverage Δ
Client	`79.25% <ø> (∅)`
Core	`84.06% <90.65%> (+0.07%)`	⬆️
Go	`75.77% <ø> (∅)`

Files with missing lines	Coverage Δ
internal/core/src/index/JsonInvertedIndexTest.cpp	`100.00% <100.00%> (ø)`
internal/core/src/index/Utils.h	`85.10% <ø> (ø)`
internal/core/src/index/Utils.cpp	`56.29% <96.15%> (+12.44%)`	⬆️
internal/core/src/index/InvertedIndexTantivy.cpp	`70.94% <40.00%> (+0.06%)`	⬆️
internal/core/src/index/JsonInvertedIndex.cpp	`31.57% <0.00%> (-0.34%)`	⬇️

... and 1475 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Signed-off-by: sunby <sunbingyi1992@gmail.com>

sunby · 2026-03-26T02:38:05Z

/ci-rerun-e2e-default
/ci-rerun-e2e-arm
/ci-rerun-ut-cpp

sunby · 2026-03-26T03:07:30Z

/ci-rerun-e2e-arm
/ci-rerun-e2e-default

sunby · 2026-03-26T03:34:07Z

/ci-rerun-e2e-default

…and fix test The fill_null_offsets lambda in LoadIndexMetas used resize+memcpy from position 0 on each call, which overwrites previous data when iterating over multiple slices. Fix by appending each slice's data after the previous data. Same fix applied to fill_non_exist_offset in JsonInvertedIndex. Fix TestSlicedOffsetFilesLoadIndependently: the test used non-nullable JSON data with {"a": null} expecting null_offset entries, but ProcessJsonFieldData only calls null_adder when schema.nullable()==true and the row is field-level null. Changed test to use a nullable field with explicit validity bitmap to correctly populate null_offset_. Updated non_exist_offset size expectation to account for both null rows (which also get non_exist_adder) and non-existent key rows. Signed-off-by: sunby <sunbingyi1992@gmail.com>

sunby · 2026-03-26T09:34:05Z

/ci-rerun-integration-test

sunby · 2026-03-26T10:04:04Z

/ci-rerun-e2e-arm
/ci-rerun-integration-test

sunby · 2026-03-26T10:34:24Z

/ci-rerun-e2e-arm
/ci-rerun-integration-test

sunby · 2026-03-26T11:04:00Z

/ci-rerun-e2e-arm
/ci-rerun-go-sdk
/ci-rerun-integration-test

sunby · 2026-03-26T11:34:02Z

/ci-rerun-go-sdk
/ci-rerun-integration-test

sunby · 2026-03-26T12:04:04Z

/ci-rerun-e2e-default
/ci-rerun-go-sdk
/ci-rerun-integration-test

sunby · 2026-03-26T12:33:59Z

/ci-rerun-e2e-default
/ci-rerun-go-sdk
/ci-rerun-integration-test

sunby · 2026-03-26T13:04:59Z

/ci-rerun-e2e-default
/ci-rerun-go-sdk
/ci-rerun-integration-test

sunby · 2026-03-26T13:34:10Z

/ci-rerun-e2e-default
/ci-rerun-go-sdk
/ci-rerun-integration-test

sunby · 2026-03-26T14:04:24Z

/ci-rerun-go-sdk
/ci-rerun-integration-test

sunby · 2026-03-26T14:34:13Z

/ci-rerun-go-sdk
/ci-rerun-integration-test

sunby · 2026-03-26T15:04:18Z

/ci-rerun-go-sdk
/ci-rerun-integration-test

sunby · 2026-03-26T15:34:11Z

/ci-rerun-e2e-default
/ci-rerun-go-sdk
/ci-rerun-integration-test

sunby · 2026-03-26T16:04:09Z

/ci-rerun-integration-test

sunby · 2026-03-26T17:05:07Z

/ci-rerun-integration-test

sunby · 2026-03-26T17:34:09Z

/ci-rerun-e2e-default
/ci-rerun-integration-test

sunby · 2026-03-26T18:04:11Z

/ci-rerun-integration-test

sunby · 2026-03-26T18:34:06Z

/ci-rerun-integration-test

sunby · 2026-03-26T19:04:08Z

/ci-rerun-e2e-default
/ci-rerun-integration-test

fix: Fix inverted index load failure caused by sliced files

3e0fb37

In CompactIndexDatas, slice meta may contain files that are not in index_datas. So compact will fail. Now we specify key when loading inverted index to avoid it. Signed-off-by: sunby <sunbingyi1992@gmail.com>

sre-ci-robot requested review from czs007 and xiaocai2333 March 25, 2026 09:35

sre-ci-robot added the size/L Denotes a PR that changes 100-499 lines. label Mar 25, 2026

mergify bot added dco-passed DCO check passed. kind/bug Issues or changes related a bug labels Mar 25, 2026

sre-ci-robot added the low-code-coverage add test-label from zhikun, diff coverage > 80% label Mar 25, 2026

fix: resolve padded_string ambiguity and apply clang-format

9b96121

Signed-off-by: sunby <sunbingyi1992@gmail.com>

sunby mentioned this pull request Mar 26, 2026

fix: [2.6] Fix inverted index load failure caused by sliced files #48542

Open

sre-ci-robot removed the low-code-coverage add test-label from zhikun, diff coverage > 80% label Mar 26, 2026

Conversation

sunby commented Mar 25, 2026

Uh oh!

sre-ci-robot commented Mar 25, 2026

Uh oh!

sre-ci-robot commented Mar 25, 2026

Uh oh!

codecov bot commented Mar 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

sunby commented Mar 26, 2026

Uh oh!

sunby commented Mar 26, 2026

Uh oh!

sunby commented Mar 26, 2026

Uh oh!

sunby commented Mar 26, 2026

Uh oh!

sunby commented Mar 26, 2026

Uh oh!

sunby commented Mar 26, 2026

Uh oh!

sunby commented Mar 26, 2026

Uh oh!

sunby commented Mar 26, 2026

Uh oh!

sunby commented Mar 26, 2026

Uh oh!

sunby commented Mar 26, 2026

Uh oh!

sunby commented Mar 26, 2026

Uh oh!

sunby commented Mar 26, 2026

Uh oh!

sunby commented Mar 26, 2026

Uh oh!

sunby commented Mar 26, 2026

Uh oh!

sunby commented Mar 26, 2026

Uh oh!

sunby commented Mar 26, 2026

Uh oh!

sunby commented Mar 26, 2026

Uh oh!

sunby commented Mar 26, 2026

Uh oh!

sunby commented Mar 26, 2026

Uh oh!

sunby commented Mar 26, 2026

Uh oh!

sunby commented Mar 26, 2026

Uh oh!

sunby commented Mar 26, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

codecov bot commented Mar 25, 2026 •

edited

Loading