DAOS-18368 rebuild: fix bug of ec_agg_boundary and agg peer update #17324

liuxuezhao · 2025-12-29T09:18:41Z

1. fix a bug of using ec_agg_boundary before checking its valid
2. add some more logs for rebuild fetch getting zero iod_size,
   to provide some hints for layout information.
3. fix a bug of EC agg peer update, some failed update need to be retried
    to avoid data corruption.
4. refine some detailed process of dtx_resync wating for rebuild scan.

 Signed-off-by: Xuezhao Liu <[email protected]>

Steps for the author:

Commit message follows the guidelines.
Appropriate Features or Test-tag pragmas were used.
Appropriate Functional Test Stages were run.
At least two positive code reviews including at least one code owner from each category referenced in the PR.
Testing is complete. If necessary, forced-landing label added and a reason added in a comment.

After all prior steps are complete:

Gatekeeper requested (daos-gatekeeper added as a reviewer).

github-actions · 2025-12-29T09:18:57Z

Ticket title is 'Data corruption observed with master branch under MDonSSD environment.'
Status is 'In Progress'
Labels: 'md_on_ssd,scrubbed_2.8'
https://daosio.atlassian.net/browse/DAOS-18368

daosbuild3 · 2025-12-29T10:40:43Z

Test stage NLT on EL 8.8 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17324/2/display/redirect

daosbuild3 · 2025-12-31T06:58:07Z

Test stage NLT on EL 8.8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-17324/4/testReport/

daosbuild3 · 2025-12-31T12:45:24Z

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17324/4/execution/node/1313/log

daosbuild3 · 2025-12-31T16:00:59Z

Test stage Functional Hardware Medium Verbs Provider MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17324/4/execution/node/1323/log

liuxuezhao · 2026-01-04T10:47:06Z

just refresh to change a few logs.

daosbuild3 · 2026-01-04T12:26:50Z

Test stage NLT on EL 8.8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-17324/7/testReport/

daosbuild3 · 2026-01-07T10:37:12Z

Test stage NLT on EL 8.8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-17324/8/testReport/

daosbuild3 · 2026-01-07T12:32:00Z

Test stage Functional on EL 8.8 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17324/8/execution/node/1075/log

src/rebuild/srv.c

kccain

rebuild/ source file changes LGTM.

daosbuild3 · 2026-01-08T23:05:05Z

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17324/12/execution/node/1277/log

daosbuild3 · 2026-01-09T01:30:43Z

Test stage Functional Hardware Medium Verbs Provider MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17324/12/execution/node/1318/log

1. fix a bug of using ec_agg_boundary before checking its valid 2. add some more logs for rebuild fetch getting zero iod_size, to provide some hints for layout information. Signed-off-by: Xuezhao Liu <[email protected]>

Signed-off-by: Xuezhao Liu <[email protected]>

Some failures need to be retried. Signed-off-by: Xuezhao Liu <[email protected]>

For reint ranks is excluded from rebuild/reclaim if the co_in_ver exceed rebuild ver. Should set its completion in rebuild leader to avoid possible stuck. Refine dtx_resync wait handling, need not wait anymore if previously already resynced. Add some log. Signed-off-by: Xuezhao Liu <[email protected]>

liuxuezhao requested review from a team as code owners December 29, 2025 09:18

liuxuezhao mentioned this pull request Dec 29, 2025

DAOS-17843 rebuild: add some logs for rebuild enumerate #17309

Merged

6 tasks

liuxuezhao requested review from NiuYawei, gnailzenh and wangshilong December 29, 2025 09:25

liuxuezhao force-pushed the lxz/rb_fix_ec_agg_eph branch 2 times, most recently from 122e9e8 to 10fa58e Compare December 29, 2025 10:39

NiuYawei previously approved these changes Dec 30, 2025

View reviewed changes

liuxuezhao force-pushed the lxz/rb_fix_ec_agg_eph branch from 10fa58e to 66f44a9 Compare December 31, 2025 06:06

liuxuezhao dismissed NiuYawei’s stale review via 30a8331 January 4, 2026 10:46

liuxuezhao requested a review from NiuYawei January 4, 2026 10:47

liuxuezhao force-pushed the lxz/rb_fix_ec_agg_eph branch 2 times, most recently from 32db84f to e1c08ea Compare January 4, 2026 11:00

NiuYawei previously approved these changes Jan 5, 2026

View reviewed changes

wangshilong previously approved these changes Jan 5, 2026

View reviewed changes

liuxuezhao dismissed stale reviews from wangshilong and NiuYawei via c474fe4 January 7, 2026 07:31

liuxuezhao force-pushed the lxz/rb_fix_ec_agg_eph branch from e1c08ea to c474fe4 Compare January 7, 2026 07:31

liuxuezhao requested review from NiuYawei and wangshilong January 7, 2026 07:33

liuxuezhao changed the title ~~DAOS-18368 rebuild: fix use before check of ec_agg_boundary~~ DAOS-18368 rebuild: fix bug of ec_agg_boundary usage and reintegrate Jan 7, 2026

liuxuezhao force-pushed the lxz/rb_fix_ec_agg_eph branch from c474fe4 to 3b13f7d Compare January 7, 2026 12:30

liuxuezhao requested a review from kccain January 7, 2026 14:03

NiuYawei previously approved these changes Jan 8, 2026

View reviewed changes

src/rebuild/srv.c Outdated Show resolved Hide resolved

src/rebuild/srv.c Show resolved Hide resolved

NiuYawei reviewed Jan 8, 2026

View reviewed changes

src/rebuild/srv.c Show resolved Hide resolved

liuxuezhao dismissed NiuYawei’s stale review via 1ba9f49 January 8, 2026 12:36

liuxuezhao force-pushed the lxz/rb_fix_ec_agg_eph branch from 3b13f7d to 1ba9f49 Compare January 8, 2026 12:36

liuxuezhao changed the title ~~DAOS-18368 rebuild: fix bug of ec_agg_boundary usage and reintegrate~~ DAOS-18368 rebuild: fix bug of ec_agg_boundary and agg peer update Jan 8, 2026

liuxuezhao requested a review from NiuYawei January 8, 2026 12:54

liuxuezhao force-pushed the lxz/rb_fix_ec_agg_eph branch 2 times, most recently from 56751dc to f4bc272 Compare January 8, 2026 13:00

NiuYawei previously approved these changes Jan 8, 2026

View reviewed changes

wangshilong previously approved these changes Jan 8, 2026

View reviewed changes

kccain previously approved these changes Jan 8, 2026

View reviewed changes

liuxuezhao added 5 commits January 9, 2026 10:29

DAOS-18368 rebuild: fix use before check of ec_agg_boundary

6b8dab7

1. fix a bug of using ec_agg_boundary before checking its valid 2. add some more logs for rebuild fetch getting zero iod_size, to provide some hints for layout information. Signed-off-by: Xuezhao Liu <[email protected]>

DAOS-18368 rebuild: a few log change

df7d4e7

Signed-off-by: Xuezhao Liu <[email protected]>

DAOS-18368 rebuild: refine rebuild_leader_status_check

c4ecd2d

Signed-off-by: Xuezhao Liu <[email protected]>

DAOS-18368 object: refine EC agg peer update

d3a768d

Some failures need to be retried. Signed-off-by: Xuezhao Liu <[email protected]>

liuxuezhao dismissed stale reviews from kccain, wangshilong, and NiuYawei via 17ef124 January 9, 2026 10:51

liuxuezhao force-pushed the lxz/rb_fix_ec_agg_eph branch from f4bc272 to 17ef124 Compare January 9, 2026 10:51

liuxuezhao requested a review from Nasf-Fan January 9, 2026 10:56

DAOS-18368 rebuild: fix bug of ec_agg_boundary and agg peer update #17324

Are you sure you want to change the base?

DAOS-18368 rebuild: fix bug of ec_agg_boundary and agg peer update #17324

Conversation

liuxuezhao commented Dec 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Steps for the author:

After all prior steps are complete:

Uh oh!

github-actions bot commented Dec 29, 2025

Uh oh!

daosbuild3 commented Dec 29, 2025

Uh oh!

daosbuild3 commented Dec 31, 2025

Uh oh!

daosbuild3 commented Dec 31, 2025

Uh oh!

daosbuild3 commented Dec 31, 2025

Uh oh!

liuxuezhao commented Jan 4, 2026

Uh oh!

daosbuild3 commented Jan 4, 2026

Uh oh!

daosbuild3 commented Jan 7, 2026

Uh oh!

daosbuild3 commented Jan 7, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kccain left a comment

Choose a reason for hiding this comment

Uh oh!

daosbuild3 commented Jan 8, 2026

Uh oh!

daosbuild3 commented Jan 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

6 participants

liuxuezhao commented Dec 29, 2025 •

edited

Loading