DAOS-15993 rebuild: for manual rebuilds do not eval self_heal #17345

kccain · 2026-01-06T20:44:20Z

Consider a quick maintenance scenario in which a daos_engine is stopped briefly, and the administrator does not wish to have the DAOS automatic recovery / rebuild mechanism occur. That is, a pool map update (targets from UP_IN to DOWN) is to occur, the pool to enter a degraded mode (still allowing ongoing I/O), and NO rebuild to be triggered during this brief time window.

The above can be arranged by modifying the system or pool-specific self_heal property value (to not set the rebuild bit), and then stopping the engine.

Now also consider the conclusion of the maintenance that involes re-starting the engine, and reintegrating that rank back into the pool. It is most convenient to directly issue a dmg pool reintegrate command from the maintenance state.

Before this change, manual administration commands such as dmg pool exclude/reintegrate were prevented from triggering rebuilds due to the pool self_heal property setting. However, the intention of the self_heal (aka auto recovery) feature is to only apply to automatic rebuilds.

With this change, the is_pool_rebuild_allowed() function is updated to accept an indication of whether the self_heal checks are applicable. Manual pool map update and rebuild cases supply false for this argument (allowing those cases to result in a rebuild being scheduled).

Features: rebuild pool

Steps for the author:

Commit message follows the guidelines.
Appropriate Features or Test-tag pragmas were used.
Appropriate Functional Test Stages were run.
At least two positive code reviews including at least one code owner from each category referenced in the PR.
Testing is complete. If necessary, forced-landing label added and a reason added in a comment.

After all prior steps are complete:

Gatekeeper requested (daos-gatekeeper added as a reviewer).

Consider a quick maintenance scenario in which a daos_engine is stopped briefly, and the administrator does not wish to have the DAOS automatic recovery / rebuild mechanism occur. That is, a pool map update (targets from UP_IN to DOWN) is to occur, the pool to enter a degraded mode (still allowing ongoing I/O), and NO rebuild to be triggered during this brief time window. The above can be arranged by modifying the system or pool-specific self_heal property value (to not set the rebuild bit), and then stopping the engine. Now also consider the conclusion of the maintenance that involes re-starting the engine, and reintegrating that rank back into the pool. It is most convenient to directly issue a dmg pool reintegrate command from the maintenance state. Before this change, manual administration commands such as dmg pool exclude/reintegrate were prevented from triggering rebuilds due to the pool self_heal property setting. However, the intention of the self_heal (aka auto recovery) feature is to only apply to automatic rebuilds. With this change, the is_pool_rebuild_allowed() function is updated to accept an indication of whether the self_heal checks are applicable. Manual pool map update and rebuild cases supply false for this argument (allowing those cases to result in a rebuild being scheduled). Features: rebuild pool Signed-off-by: Kenneth Cain <[email protected]>

github-actions · 2026-01-06T20:47:20Z

Ticket title is 'pool reintegrate issue when pool property self_heal:exclude (no rebuild)'
Status is 'In Progress'
Labels: 'scrubbed_2.8,triaged'
https://daosio.atlassian.net/browse/DAOS-15993

daosbuild3 · 2026-01-06T21:02:14Z

Test stage Unit Test bdev with memcheck on EL 8.8 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17345/1/display/redirect

daosbuild3 · 2026-01-06T21:35:47Z

Test stage NLT on EL 8.8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-17345/1/testReport/

liw

Looks good in general; one "[question]" needs an answer before I approve this PR.

liw · 2026-01-07T02:45:19Z

src/include/daos_srv/pool.h


 static inline bool
-is_pool_rebuild_allowed(struct ds_pool *pool, bool check_delayed_rebuild)
+is_pool_rebuild_allowed(struct ds_pool *pool, uint64_t self_heal, bool self_heal_applicable,


[Nit] Would it be any clearer to name self_heal_applicable auto?

Yes I think so. Changed to auto_recovery.

liw · 2026-01-07T02:48:37Z

src/include/daos_srv/rebuild.h

 int
     ds_rebuild_regenerate_task(struct ds_pool *pool, daos_prop_t *prop, uint64_t sys_self_heal,
-				uint64_t delay_sec);
+				bool self_heal_applicable, uint64_t delay_sec);


[Nit] self_heal_applicable or perhaps auto? No strong opinion though.

Changed to auto_recovery.

liw · 2026-01-07T02:49:46Z

src/pool/srv_pool.c


-	rc = ds_rebuild_regenerate_task(svc->ps_pool, prop, sys_self_heal, 0);
+	rc = ds_rebuild_regenerate_task(svc->ps_pool, prop, sys_self_heal,
+					true /* self_heal_applicable */, 0 /* delay_sec*/);


[Nit] A missing space between delay_sec and */.

Thanks, fixed.

liw · 2026-01-07T02:53:47Z

src/pool/srv_pool.c

+	self_heal_applicable = (opc == MAP_EXCLUDE && src == MUS_SWIM);

-	if (sys_self_heal_applicable) {
+	/* do not update pool map if system.self_heal is applicable but does not enable exclude */


[Nit] The comment isn't that helpful; if one must be added, I think a conciser one like "If applicable, check the system self-heal policy." might be more helpful.

Thanks, simplified.

liw · 2026-01-07T03:00:22Z

src/pool/srv_pool.c

 		}
 	}

+	/* Update pool map if pool.self_heal is applicable and enables exclude. */


[Nit] Could we say "The pool self-heal policy is checked by the following call." instead? Considering that the call performs many other checks too...

Agree, fixed.

liw · 2026-01-07T03:01:40Z

src/pool/srv_pool.c

 	d_freeenv_str(&env);

-	if (sys_self_heal_applicable && !(sys_self_heal & DS_MGMT_SELF_HEAL_POOL_REBUILD)) {
+	/* Do not trigger rebuild if system.self_heal is applicable but does not enable rebuild. */


[Nit] The log message has already said it.

Removed to de-duplicate the information in the source code.

liw · 2026-01-07T03:02:09Z

src/pool/srv_pool.c

 	}

-	if (!is_pool_rebuild_allowed(svc->ps_pool, true)) {
+	/* Do not trigger rebuild if pool.self_heal is applicable but does not enable rebuild. */


[Nit] The log message has already said it.

Removed to de-duplicate the information in the source code.

liw · 2026-01-07T04:42:55Z

src/include/daos_srv/pool.h

 	if (pool->sp_disable_rebuild)
 		return false;
-	if (!(pool->sp_self_heal & flags))
+	if (self_heal_applicable && !(self_heal & flags))


[Question] Does this change affect the delayed rebuild case? I don't know the answer; just making sure this has been considered.

I don't think so. What I've done is remove that argument, since no callers currently specify anything other than true.

Also I did experiment with some manual testing of a pool whose self_heal property value was "exclude;delay_rebuild" and it seemed to work as expected (no exclude rebuilds occur, deferring until a subsequent reintegrate).

daosbuild3 · 2026-01-07T13:12:47Z

Test stage NLT on EL 8.8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-17345/2/testReport/

Signed-off-by: Kenneth Cain <[email protected]>

daosbuild3 · 2026-01-07T23:06:16Z

Test stage Functional Hardware Large MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17345/2/execution/node/1100/log

daosbuild3 · 2026-01-07T23:06:33Z

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17345/2/execution/node/1141/log

daosbuild3 · 2026-01-08T19:16:50Z

Test stage Functional Hardware Medium Verbs Provider MD on SSD completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-17345/3/testReport/

daosbuild3 · 2026-01-09T12:58:48Z

Test stage Unit Test bdev with memcheck on EL 8.8 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17345/5/display/redirect

daosbuild3 · 2026-01-09T14:08:41Z

Test stage NLT on EL 8.8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-17345/6/testReport/

liw reviewed Jan 7, 2026

View reviewed changes

kccain added 2 commits January 7, 2026 18:00

Merge branch 'master' into kccain/daos_15993

f134ead

address preliminary review feedback by liwei.

162e478

Signed-off-by: Kenneth Cain <[email protected]>

DAOS-15993 rebuild: for manual rebuilds do not eval self_heal #17345

Are you sure you want to change the base?

DAOS-15993 rebuild: for manual rebuilds do not eval self_heal #17345

Conversation

kccain commented Jan 6, 2026

Steps for the author:

After all prior steps are complete:

Uh oh!

github-actions bot commented Jan 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

daosbuild3 commented Jan 6, 2026

Uh oh!

daosbuild3 commented Jan 6, 2026

Uh oh!

liw left a comment

Choose a reason for hiding this comment

Uh oh!

liw Jan 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

liw Jan 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

daosbuild3 commented Jan 7, 2026

Uh oh!

daosbuild3 commented Jan 7, 2026

Uh oh!

daosbuild3 commented Jan 7, 2026

Uh oh!

daosbuild3 commented Jan 8, 2026

Uh oh!

daosbuild3 commented Jan 9, 2026

Uh oh!

daosbuild3 commented Jan 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

4 participants

github-actions bot commented Jan 6, 2026 •

edited

Loading

liw Jan 7, 2026 •

edited

Loading

liw Jan 7, 2026 •

edited

Loading