DAOS-18347 control: Add rebuild states to pool query #17322

tanabarr · 2025-12-29T05:40:55Z

Add intermediate "derived" rebuild state field to indicate temporal
pool rebuild conditions. Preserve rebuild state value (idle/done/busy)
whilst adding intermediate states in derived_state field
(stopped/stopping/failed/failing) to better inform administrator.

Features: control

Steps for the author:

Commit message follows the guidelines.
Appropriate Features or Test-tag pragmas were used.
Appropriate Functional Test Stages were run.
At least two positive code reviews including at least one code owner from each category referenced in the PR.
Testing is complete. If necessary, forced-landing label added and a reason added in a comment.

After all prior steps are complete:

Gatekeeper requested (daos-gatekeeper added as a reviewer).

github-actions · 2025-12-29T05:41:20Z

Ticket title is 'Rebuild state reported in pool query human-readable output needs refinement'
Status is 'In Review'
Labels: 'scrubbed_2.8'
https://daosio.atlassian.net/browse/DAOS-18347

daosbuild3 · 2025-12-29T05:47:34Z

Test stage Build on Leap 15.5 with Intel-C and TARGET_PREFIX completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17322/1/execution/node/301/log

daosbuild3 · 2025-12-29T05:48:01Z

Test stage Build on EL 9.6 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17322/1/execution/node/309/log

daosbuild3 · 2025-12-29T05:48:02Z

Test stage Build on EL 8.8 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17322/1/execution/node/317/log

daosbuild3 · 2025-12-29T05:55:47Z

Test stage Build on Leap 15.5 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17322/1/execution/node/405/log

Features: control Signed-off-by: Tom Nabarro <[email protected]>

daosbuild3 · 2025-12-29T13:36:31Z

Test stage Build on Leap 15.5 with Intel-C and TARGET_PREFIX completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17322/2/execution/node/301/log

daosbuild3 · 2025-12-29T13:37:02Z

Test stage Build on EL 8.8 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17322/2/execution/node/317/log

daosbuild3 · 2025-12-29T13:37:11Z

Test stage Build on EL 9.6 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17322/2/execution/node/311/log

daosbuild3 · 2025-12-29T13:44:25Z

Test stage Build on Leap 15.5 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17322/2/execution/node/405/log

Features: control Signed-off-by: Tom Nabarro <[email protected]>

coverage Features: control Signed-off-by: Tom Nabarro <[email protected]>

…build-states Features: control Signed-off-by: Tom Nabarro <[email protected]>

daosbuild3 · 2026-01-06T13:20:25Z

Test stage NLT on EL 8.8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-17322/5/testReport/

kccain

I thought I had submitted this feedback earlier, but github still shows it as pending. Trying again. Sorry for the inadvertent delay.

kccain · 2026-01-02T15:31:50Z

src/proto/mgmt/pool.proto

+		FAILING  = 5;
+		FAILED   = 6;
 	}
 	State state = 2;


To avoid re-assigning the official status/state before output of human-readable or JSON (i.e., to always have alignment with what a libdaos API caller would see), a solution could be to add here a derived_state used only by the tools. Both human-readable and JSON output could consistently show all 3 values (e.g., "derived_state (state, status)".

Regardless of whether we adopt the above suggestion, let's inform @jamesanunez and @daltonbohning of the potential changes here in case functional testing (that uses dmg/daos pool query mostly, and less-so the libdaos API).

In the proposal, state would only ever be busy/idle/done and would never be translated to a new value. status (errno) would similarly never be manipulated. It will have that special -DER_OP_CANCELED value for the stopping/stopped conditions and it would always be presented to the caller of the dmg/daos pool query utilities.

derived_state could take on any of the above State values (busy/idle/done if there is no derived condition such as stopping or stopped or failing or failed. If there is a further derived condition, derived_state could take on one of the new State values to provide the qualifying detail. In that latter case, stopping and failing are considered modifiers to the busy state, stopped is a modifier to idle, and failed is a modifier to done.

@kccain @daltonbohning changes applied, please review and verify when you can. TIA

So if state and status are never manipulated then this seems to be backward compatible with existing tests. Using the new derived_state probably should be a separate PR because the rebuild code in ftest that detects that is pretty messy currently.

kccain · 2026-01-02T15:32:09Z

src/proto/mgmt/pool.proto

+		IDLE     = 1;
+		DONE     = 2;
+		STOPPING = 3;
+		StOPPED  = 4;


(change to all upper case letters)

Suggested change

StOPPED = 4;

STOPPED = 4;

Signed-off-by: Tom Nabarro <[email protected]>

Features: control Signed-off-by: Tom Nabarro <[email protected]>

…build-states Features: control Signed-off-by: Tom Nabarro <[email protected]>

daosbuild3 · 2026-01-12T12:55:27Z

Test stage NLT on EL 8.8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-17322/7/testReport/

daosbuild3 · 2026-01-12T17:28:26Z

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17322/7/display/redirect

daosbuild3 · 2026-01-12T17:28:35Z

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17322/7/display/redirect

daosbuild3 · 2026-01-12T17:28:50Z

Test stage Functional Hardware Large MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17322/7/display/redirect

daosbuild3 · 2026-01-12T17:28:56Z

Test stage Functional Hardware Medium Verbs Provider MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17322/7/display/redirect

daosbuild3 · 2026-01-12T19:13:06Z

Test stage Functional Hardware Large MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17322/8/execution/node/507/log

daosbuild3 · 2026-01-12T23:32:52Z

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17322/8/execution/node/482/log

daosbuild3 · 2026-01-12T23:35:10Z

Test stage Functional Hardware Medium Verbs Provider MD on SSD completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-17322/8/testReport/

…build-states Features: control Signed-off-by: Tom Nabarro <[email protected]>

daosbuild3 · 2026-01-14T01:16:19Z

Test stage NLT on EL 8.8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-17322/9/testReport/

daosbuild3 · 2026-01-14T13:20:31Z

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17322/9/execution/node/1282/log

tanabarr · 2026-01-14T15:10:01Z

@daltonbohning I need to make some test changes to adjust for the addition of derived_state (https://jenkins.daos.hpc.amslabs.hpecorp.net/blue/organizations/jenkins/daos-stack%2Fdaos/detail/PR-17322/9/tests). Any idea whether the necessary changes will be in many multiple places or whether it will just be a few helper functions?

daltonbohning · 2026-01-14T15:50:55Z

@daltonbohning I need to make some test changes to adjust for the addition of derived_state (https://jenkins.daos.hpc.amslabs.hpecorp.net/blue/organizations/jenkins/daos-stack%2Fdaos/detail/PR-17322/9/tests). Any idea whether the necessary changes will be in many multiple places or whether it will just be a few helper functions?

For control/dmg_pool_query_test.py you can update these expected values:

daos/src/tests/ftest/control/dmg_pool_query_test.py

Line 66 in 8a24f84

exp_info = {

Most of the tests are failing with errors like this that I don't understand

2026/01/14 12:31:45 DEBUG                log_result_data:   opa-[102-103] (rc=1): find: \x2018/mnt/share/tmp.lwK2QyOveh\x2019: No such file or directory
2026/01/14 12:31:45 ERROR                      fail_test: Error determining if /mnt/share/tmp.lwK2QyOveh/valgrind* files exist on opa-[102-103]

Features: control Signed-off-by: Tom Nabarro <[email protected]>

…build-states Features: control Signed-off-by: Tom Nabarro <[email protected]>

daltonbohning

ftest LGTM

daosbuild3 · 2026-01-16T22:00:00Z

Test stage NLT on EL 8.8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-17322/11/testReport/

tanabarr · 2026-01-18T13:54:04Z

awaiting reviews

knard38 · 2026-01-19T09:28:11Z

Nice update which should be very helpful for support tasks 👍

tanabarr requested a review from kccain December 29, 2025 05:40

tanabarr self-assigned this Dec 29, 2025

tanabarr requested review from a team as code owners December 29, 2025 05:40

DAOS-18347 control: Add rebuild states to pool query

61cc7aa

Features: control Signed-off-by: Tom Nabarro <[email protected]>

tanabarr force-pushed the tanabarr/control-rebuild-states branch from 6eae49c to 61cc7aa Compare December 29, 2025 13:30

tanabarr added 3 commits January 5, 2026 15:14

adjust states before drpc response

0c5727e

Features: control Signed-off-by: Tom Nabarro <[email protected]>

update state definitions in src/control/lib/daos, increase code

1e079cd

coverage Features: control Signed-off-by: Tom Nabarro <[email protected]>

Merge remote-tracking branch 'origin/master' into tanabarr/control-re…

038e0d0

…build-states Features: control Signed-off-by: Tom Nabarro <[email protected]>

kccain reviewed Jan 6, 2026

View reviewed changes

tanabarr added 3 commits January 6, 2026 14:55

minor test corrections

1046349

Signed-off-by: Tom Nabarro <[email protected]>

add derived_state to pool rebuild status

c006389

Features: control Signed-off-by: Tom Nabarro <[email protected]>

Merge remote-tracking branch 'origin/master' into tanabarr/control-re…

b509a41

…build-states Features: control Signed-off-by: Tom Nabarro <[email protected]>

tanabarr requested review from kccain, kjacque and knard38 January 12, 2026 12:05

Merge remote-tracking branch 'origin/master' into tanabarr/control-re…

71f5621

…build-states Features: control Signed-off-by: Tom Nabarro <[email protected]>

add derived_state in dmg_pool_query_test.py

336aee5

Features: control Signed-off-by: Tom Nabarro <[email protected]>

tanabarr requested review from a team as code owners January 16, 2026 17:28

Merge remote-tracking branch 'origin/master' into tanabarr/control-re…

3644daf

…build-states Features: control Signed-off-by: Tom Nabarro <[email protected]>

daltonbohning reviewed Jan 16, 2026

View reviewed changes

tanabarr requested a review from daltonbohning January 18, 2026 13:53

knard38 approved these changes Jan 19, 2026

View reviewed changes

DAOS-18347 control: Add rebuild states to pool query #17322

Are you sure you want to change the base?

DAOS-18347 control: Add rebuild states to pool query #17322

Conversation

tanabarr commented Dec 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Steps for the author:

After all prior steps are complete:

Uh oh!

github-actions bot commented Dec 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

daosbuild3 commented Dec 29, 2025

Uh oh!

daosbuild3 commented Dec 29, 2025

Uh oh!

daosbuild3 commented Dec 29, 2025

Uh oh!

daosbuild3 commented Dec 29, 2025

Uh oh!

daosbuild3 commented Dec 29, 2025

Uh oh!

daosbuild3 commented Dec 29, 2025

Uh oh!

daosbuild3 commented Dec 29, 2025

Uh oh!

daosbuild3 commented Dec 29, 2025

Uh oh!

daosbuild3 commented Jan 6, 2026

Uh oh!

kccain left a comment

Choose a reason for hiding this comment

Uh oh!

kccain Jan 2, 2026

Choose a reason for hiding this comment

Uh oh!

tanabarr Jan 12, 2026

Choose a reason for hiding this comment

Uh oh!

daltonbohning Jan 12, 2026

Choose a reason for hiding this comment

Uh oh!

kccain Jan 2, 2026

Choose a reason for hiding this comment

Uh oh!

daosbuild3 commented Jan 12, 2026

Uh oh!

daosbuild3 commented Jan 12, 2026

Uh oh!

daosbuild3 commented Jan 12, 2026

Uh oh!

daosbuild3 commented Jan 12, 2026

Uh oh!

daosbuild3 commented Jan 12, 2026

Uh oh!

daosbuild3 commented Jan 12, 2026

Uh oh!

daosbuild3 commented Jan 12, 2026

Uh oh!

daosbuild3 commented Jan 12, 2026

Uh oh!

daosbuild3 commented Jan 14, 2026

Uh oh!

daosbuild3 commented Jan 14, 2026

Uh oh!

tanabarr commented Jan 14, 2026

Uh oh!

daltonbohning commented Jan 14, 2026

Uh oh!

daltonbohning left a comment

Choose a reason for hiding this comment

Uh oh!

daosbuild3 commented Jan 16, 2026

Uh oh!

tanabarr commented Jan 18, 2026

Uh oh!

knard38 commented Jan 19, 2026

Uh oh!

Reviewers

Assignees

tanabarr commented Dec 29, 2025 •

edited

Loading

github-actions bot commented Dec 29, 2025 •

edited

Loading