DAOS-18431 bio: Set power management register on NVMe #17355
Conversation
Signed-off-by: Tom Nabarro <[email protected]>
|
Ticket title is 'NVMe power control feature' |
|
Test stage NLT on EL 8.8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-17355/1/testReport/ |
tanabarr
force-pushed
the
tanabarr/bio-powmanage-patch
branch
from
5e0062d to
0be43f7
Compare
Signed-off-by: Tom Nabarro <[email protected]>
tanabarr
force-pushed
the
tanabarr/bio-powmanage-patch
branch
from
0be43f7 to
8b31c49
Compare
kjacque
left a comment
kjacque
left a comment
There was a problem hiding this comment.
Overall looks reasonable. Just trying to understand the right way to interact with the spdk stuff.
If we're letting users configure this generally, it might be good to add an explicit parameter to the config file with more intuitively named values, based on whatever the 0-4 stand for. Not necessary in this PR, but something to think about.
| if (get_bdev_type(bdev) != BDEV_CLASS_NVME) { | ||
| D_DEBUG(DB_MGMT, "Device %s is not NVMe, skipping power management\n", | ||
| d_bdev->bb_name); | ||
| return 0; | ||
| } |
There was a problem hiding this comment.
Seems like returning an error like DER_NOTSUPPORTED might be meaningful here. Are we calling this for simulated devices?
| set_power_mgmt_completion(struct spdk_bdev_io *bdev_io, bool success, void *cb_arg) | ||
| { | ||
| struct bio_bdev *d_bdev = cb_arg; | ||
| int sc, sct; |
There was a problem hiding this comment.
nit - in our coding guidelines we mention defining 1 variable per line.
| D_ASSERT(channel != NULL); | ||
| bb->bb_dev_health.bdh_io_channel = channel; | ||
|
|
||
| /* Set NVMe power management to 0x1 */ |
There was a problem hiding this comment.
minor - looks like the value is set by an environment variable.
| /* Set NVMe power management to 0x1 */ | |
| /* Set up NVMe power management */ |
| memset(&cmd, 0, sizeof(cmd)); | ||
| cmd.opc = SPDK_NVME_OPC_SET_FEATURES; | ||
| cmd.cdw10 = SPDK_NVME_FEAT_POWER_MANAGEMENT; | ||
| cmd.cdw11 = bio_spdk_power_mgmt_val; |
There was a problem hiding this comment.
union spdk_nvme_cmd_cdw11 has a member union spdk_nvme_feat_power_management feat_power_management that has multiple fields, not just the power state. So I think we should be using that to make sure we're setting the value correctly.
IMO we should also check the value taken from the env variable is in an acceptable range, preferably when we first ingest it. I had some trouble finding definitions for the power state values. Does SPDK define those somewhere?
| d_getenv_uint("DAOS_NVME_POWER_MGMT", &bio_spdk_power_mgmt_val); | ||
| if (bio_spdk_power_mgmt_val != UINT32_MAX) | ||
| D_INFO("NVMe power management setting to be applied is %u\n", | ||
| bio_spdk_power_mgmt_val); |
There was a problem hiding this comment.
So if I understand the flow correctly:
- Environment variable is set (probably in the daos_server config file).
- Daos_engine comes up and this function saves off the environment variable value.
- In
bio_init_health_monitoring, we apply this value, or do nothing if it was unset.
Do we know for sure that the value doesn't persist in the drives between restarts of the daos_engine? Obviously the engine is not maintaining the value, I understand that, but do the drives reset this kind of thing themselves every time an engine restarts?
I imagine a case where someone stops the system, unsets the environment variable, and restarts. Do the drives have the same power state as before the restart, or do they auto-reset to the default without DAOS code doing anything? If not the latter, maybe we should define a sane default.
Set NVMe power management values for SSDs by setting the new
engine DAOS_NVME_POWER_MGMT environment variable to an integer
normally (0-4). Value will be applied by SPDK on devices attached to
an engine process. The value will not be reset on engine exit.
Steps for the author:
After all prior steps are complete: