feat: Automatic NVOS image updates with restart-safe credential recovery

[jayzhudev]

Is this a new feature, an enhancement, or a change to existing functionality?

New Feature

How would you describe the priority of this feature request

Medium

Please provide a clear description of problem this feature solves

Extend automatic rack maintenance to update switch NVOS images using the SOT document configured by RackProfile.firmware_object.

NICo already supports explicit NVOS maintenance requests. The new feature adds automatic NVOS updates to the maintenance sequence and makes post-update credential recovery resumable across service restarts.

Feature Description

Extend automatic rack maintenance to update switch NVOS images using the SOT document configured by RackProfile.firmware_object.

The automatic maintenance sequence becomes:

FirmwareUpgrade -> NVOSUpdate -> ConfigureNmxCluster

The feature also verifies or restores each switch's desired NVOS admin credential after the image operation and preserves recovery progress across NICo and RMS restarts.

NVOS updates

Automatic rack maintenance uses the default full-rack scope. When the rack profile defines firmware_object, NICo:

• Fetches the configured SOT URL separately for the firmware and NVOS phases.
• Submits the complete SOT document to RMS with NOAUTH.
• Updates every switch in the rack.
• Lets RMS select the matching switch system image from the SOT document.
• Persists the RMS parent and per-switch job IDs.
• Reports progress through the existing per-switch NVOS status.

Credential recovery

An NVOS image update may change the switch admin credential. After all image jobs complete or fail, NICo submits UpdateSwitchSystemPassword for each switch in the NVOS job to verify or restore its desired SwitchNvosAdmin credential.

NICo:

• Persists each password job ID and state.
• Resumes password polling after a NICo restart.
• Resubmits a password job reported as failed, missing, or unknown.
• Runs credential recovery after both successful and failed image jobs.
• Keeps the NVOS phase in progress until every password job completes.
• Advances maintenance only when all image jobs succeeded.
• Transitions the rack to Error after credential recovery when any image job failed.

If RMS loses an image job, NICo records the image job as failed and does not submit another image update because the applied-image state is unknown. Credential recovery still runs before NICo reports the failure.

Describe your ideal solution

No response

Describe any alternatives you have considered

No response

Additional context

No response

Code of Conduct

• I agree to follow NVIDIA Infra Controller's Code of Conduct
• I have searched the open feature requests and have found no duplicates for this feature request