Module graceful shutdown support #255

rameshraghupathy · 2025-05-14T01:56:43Z

Provide support for SmartSwitch DPU module graceful shutdown.

Description:

Single source of truth for transitions
- All components now use sonic_platform_base.module_base.ModuleBase helpers:
  - set_module_state_transition(db, name, transition_type)
  - clear_module_state_transition(db, name)
  - get_module_state_transition(db, name) -> dict
  - is_module_state_transition_timed_out(db, name, timeout_secs) -> bool
- Eliminates duplicated logic and race-prone direct Redis writes.
Correct table everywhere
- Standardized on CHASSIS_MODULE_TABLE (replaces CHASSIS_MODULE_INFO_TABLE).
- HLD mismatch addressed in code (HLD fix tracked separately).
Ownership & lifecycle
- The initiator of an operation (startup/shutdown/reboot) sets:
  - state_transition_in_progress=True
  - transition_type=<op>
  - transition_start_time=<utc-iso8601>
- The platform (set_admin_state()) is responsible for clearing:
  - state_transition_in_progress=False
  - optionally transition_end_time=<epoch> (or similar end stamp).
- CLI pre-clears only when a prior transition is timed out.
Timeouts & policy
- Platform JSON path only: /usr/share/sonic/device/{plat}/platform.json; else constants.
- Typical production values used:
  - startup: 180s, shutdown: 180s (≈ graceful_wait 60s + power 120s), reboot: 120s.
- Graceful wait (e.g., waiting for “Graceful shutdown complete”) is a platform policy and implemented inside platform set_admin_state()—not in ModuleBase.
Boot behavior
- chassisd on start:
  1. Clears stale flags once (centralized sweep).
  2. Runs set_initial_dpu_admin_state() which marks transitions via ModuleBase before calling platform set_admin_state().
  3. Leaves clearing to the platform or to well-defined status transitions (ONLINE/OFFLINE) where appropriate.
gNOI shutdown daemon
- Listens on CHASSIS_MODULE_TABLE and triggers only when:
  - state_transition_in_progress=True and transition_type=shutdown.
- Never clears the flag (ownership stays with the platform).
- Bounded RPC timeouts and robust Redis access (swsssdk/swsscommon).
CLI (config chassis modules …)
- Uses ModuleBase APIs for all set/get/timeout checks.
- If a previous transition is stuck, is_module_state_transition_timed_out() → auto-clear then proceed.
- Sets transition at the start of startup/shutdown; platform clears on completion.
- Fabric card flow retained; edits are surgical.
Redis robustness
- Helpers handle both stacks (swsssdk/swsscommon); no hset(mapping=...) usage.
- Consistent HGETALL/HSET paths; resilient to connector differences.
Race reduction & consistency
- Centralized writes prevent multi-writer races.
- All transition writes include transition_start_time; clears may add an end stamp.
- Existing PCI/file-lock logic left intact; unrelated behavior unchanged.
Change scope
- Minimal, targeted diffs.
- No background tasks added, no broad refactors beyond transition handling.
- Behavior changes are limited to making transition semantics correct and uniform across repos.

HLD: # 1991 sonic-net/SONiC#1991
sonic-platform-common: #567 sonic-net/sonic-platform-common#567
sonic-utilities: sonic-net/sonic-utilities#4031
sonic-platform-daemons: sonic-net/sonic-platform-daemons#667

How to verify it
Issue the "config chassis modules shutdown DPUx" command
Verify the DPU module is gracefully shut by checking the logs in /var/log/syslog on both NPU and DPU

mssonicbld · 2025-05-14T01:56:51Z

/azp run

azure-pipelines · 2025-05-14T01:57:00Z

Azure Pipelines successfully started running 1 pipeline(s).

hdwhdw · 2025-05-19T20:36:20Z

Do you mind pasting the steps and output for testing (commands) in the PR description

data/debian/sonic-host-services-data.gnoi-reboot.service

scripts/gnoi-reboot-daemon

mssonicbld · 2025-05-21T16:34:52Z

/azp run

azure-pipelines · 2025-05-21T16:35:01Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2025-05-21T17:28:44Z

/azp run

azure-pipelines · 2025-05-21T17:28:53Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2025-05-21T17:34:20Z

/azp run

azure-pipelines · 2025-05-21T17:34:30Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2025-05-21T17:46:58Z

/azp run

azure-pipelines · 2025-05-21T17:47:07Z

Azure Pipelines successfully started running 1 pipeline(s).

scripts/check_platform.sh

scripts/gnoi-reboot-daemon

mssonicbld · 2025-07-07T17:09:26Z

/azp run

azure-pipelines · 2025-07-07T17:09:34Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2025-07-10T17:40:18Z

/azp run

azure-pipelines · 2025-07-10T17:40:27Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2025-11-19T20:11:57Z

/azp run

azure-pipelines · 2025-11-19T20:12:07Z

Azure Pipelines successfully started running 1 pipeline(s).

vvolam

LGTM, Thank you

mssonicbld · 2025-11-19T23:11:04Z

/azp run

azure-pipelines · 2025-11-19T23:11:14Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2025-11-19T23:25:28Z

/azp run

azure-pipelines · 2025-11-19T23:25:37Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2025-11-20T19:56:24Z

/azp run

azure-pipelines · 2025-11-20T19:56:34Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2025-11-20T21:13:54Z

/azp run

azure-pipelines · 2025-11-20T21:14:04Z

Azure Pipelines successfully started running 1 pipeline(s).

saiarcot895 · 2025-11-25T00:14:48Z

@rameshraghupathy Submodule update is blocked due to loganalyzer failures resulting from this PR, I think. Please take a look and help fix:

Blocked submodule update: sonic-net/sonic-buildimage#24404

E               Failed: Got matched syslog in processes "analyze_logs--<MultiAsicSonicHost vlab-01>" exit code:"1"
E               match: 2
E               expected_match: 0
E               expected_missing_match: 0
E               
E               Match Messages:
E               2025 Nov 24 10:26:30.265154 vlab-01 ERR systemd[1]: Failed to start gnoi-shutdown.service - gNOI based DPU Graceful Shutdown Daemon.
E               
E               2025 Nov 24 10:26:35.492174 vlab-01 ERR systemd[1]: Failed to start gnoi-shutdown.service - gNOI based DPU Graceful Shutdown Daemon.

vvolam · 2025-11-25T04:42:08Z

@rameshraghupathy Here is more information on the error path

2025 Nov 24 10:22:21.080953 vlab-01 INFO systemd[1]: gnoi-shutdown.service: Scheduled restart job, restart counter is at 203.
2025 Nov 24 10:22:21.081472 vlab-01 INFO systemd[1]: Stopped gnoi-shutdown.service - gNOI based DPU Graceful Shutdown Daemon.
2025 Nov 24 10:22:21.097836 vlab-01 INFO systemd[1]: Starting gnoi-shutdown.service - gNOI based DPU Graceful Shutdown Daemon...
2025 Nov 24 10:22:21.201959 vlab-01 NOTICE systemd[1]: gnoi-shutdown.service: Control process exited, code=exited, status=1/FAILURE
2025 Nov 24 10:22:21.202231 vlab-01 WARNING systemd[1]: gnoi-shutdown.service: Failed with result 'exit-code'.
2025 Nov 24 10:22:21.202411 vlab-01 ERR systemd[1]: Failed to start gnoi-shutdown.service - gNOI based DPU Graceful Shutdown Daemon.

mssonicbld · 2025-12-03T21:21:07Z

Cherry-pick PR to 202511: #324

…nic-net#255 Why I did it The gNOI shutdown daemon service was causing loganalyzer test failures on non-SmartSwitch platforms (e.g., vlab-01). The service attempted to start via ExecStartPre=/usr/local/bin/check_platform.py, which exited with code 1 on incompatible platforms. This caused systemd to log ERROR messages like: ERR systemd[1]: Failed to start gnoi-shutdown.service - gNOI based DPU Graceful Shutdown Daemon These errors blocked CI/CD submodule updates due to loganalyzer failures. How I did it Changed the service file to use ExecCondition= instead of ExecStartPre= for platform checking: ExecCondition=/usr/bin/python3 /usr/local/bin/check_platform.py runs before service start When check_platform.py returns exit code 1 on non-SmartSwitch platforms, systemd treats this as a condition not met rather than a failure Service is gracefully skipped without error logs on incompatible platforms Changed Restart=always to Restart=on-failure to avoid unnecessary restart attempts when conditions aren't met How to verify it On SmartSwitch NPU platform: Service starts normally and handles DPU graceful shutdown sonic-net/sonic-buildimage#24609 is run with this change Which release branch to backport [x]202511

… (#333) Why I did it The gNOI shutdown daemon service was causing loganalyzer test failures on non-SmartSwitch platforms (e.g., vlab-01). The service attempted to start via ExecStartPre=/usr/local/bin/check_platform.py, which exited with code 1 on incompatible platforms. This caused systemd to log ERROR messages like: ERR systemd[1]: Failed to start gnoi-shutdown.service - gNOI based DPU Graceful Shutdown Daemon These errors blocked CI/CD submodule updates due to loganalyzer failures. How I did it Changed the service file to use ExecCondition= instead of ExecStartPre= for platform checking: ExecCondition=/usr/bin/python3 /usr/local/bin/check_platform.py runs before service start When check_platform.py returns exit code 1 on non-SmartSwitch platforms, systemd treats this as a condition not met rather than a failure Service is gracefully skipped without error logs on incompatible platforms Changed Restart=always to Restart=on-failure to avoid unnecessary restart attempts when conditions aren't met How to verify it On SmartSwitch NPU platform: Service starts normally and handles DPU graceful shutdown sonic-net/sonic-buildimage#24609 is run with this change Which release branch to backport [x]202511

oleksandrivantsiv · 2026-01-16T22:11:14Z

scripts/wait-for-sonic-core.sh

+# Hard dep we expect to be up before we start: swss
+if ! systemctl is-active --quiet swss.service; then
+  log "Waiting for swss.service to become active…"
+  systemctl --no-pager --full status swss.service || true


@vvolam, @rameshraghupathy, @qiluo-msft
Why is this needed? Why don’t we use dependencies in the data/debian/sonic-host-services-data.gnoi-shutdown.service file? Why do we need to create an entire script for something that can be handled directly in the service file?

[Unit] Description=gNOI based DPU Graceful Shutdown Daemon Requires=database.service Wants=network-online.target After=network-online.target database.service swss.service gnmi.service pmon.service [Service] Type=simple ExecStartPre=/usr/bin/python3 /usr/local/bin/check_platform.py ExecStartPre=/bin/bash /usr/local/bin/wait-for-sonic-core.sh ExecStart=/usr/bin/python3 /usr/local/bin/gnoi_shutdown_daemon.py Restart=always RestartSec=5 [Install] WantedBy=multi-user.target

Did the instrumentation for gnoi-reboot.service

4e3a096

This was referenced May 14, 2025

Module graceful shutdown support sonic-net/sonic-platform-common#567

Open

Module graceful shutdown support sonic-net/sonic-buildimage#22611

Draft

hdwhdw requested review from hdwhdw and vvolam May 19, 2025 20:33

vvolam reviewed May 20, 2025

View reviewed changes

data/debian/sonic-host-services-data.gnoi-reboot.service Outdated Show resolved Hide resolved

scripts/gnoi-reboot-daemon Outdated Show resolved Hide resolved

scripts/gnoi-reboot-daemon Outdated Show resolved Hide resolved

scripts/gnoi-reboot-daemon Outdated Show resolved Hide resolved

rameshraghupathy mentioned this pull request May 21, 2025

HLD for SmsrtSwitch DPU graceful shutdown sonic-net/SONiC#1991

Merged

Modified based on the Redis based IPC

4a7e6bf

Modified based on the Redis based IPC

c2f9cb8

made check_platform.sh executable

db7848f

Did some cleanup

f946e72

vvolam reviewed May 22, 2025

View reviewed changes

scripts/check_platform.sh Outdated Show resolved Hide resolved

scripts/gnoi-reboot-daemon Outdated Show resolved Hide resolved

hdwhdw reviewed May 23, 2025

View reviewed changes

scripts/gnoi-reboot-daemon Outdated Show resolved Hide resolved

scripts/gnoi-reboot-daemon Outdated Show resolved Hide resolved

Draft version. Need to test again

4434463

Fixing test failure

91897ed

return early when gnoi command fails

9c3bf08

vvolam approved these changes Nov 19, 2025

View reviewed changes

added specific exception type

9551a7f

Addressed review comments

dd8fe58

Addressed review comments

8d1358e

Addressed review comments

13592f4

vvolam added the Request for 202511 Branch label Nov 21, 2025

qiluo-msft approved these changes Nov 21, 2025

View reviewed changes

qiluo-msft merged commit ee9ef3f into sonic-net:master Nov 21, 2025
5 checks passed

vmittal-msft added the Approved for 202511 Branch label Dec 3, 2025

mssonicbld mentioned this pull request Dec 3, 2025

[action] [PR:255] Module graceful shutdown support #324

Merged

mssonicbld added the Created PR to 202511 Branch label Dec 3, 2025

mssonicbld added Included in 202511 Branch and removed Created PR to 202511 Branch labels Dec 3, 2025

oleksandrivantsiv reviewed Jan 16, 2026

View reviewed changes

Module graceful shutdown support #255

Module graceful shutdown support #255

Uh oh!

Conversation

rameshraghupathy commented May 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description:

Uh oh!

mssonicbld commented May 14, 2025

Uh oh!

azure-pipelines bot commented May 14, 2025

Uh oh!

hdwhdw commented May 19, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mssonicbld commented May 21, 2025

Uh oh!

azure-pipelines bot commented May 21, 2025

Uh oh!

mssonicbld commented May 21, 2025

Uh oh!

azure-pipelines bot commented May 21, 2025

Uh oh!

mssonicbld commented May 21, 2025

Uh oh!

azure-pipelines bot commented May 21, 2025

Uh oh!

mssonicbld commented May 21, 2025

Uh oh!

azure-pipelines bot commented May 21, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mssonicbld commented Jul 7, 2025

Uh oh!

azure-pipelines bot commented Jul 7, 2025

Uh oh!

mssonicbld commented Jul 10, 2025

Uh oh!

azure-pipelines bot commented Jul 10, 2025

Uh oh!

mssonicbld commented Nov 19, 2025

Uh oh!

azure-pipelines bot commented Nov 19, 2025

Uh oh!

vvolam left a comment

Choose a reason for hiding this comment

Uh oh!

mssonicbld commented Nov 19, 2025

Uh oh!

azure-pipelines bot commented Nov 19, 2025

Uh oh!

mssonicbld commented Nov 19, 2025

Uh oh!

azure-pipelines bot commented Nov 19, 2025

Uh oh!

mssonicbld commented Nov 20, 2025

Uh oh!

azure-pipelines bot commented Nov 20, 2025

Uh oh!

mssonicbld commented Nov 20, 2025

Uh oh!

azure-pipelines bot commented Nov 20, 2025

Uh oh!

Uh oh!

saiarcot895 commented Nov 25, 2025

Uh oh!

vvolam commented Nov 25, 2025

Uh oh!

mssonicbld commented Dec 3, 2025

Uh oh!

oleksandrivantsiv Jan 16, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

rameshraghupathy commented May 14, 2025 •

edited

Loading