Module graceful shutdown support #255

rameshraghupathy · 2025-05-14T01:56:43Z

Provide support for SmartSwitch DPU module graceful shutdown.

Description:

Single source of truth for transitions
- All components now use sonic_platform_base.module_base.ModuleBase helpers:
  - set_module_state_transition(db, name, transition_type)
  - clear_module_state_transition(db, name)
  - get_module_state_transition(db, name) -> dict
  - is_module_state_transition_timed_out(db, name, timeout_secs) -> bool
- Eliminates duplicated logic and race-prone direct Redis writes.
Correct table everywhere
- Standardized on CHASSIS_MODULE_TABLE (replaces CHASSIS_MODULE_INFO_TABLE).
- HLD mismatch addressed in code (HLD fix tracked separately).
Ownership & lifecycle
- The initiator of an operation (startup/shutdown/reboot) sets:
  - state_transition_in_progress=True
  - transition_type=<op>
  - transition_start_time=<utc-iso8601>
- The platform (set_admin_state()) is responsible for clearing:
  - state_transition_in_progress=False
  - optionally transition_end_time=<epoch> (or similar end stamp).
- CLI pre-clears only when a prior transition is timed out.
Timeouts & policy
- Platform JSON path only: /usr/share/sonic/device/{plat}/platform.json; else constants.
- Typical production values used:
  - startup: 180s, shutdown: 180s (≈ graceful_wait 60s + power 120s), reboot: 120s.
- Graceful wait (e.g., waiting for “Graceful shutdown complete”) is a platform policy and implemented inside platform set_admin_state()—not in ModuleBase.
Boot behavior
- chassisd on start:
  1. Clears stale flags once (centralized sweep).
  2. Runs set_initial_dpu_admin_state() which marks transitions via ModuleBase before calling platform set_admin_state().
  3. Leaves clearing to the platform or to well-defined status transitions (ONLINE/OFFLINE) where appropriate.
gNOI shutdown daemon
- Listens on CHASSIS_MODULE_TABLE and triggers only when:
  - state_transition_in_progress=True and transition_type=shutdown.
- Never clears the flag (ownership stays with the platform).
- Bounded RPC timeouts and robust Redis access (swsssdk/swsscommon).
CLI (config chassis modules …)
- Uses ModuleBase APIs for all set/get/timeout checks.
- If a previous transition is stuck, is_module_state_transition_timed_out() → auto-clear then proceed.
- Sets transition at the start of startup/shutdown; platform clears on completion.
- Fabric card flow retained; edits are surgical.
Redis robustness
- Helpers handle both stacks (swsssdk/swsscommon); no hset(mapping=...) usage.
- Consistent HGETALL/HSET paths; resilient to connector differences.
Race reduction & consistency
- Centralized writes prevent multi-writer races.
- All transition writes include transition_start_time; clears may add an end stamp.
- Existing PCI/file-lock logic left intact; unrelated behavior unchanged.
Change scope
- Minimal, targeted diffs.
- No background tasks added, no broad refactors beyond transition handling.
- Behavior changes are limited to making transition semantics correct and uniform across repos.

HLD: # 1991 sonic-net/SONiC#1991
sonic-platform-common: #567 sonic-net/sonic-platform-common#567
sonic-utilities: sonic-net/sonic-utilities#4031
sonic-platform-daemons: sonic-net/sonic-platform-daemons#667

How to verify it
Issue the "config chassis modules shutdown DPUx" command
Verify the DPU module is gracefully shut by checking the logs in /var/log/syslog on both NPU and DPU

mssonicbld · 2025-05-14T01:56:51Z

/azp run

azure-pipelines · 2025-05-14T01:57:00Z

Azure Pipelines successfully started running 1 pipeline(s).

hdwhdw · 2025-05-19T20:36:20Z

Do you mind pasting the steps and output for testing (commands) in the PR description

data/debian/sonic-host-services-data.gnoi-reboot.service

scripts/gnoi-reboot-daemon

mssonicbld · 2025-05-21T16:34:52Z

/azp run

azure-pipelines · 2025-05-21T16:35:01Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2025-05-21T17:28:44Z

/azp run

azure-pipelines · 2025-05-21T17:28:53Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2025-05-21T17:34:20Z

/azp run

azure-pipelines · 2025-05-21T17:34:30Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2025-05-21T17:46:58Z

/azp run

azure-pipelines · 2025-05-21T17:47:07Z

Azure Pipelines successfully started running 1 pipeline(s).

scripts/check_platform.sh

scripts/gnoi-reboot-daemon

mssonicbld · 2025-07-07T17:09:26Z

/azp run

azure-pipelines · 2025-07-07T17:09:34Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2025-07-10T17:40:18Z

/azp run

azure-pipelines · 2025-07-10T17:40:27Z

Azure Pipelines successfully started running 1 pipeline(s).

Copilot

Pull Request Overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 15 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

scripts/gnoi_shutdown_daemon.py

scripts/wait-for-sonic-core.sh

scripts/gnoi_shutdown_daemon.py

data/debian/sonic-host-services-data.gnoi-shutdown.service

scripts/check_platform.py

scripts/wait-for-sonic-core.sh

scripts/check_platform.py

setup.py

scripts/gnoi_shutdown_daemon.py

Co-authored-by: Copilot <[email protected]>

mssonicbld · 2025-11-13T19:39:48Z

/azp run

azure-pipelines · 2025-11-13T19:39:58Z

Azure Pipelines successfully started running 1 pipeline(s).

Co-authored-by: Copilot <[email protected]>

mssonicbld · 2025-11-13T20:01:41Z

/azp run

azure-pipelines · 2025-11-13T20:01:51Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2025-11-13T20:22:40Z

/azp run

azure-pipelines · 2025-11-13T20:22:50Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2025-11-13T20:33:39Z

/azp run

azure-pipelines · 2025-11-13T20:33:50Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2025-11-14T00:00:28Z

/azp run

azure-pipelines · 2025-11-14T00:00:49Z

Azure Pipelines successfully started running 1 pipeline(s).

vvolam · 2025-11-14T18:07:15Z

@qiluo-msft could you please review this new service addition to sonic-host-services?

hdwhdw · 2025-11-14T23:19:22Z

scripts/check_platform.py

+            capture_output=True,
+            text=True,
+            timeout=5
+        )


Maybe use native python for this purpose?

from swsscommon import swsscommon config_db = swsscommon.ConfigDBConnector() config_db.connect() entry = config_db.get_entry('DEVICE_METADATA', 'localhost') subtype = entry.get('subtype') if entry else None If you need the exact same behavior as the subprocess (string output): from swsscommon import swsscommon config_db = swsscommon.ConfigDBConnector() config_db.connect() entry = config_db.get_entry('DEVICE_METADATA', 'localhost') result = entry.get('subtype', '') if entry else '' With error handling like the original subprocess: from swsscommon import swsscommon try: config_db = swsscommon.ConfigDBConnector() config_db.connect() entry = config_db.get_entry('DEVICE_METADATA', 'localhost') subtype = entry.get('subtype') if entry else None except Exception: subtype = None

@hdwhdw I have simplified it even further please take a look.

hdwhdw · 2025-11-14T23:20:57Z

scripts/gnoi_shutdown_daemon.py

+# gNOI helpers
+# ############
+
+def execute_gnoi_command(command_args, timeout_sec=REBOOT_RPC_TIMEOUT_SEC):


@rameshraghupathy in that case maybe at least rename the function to execute_command instead of execute_gnoi_command?

vvolam · 2025-11-14T23:24:58Z

setup.py

        'scripts/determine-reboot-cause',
        'scripts/process-reboot-cause',
+        'scripts/check_platform.py',
+        'scripts/wait-for-sonic-core.sh',


@rameshraghupathy why are we not copying gnoi_shutdown_daemon.py? Is the PR tested end to end? If yes, could you share the gnmi.logs with reboot call log samples?

@vvolum Looks like it is missed, thanks! Tested it locally, please find below the results.

gNOI halt request failing case: using DPU1

2025 Nov 15 09:04:11.933165 sonic NOTICE gnoi-shutdown-daemon[3931659]: DPU1: Admin shutdown detected, initiating gNOI HALT
2025 Nov 15 09:04:11.933340 sonic INFO python3[3931659]: NOTICE:gnoi-shutdown-daemon:DPU1: Admin shutdown detected, initiating gNOI HALT
2025 Nov 15 09:04:11.933439 sonic NOTICE gnoi-shutdown-daemon[3931659]: DPU1: Starting gNOI shutdown sequence
2025 Nov 15 09:04:11.933557 sonic INFO python3[3931659]: NOTICE:gnoi-shutdown-daemon:DPU1: Starting gNOI shutdown sequence
2025 Nov 15 09:04:12.352564 sonic NOTICE gnoi-shutdown-daemon[3931659]: DPU1: PCI detach complete, proceeding for halting services via gNOI
2025 Nov 15 09:04:12.352701 sonic INFO python3[3931659]: NOTICE:gnoi-shutdown-daemon:DPU1: PCI detach complete, proceeding for halting services via gNOI
2025 Nov 15 09:05:29.396271 sonic INFO python3[3931659]: NOTICE:gnoi-shutdown-daemon:DPU1: gNOI sequence failed

gNOI halt request normal passing case: using DPU2

2025 Nov 15 10:02:52.643878 sonic NOTICE gnoi-shutdown-daemon[3931659]: DPU2: Admin shutdown detected, initiating gNOI HALT
2025 Nov 15 10:02:52.644048 sonic NOTICE gnoi-shutdown-daemon[3931659]: DPU2: Starting gNOI shutdown sequence
2025 Nov 15 10:02:52.644156 sonic INFO python3[3931659]: NOTICE:gnoi-shutdown-daemon:DPU2: Admin shutdown detected, initiating gNOI HALT
2025 Nov 15 10:02:52.644212 sonic INFO python3[3931659]: NOTICE:gnoi-shutdown-daemon:DPU2: Starting gNOI shutdown sequence
2025 Nov 15 10:02:53.030039 sonic NOTICE gnoi-shutdown-daemon[3931659]: DPU2: PCI detach complete, proceeding for halting services via gNOI
2025 Nov 15 10:02:53.030204 sonic INFO python3[3931659]: NOTICE:gnoi-shutdown-daemon:DPU2: PCI detach complete, proceeding for halting services via gNOI
2025 Nov 15 10:03:47.303558 sonic NOTICE gnoi-shutdown-daemon[3931659]: DPU2: boot halt success rc: 0 out_s:System RebootStatus#012{"reason":"Halt reboot completed","count":1,"method":3,"status":{"status":1}}
2025 Nov 15 10:03:47.303710 sonic INFO python3[3931659]: NOTICE:gnoi-shutdown-daemon:DPU2: boot halt success rc: 0 out_s:System RebootStatus
2025 Nov 15 10:03:47.304954 sonic NOTICE gnoi-shutdown-daemon[3931659]: DPU2: gNOI sequence completed
2025 Nov 15 10:03:47.305021 sonic INFO python3[3931659]: NOTICE:gnoi-shutdown-daemon:DPU2: gNOI sequence completed

Name Description Physical-Slot Oper-Status Admin-Status Serial

DPU0 N/A N/A Offline down N/A
DPU1 AMD Pensando DSC N/A Offline down FLM281704EZ-1
DPU2 AMD Pensando DSC N/A Offline down FLM281704EK-0
DPU3 N/A N/A Offline down N/A
DPU4 AMD Pensando DSC N/A Online up FLM281704EM-0
DPU5 AMD Pensando DSC N/A Online up FLM281704EM-1
DPU6 AMD Pensando DSC N/A Online up FLM281704EU-0
DPU7 N/A N/A Offline down N/A

Note:

Make sure

gnoi_shutdown_daemon.py is present in /usr/local/bin

Make sure the daemon is running

For MtFuji the platform module.py should have the following until this feature is committed.
def module_pre_shutdown(self):
return True

mssonicbld · 2025-11-15T16:25:34Z

/azp run

azure-pipelines · 2025-11-15T16:25:43Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2025-11-15T19:32:24Z

/azp run

azure-pipelines · 2025-11-15T19:32:33Z

Azure Pipelines successfully started running 1 pipeline(s).

Did the instrumentation for gnoi-reboot.service

4e3a096

This was referenced May 14, 2025

Module graceful shutdown support sonic-net/sonic-platform-common#567

Open

Module graceful shutdown support sonic-net/sonic-buildimage#22611

Draft

hdwhdw requested review from hdwhdw and vvolam May 19, 2025 20:33

vvolam reviewed May 20, 2025

View reviewed changes

data/debian/sonic-host-services-data.gnoi-reboot.service Outdated Show resolved Hide resolved

scripts/gnoi-reboot-daemon Outdated Show resolved Hide resolved

scripts/gnoi-reboot-daemon Outdated Show resolved Hide resolved

scripts/gnoi-reboot-daemon Outdated Show resolved Hide resolved

rameshraghupathy mentioned this pull request May 21, 2025

HLD for SmsrtSwitch DPU graceful shutdown sonic-net/SONiC#1991

Merged

Modified based on the Redis based IPC

4a7e6bf

Modified based on the Redis based IPC

c2f9cb8

made check_platform.sh executable

db7848f

Did some cleanup

f946e72

vvolam reviewed May 22, 2025

View reviewed changes

scripts/check_platform.sh Outdated Show resolved Hide resolved

scripts/gnoi-reboot-daemon Outdated Show resolved Hide resolved

hdwhdw reviewed May 23, 2025

View reviewed changes

scripts/gnoi-reboot-daemon Outdated Show resolved Hide resolved

scripts/gnoi-reboot-daemon Outdated Show resolved Hide resolved

Draft version. Need to test again

4434463

Fixing test failure

91897ed

Copilot started reviewing on behalf of vvolam November 13, 2025 18:17 View session

Copilot finished reviewing on behalf of vvolam November 13, 2025 18:20

Copilot AI reviewed Nov 13, 2025

View reviewed changes

rameshraghupathy and others added 2 commits November 13, 2025 11:30

Update scripts/gnoi_shutdown_daemon.py

d25029a

Co-authored-by: Copilot <[email protected]>

Update scripts/gnoi_shutdown_daemon.py

0410028

Co-authored-by: Copilot <[email protected]>

Update scripts/check_platform.py

e6c71c4

Co-authored-by: Copilot <[email protected]>

addressed review comments

7f86b98

Fix test failures - add returncode mocking and fix exception test

183c337

addressed review comments

6ff92ad

vvolam requested a review from qiluo-msft November 14, 2025 18:06

hdwhdw approved these changes Nov 14, 2025

View reviewed changes

vvolam reviewed Nov 14, 2025

View reviewed changes

hdwhdw self-requested a review November 14, 2025 23:25

added gnoi_shutdown_daemon.py to setup

fd0037b

Addressed review comments

565294c

Module graceful shutdown support #255

Are you sure you want to change the base?

Module graceful shutdown support #255

Uh oh!

Conversation

rameshraghupathy commented May 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description:

Uh oh!

mssonicbld commented May 14, 2025

Uh oh!

azure-pipelines bot commented May 14, 2025

Uh oh!

hdwhdw commented May 19, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mssonicbld commented May 21, 2025

Uh oh!

azure-pipelines bot commented May 21, 2025

Uh oh!

mssonicbld commented May 21, 2025

Uh oh!

azure-pipelines bot commented May 21, 2025

Uh oh!

mssonicbld commented May 21, 2025

Uh oh!

azure-pipelines bot commented May 21, 2025

Uh oh!

mssonicbld commented May 21, 2025

Uh oh!

azure-pipelines bot commented May 21, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mssonicbld commented Jul 7, 2025

Uh oh!

azure-pipelines bot commented Jul 7, 2025

Uh oh!

mssonicbld commented Jul 10, 2025

Uh oh!

azure-pipelines bot commented Jul 10, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mssonicbld commented Nov 13, 2025

Uh oh!

azure-pipelines bot commented Nov 13, 2025

Uh oh!

mssonicbld commented Nov 13, 2025

Uh oh!

azure-pipelines bot commented Nov 13, 2025

Uh oh!

mssonicbld commented Nov 13, 2025

Uh oh!

azure-pipelines bot commented Nov 13, 2025

Uh oh!

mssonicbld commented Nov 13, 2025

Uh oh!

azure-pipelines bot commented Nov 13, 2025

Uh oh!

mssonicbld commented Nov 14, 2025

Uh oh!

azure-pipelines bot commented Nov 14, 2025

rameshraghupathy commented May 14, 2025 •

edited

Loading

rameshraghupathy Nov 15, 2025 •

edited

Loading