Liquid Cooling leakage detection in SONiC

Signed-off-by: Yuanzhe, Liu <[email protected]>

Author: yuazhe

doc/bmc/leakage_detection_hld.md

@@ -0,0 +1,155 @@
+# Liquid Cooling leakage detection in SONiC
+
+## 1. Overview
+
+Due to the excessive heat generated by the equipment, traditional air-cooling methods are no longer sufficient for effective heat dissipation. Therefore, liquid cooling technology has become a necessary choice for more efficiently cooling the equipment and ensuring its proper operation. Given the potential fatality of liquid cooling leakage, implementing a mechanism to monitor and instantly alert the system when such an event occurs is crucial.
+
+## 2. Requirements
+1. Monitoring the liquid colling leakge detection sensor, and alarm accordingly.
+2. For platform that doesn't support liquid cooling at all, there should be no further performance overheading.
+
+## 3. Detection and alarm flow
+The leak alarm process is straightforward. The platform API first acquires the status of the leak detection sensors. Then, thermalctld has a thread that calling the API and in turn notifies system health monitor, who ultimately sends out a gNMI event.
+
+![LCflow chart](https://github.com/sonic-net/SONiC/blob/73f11eb7ad058b214d745e9ef728b8319574edbe/images/bmc/leakage_detection_flow.png)
+
+## 4. Platform API
+A new object `LiquidCollingBase` will be added to the `sonic-platform-common` to reflect the new liquid cooling device
+
+```
+Class LiquidcollingBase(ojbect):
+    leakge_sensors_num = 0
+    leakage_sensors = {}
+
+    def get_leak_sensor_num(self):
+        """
+        Retrieves the number of leakage sensors
+
+        Returns:
+            int: The number of leakage sensors
+        """
+        return self.leakge_sensors_num
+
+    def get_leak_sensor_list(self):
+        """
+        Retrieves the list of leakage sensors
+
+        Returns:
+            list: A list of leakage sensor names
+        """
+        return self.leakage_sensors
+
+    def get_leak_sensor_status(self):
+        """
+        Retrieves the leak status of the sensors
+
+        Returns:
+            list: A list of leakage sensor names that are leaking, empty list if no leakage
+        """
+        leaking_sensors = []
+        for sensor in self.leakage_sensors:
+            if sensor.is_leak():
+                leaking_sensors.add(sensors)
+        return leaking_sensors
+
+Class LeakageSensor(sensor_base):
+    "" there might be mutiple leakge detection sensors, to let user better find the location, 
+    name = ""
+    leaking = 0
+
+    "" string return to get the sensor's name
+    def get_name():
+
+    "" boolean returen to indicate whether there is a leak detected
+    def is_leak():
+    
+```
+
+## 5. Thermal control daemon
+A new object `LiquidCoolingUpdater` will be added to the Thermal Control daemon that dedicated monitoring the liquid device status.
+
+During initialization, a separate thread will be launched to periodically call the `get_leak_sensor_status` API. The reason for separating this thread from the main thermal control daemon process is that the main process has a period of 1 minute, which is reasonable for normal thermal device updates but not suitable for leak events. Leak events are critical and need to be reported as soon as possible. However, these events are also abnormal and rare, so we don't need to involve them in the main thread and increase the main thread period to 1 second, which would be overkill and lower performance.
+
+New configuration will be added to pmon_daemon_control.json, to indicate whether the system has liquid cooling system, if not, the object and thread will not be created in the initialization of thermalctld at all to avoid performance overheading.
+```
+# to enable the seperate thread for liquid cooling monitor
+enable_liquid_cooling: true,
+# set the interval to update the leakage status, default 0.5
+liquid_cooling_update_interval: 0.5
+```
+
+Once the leakage event has been detected, the thread will write it to state db to notify the system health monitor. Meanwhile, the syslog error message will be printed out.
+"Liquid cooling leakge has been detected on sensor{}"
+
+```
+class LiquidCoolingUpdater():
+
+    def update():
+        _refresh_leak_status_update
+        _refresh_other_status_update
+    def _refresh_leak_status_update():
+        liquidCoolingOjbect = chassis.get_liquid_cooling_device
+        liquidCoolingOjbect.get_leak_sensor_status is not None
+            update the state db accordingly
+```
+
+### stat_db data schema 
+the `LIQUID_COOLING_DEVICE` table stores all the date gathered by thermal control deamon, currtenly, it will have only `leakage_sensors`
+
+```
+Defines a logical structure for liquid cooling devices, with keys for various sensors.
+
+key                       = LIQUID_COOLING_DEVICE|leakage_sensors{X}
+ ; field                  = value
+name                      = STR     ; sensor name
+leaking                   = STR     ; Yes or No to indicate leakage status
+```
+
+## 6. system health monitor
+A new function named `_check_liquid_cooling_status(self, config)` will be added to the system health monitor hardware_chekcer.py, used to monitoring the leakage detection state db value, and once it is detected, a gnmi event will be sent out.
+It worth to note that both change from NOTleak to leak and leak to NOTleak will trigger an event.
+
+```
+def publish_events(self, leakge_sensor_list):
+    params = swsscommon.FieldValueMap()
+    for leakage_sensor in leakge_sensor_list:
+        swsscommon.event_publish(self.events_handle, EVENTS_PUBLISHER_TAG, params)
+```
+
+### the GNMI event model
+EVENTS_PUBLISHER_SOURCE = "sonic-events-host"
+
+EVENTS_PUBLISHER_TAG = "liquid-cooling-leak"
+
+## 7. CLI management
+A new command `show platform leakage status` will be added to let user know current leak sensor status, the data will come from the state db
+```
+Name       Leak  
+-----------------
+leak_sensors1    NO   
+leak_sensors2    NO   
+...
+leak_sensorsX    Yes   
+```
+
+As the new functionality will be added to system health monitor, the relevant command `show system-health detail` will be updated.
+```
+System services and devices monitor list
+Name                     Status    Type
+-----------------------  --------  ----------
+...
+leak_sensors1            OK        LiquidCooling
+leak_sensors2            OK        LiquidCooling
+...
+leak_sensors3            Not OK    LiquidCooling
+```
+
+## 8. Performance
+Seperate thread will be lunched in thermal contorl daemon keep monitoring entire liquid cooling device status within 0.5s interval
+
+## 9. Testing
+A mock testing should be created to demonstrate the functionality of this implementation. Once simulated a leaking event, these things need to be checked:
+1. correct sensors number had been indicated in the syslog messge
+2. state db is rightly updated 
+3. GNMI event had been sent out
+4. `show platform leakage status` command output is correct