PFC Watchdog Design
| Rev | Date | Author | Change Description |
|---|---|---|---|
| 0.1 | Marian Pritsak | Initial version |
This document provides general information about the PFC Watchdog feature implementation in SONiC. PFC watchdog is designed to detect and mitigate PFC storm received for each port. PFC pause frames are used in lossless Ethernet to pause the link partner from sending packets. Such back-pressure mechanism could propagate to the whole network and cause the network stop forwarding traffic. PFC watchdog is to detect abnormal back-pressure caused by receiving excessive PFC pause frames, and mitigate such situation by disabling PFC caused pause temporarily. PFC watchdog has three functional blocks, i.e. detection, mitigation and restoration.
This document describes the high level design of the PFC WD feature.
| Definitions/Abbreviation | Description |
|---|---|
| PFC | Priority Flow Control |
| WD | Watchdog |
| ACL | Access Control List |
- Support PAUSE duration counters (M)
- Support egress ACL for drop mitigation action (S)
; Defines PFC WD configuration on a port
key = PFC_WD_TABLE:ifname ; configuration for watchdog on port
; field = value
state = "enabled"/"disabled" ; state of WD as configured by user
t0 = 1*3DIGIT ; t0 pause duration in msecs time after which
; queue is considered to be in PFC
; storm, and watchdog action is triggered.
t1 = 1*3DIGIT ; t1 time in msecs after which
; queue in PFC storm state is
; restored if no pause frames were received.
action = "drop"/"forward" ; action taken by watchdog in case
; of PFC storm detection.
Orchagent can have different strategies or criteria for storm detection, hence it needs to tell syncd which counters to poll.
; Defines schema for queues that must be polled by syncd, and corresponding counters
key = "PFC_WD_STATE":""queueId" ; WD queue entry
; field = value
state = "enabeld"/"disabled" ; WD state
PORT_COUNTER_ID_LIST = 1*64VCHAR ; list of ',' separated counter IDs
QUEUE_COUNTER_ID_LIST = 1*64VCHAR ; list of ',' separated counter IDs
As different vendors support different counters, there must be a way to let every ASIC vendor decide how to tell if queue is stormed. For instance, a possible criteria could be one based on pause duration counter:
(SAI_QUEUE_STAT_CURR_OCCUPANCY_BYTES > 0 && SAI_QUEUE_STAT_PACKETS.current - SAI_QUEUE_STAT_PACKETS.last == 0 && SAI_PORT_STAT_PFC_[queue]_RX_PKT.current - SAI_PORT_STAT_PFC_[queue]_RX_PKT > 0)
||
(SAI_QUEUE_STAT_CURR_OCCUPANCY_BYTES == 0 && SAI_QUEUE_STAT_PACKETS.current - SAI_QUEUE_STAT_PACKETS.last == 0 && SAI_PORT_STAT_PFC_[queue]_RX_PAUSE_DURATION.current == SAI_PORT_STAT_PFC_[queue]_RX_PAUSE_DURATION.last + t0 * delta)
// delta is a percentage of time that queue had to be paused (e. g. 0.9)Those criteria are coded in .lua script for Redis DB and called periodically based on configured timers by orchagent.
As different platforms might choose different actions to be performed for mitigation and restoration, PfcWdAction class is defined to provide common interface for these handlers:
class PfcWdAction
{
virtual sai_status_t MitigateHandler(sai_object_t port, uint32_t queueId);
virtual sai_status_t RestoreHandler(sai_object_t port, uint32_t queueId);
};Currently drop and forward actions are defined and provide corresponding mark and unmark handlers.
Watchdog settings are specified per port, in the device minigraph under DeviceInfos/DeviceInfo/EthernetInterfaces/EthernetInterface.
All configuration goes under DeviceInfos/DeviceInfo/EthernetInterfaces/EthernetInterface/PfcWatchdog tag. if its absent, watchdog is considered to be disabled for that port.
DeviceInfos/DeviceInfo/EthernetInterfaces/EthernetInterface/PfcWatchdog/Action defines mitigation action to be taken in case of pause storm.
DeviceInfos/DeviceInfo/EthernetInterfaces/EthernetInterface/PfcWatchdog/DetectionTime defines time interval for storm detection.
DeviceInfos/DeviceInfo/EthernetInterfaces/EthernetInterface/PfcWatchdog/RestorationTime defines time interval for restoration queue from storm.
...
<DeviceInfos>
...
<DeviceInfo>
<EthernetInterfaces>
...
<EthernetInterface>
<InterfaceName>Ethernet1</InterfaceName>
...
<PfcWatchdog>
<Action>drop</Action>
<DetectionTime>200</DetectionTime>
<RestorationTime>5000</RestorationTime>
</PfcWatchdog>
...
</EthernetInterface>
...
</EthernetInterfaces>
</DeviceInfo>
</DeviceInfos>
There is a set of external events that can fully or partially invalidate current state of watchdog.
In case of resetting counters PFC WD is in undefined state and should skip one polling interval. SONiC does not use API to reset counters on ASIC, so this event is ignored.
In case if port's state changes to DOWN, all queues marked as stormed, are restored.
Queue configuration must be going through PFC WD proxy in order to make it update its internal state. If PFC is disabled on a queue that was marked as stormed, queue will be restored.
PFC WD mitigation action can make different configuration settings to port/queue. In order to avoid cases in which other orchagents will apply their configuration on top of watchdog, such calls to SAI API must go through PFC proxy.
All port counters are stored in Counters DB. In order to have all required information from hardware, watchdog needs to extend it by making syncd read following values from SAI: SAI_PORT_STAT_PFC_[queue]TX_PAUSE_DURATION, SAI_PORT_STAT_PFC[queue]_RX_PKTS, SAI_QUEUE_STAT_CURR_OCCUPANCY_BYTES, SAI_QUEUE_STAT_PACKETS.
Watchdog is a separate thread running as part of orchagent daemon. It is responsible for three main tasks - PFC storm detection, mitigation and restoration.
Following mitigation handler disables PFC on marked queue and sets its reserved buffer to 0.
Following restoration handler returns reserved buffer value to initial value and enables PFC on unmarked queue.
Following mitigation handler disables PFC on stormed queue. It will no more respect pause frames from link partner, and forward all packets.
Following restore handler will reenable PFC on a queue so that it will continue to work in lossless mode as configured by user.
