[DASH] [SmartSwitch]: Counter Identification, Definition and Standardization #2139
Description
We have a need to standardize and enhance telemetry counters, with all participants to contribute their internal counters, clarify counter definitions, and ensure comprehensive coverage across vendors for improved diagnostics and monitoring.
.
Current Counter Limitations: Existing telemetry counters are basic and lack sufficient visibility for packet analysis, with many valuable internal counters not exposed or standardized across vendors, making diagnostics and monitoring challenging.
.
Call for Community Contributions: Request for all companies in the community contribute their own counters, especially those related to DPU and ENI, to a shared list, and participate in standardizing which counters should be exposed for unified analysis. Current comments are often unclear or simply repeat enum names, making it difficult to interpret counter meanings and usage.
.
Tracking: Create a comprehensive table listing all required counters, with columns for each supplier to indicate support, and color-coding must-have counters, enabling tracking of implementation status and identifying gaps.
DASH ENI
SAI Port
4.7 Counters written by Riff
.
Telemetry Counter Update Frequency and Data Flow: Update intervals for telemetry counters; the polling frequency is configurable per counter type and typically ranges from one to five seconds, with a Lua script handling data retrieval from hardware SDKs. The polling interval for counters such as port stats and ENI stats is configurable, with common settings being one second for port stats and five seconds for queue stats, and that this interval can be adjusted as needed. Can use the script that runs at the configured polling interval to fetch counter data from the hardware SDK and update the counter database, ensuring timely and consistent telemetry data collection.
.
Performance Considerations: Noted that the frequency of counter updates could impact performance; Michal confirmed that an interval of 1 to 5 seconds is sufficient for their needs, as backend systems aggregate the data.
Place .md table below:
| Counter Name | Description | SAI Counter Name | Notes / Questions |
|---|---|---|---|
| NicBytesTx | Total bytes Transmitted on ENI (overall/outbound/inbound) pipeline | SAI_ENI_STAT_(/OUTBOUND_/INBOUND_)TX_BYTES | |
| NicBytesRx | Total number of Bytes received on ENI (overall/outbound/inbound) pipeline | SAI_ENI_STAT_(/OUTBOUND_/INBOUND_)RX_BYTES | |
| NicPacketsTx | Total packets sent by ENI (overall/outbound/inbound) pipeline | SAI_ENI_STAT_(/OUTBOUND_/INBOUND_)TX_PACKETS | |
| NicPacketsRx | Total number of packets received by ENI (overall/outbound/inbound) pipeline | SAI_ENI_STAT_(/OUTBOUND_/INBOUND_)RX_PACKETS | |
| NicActiveSessions | Total Number of sessions currently active on ENI | ||
| NicDropAll | Dropped Packets Total per ENI | ||
| NicDrops | Drops (other) per ENI | ||
| NicDropSourceTepCheckFailure | Dropped because source PA validation fails | ||
| NicDropForwarding | Dropped caused by forwarding problems (mapping & route lookup misses) | ||
| NicDropPolicy | Dropped because of Policy/ACLing | <add the names> | |
| NicDropTcpNonSynFlowMiss | Drops caused by tcp nonsyn flow miss | ||
| NicDropSessionLimitExceeded | Drops when max number of sessions is reached | ||
| NicDropUnexpectedSessionState | Drops caused by unexpected session state | ||
| NicDropUnsupportedProtocol | Drops caused by receiving unsupported protocol | ||
| NicDropInvalidVni | Drops caused by Invalid VNI | ||
| NicDropVpcPeerCheckFailure | Drops caused by unpeered/unauthorized VNI | ||
| NicDropVNicVipIdMismatch | Drop caused by VIP ID mismatch | ||
| NicDropTxPolicer | Drops caused by TX policer (e.g., CPS limit hit) | ||
| NicDropMyTepCheckFailure | Drops due to MyTEP check failure due to wrong TEP IP on received packets | ||
| NicDropResponderFlowFlowSync | Rflow flowsync drops | ||
| NicDropCopp | Drops due to slow path policer drops (CPS) | ||
| NicMaxBandwidth | Bandwidth set on TX Policer | ||
| NicMaxConnectionsPerSeconds | CPS set on the TX Policer | ||
| NicMaxNumberOfSessions | Session limit per ENI | ||
| NicIcmpRedirects | ICMP redirect packets received | ||
| NicIcmpRedirectDrops | Drops caused by ICMP redirect packets | ||
| DropSiriusNicData | ??? | Not sure of use of this counter | |
| NicDropInconsistentFlowEntry | Drops due to inconsistent TCP state | ||
| NicDropLocalMappingMiss | Local mapping miss drops | ||
| NicDropPipelinePacketLoop | Internal pipeline packet loop drops | ||
| NicDropUnsupportedProtocolTowardsNvaTunnel | Unsupported protocol drops | ||
| NicMaxCpsRx | Max CPS on Rx packets | ||
| NicMaxCpsTx | Max CPS on Tx Packets | ||
| NicRxFlowSyncAck | Flow Sync Ack Received | ||
| NicRxFlowSyncRedirectPackets | Flow Sync Redirected packets to the vnic owner | ||
| NicRxTcpFinPackets | Number of Rx TCP Fin packets | ||
| NicRxTcpRstPackets | Number of RX TCP Rst packets | ||
| NicRxTcpSessionsClosedByFin | Number of sessions closed by FIN | Number of RX FIN per ENI counter exists | |
| NicRxTcpSessionsClosedByRst | Number of sessions closed by Reset | Number of RX RST per ENI counter exists | |
| NicRxTcpSessionsEstablished | Number of TCP sessions established | ||
| NicRxTcpSynAckPackets | No of TCP SyncAck packets received | ||
| NicRxTcpSynPackets | No of TCP Syn packets received | ||
| NicTcpRstInjected | Number of injected RST packets | ||
| NicTxFlowSyncRequest | Number of FlowSync Request transmitted | ||
| NicTxTcpFinPackets | Number of Fin packets transmitted | ||
| NicTxTcpRstPackets | Number of RST packets transmitted | ||
| NicTxTcpSessionsClosedByFin | Number of Tx sessions closed FIN packets | Number of Tx Fin per ENI counter exists | |
| NicTxTcpSessionsClosedByRst | Number of Tx sessions closed by RST packets | Number of Tx RST per ENI counter exists | |
| NicTxTcpSessionsEstablished | Number of Tx sessions established ESTABLISHED packets | ||
| NicTxTcpSynAckPackets | Number of Tx syn ack packets PACKETS packets | ||
| NicTxTcpSynPackets | Number of Tx syn packets PACKETS packets | ||
| MirrorTxBytes | Number of Tx BYTES mirror packets | ||
| MirrorRxBytes | Number of Rx BYTES mirror packets | ||
| MirrorTxPackets | Number of Tx PACKETS for mirror session | ||
| MirrorRxPackets | Number of Rx PACKETS for mirror session | ||
| MirrorTxBytesSkipped | Number of Tx Bytes skipped for vTAP mirror session | ||
| MirrorRxBytesSkipped | Number of Rx Bytes skipped for vTAP mirror session | ||
| MirrorTxPacketsSkipped | Number of Tx Packets skipped for vTAP mirror session | ||
| MirrorRxPacketsSkipped | Number of Rx packets skipped for vTAP mirror session | ||
| ENI Traffic Level Counter 4.7.3.1 (RX_Bytes) | Total bytes received on ENI (overall/outbound/inbound) pipeline | SAI_ENI_STAT_(/OUTBOUND_/INBOUND_)RX_BYTES | Duplicate of row 3 |
| ENI Traffic Level Counter 4.7.3.1 (RX Packets) | Total number of packets received on ENI (overall/outbound/inbound) pipeline | SAI_ENI_STAT_(/OUTBOUND_/INBOUND_)RX_PACKETS | Duplicate of row 5 |
| ENI Traffic Level Counter 4.7.3.1 (TX_Bytes) | Total bytes sent by ENI (overall/outbound/inbound) pipeline | SAI_ENI_STAT_(/OUTBOUND_/INBOUND_)TX_BYTES | Duplicate of row 2 |
| ENI Traffic Level Counter 4.7.3.1 (TX_Packets) | Total number of packets sent by ENI (overall/outbound/inbound) pipeline | SAI_ENI_STAT_(/OUTBOUND_/INBOUND_)TX_PACKETS | Duplicate of row 1 |
| ENI-level flow operation counters 4.7.3.2 | Total flow created on ENI | SAI_ENI_STAT_FLOW_CREATED | Total sessions (x2 flows) created per ENI on CPS stats |
| ENI-level flow operation counters 4.7.3.2 | Total flow failed to create on ENI | SAI_ENI_STAT_FLOW_CREATE_FAILED | Total Flows Failed on Datapath assist counters and not per ENI |
| ENI-level flow operation counters 4.7.3.2 | Total flow updated on ENI | SAI_ENI_STAT_FLOW_UPDATED | Total sessions updated per ENI on CPSStats |
| ENI-level flow operation counters 4.7.3.2 | Total flow failed to update on ENI | SAI_ENI_STAT_FLOW_UPDATE_FAILED | Total Flows failed to Updated on VPP counters and not per ENI |
| ENI-level flow operation counters 4.7.3.2 | Total flow deleted on ENI | SAI_ENI_STAT_FLOW_DELETED | Total sessions deleted per ENI on CPSstats metric |
| ENI-level flow operation counters 4.7.3.2 | Total flow failed to delete on ENI | SAI_ENI_STAT_FLOW_DELETE_FAILED | Total flows failed to age on VPP counters and not per ENI |
| ENI-level flow operation counters 4.7.3.2 | Total flow aged out on ENI. A flow aged does not necessarily mean deleted; it may be marked pending deletion. | SAI_ENI_STAT_FLOW_AGED | Total flows aged on VPP counters and not per ENI |
| ENI-level flow sync packet counters 4.7.3.3 | The bytes of inline/timed flow sync packets received by the ENI | SAI_ENI_STAT_(INLINE/TIMED)_FLOW_SYNC_PACKET_RX_BYTES | Flow Sync Rx Bytes not there |
| ENI-level flow sync packet counters 4.7.3.3 | The number of inline/timed flow sync packets received by the ENI | SAI_ENI_STAT_(INLINE/TIMED)_FLOW_SYNC_PACKET_RX_PACKETS | Flow Sync Rx Packets per ENI is there |
| ENI-level flow sync packet counters 4.7.3.3 | The bytes of inline/timed flow sync packet that this ENI sent | SAI_ENI_STAT_(INLINE/TIMED)_FLOW_SYNC_PACKET_TX_BYTES | Flow sync Tx bytes not there |
| ENI-level flow sync packet counters 4.7.3.3 | The number of inline/timed flow sync packets that this ENI sent | SAI_ENI_STAT_(INLINE/TIMED)_FLOW_SYNC_PACKET_TX_PACKETS | Flow Sync Tx Packets per ENI is there |
| ENI-level flow sync operations counters 4.7.3.4 | The number of inline/timed flow create/update/delete request that the ENI sent | SAI_ENI_STAT_(INLINE/TIMED)FLOW(CREATE/UPDATE/DELETE)_REQ_SENT | Notes |
| ENI-level flow sync operations counters 4.7.3.4 | The number of inline/timed flow create/update/delete request that the ENI received | SAI_ENI_STAT_(INLINE/TIMED)FLOW(CREATE/UPDATE/DELETE)_REQ_RECV | Notes |
| ENI-level flow sync operations counters 4.7.3.4 | The number of inline/timed flow create/update/delete request that the ENI received but failed to process | SAI_ENI_STAT_(INLINE/TIMED)FLOW(CREATE/UPDATE/DELETE)_REQ_FAILED | Notes |
| ENI-level flow sync operations counters4.7.3.4 | The number of inline/timed flow create/update/delete request that the ENI received but its flow operation is processed as ignored | SAI_ENI_STAT_(INLINE/TIMED)FLOW(CREATE/UPDATE/DELETE)_REQ_IGNORED | Notes |
| ENI-level flow sync operations counters 4.7.3.4 | The number of inline/timed flow create/update/delete ack that the ENI is received | SAI_ENI_STAT_(INLINE/TIMED)FLOW(CREATE/UPDATE/DELETE)_ACK_RECV | Notes |
| ENI-level flow sync operations counters 4.7.3.4 | The number of inline/timed flow create/update/delete ack that the ENI is received but failed to process | SAI_ENI_STAT_(INLINE/TIMED)FLOW(CREATE/UPDATE/DELETE)_ACK_FAILED | Notes |
| ENI-level flow sync operations counters 4.7.3.4 | The number of inline/timed flow create/update/delete ack that the ENI is received but its flow operation is processed as ignored | SAI_ENI_STAT_(INLINE/TIMED)FLOW(CREATE/UPDATE/DELETE)_ACK_IGNORED | Notes |
| ENI-level drop counters 4.7.3.5 | Number of packets that are dropped due to outbound routing entry not found | SAI_ENI_STAT_OUTBOUND_ROUTING_ENTRY_MISS_DROP_PACKETS | Both Routing/Mapping miss drops and not specific to only Routing |
| ENI-level drop counters 4.7.3.5 | Number of packets that are dropped due to outbound routing entry not found | SAI_ENI_STAT_OUTBOUND_CA_PA_ENTRY_MISS_DROP_PACKETS | Both Routing/Mapping miss drops and not specific to only Routing |
| ENI-level drop counters 4.7.3.5 | Number of packets that are dropped due to outbound routing entry not found | SAI_ENI_STAT_TUNNEL_MISS_DROP_PACKETS | Forwarding drops due to tunnel miss is counted with mapping/routing miss also |
| ENI-level drop counters 4.7.3.5 | Number of packets that are dropped due to outbound routing entry not found | SAI_ENI_STAT_INBOUND_ROUTING_MISS_DROP_PACKETS | Notes |
| SAI_PORT_STAT_ENI_MISS_DROP_PACKETS | Number of packets that are dropped due to ENI not found. | SAI_PORT_STAT_ENI_MISS_DROP_PACKETS | |
| SAI_PORT_STAT_VIP_MISS_DROP_PACKETS | Number of packets that are dropped due to VIP not found. | SAI_PORT_STAT_VIP_MISS_DROP_PACKETS | |
| SAI_HA_SET_STAT_DP_PROBE_(REQ/ACK)_RX_BYTES | The bytes of data plane probes that this HA set received. | SAI_HA_SET_STAT_DP_PROBE_(REQ/ACK)_RX_BYTES | |
| SAI_HA_SET_STAT_DP_PROBE_(REQ/ACK)_RX_PACKETS | The number of packets of data plane probes that this HA set received. | SAI_HA_SET_STAT_DP_PROBE_(REQ/ACK)_RX_PACKETS | |
| SAI_HA_SET_STAT_DP_PROBE_(REQ/ACK)_TX_BYTES | The bytes of data plane probes that this HA set sent. | SAI_HA_SET_STAT_DP_PROBE_(REQ/ACK)_TX_BYTES | |
| SAI_HA_SET_STAT_DP_PROBE_(REQ/ACK)_TX_PACKETS | The number of packets of data plane probes that this HA set sent. | SAI_HA_SET_STAT_DP_PROBE_(REQ/ACK)_TX_PACKETS | |
| SAI_HA_SET_STAT_DP_PROBE_FAILED | The number of probes that failed. Failure rate = failed probes / tx packets. | SAI_HA_SET_STAT_DP_PROBE_FAILED | |
| SAI_HA_SET_STAT_CP_DATA_CHANNEL_CONNECT_ATTEMPTED | Number of connect calls to establish the data channel. | SAI_HA_SET_STAT_CP_DATA_CHANNEL_CONNECT_ATTEMPTED | |
| SAI_HA_SET_STAT_CP_DATA_CHANNEL_CONNECT_RECEIVED | Number of connect calls received for the data channel. | SAI_HA_SET_STAT_CP_DATA_CHANNEL_CONNECT_RECEIVED | |
| SAI_HA_SET_STAT_CP_DATA_CHANNEL_CONNECT_SUCCEEDED | Number of connect calls that succeeded. | SAI_HA_SET_STAT_CP_DATA_CHANNEL_CONNECT_SUCCEEDED | |
| SAI_HA_SET_STAT_CP_DATA_CHANNEL_CONNECT_FAILED | Number of connect calls that failed (non-timeout/unreachable). | SAI_HA_SET_STAT_CP_DATA_CHANNEL_CONNECT_FAILED | |
| SAI_HA_SET_STAT_CP_DATA_CHANNEL_CONNECT_REJECTED | Connect calls rejected due to certificate or auth issues. | SAI_HA_SET_STAT_CP_DATA_CHANNEL_CONNECT_REJECTED | |
| SAI_HA_SET_STAT_CP_DATA_CHANNEL_TIMEOUT_COUNT | Number of connect calls that failed due to timeout/unreachable. | SAI_HA_SET_STAT_CP_DATA_CHANNEL_TIMEOUT_COUNT | |
| SAI_HA_SET_STAT_BULK_SYNC_MESSAGE_RECEIVED | Number of bulk sync messages received via the data channel. | SAI_HA_SET_STAT_BULK_SYNC_MESSAGE_RECEIVED | |
| SAI_HA_SET_STAT_BULK_SYNC_MESSAGE_SENT | Number of bulk sync messages sent via the data channel. | SAI_HA_SET_STAT_BULK_SYNC_MESSAGE_SENT | |
| SAI_HA_SET_STAT_BULK_SYNC_MESSAGE_SEND_FAILED | Number of bulk sync messages that failed to send. | SAI_HA_SET_STAT_BULK_SYNC_MESSAGE_SEND_FAILED | |
| SAI_HA_SET_STAT_BULK_SYNC_FLOW_RECEIVED | Number of flows received via bulk sync messages (each may contain many flows). | SAI_HA_SET_STAT_BULK_SYNC_FLOW_RECEIVED | |
| SAI_HA_SET_STAT_BULK_SYNC_FLOW_SENT | Number of flows sent via bulk sync messages (each may contain many flows). | SAI_HA_SET_STAT_BULK_SYNC_FLOW_SENT |