From 344aabe117d23cc5fef9d3fb7a80b8755d2b850a Mon Sep 17 00:00:00 2001 From: Vivek Verma <137406113+vivekverma-arista@users.noreply.github.com> Date: Tue, 19 Nov 2024 12:32:15 +0530 Subject: [PATCH 1/7] Create aggregate_voq_counters.md --- doc/voq/aggregate_voq_counters.md | 40 +++++++++++++++++++++++++++++++ 1 file changed, 40 insertions(+) create mode 100644 doc/voq/aggregate_voq_counters.md diff --git a/doc/voq/aggregate_voq_counters.md b/doc/voq/aggregate_voq_counters.md new file mode 100644 index 00000000000..a8a3d90a6bb --- /dev/null +++ b/doc/voq/aggregate_voq_counters.md @@ -0,0 +1,40 @@ +# Aggregate VOQ Counters in SONiC # +#### Rev 1.0 + +## Table of Content + * [Revision](#revision) + * [Overview](#overview) + * [Requirements](#requirements) + * [Architecture Design](#architecture-design) + * [High-Level Design](#high-level-design) + * [Database changes](#database-changes) + * [Repositories that need to be changed](#repositories-that-need-to-be-changed) + * [SAI API](#sai-api) + * [Configuration and management](#configuration-and-management) + * [CLI](#cli) + * [Testing Requirements/Design](#testing-requirementsdesign) + * [System Test cases](#system-test-cases) + +### Revision +| Rev | Date | Author | Change Description | +|:---:|:-----------:|:----------------------------------------------------------------------------------:|-----------------------------------| +| 1.0 | 19-Nov-2024 | Harsis Yadav, Pandurangan R S, Vivek Kumar Verma (Arista Networks) | Initial public version | + +### Overview + +In a [distributed VOQ architecture](https://github.com/sonic-net/SONiC/blob/master/doc/voq/architecture.md) corresponding to each output VOQ present on an ASIC, there are VOQs present on every ASIC in the system. Each ASIC has its own set of VOQ stats maintained in the FSI that have to be gathered independently and can be hard to visualize, providing a non-cohesive experience. + +### Requirements + +Provide aggregate VOQ counters in a distributed VOQ architecture. + +### Architecture Design + +No new architecture changes are required to SONiC. + +### High-Level Design + + + + + From 2062517cf87bbc34ab734f9629d285c8b5c5f872 Mon Sep 17 00:00:00 2001 From: Vivek Verma <137406113+vivekverma-arista@users.noreply.github.com> Date: Tue, 8 Apr 2025 12:50:30 +0530 Subject: [PATCH 2/7] Update aggregate_voq_counters.md Add high level design --- doc/voq/aggregate_voq_counters.md | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-) diff --git a/doc/voq/aggregate_voq_counters.md b/doc/voq/aggregate_voq_counters.md index a8a3d90a6bb..417f0718c6c 100644 --- a/doc/voq/aggregate_voq_counters.md +++ b/doc/voq/aggregate_voq_counters.md @@ -7,7 +7,6 @@ * [Requirements](#requirements) * [Architecture Design](#architecture-design) * [High-Level Design](#high-level-design) - * [Database changes](#database-changes) * [Repositories that need to be changed](#repositories-that-need-to-be-changed) * [SAI API](#sai-api) * [Configuration and management](#configuration-and-management) @@ -34,6 +33,23 @@ No new architecture changes are required to SONiC. ### High-Level Design +On multi-asic systems (fixed system or linecard) the redis databases corresponding to each namespace are exposed on the docker network (non-loopback IP): https://github.com/sonic-net/sonic-buildimage/blob/master/dockers/docker-database/docker-database-init.sh. And when the IP to be bound to redis instance is not loopback IP we start redis server in unprotected mode: https://github.com/sonic-net/sonic-buildimage/blob/master/dockers/docker-database/supervisord.conf.j2#L38 + +We can leverage this property to expose redis instances that correspond to each ASIC over midplane IP addresses in addition to docker network in case of a chassis based system using the shell script docker-database.sh by simple redis-commands. + +``` +redis-cli -h $ip -p $port config set bind "$bound_ips $midplane_ip" +redis-cli -h $ip -p $port config rewrite +``` + +This gives us access to each ASIC's redis instance from the supervisor. Then queuestat script can access the counters data and provide the user an aggregated view of the VOQ counters. + + + + + + + From c8010d66c509c3a998fa84cd713d8ee5b367deaf Mon Sep 17 00:00:00 2001 From: Vivek Verma <137406113+vivekverma-arista@users.noreply.github.com> Date: Tue, 8 Apr 2025 13:34:27 +0530 Subject: [PATCH 3/7] Update aggregate_voq_counters.md --- doc/voq/aggregate_voq_counters.md | 68 +++++++++++++++++++++++++++++++ 1 file changed, 68 insertions(+) diff --git a/doc/voq/aggregate_voq_counters.md b/doc/voq/aggregate_voq_counters.md index 417f0718c6c..e61572adae1 100644 --- a/doc/voq/aggregate_voq_counters.md +++ b/doc/voq/aggregate_voq_counters.md @@ -44,7 +44,75 @@ redis-cli -h $ip -p $port config rewrite This gives us access to each ASIC's redis instance from the supervisor. Then queuestat script can access the counters data and provide the user an aggregated view of the VOQ counters. +#### sonic-buildimage changes +`docker_image_ctl.j2` needs to be modified to encorporate changes in order expose namespace redis instances on linecards over midplane network. +PR: https://github.com/sonic-net/sonic-buildimage/pull/20803 +#### sonic-swss-common changes +A new API will be added to sonicv2connector.cpp which can take the db name and host IP as an argument and connect us to the redis instance. The existing [API](https://github.com/sonic-net/sonic-swss-common/blob/202411/common/sonicv2connector.cpp#L18-L30) needs the db_name and the host_ip and port (or unix_socket) is decoded using [database_config.json](https://github.com/sonic-net/sonic-buildimage/blob/master/dockers/docker-database/database_config.json.j2). This API is tailored for use cases when you want to connect to the namespace redis instances from the same device. Our use case involves connecting to a redis instances over midplane IP hence the new API is needed. + +#### sonic-py-swsssdk changes +Same rationale as sonic-swss-common applies here as well. Also we would like to maintain parity between dbconnector.py between swsscommon and swsssdk so that we can write a unit test for this feature or even otherwise. + +#### sonic-utilities changes +We would leverage the existing `show queue counters --voq` on the supervisor to connect to various forwarding ASIC's database instances and do a summation of VOQ counters corresponding to each system port. +PR: https://github.com/sonic-net/sonic-utilities/pull/3617 + +$ show VOQ counters [interface] --voq + +From linecard - cmp217-5 for asic0 (existing CLI) +``` +admin@cmp217-5:~$ show queue counters -n asic0 "cmp217-5|asic1|Ethernet256" --voq +For namespace asic0: + Port Voq Counter/pkts Counter/bytes Drop/pkts Drop/bytes Credit-WD-Del/pkts +-------------------------- ----- -------------- --------------- ----------- ------------ -------------------- +cmp217-5|asic1|Ethernet256 VOQ0 123 6150 0 0 0 +cmp217-5|asic1|Ethernet256 VOQ1 12 600 0 0 0 +cmp217-5|asic1|Ethernet256 VOQ2 1000 50000 0 0 0 +cmp217-5|asic1|Ethernet256 VOQ3 456 22800 0 0 0 +cmp217-5|asic1|Ethernet256 VOQ4 211 10550 0 0 0 +cmp217-5|asic1|Ethernet256 VOQ5 45 2250 0 0 0 +cmp217-5|asic1|Ethernet256 VOQ6 24 1200 0 0 0 +cmp217-5|asic1|Ethernet256 VOQ7 0 0 0 0 0 +``` + +From linecard - cmp217-5 for asic1 (existing CLI) +``` +admin@cmp217-5:~$ show queue counters -n asic1 "cmp217-5|asic1|Ethernet256" --voq +For namespace asic1: + Port Voq Counter/pkts Counter/bytes Drop/pkts Drop/bytes Credit-WD-Del/pkts +-------------------------- ----- -------------- --------------- ----------- ------------ -------------------- +cmp217-5|asic1|Ethernet256 VOQ0 1111 55550 0 0 0 +cmp217-5|asic1|Ethernet256 VOQ1 45 2250 0 0 0 +cmp217-5|asic1|Ethernet256 VOQ2 9 450 0 0 0 +cmp217-5|asic1|Ethernet256 VOQ3 91 4550 0 0 0 +cmp217-5|asic1|Ethernet256 VOQ4 55 2750 0 0 0 +cmp217-5|asic1|Ethernet256 VOQ5 88 4400 0 0 0 +cmp217-5|asic1|Ethernet256 VOQ6 21 1050 0 0 0 +cmp217-5|asic1|Ethernet256 VOQ7 48 14437 0 0 0 + +``` + +From supervisor (same command extended for sup.) + +``` +admin@cmp217:~$ show queue counters "cmp217-5|asic1|Ethernet256" --voq + Port Voq Counter/pkts Counter/bytes Drop/pkts Drop/bytes Credit-WD-Del/pkts +-------------------------- ----- -------------- --------------- ----------- ------------ -------------------- +cmp217-5|asic1|Ethernet256 VOQ0 1234 61700 0 0 0 +cmp217-5|asic1|Ethernet256 VOQ1 57 2850 0 0 0 +cmp217-5|asic1|Ethernet256 VOQ2 1009 50450 0 0 0 +cmp217-5|asic1|Ethernet256 VOQ3 547 27350 0 0 0 +cmp217-5|asic1|Ethernet256 VOQ4 266 13300 0 0 0 +cmp217-5|asic1|Ethernet256 VOQ5 133 6650 0 0 0 +cmp217-5|asic1|Ethernet256 VOQ6 45 2250 0 0 0 +cmp217-5|asic1|Ethernet256 VOQ7 52 15809 0 0 0 + +``` + +### Testing Requirements/Design +#### System Test cases +Send traffic across different ASICs and ensure aggregate counters are correctly displayed. From 190e35531537ac1782b68bae5cf92158b989f700 Mon Sep 17 00:00:00 2001 From: Vivek Verma <137406113+vivekverma-arista@users.noreply.github.com> Date: Tue, 8 Apr 2025 13:35:18 +0530 Subject: [PATCH 4/7] Update aggregate_voq_counters.md --- doc/voq/aggregate_voq_counters.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/doc/voq/aggregate_voq_counters.md b/doc/voq/aggregate_voq_counters.md index e61572adae1..dcfbdf7aa82 100644 --- a/doc/voq/aggregate_voq_counters.md +++ b/doc/voq/aggregate_voq_counters.md @@ -58,7 +58,8 @@ Same rationale as sonic-swss-common applies here as well. Also we would like to We would leverage the existing `show queue counters --voq` on the supervisor to connect to various forwarding ASIC's database instances and do a summation of VOQ counters corresponding to each system port. PR: https://github.com/sonic-net/sonic-utilities/pull/3617 -$ show VOQ counters [interface] --voq +### Configuration and management +#### CLI From linecard - cmp217-5 for asic0 (existing CLI) ``` From 23f1a89cecfc3967df2f8c4d19d56427248037b2 Mon Sep 17 00:00:00 2001 From: Vivek Verma <137406113+vivekverma-arista@users.noreply.github.com> Date: Tue, 8 Apr 2025 13:39:00 +0530 Subject: [PATCH 5/7] Update aggregate_voq_counters.md --- doc/voq/aggregate_voq_counters.md | 20 +++++++++----------- 1 file changed, 9 insertions(+), 11 deletions(-) diff --git a/doc/voq/aggregate_voq_counters.md b/doc/voq/aggregate_voq_counters.md index dcfbdf7aa82..5d73e205f01 100644 --- a/doc/voq/aggregate_voq_counters.md +++ b/doc/voq/aggregate_voq_counters.md @@ -8,7 +8,6 @@ * [Architecture Design](#architecture-design) * [High-Level Design](#high-level-design) * [Repositories that need to be changed](#repositories-that-need-to-be-changed) - * [SAI API](#sai-api) * [Configuration and management](#configuration-and-management) * [CLI](#cli) * [Testing Requirements/Design](#testing-requirementsdesign) @@ -45,7 +44,8 @@ redis-cli -h $ip -p $port config rewrite This gives us access to each ASIC's redis instance from the supervisor. Then queuestat script can access the counters data and provide the user an aggregated view of the VOQ counters. #### sonic-buildimage changes -`docker_image_ctl.j2` needs to be modified to encorporate changes in order expose namespace redis instances on linecards over midplane network. +`docker_image_ctl.j2` needs to be modified to incorporate changes in order expose namespace redis instances on linecards over midplane network. + PR: https://github.com/sonic-net/sonic-buildimage/pull/20803 #### sonic-swss-common changes @@ -56,8 +56,15 @@ Same rationale as sonic-swss-common applies here as well. Also we would like to #### sonic-utilities changes We would leverage the existing `show queue counters --voq` on the supervisor to connect to various forwarding ASIC's database instances and do a summation of VOQ counters corresponding to each system port. + PR: https://github.com/sonic-net/sonic-utilities/pull/3617 +#### Repositories that need to be changed + * sonic-buildimage + * sonic-swss-common + * sonic-py-swsssdk + * sonic-utilities + ### Configuration and management #### CLI @@ -114,12 +121,3 @@ cmp217-5|asic1|Ethernet256 VOQ7 52 15809 0 ### Testing Requirements/Design #### System Test cases Send traffic across different ASICs and ensure aggregate counters are correctly displayed. - - - - - - - - - From 0caa5e1857eb2afa9f03a3c1914b7c1e71a4d8c7 Mon Sep 17 00:00:00 2001 From: Vivek Verma <137406113+vivekverma-arista@users.noreply.github.com> Date: Tue, 8 Apr 2025 13:40:03 +0530 Subject: [PATCH 6/7] Update aggregate_voq_counters.md --- doc/voq/aggregate_voq_counters.md | 1 - 1 file changed, 1 deletion(-) diff --git a/doc/voq/aggregate_voq_counters.md b/doc/voq/aggregate_voq_counters.md index 5d73e205f01..653e47e95bd 100644 --- a/doc/voq/aggregate_voq_counters.md +++ b/doc/voq/aggregate_voq_counters.md @@ -9,7 +9,6 @@ * [High-Level Design](#high-level-design) * [Repositories that need to be changed](#repositories-that-need-to-be-changed) * [Configuration and management](#configuration-and-management) - * [CLI](#cli) * [Testing Requirements/Design](#testing-requirementsdesign) * [System Test cases](#system-test-cases) From 23399e4e6a0ec1ea6404833ff8f5e6315f9559ac Mon Sep 17 00:00:00 2001 From: Vivek Verma <137406113+vivekverma-arista@users.noreply.github.com> Date: Thu, 10 Apr 2025 21:18:09 +0530 Subject: [PATCH 7/7] Update aggregate_voq_counters.md --- doc/voq/aggregate_voq_counters.md | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/doc/voq/aggregate_voq_counters.md b/doc/voq/aggregate_voq_counters.md index 653e47e95bd..e2c97d87a6a 100644 --- a/doc/voq/aggregate_voq_counters.md +++ b/doc/voq/aggregate_voq_counters.md @@ -10,7 +10,8 @@ * [Repositories that need to be changed](#repositories-that-need-to-be-changed) * [Configuration and management](#configuration-and-management) * [Testing Requirements/Design](#testing-requirementsdesign) - * [System Test cases](#system-test-cases) + * [System Test cases](#system-test-cases) + * [Limitations and future work](#limitations-and-future-work) ### Revision | Rev | Date | Author | Change Description | @@ -50,9 +51,13 @@ PR: https://github.com/sonic-net/sonic-buildimage/pull/20803 #### sonic-swss-common changes A new API will be added to sonicv2connector.cpp which can take the db name and host IP as an argument and connect us to the redis instance. The existing [API](https://github.com/sonic-net/sonic-swss-common/blob/202411/common/sonicv2connector.cpp#L18-L30) needs the db_name and the host_ip and port (or unix_socket) is decoded using [database_config.json](https://github.com/sonic-net/sonic-buildimage/blob/master/dockers/docker-database/database_config.json.j2). This API is tailored for use cases when you want to connect to the namespace redis instances from the same device. Our use case involves connecting to a redis instances over midplane IP hence the new API is needed. +PR: https://github.com/sonic-net/sonic-swss-common/pull/1003 + #### sonic-py-swsssdk changes Same rationale as sonic-swss-common applies here as well. Also we would like to maintain parity between dbconnector.py between swsscommon and swsssdk so that we can write a unit test for this feature or even otherwise. +PR: https://github.com/sonic-net/sonic-py-swsssdk/pull/147 + #### sonic-utilities changes We would leverage the existing `show queue counters --voq` on the supervisor to connect to various forwarding ASIC's database instances and do a summation of VOQ counters corresponding to each system port. @@ -120,3 +125,7 @@ cmp217-5|asic1|Ethernet256 VOQ7 52 15809 0 ### Testing Requirements/Design #### System Test cases Send traffic across different ASICs and ensure aggregate counters are correctly displayed. + +### Limitations and future work +1. Currently we are not exposing redis instance over midplane IP for single ASIC linecards as redis runs in protected mode. +2. Clear functionality is not supported as of now for aggregate VOQ counters.