-
Notifications
You must be signed in to change notification settings - Fork 708
VPP AArch64
Dave Wallace edited this page Apr 21, 2026
·
1 revision
- 1 Get Involved
- 2 Release Milestones
- 3 Machines
- 4 CI
- 5 CSIT
- 6 Contiv-VPP
- 7 Porting and Tuning Roadmap
- 8 Known Issues
- 9 Activity
- 10 Meeting Minutes
- 11 Status Report Ligato/Contiv
- Regular AArch64 meeting: 1st and 3rd Tuesdays of every month at 06:00 PT (Pacific Time) (biweekly). Convert to your timezone.
#fdio-arm on freenode.net
Request invitation at https://slack.fd.io/
[Jira issues with ARM64 label](https://jira.fd.io/browse/VPP-1103?jql=labels = ARM64)
- The path to Fast Data on Arm [pdf] - FD.io Mini-Summit at KC+CNC EU 2018
- Vector Packet Processing (VPP) Arm Story: Now and Beyond [youtube] - FD.io Mini-summit at KC+CNC NA 2018
18.10
18.07
18.04
- CI
- Upstream patch verification on ARMv8 machines
- .deb packages
The FD.io lab is hosted at VEXXHOST colocation centre in Montreal Québec, Canada.
-
CI build servers integrated into Jenkins
| Platform | Role | Status | Hostname | IP | IPMI | Cores | RAM | Ethernet | Distro |
|---|---|---|---|---|---|---|---|---|---|
| Marvell ThunderX | VPP dev debug server | Running | vpp-marvell-dev | 10.30.51.38 | 10.30.50.38 | 96 | 128GB | 3x40GbE QSFP+ / 4x10GbE SFP+ | Ubuntu 18.04.4 |
| CI build server | Running in Nomad | s53-nomad | 10.30.51.39 | 10.30.50.39 | 96 | 128GB | 3x40GbE QSFP+ / 4x10GbE SFP+ | Ubuntu 18.04.4 | |
| CI build server | Running in Nomad | s54-nomad | 10.30.51.40 | 10.30.50.40 | 96 | 128GB | 3x40GbE QSFP+ / 4x10GbE SFP+ | Ubuntu 18.04.4 | |
| CI build server | Running in Nomad | s52-nomad | 10.30.51.65 | 10.30.50.65 | 96 | 256GB | 2xQSFP+ / USB Ethernet | Ubuntu 18.04.4 | |
| CI build server | Running in Nomad | s51-nomad | 10.30.51.66 | 10.30.50.66 | 96 | 256GB | 2xQSFP+ / USB Ethernet | Ubuntu 18.04.4 | |
| CI build server | Running in Nomad | s49-nomad | 10.30.51.67 | 10.30.50.67 | 96 | 256GB | 2xQSFP+ / USB Ethernet | Ubuntu 18.04.4 | |
| CI build server | Running in Nomad | s50-nomad | 10.30.51.68 | 10.30.50.68 | 96 | 256GB | 2xQSFP+ / USB Ethernet | Ubuntu 18.04.4 | |
| Marvell ThunderX2 | Perf DUT candidate | Running | s27-t13-sut1 | 10.30.51.69 | 10.30.50.69 | 224 | 128GB | 3x40GbE QSFP+ XL710-QDA2 | Ubuntu 18.04.2 |
| VPP device server | Running in Nomad | s55-t36-sut1 | 10.30.51.70 | 10.30.50.70 | 256 | 256GB | 2x40GbE QSFP+ XL710-QDA2 / 2x10/25GE SFP+ ConnectX5 | Ubuntu 18.04.4 | |
| VPP device server | Running in Nomad | s56-t37-sut1 | 10.30.51.71 | 10.30.50.71 | 256 | 256GB | 2x40GbE QSFP+ XL710-QDA2 / 2x10/25GE SFP+ ConnectX5 | Ubuntu 18.04.4 | |
| Huawei TaiShan 2280 | CSIT testbed | Running in CI | s17-t33-sut1 | 10.30.51.36 | 10.30.50.36 | 64 | 128GB | 2x10GbE SFP+ Intel X520-DA2 / 2x25GbE SFP28 Mellanox CX-4 | 18.04.1 |
| CSIT testbed | Running in CI | s18-t33-sut2 | 10.30.51.37 | 10.30.50.37 | 64 | 128GB | 2x10GbE SFP+ Intel X520-DA2 / 2x25GbE SFP28 Mellanox CX-4 | 18.04.1 | |
| Marvell MACCHIATObin | N/A | Decommissioned | s20-t34-sut1 | 10.30.51.41 | 10.30.51.49, then connect to /dev/ttyUSB0 | 4 | 16GB | 2x10GbE SFP+ | Ubuntu 16.04.4 |
| N/A | Decommissioned | s21-t34-sut2 | 10.30.51.42 | 10.30.51.49, then connect to /dev/ttyUSB1 | 4 | 16GB | 2x10GbE SFP+ | Ubuntu 16.04.5 | |
| N/A | Decommissioned | fdio-mcbin3 | 10.30.51.43 | 10.30.51.49, then connect to /dev/ttyUSB2 | 4 | 16GB | 2x10GbE SFP+ | Ubuntu 16.04.5 | |
| Power Cycler | Operational | 10.30.50.80 | |||||||
| SoftIron OverDrive 1000 | N/A | Decommissioned | softiron-1 | 10.30.51.12 | N/A | 4 | 8GB | openSUSE | |
| N/A | Decommissioned | softiron-2 | 10.30.51.13 | N/A | 4 | 8GB | openSUSE | ||
| N/A | Decommissioned | softiron-3 | 10.30.51.14 | N/A | 4 | 8GB | openSUSE |
Note: to get lab access, create a gpg key, upload it to keyserver, have it signed by a trusted anchor in a video call (fingerprint will be needed) and then an ARM authority (Tina) needs to send an e-mail to email with your name, e-mail, keygrip and key fingerprint
Covers automated build, unit test, and packaging for various Linux distros on ARMv8 machines.
| Jenkins job | Status | Description |
|---|---|---|
| vpp-arm-verify-master-ubuntu1604 | Running | xxx |
| vpp-arm-merge-master-ubuntu1604 | Running | xxx |
| vpp-arm-verify-1804-ubuntu1604 | Running | xxx |
| vpp-arm-merge-1804-ubuntu1604 | Running | xxx |
Next steps:
-
make testadded toverifyjobs - Clang build
- openSUSE Leap 15 | CentOS 7 | Ubuntu 18.04
-
vpp-csit-verify-virl-masteror equivalent CSIT functional testing
Covers automated functional and performance integration testing on ARMv8 3-node and 2-node testbeds.
https://wiki.fd.io/view/CSIT/AArch64
This Kubernetes network plugin uses FD.io VPP to provide network connectivity between PODs.
The installation guide of Contiv-VPP on Arm64 platform is
https://github.com/contiv/vpp/blob/master/docs/arm64/MANUAL_INSTALL_ARM64.md
- VPP Vectorization: Expanding the Neon Library for IPv4 forwarding code path - Sirshak/Lijian
- Tuning the quad loop/dual loop for small cores - Lijian
- General performance analysis and tuning of various graph nodes for IPv4 forwarding test case - Sirshak/Lijian
- Memory Ordering - Sirshak
- CSIT Performance Test - Khemendra
- CSIT Device Test - Juraj
- CSIT Path Test - Juraj
GCC 5.3 ICEs during FP register allocation. Please use GCC 5.4 or newer.
11/21/2023
-
Attendees
- Lijian Zhang
- Juraj Linkes
- Niyaz Murshed
- Jieqiang Wang
-
CSIT
- Status
- Dave Wallace help monitor the AArch64 CI/CD status, which looks fine
- Replace old thunderX2 with Ampera Altra, bugdets got approved, still in progress
- Sync with CSIT folks in the call when possible -- Juraj
- Maciek asked about the availability of N2-based hardwares
- Plans to ship N2-based servers(Nvidia Grace(V2)/Ampere One(in-house design by Ampere)) to FD.io lab in next year
- Timeline TBD
- IPSec test cases
- Patch already merged
- QAT cards in Austin labs, plan to ship them to FD.io lab
- RDMA test cases
- MLX DPDK test cases are enabled, RDMA are not on AArch64
- Status
-
VPP
- Detailed planning for VPP projects in the next call
- Refactor OpenSSL usage in VPP IPsec -- Lijian
- Move key generation and initialization steps out of data plane to control plane, see performance boost
- Investigate make test framework in VPP -- Lijian
- Patch broke wireguard test cases so need to figure out the work flow
- VPP ramp-up -- Niyaz
- Investigate VPP graph node mechanism and how to add nodes to the group
- IPSec scalability tests -- Jieqiang
- Try to figure out dpdk-rss-flows.py and how to generate balanced rss flows for IPSec tests
07/18/2023
- Attendees
- Jieqiang Wang
- Tianyu Li
- Juraj Linkes
- CSIT
- Timeout issue happens preriodically on Taishan server, even in release testing.
- Setting CPU affinity only after VMs boot up fully.
- Another issue maybe related with Taishan NUMA topology
- Increase timeout will bypass the issue and have no effecton VPP VM perf
- IPSec & VxLAN performance drop issue on Ampere Altra
- QAT cards are planned to be shipped
- need to pay attention to the execution time for IPSec release testing
- Need to investigate further on performance degradation issue
- Verify job, Merge Job, Device Testing, and release testing is so far so good.
- RDMA PMD claims ConnectX4/5 support; Whether ConnectX6/7 is supported or not?
- Will have a debug meeting with RDMA maintainers on the issues.
- ConnectX6 NIC info will be updated in doc first
- Decide which test cases to be run on the testbeds(time consideration/iterative test/coverage test)
- MRR failed cases
- Probably due to latest DPDK upgrade, not an arm-specific issue.
- New test cases list on 3n-alt
- NAT tests cannot be added because they are running on 2-node testbed only
- enable IPSec flow cache(arm)/IPSec SPD fast path feature
- Release testing
- 23.06 release testing is done
- New CSIT page https://csit.fd.io/
- Plan to replace TX2 with Altra as VPP device testing testbed
- Timeout issue happens preriodically on Taishan server, even in release testing.
06/20/2023
- Attendees
- Lijian Zhang
- Jieqiang Wang
- Tianyu Li
- Juraj Linkes
- CSIT
- Timeout issue happens preriodically on Taishan server, even in release testing.
- Setting CPU affinity only after VMs boot up fully.
- Another issue maybe related with Taishan NUMA topology
- Increase timeout will bypass the issue and have no effecton VPP VM perf
- IPSec & VxLAN performance drop issue on Ampere Altra
- QAT cards are planned to be shipped
- need to pay attention to the execution time for IPSec release testing
- Need to investigate further on performance degradation issue
- Verify job, Merge Job, Device Testing, and release testing is so far so good.
- RDMA PMD claims ConnectX4/5 support; Whether ConnectX6/7 is supported or not?
- Will have a debug meeting with RDMA maintainers on the issues.
- ConnectX6 NIC info will be updated in doc first
- Decide which test cases to be run on the testbeds(time consideration/iterative test/coverage test)
- MRR failed cases
- Probably due to latest DPDK upgrade, not an arm-specific issue.
- New test cases list on 3n-alt
- NAT tests cannot be added because they are running on 2-node testbed only
- enable IPSec flow cache(arm)/IPSec SPD fast path feature
- Timeout issue happens preriodically on Taishan server, even in release testing.
05/16/2023
- Attendees
- Lijian Zhang
- Jieqiang Wang
- Tianyu Li
- Juraj
- CSIT
- Timeout issue happens preriodically on Taishan server, even in release testing.
- Setting CPU affinity only after VMs boot up fully.
- Another issue maybe related with Taishan NUMA topology
- Increase timeout will bypass the issue and have no effecton VPP VM perf
- The link issue in DPDK testpmd test cases on Ampere Altra is still there.
- Try cable switch while upgrading NIC firmeare and drivers
- Try to reproduce the tests after the NIC firmware
- Try different port pairs of the same two NICs
- IPSec & VxLAN performance drop issue on Ampere Altra
- QAT cards are planned to be shipped
- need to pay attention to the execution time for IPSec release testing
- Need to investigate further on performance degradation issue
- Verify job, Merge Job, Device Testing, and release testing is so far so good.
- RDMA PMD claims ConnectX4/5 support; Whether ConnectX6/7 is supported or not?
- Will have a debug meeting with RDMA maintainers on the issues.
- ConnectX6 NIC info will be updated in doc first
- Decide which test cases to be run on the testbeds(time consideration/iterative test/coverage test)
- MRR failed cases
- Probably due to latest DPDK upgrade, not an arm-specific issue.
- Timeout issue happens preriodically on Taishan server, even in release testing.
- VPP
04/18/2023
- Attendees
- Lijian Zhang
- Jieqiang Wang
- Tianyu Li
- Juraj
- CSIT
- Timeout issue happens preriodically on Taishan server, even in release testing.
- Setting CPU affinity only after VMs boot up fully.
- Another issue maybe related with Taishan NUMA topology
- The link issue in DPDK testpmd test cases on Ampere Altra is still there.
- IPSec & VxLAN performance drop issue on Ampere Altra
- QAT cards are planned to be shipped
- need to pay attention to the execution time for IPSec release testing
- Need to investigate further on performance degradation issue
- Verify job, Merge Job, Device Testing, and release testing is so far so good.
- RDMA PMD claims ConnectX4/5 support; Whether ConnectX6/7 is supported or not?
- Will have a debug meeting with RDMA maintainers on the issues.
- ConnectX6 NIC info will be updated in doc first
- Timeout issue happens preriodically on Taishan server, even in release testing.
- VPP
04/04/2023
- Attendees
- Lijian Zhang
- Jieqiang Wang
- Tianyu Li
- Juraj
- CSIT
- Timeout issue happens preriodically on Taishan server, even in release testing.
- Setting CPU affinity only after VMs boot up fully.
- Another issue maybe related with Taishan NUMA topology
- The link issue in DPDK testpmd test cases on Ampere Altra is still there.
- IPSec & VxLAN performance drop issue on Ampere Altra
- Verify job, Merge Job, Device Testing, and release testing is so far so good.
- RDMA PMD claims ConnectX4/5 support; Whether ConnectX6/7 is supported or not?
- Will have a debug meeting with RDMA maintainers on the issues.
- Timeout issue happens preriodically on Taishan server, even in release testing.
- VPP
03/07/2023
- Attendees
- Lijian Zhang
- Jieqiang Wang
- Tianyu Li
- Juraj
- CSIT
- Timeout issue happens preriodically on Taishan server, even in release testing.
- The link issue in DPDK testpmd test cases on Ampere Altra is still there.
- Verify job, Merge Job, Device Testing, and release testing is so far so good.
- RDMA PMD claims ConnectX4/5 support; Whether ConnectX6/7 is supported or not?
- VPP
2/21/2023
-
Attendees
- Lijian Zhang
- Jieqiang Wang
- Tianyu Li
- Juraj
-
Miscellaneous
- Reschedule the meeting to 8:30 am for Juraj and 3:30 pm for Shanghai folks
-
CSIT
- VPP Performance Test
- https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-3n-tsh/
- https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-report-iterative-2106-3n-tsh/
-
https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-3n-alt/
-
- DPDK testpmd XL710 interface not up failure(2 ampere back to back) - VPP and other apps, the same interface works fine.
- Local setup not reproduced - need to schedule a debug session to reproduce the issue in FD.io lab.
- Confirm with Vexxhost people if replacing intel NICs is feasible
- Tried to wait more time, interface still not up, restart is not enough either - Need to figure out reliable workaround.
- Tried old version of DPDK - 21.08 does not work. May need to try older version.
- Will it related to NIC's Speed, Duplex and Auto negotiation configuration?
- May try to upgrade the NIC's firmware. - check local xl710 firmware version
- Will talk to dpdk i40e maintainer to seek their help
- Dpdk Port/link status broken - l3fwd have the some issue
- Sent detail email to i40e maintainer in dpdk-dev mail list waiting for response
- DPDK testpmd XL710 interface not up failure(2 ampere back to back) - VPP and other apps, the same interface works fine.
- New links for VPP perf trending/report pages
- Daily trending: https://s3-docs.fd.io/csit/master/trending/
- Release report: https://s3-docs.fd.io/csit/master/report/
- Need to investigate 22.10 release testing result
- Compiler version change seems to be one of factors for perf degradation
- Old version: clang 10.0.0-4ubuntu1, gcc Ubuntu 9.4.0-1ubuntu1~20.04.1
- New version: clang 14.0.0-1ubuntu1, gcc Ubuntu 11.3.0-1ubuntu1~22.04
- Compiler version change seems to be one of factors for perf degradation
- VM testcase timeout issue on 3-tsh testbed
- Timeout issue occured when starting VPP inside VM, but not for starting testpmd
- Config isolcpu in kernel boot parameter is deprecated, Tianyu proposed a solution that Juraj would try
- isolcpus seems to be working fine
- still need to root cause the timeout issue- sometimes slower
- run dpdk build, just use the non-isolated cores for build
- both VM and VPP start slower than before
- VPP loading plugins and timeout happens
- Is VPP crashing? - not crash
- Is the VM bound with isolated core? - need to check
- Will set up a live debug session for Tianyu and Juraj
-
- VPP Performance Test
-
- VPP Path
- Voting and working fine.
- CentOS-8 jobs have been removed.
- VPP Path
-
- VPP Device
- Enable VPP device testing per patch
- Voting right for VPP device testing on Arm is enabled
- VPP device testing on Arm runs per VPP/CSIT patch
- Enable VPP device testing per patch
- QAT cards
- Govind will ship another 2x QAT from Austin to FD.io lab
- Will procure 2x QAT cards and verify them internally firstly.
- The existing QAT cards will be removed and returned to Vexxhost/FD.io lab
- QAT cards can be seen with new kernel update
- MLX NICs Planning
- CX6 and CX7 - CX7 is hard to get on market - MLX Nics will be used and reported
- CX6 vpp native rdma driver has issues, dpdk mlx driver is fine.
- VPP Device
-
VPP
- VPP SVE implementation - Lijian
- SVE validation on FPGA platform - Confluence page ready
- Run L3 traffic testing with NEON/SVE-VLA/SVE-VLS VPP version
- Performance number: NEON>SVE-VLA>SVE-VLS on arm internal platform
- Investigate SVE vs NEON packet checksum comparison
- Investigate VPP cross compilation with buildroot - for running VPP on FVP - distro like ubuntu is slower than buildroot - Lijian
- Depends on some libraries, dpdk, ipsec_mb, rdma-core and nasm etc - optional
- Investigate One Terabit throughput test on Arm platform
- Investigate automate rfc2544 no drop rate throughput test with Ixia on N1 platform - Tianyu
- Kernel cmdline may impact on NDR PDR results - Jieqiang
- Intern help to benchmark VPP using 100G MLX5 NIC - scalabilty test
- VPP IPsec on Arm - Govind
- Work on IPsec input - Zach
- IPSec input node/decryption flow cache implemented in a separate patch - Zach
- Upstreamed https://gerrit.fd.io/r/c/vpp/+/32903
- Review SPD flow cache patch from Intel folks - Zach
-
- Depends on kernel patch
- Kernel with aarch64 patch is expected to release soon
- Depends on kernel patch
- Building Intel QAT driver on arm to test IPsec crypt - Picchi
- Run IPsec with QAT on skylake this week, run the same setup on Ampere is next step
- Patch to resolve iommu issue for mlx NIC when using with QAT card
- Benchmark IPSec test case with QAT card/mlx NIC(single-core/multi-core)
- Needs a kernel patch to resolve crash issue for QAT card
- Patch made by Yoan is upstream and waits for review
- Try patched VPP to verify QAT card usage
- Investigating crash issue with 90% linerate IPSec traffic with QAT card
- VPP rdma native driver only - vpp meta data corrupt - may related to memory barrier
- QAT single core test done - investigate multiple core QAT case
- IPSec two core test with QAT offload, performance is poor on Ampere - need to investigate
-
- VPP SVE implementation - Lijian
2/7/2023
-
Attendees
- Lijian Zhang
- Jieqiang Wang
- Tianyu Li
- Juraj
-
Miscellaneous
- Reschedule the meeting to 8:30 am for Juraj and 3:30 pm for Shanghai folks
-
CSIT
- VPP Performance Test
- https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-3n-tsh/
- https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-report-iterative-2106-3n-tsh/
-
https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-3n-alt/
-
- DPDK testpmd XL710 interface not up failure(2 ampere back to back) - VPP and other apps, the same interface works fine.
- Local setup not reproduced - need to schedule a debug session to reproduce the issue in FD.io lab.
- Confirm with Vexxhost people if replacing intel NICs is feasible
- Tried to wait more time, interface still not up, restart is not enough either - Need to figure out reliable workaround.
- Tried old version of DPDK - 21.08 does not work. May need to try older version.
- Will it related to NIC's Speed, Duplex and Auto negotiation configuration?
- May try to upgrade the NIC's firmware. - check local xl710 firmware version
- Will talk to dpdk i40e maintainer to seek their help
- Dpdk Port/link status broken - l3fwd have the some issue
- Sent detail email to i40e maintainer in dpdk-dev mail list waiting for response
- DPDK testpmd XL710 interface not up failure(2 ampere back to back) - VPP and other apps, the same interface works fine.
- New links for VPP perf trending/report pages
- Daily trending: https://s3-docs.fd.io/csit/master/trending/
- Release report: https://s3-docs.fd.io/csit/master/report/
- Need to investigate 22.10 release testing result
- Compiler version change seems to be one of factors for perf degradation
- Old version: clang 10.0.0-4ubuntu1, gcc Ubuntu 9.4.0-1ubuntu1~20.04.1
- New version: clang 14.0.0-1ubuntu1, gcc Ubuntu 11.3.0-1ubuntu1~22.04
- Compiler version change seems to be one of factors for perf degradation
- VM testcase timeout issue on 3-tsh testbed
- Timeout issue occured when starting VPP inside VM, but not for starting testpmd
- Config isolcpu in kernel boot parameter is deprecated, Tianyu proposed a solution that Juraj would try
- isolcpus seems to be working fine
- still need to root cause the timeout issue- sometimes slower
- run dpdk build, just use the non-isolated cores for build
- both VM and VPP start slower than before
- VPP loading plugins and timeout happens
- Is VPP crashing? - not crash
- Is the VM bound with isolated core? - need to check
-
- VPP Performance Test
-
- VPP Path
- Voting and working fine.
- CentOS-8 jobs have been removed.
- VPP Path
-
- VPP Device
- Enable VPP device testing per patch
- Voting right for VPP device testing on Arm is enabled
- VPP device testing on Arm runs per VPP/CSIT patch
- Enable VPP device testing per patch
- QAT cards
- Govind will ship another 2x QAT from Austin to FD.io lab
- Will procure 2x QAT cards and verify them internally firstly.
- The existing QAT cards will be removed and returned to Vexxhost/FD.io lab
- QAT cards can be seen with new kernel update
- MLX NICs Planning
- CX6 and CX7 - CX7 is hard to get on market - MLX Nics will be used and reported
- CX6 vpp native rdma driver has issues, dpdk mlx driver is fine.
- VPP Device
-
VPP
- VPP SVE implementation - Lijian
- SVE validation on FPGA platform - Confluence page ready
- Run L3 traffic testing with NEON/SVE-VLA/SVE-VLS VPP version
- Performance number: NEON>SVE-VLA>SVE-VLS on arm internal platform
- Investigate SVE vs NEON packet checksum comparison
- Investigate VPP cross compilation with buildroot - for running VPP on FVP - distro like ubuntu is slower than buildroot - Lijian
- Depends on some libraries, dpdk, ipsec_mb, rdma-core and nasm etc - optional
- Investigate One Terabit throughput test on Arm platform
- Investigate automate rfc2544 no drop rate throughput test with Ixia on N1 platform - Tianyu
- Kernel cmdline may impact on NDR PDR results - Jieqiang
- Intern help to benchmark VPP using 100G MLX5 NIC - scalabilty test
- VPP IPsec on Arm - Govind
- Work on IPsec input - Zach
- IPSec input node/decryption flow cache implemented in a separate patch - Zach
- Upstreamed https://gerrit.fd.io/r/c/vpp/+/32903
- Review SPD flow cache patch from Intel folks - Zach
-
- Depends on kernel patch
- Kernel with aarch64 patch is expected to release soon
- Depends on kernel patch
- Building Intel QAT driver on arm to test IPsec crypt - Picchi
- Run IPsec with QAT on skylake this week, run the same setup on Ampere is next step
- Patch to resolve iommu issue for mlx NIC when using with QAT card
- Benchmark IPSec test case with QAT card/mlx NIC(single-core/multi-core)
- Needs a kernel patch to resolve crash issue for QAT card
- Patch made by Yoan is upstream and waits for review
- Try patched VPP to verify QAT card usage
- Investigating crash issue with 90% linerate IPSec traffic with QAT card
- VPP rdma native driver only - vpp meta data corrupt - may related to memory barrier
- QAT single core test done - investigate multiple core QAT case
- IPSec two core test with QAT offload, performance is poor on Ampere - need to investigate
-
- VPP SVE implementation - Lijian
1/17/2023
-
Attendees
- Lijian Zhang
- Jieqiang Wang
- Tianyu Li
- Tina Tsou
-
Miscellaneous
- Reschedule the meeting to 8:30 am for Juraj and 3:30 pm for Shanghai folks
-
CSIT
- VPP Performance Test
- https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-3n-tsh/
- https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-report-iterative-2106-3n-tsh/
-
https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-3n-alt/
-
- DPDK testpmd XL710 interface not up failure(2 ampere back to back) - VPP and other apps, the same interface works fine.
- Local setup not reproduced - need to schedule a debug session to reproduce the issue in FD.io lab.
- Confirm with Vexxhost people if replacing intel NICs is feasible
- Tried to wait more time, interface still not up, restart is not enough either - Need to figure out reliable workaround.
- Tried old version of DPDK - 21.08 does not work. May need to try older version.
- Will it related to NIC's Speed, Duplex and Auto negotiation configuration?
- May try to upgrade the NIC's firmware. - check local xl710 firmware version
- Will talk to dpdk i40e maintainer to seek their help
- DPDK testpmd XL710 interface not up failure(2 ampere back to back) - VPP and other apps, the same interface works fine.
- New links for VPP perf trending/report pages
- Daily trending: https://s3-docs.fd.io/csit/master/trending/
- Release report: https://s3-docs.fd.io/csit/master/report/
- Need to investigate 22.10 release testing result
- Compiler version change seems to be one of factors for perf degradation
- Old version: clang 10.0.0-4ubuntu1, gcc Ubuntu 9.4.0-1ubuntu1~20.04.1
- New version: clang 14.0.0-1ubuntu1, gcc Ubuntu 11.3.0-1ubuntu1~22.04
- Compiler version change seems to be one of factors for perf degradation
- VM testcase timeout issue on 3-tsh testbed
- Timeout issue occured when starting VPP inside VM, but not for starting testpmd
- Config isolcpu in kernel boot parameter is deprecated, Tianyu proposed a solution that Juraj would try
-
- VPP Performance Test
-
- VPP Path
- Voting and working fine.
- CentOS-8 jobs have been removed.
- VPP Path
-
- VPP Device
- Enable VPP device testing per patch
- Voting right for VPP device testing on Arm is enabled
- VPP device testing on Arm runs per VPP/CSIT patch
- Enable VPP device testing per patch
- QAT cards
- Govind will ship another 2x QAT from Austin to FD.io lab
- Will procure 2x QAT cards and verify them internally firstly.
- The existing QAT cards will be removed and returned to Vexxhost/FD.io lab
- QAT cards can be seen with new kernel update
- VPP Device
-
VPP
- VPP SVE implementation - Lijian
- SVE validation on FPGA platform - Confluence page ready
- Run L3 traffic testing with NEON/SVE-VLA/SVE-VLS VPP version
- Performance number: NEON>SVE-VLA>SVE-VLS on arm internal platform
- Investigate SVE vs NEON packet checksum comparison
- Investigate VPP cross compilation with buildroot - for running VPP on FVP - distro like ubuntu is slower than buildroot - Lijian
- Depends on some libraries, dpdk, ipsec_mb, rdma-core and nasm etc - optional
- Investigate One Terabit throughput test on Arm platform
- Investigate automate rfc2544 no drop rate throughput test with Ixia on N1 platform - Tianyu
- Kernel cmdline may impact on NDR PDR results - Jieqiang
- Intern help to benchmark VPP using 100G MLX5 NIC - scalabilty test
- VPP IPsec on Arm - Govind
- Work on IPsec input - Zach
- IPSec input node/decryption flow cache implemented in a separate patch - Zach
- Upstreamed https://gerrit.fd.io/r/c/vpp/+/32903
- Review SPD flow cache patch from Intel folks - Zach
-
- Depends on kernel patch
- Kernel with aarch64 patch is expected to release soon
- Depends on kernel patch
- Building Intel QAT driver on arm to test IPsec crypt - Picchi
- Run IPsec with QAT on skylake this week, run the same setup on Ampere is next step
- Patch to resolve iommu issue for mlx NIC when using with QAT card
- Benchmark IPSec test case with QAT card/mlx NIC(single-core/multi-core)
- Needs a kernel patch to resolve crash issue for QAT card
- Patch made by Yoan is upstream and waits for review
- Try patched VPP to verify QAT card usage
- Investigating crash issue with 90% linerate IPSec traffic with QAT card
- VPP rdma native driver only - vpp meta data corrupt - may related to memory barrier
- QAT single core test done - investigate multiple core QAT case
- IPSec two core test with QAT offload, performance is poor on Ampere - need to investigate
-
- VPP SVE implementation - Lijian
12/20/2022
-
Attendees
- Lijian Zhang
- Jieqiang Wang
- Tianyu Li
- Tina Tsou
-
Miscellaneous
- Reschedule the meeting to 8:30 am for Juraj and 3:30 pm for Shanghai folks
-
CSIT
- VPP Performance Test
- https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-3n-tsh/
- https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-report-iterative-2106-3n-tsh/
-
https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-3n-alt/
-
- DPDK testpmd XL710 interface not up failure(2 ampere back to back) - VPP and other apps, the same interface works fine.
- Local setup not reproduced - need to schedule a debug session to reproduce the issue in FD.io lab.
- Confirm with Vexxhost people if replacing intel NICs is feasible
- Tried to wait more time, interface still not up, restart is not enough either - Need to figure out reliable workaround.
- Tried old version of DPDK - 21.08 does not work. May need to try older version.
- Will it related to NIC's Speed, Duplex and Auto negotiation configuration?
- May try to upgrade the NIC's firmware. - check local xl710 firmware version
- Will talk to dpdk i40e maintainer to seek their help
- DPDK testpmd XL710 interface not up failure(2 ampere back to back) - VPP and other apps, the same interface works fine.
- New links for VPP perf trending/report pages
- Daily trending: https://s3-docs.fd.io/csit/master/trending/
- Release report: https://s3-docs.fd.io/csit/master/report/
- Need to investigate 22.10 release testing result
- Compiler version change seems to be one of factors for perf degradation
- Old version: clang 10.0.0-4ubuntu1, gcc Ubuntu 9.4.0-1ubuntu1~20.04.1
- New version: clang 14.0.0-1ubuntu1, gcc Ubuntu 11.3.0-1ubuntu1~22.04
- Compiler version change seems to be one of factors for perf degradation
- VM testcase timeout issue on 3-tsh testbed
- Timeout issue occured when starting VPP inside VM, but not for starting testpmd
- Config isolcpu in kernel boot parameter is deprecated, Tianyu proposed a solution that Juraj would try
-
- VPP Performance Test
-
- VPP Path
- Voting and working fine.
- CentOS-8 jobs have been removed.
- VPP Path
-
- VPP Device
- Enable VPP device testing per patch
- Voting right for VPP device testing on Arm is enabled
- VPP device testing on Arm runs per VPP/CSIT patch
- Enable VPP device testing per patch
- QAT cards
- Govind will ship another 2x QAT from Austin to FD.io lab
- Will procure 2x QAT cards and verify them internally firstly.
- The existing QAT cards will be removed and returned to Vexxhost/FD.io lab
- QAT cards can be seen with new kernel update
- VPP Device
-
VPP
- VPP SVE implementation - Lijian
- SVE validation on FPGA platform - Confluence page ready
- Run L3 traffic testing with NEON/SVE-VLA/SVE-VLS VPP version
- Performance number: NEON>SVE-VLA>SVE-VLS on arm internal platform
- Investigate SVE vs NEON packet checksum comparison
- Investigate VPP cross compilation with buildroot - for running VPP on FVP - distro like ubuntu is slower than buildroot - Lijian
- Depends on some libraries, dpdk, ipsec_mb, rdma-core and nasm etc - optional
- Investigate One Terabit throughput test on Arm platform
- Investigate automate rfc2544 no drop rate throughput test with Ixia on N1 platform - Tianyu
- Kernel cmdline may impact on NDR PDR results - Jieqiang
- Intern help to benchmark VPP using 100G MLX5 NIC - scalabilty test
- VPP IPsec on Arm - Govind
- Work on IPsec input - Zach
- IPSec input node/decryption flow cache implemented in a separate patch - Zach
- Upstreamed https://gerrit.fd.io/r/c/vpp/+/32903
- Review SPD flow cache patch from Intel folks - Zach
-
- Depends on kernel patch
- Kernel with aarch64 patch is expected to release soon
- Depends on kernel patch
- Building Intel QAT driver on arm to test IPsec crypt - Picchi
- Run IPsec with QAT on skylake this week, run the same setup on Ampere is next step
- Patch to resolve iommu issue for mlx NIC when using with QAT card
- Benchmark IPSec test case with QAT card/mlx NIC(single-core/multi-core)
- Needs a kernel patch to resolve crash issue for QAT card
- Patch made by Yoan is upstream and waits for review
- Try patched VPP to verify QAT card usage
- Investigating crash issue with 90% linerate IPSec traffic with QAT card
- VPP rdma native driver only - vpp meta data corrupt - may related to memory barrier
- QAT single core test done - investigate multiple core QAT case
- IPSec two core test with QAT offload, performance is poor on Ampere - need to investigate
-
- VPP SVE implementation - Lijian
12/06/2022
-
Attendees
- Juraj Linkes
- Lijian Zhang
- Jieqiang Wang
- Tianyu Li
-
Miscellaneous
- Reschedule the meeting to 8:30 am for Juraj and 3:30 pm for Shanghai folks
-
CSIT
- VPP Performance Test
- https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-3n-tsh/
- https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-report-iterative-2106-3n-tsh/
-
https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-3n-alt/
-
- DPDK testpmd XL710 interface not up failure(2 ampere back to back) - VPP and other apps, the same interface works fine.
- Local setup not reproduced - need to schedule a debug session to reproduce the issue in FD.io lab.
- Confirm with Vexxhost people if replacing intel NICs is feasible
- Tried to wait more time, interface still not up, restart is not enough either - Need to figure out reliable workaround.
- Tried old version of DPDK - 21.08 does not work. May need to try older version.
- Will it related to NIC's Speed, Duplex and Auto negotiation configuration?
- May try to upgrade the NIC's firmware. - check local xl710 firmware version
- Will talk to dpdk i40e maintainer to seek their help
- DPDK testpmd XL710 interface not up failure(2 ampere back to back) - VPP and other apps, the same interface works fine.
- New links for VPP perf trending/report pages
- Daily trending: https://s3-docs.fd.io/csit/master/trending/
- Release report: https://s3-docs.fd.io/csit/master/report/
- Need to investigate 22.10 release testing result
- Compiler version change seems to be one of factors for perf degradation
- Old version: clang 10.0.0-4ubuntu1, gcc Ubuntu 9.4.0-1ubuntu1~20.04.1
- New version: clang 14.0.0-1ubuntu1, gcc Ubuntu 11.3.0-1ubuntu1~22.04
- Compiler version change seems to be one of factors for perf degradation
- VM testcase timeout issue on 3-tsh testbed
- Timeout issue occured when starting VPP inside VM, but not for starting testpmd
- Config isolcpu in kernel boot parameter is deprecated, Tianyu proposed a solution that Juraj would try
-
- VPP Performance Test
-
- VPP Path
- Voting and working fine.
- CentOS-8 jobs have been removed.
- VPP Path
-
- VPP Device
- Enable VPP device testing per patch
- Voting right for VPP device testing on Arm is enabled
- VPP device testing on Arm runs per VPP/CSIT patch
- Enable VPP device testing per patch
- QAT cards
- Govind will ship another 2x QAT from Austin to FD.io lab
- Will procure 2x QAT cards and verify them internally firstly.
- The existing QAT cards will be removed and returned to Vexxhost/FD.io lab
- QAT cards can be seen with new kernel update
- VPP Device
-
VPP
- VPP SVE implementation - Lijian
- SVE validation on FPGA platform - Confluence page ready
- Run L3 traffic testing with NEON/SVE-VLA/SVE-VLS VPP version
- Performance number: NEON>SVE-VLA>SVE-VLS on arm internal platform
- Investigate SVE vs NEON packet checksum comparison
- Investigate VPP cross compilation with buildroot - for running VPP on FVP - distro like ubuntu is slower than buildroot - Lijian
- Depends on some libraries, dpdk, ipsec_mb, rdma-core and nasm etc - optional
- Investigate One Terabit throughput test on Arm platform
- Investigate automate rfc2544 no drop rate throughput test with Ixia on N1 platform - Tianyu
- Kernel cmdline may impact on NDR PDR results - Jieqiang
- Intern help to benchmark VPP using 100G MLX5 NIC - scalabilty test
- VPP IPsec on Arm - Govind
- Work on IPsec input - Zach
- IPSec input node/decryption flow cache implemented in a separate patch - Zach
- Upstreamed https://gerrit.fd.io/r/c/vpp/+/32903
- Review SPD flow cache patch from Intel folks - Zach
-
- Depends on kernel patch
- Kernel with aarch64 patch is expected to release soon
- Depends on kernel patch
- Building Intel QAT driver on arm to test IPsec crypt - Picchi
- Run IPsec with QAT on skylake this week, run the same setup on Ampere is next step
- Patch to resolve iommu issue for mlx NIC when using with QAT card
- Benchmark IPSec test case with QAT card/mlx NIC(single-core/multi-core)
- Needs a kernel patch to resolve crash issue for QAT card
- Patch made by Yoan is upstream and waits for review
- Try patched VPP to verify QAT card usage
- Investigating crash issue with 90% linerate IPSec traffic with QAT card
- VPP rdma native driver only - vpp meta data corrupt - may related to memory barrier
- QAT single core test done - investigate multiple core QAT case
- IPSec two core test with QAT offload, performance is poor on Ampere - need to investigate
-
- VPP SVE implementation - Lijian
11/15/2022
-
Attendees
- Juraj Linkes
- Lijian Zhang
- Jieqiang Wang
- Tianyu Li
-
Miscellaneous
- Reschedule the meeting to 9 am for Juraj and 3 pm for Shanghai folks
-
CSIT
- VPP Performance Test
- https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-3n-tsh/
-
https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-report-iterative-2106-3n-tsh/
-
https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-3n-alt/
- CSIT perf numbers VS local perf numbers
- VPP cloud image in CSIT VS native built VPP in local env
- One DPDK patch introduced perf degradation on Arm platform
- Configuration difference between CSIT env and local env(Hugepage size, startup.conf parameters and etc)
- DPDK testpmd XL710 interface not up failure(2 ampere back to back) - VPP and other apps, the same interface works fine.
- Local setup not reproduced - need to schedule a debug session to reproduce the issue in FD.io lab.
- Confirm with Vexxhost people if replacing intel NICs is feasible
- Tried to wait more time, interface still not up, restart is not enough either - Need to figure out reliable workaround.
- Tried old version of DPDK - 21.08 does not work. May need to try older version.
- Will it related to NIC's Speed, Duplex and Auto negotiation configuration?
- May try to upgrade the NIC's firmware. - check local xl710 firmware version
- CSIT perf numbers VS local perf numbers
- New links for VPP perf trending/report pages
- Daily trending: https://s3-docs.fd.io/csit/master/trending/
- Release report: https://s3-docs.fd.io/csit/master/report/
- Need to investigate 22.10 release testing result
-
https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-3n-alt/
- VPP Performance Test
-
- VPP Path
- Voting and working fine.
- CentOS-8 jobs have been removed.
- VPP Path
-
- VPP Device
- Device Testing on ThunderX2 servers
- Juraj will commit the patch to disable the failling test cases
-
https://jenkins.fd.io/view/vpp/job/vpp-csit-verify-device-master-ubuntu2004-aarch64-1n-tx2/
- Good news, No more slow down after 200 rounds of testing.
- Thunderx2 servers need to upgrade from 20.04 to 22.04 - Peter - ongoing, ETA a few days to 2 weeks
- Distro upgrade to ubuntu 22.04 is still ongoing - no ETA yet
- Server configuration will remain the same, already integrated in ansible playbook
- Re-enable voting IF no more issue with 22.04 device testing
- Submit a patch to enable voting right after meeting
- Good news, No more slow down after 200 rounds of testing.
- Test meltdown/spectre vulnerabilities
- CSIT maintainers ask for tools if existing to test vulnerabilities on Arm platform(not just limited to Arm)
- Will confirm this issue with support team - Lijian
- VM cases failed only on Arm
- Tried to increase the timeout to see it will fix the issue
- VM cases failed on 1 min timeout on creating VM (qemu cli)- tx2 node
- VM cases failed only on 3n-alt performance testbed, error log report some file missing, likely configuration issue
- Another intermit failed VM issue happens on tx2 and alt, need to figure out above case first
- Reboot server recover and monitoring
- Need to look into it, try manually
- May need to upgrade iavf driver
- Server in-accessiable
- Reboot server recover the service
- AVF interface creation issue:
- Can't create AVF interface on VFs with configured VLAN - happens with latest i40e driver on tx2
- Related to PF i40e driver, vlan stripping configured on VF, PF driver return not allowd.
- Not related to iavf driver, AVF interface - vpp native driver have this issue
- dpdk iavf ignore the error and continue initialization, while vpp abort the init process
- Intel will fix the issue from PF driver - workaround: use old i40e driver (from ubuntu 20.04)
- Race condition occur on /dev/vfio mounting
-
- Run six rounds of tests with patch https://gerrit.fd.io/r/c/csit/+/34045 and result looks good right now
- Patch has been merged
- By not mounting whole /dev/vfio, mount only /dev/vfio/xxx used
- Addressed comments, waiting Peter's review - Peter approved, patch merged and monitoring.
- Met tx2 server reboot issue when monitoring - RAS CONTROLLER: Fatal unrecoverable error detected ** NBU Error **
- Reinstall the tx2 server with older kernel version and VPP device testing works fine with Juraj's patch
- VPP Device configuration align with VPP Performance configuration - no issue yet
- Run six rounds of tests with patch https://gerrit.fd.io/r/c/csit/+/34045 and result looks good right now
- Enable VPP device testing per patch
- Monitor for a week and enable vote right then
- Ping Dave about enabling VPP device testing per patch
- Dave gave minor comments https://gerrit.fd.io/r/c/ci-management/+/34679 - comment addressed and will be merged soon
- Need to figure out how to reproduce the error - Juraj
- Periodic job will stop when per patch job enabled
-
- Can't create AVF interface on VFs with configured VLAN - happens with latest i40e driver on tx2
- Device Testing on ThunderX2 servers
- QAT cards
- Govind will ship another 2x QAT from Austin to FD.io lab
- Will procure 2x QAT cards and verify them internally firstly.
- The existing QAT cards will be removed and returned to Vexxhost/FD.io lab
- QAT cards can be seen with new kernel update
- VPP Device
-
VPP
- VPP SVE implementation - Lijian
- SVE validation on FPGA platform - Confluence page ready
- Run L3 traffic testing with NEON/SVE-VLA/SVE-VLS VPP version
- Performance number: NEON>SVE-VLA>SVE-VLS on arm internal platform
- Investigate SVE vs NEON packet checksum comparison
- Investigate VPP cross compilation with buildroot - for running VPP on FVP - distro like ubuntu is slower than buildroot - Lijian
- Depends on some libraries, dpdk, ipsec_mb, rdma-core and nasm etc - optional
- Investigate One Terabit throughput test on Arm platform
- Investigate automate rfc2544 no drop rate throughput test with Ixia on N1 platform - Tianyu
- Kernel cmdline may impact on NDR PDR results - Jieqiang
- Intern help to benchmark VPP using 100G MLX5 NIC - scalabilty test
- VPP IPsec on Arm - Govind
- Work on IPsec input - Zach
- IPSec input node/decryption flow cache implemented in a separate patch - Zach
- Upstreamed https://gerrit.fd.io/r/c/vpp/+/32903
- Review SPD flow cache patch from Intel folks - Zach
-
- Depends on kernel patch
- Kernel with aarch64 patch is expected to release soon
- Depends on kernel patch
- Building Intel QAT driver on arm to test IPsec crypt - Picchi
- Run IPsec with QAT on skylake this week, run the same setup on Ampere is next step
- Patch to resolve iommu issue for mlx NIC when using with QAT card
- Benchmark IPSec test case with QAT card/mlx NIC(single-core/multi-core)
- Needs a kernel patch to resolve crash issue for QAT card
- Patch made by Yoan is upstream and waits for review
- Try patched VPP to verify QAT card usage
- Investigating crash issue with 90% linerate IPSec traffic with QAT card
- VPP rdma native driver only - vpp meta data corrupt - may related to memory barrier
- QAT single core test done - investigate multiple core QAT case
- IPSec two core test with QAT offload, performance is poor on Ampere - need to investigate
-
- VPP SVE implementation - Lijian
10/18/2022
-
Attendees
- Juraj Linkes
- Tianyu Li
- Lijian Zhang
- Jieqiang Wang
-
Miscellaneous
- Reschedule the meeting to 9 am for Juraj and 3 pm for Shanghai folks
-
CSIT
- VPP Performance Test
- https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-3n-tsh/
-
https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-report-iterative-2106-3n-tsh/
-
https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-3n-alt/
- CSIT perf numbers VS local perf numbers
- VPP cloud image in CSIT VS native built VPP in local env
- One DPDK patch introduced perf degradation on Arm platform
- Configuration difference between CSIT env and local env(Hugepage size, startup.conf parameters and etc)
- Juraj will write down the procedures on setting up Ampere Altra setup in FD.io lab
- And the procedures of developing/developing test cases in CSIT (performance & device testing)
- Juraj should have already sent to Jieqiang previously.
- 22.06 release testing will happen soon
- NDR/PDR data difference - deep dive needed, waiting ampere folks engagement
- DPDK testpmd XL710 interface not up failure(2 ampere back to back) - VPP and other apps, the same interface works fine.
- Local setup not reproduced - need to schedule a debug session to reproduce the issue in FD.io lab.
- Tried to wait more time, interface still not up, restart is not enough either - Need to figure out reliable workaround.
- Replace XL710 NIC? - try asking tomorrow.
- Tried old version of DPDK - 21.08 does not work. May need to try older version.
- Will it related to NIC's Speed, Duplex and Auto negotiation configuration?
- May try to upgrade the NIC's firmware. - check local xl710 firmware version
- CSIT perf numbers VS local perf numbers
- New links for VPP perf trending/report pages
- Daily trending: https://s3-docs.fd.io/csit/master/trending/
- Release report: https://s3-docs.fd.io/csit/master/report/
- NUMA issue
- Will run performance report on Arm testbed onece patch to resolve NUMA issue is merged
- Dave will help merge the patch into the corresponding branches
-
https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-3n-alt/
- VPP Performance Test
-
- VPP Path
- Voting and working fine.
- CentOS-8 jobs have been removed.
- VPP Device
- Device Testing on ThunderX2 servers
- Juraj will commit the patch to disable the failling test cases
-
https://jenkins.fd.io/view/vpp/job/vpp-csit-verify-device-master-ubuntu2004-aarch64-1n-tx2/
- Good news, No more slow down after 200 rounds of testing.
- Thunderx2 servers need to upgrade from 20.04 to 22.04 - Peter - ongoing, ETA a few days to 2 weeks
- Suggest to rerun test after upgrade to 22.04
- Re-enable voting after not more issue with 22.04 device testing
- Good news, No more slow down after 200 rounds of testing.
- Device Testing on ThunderX2 servers
- VPP Path
-
-
- VM cases failed only on Arm
- Tried to increase the timeout to see it will fix the issue
- VM cases failed on 1 min timeout on creating VM (qemu cli)- tx2 node
- Reboot server recover and monitoring
- Need to look into it, try manually
- May need to upgrade iavf driver
- Server in-accessiable
- Reboot server recover the service
- AVF interface creation issue:
- Can't create AVF interface on VFs with configured VLAN - happens with latest i40e driver on tx2
- Related to PF i40e driver, vlan stripping configured on VF, PF driver return not allowd.
- Not related to iavf driver, AVF interface - vpp native driver have this issue
- dpdk iavf ignore the error and continue initialization, while vpp abort the init process
- Intel will fix the issue from PF driver - workaround: use old i40e driver (from ubuntu 20.04)
- Race condition occur on /dev/vfio mounting
-
- Run six rounds of tests with patch https://gerrit.fd.io/r/c/csit/+/34045 and result looks good right now
- Patch has been merged
- By not mounting whole /dev/vfio, mount only /dev/vfio/xxx used
- Addressed comments, waiting Peter's review - Peter approved, patch merged and monitoring.
- Met tx2 server reboot issue when monitoring - RAS CONTROLLER: Fatal unrecoverable error detected ** NBU Error **
- Reinstall the tx2 server with older kernel version and VPP device testing works fine with Juraj's patch
- VPP Device configuration align with VPP Performance configuration - no issue yet
- Run six rounds of tests with patch https://gerrit.fd.io/r/c/csit/+/34045 and result looks good right now
- Enable VPP device testing per patch
- Monitor for a week and enable vote right then
- Ping Dave about enabling VPP device testing per patch
- Dave gave minor comments https://gerrit.fd.io/r/c/ci-management/+/34679 - comment addressed and will be merged soon
- Need to figure out how to reproduce the error - Juraj
- Periodic job will stop when per patch job enabled
-
- Can't create AVF interface on VFs with configured VLAN - happens with latest i40e driver on tx2
- VM cases failed only on Arm
- QAT cards
- Govind will ship another 2x QAT from Austin to FD.io lab
- Will procure 2x QAT cards and verify them internally firstly.
- The existing QAT cards will be removed and returned to Vexxhost/FD.io lab
- QAT cards can be seen with new kernel update
-
-
VPP
- VPP SVE implementation - Lijian
- SVE validation on FPGA platform - Confluence page ready
- Run L3 traffic testing with NEON/SVE-VLA/SVE-VLS VPP version
- Performance number: NEON>SVE-VLA>SVE-VLS on arm internal platform
- Investigate SVE vs NEON packet checksum comparison
- Investigate VPP cross compilation with buildroot - for running VPP on FVP - distro like ubuntu is slower than buildroot - Lijian
- Depends on some libraries, dpdk, ipsec_mb, rdma-core and nasm etc - optional
- Investigate One Terabit throughput test on Arm platform
- Investigate automate rfc2544 no drop rate throughput test with Ixia on N1 platform - Tianyu
- Kernel cmdline may impact on NDR PDR results - Jieqiang
- Intern help to benchmark VPP using 100G MLX5 NIC - scalabilty test
- VPP IPsec on Arm - Govind
- Work on IPsec input - Zach
- IPSec input node/decryption flow cache implemented in a separate patch - Zach
- Upstreamed https://gerrit.fd.io/r/c/vpp/+/32903
- Review SPD flow cache patch from Intel folks - Zach
-
- Depends on kernel patch
- Kernel with aarch64 patch is expected to release soon
- Depends on kernel patch
- Building Intel QAT driver on arm to test IPsec crypt - Picchi
- Run IPsec with QAT on skylake this week, run the same setup on Ampere is next step
- Patch to resolve iommu issue for mlx NIC when using with QAT card
- Benchmark IPSec test case with QAT card/mlx NIC(single-core/multi-core)
- Needs a kernel patch to resolve crash issue for QAT card
- Patch made by Yoan is upstream and waits for review
- Try patched VPP to verify QAT card usage
- Investigating crash issue with 90% linerate IPSec traffic with QAT card
- VPP rdma native driver only - vpp meta data corrupt - may related to memory barrier
- QAT single core test done - investigate multiple core QAT case
- IPSec two core test with QAT offload, performance is poor on Ampere - need to investigate
-
- VPP SVE implementation - Lijian
9/20/2022
-
Attendees
- Govindarajan Mohandoss
- Juraj Linkes
- Tianyu Li
- Lijian Zhang
- Jieqiang Wang
-
CSIT
- VPP Performance Test
- https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-3n-tsh/
-
https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-report-iterative-2106-3n-tsh/
-
https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-3n-alt/
- CSIT perf numbers VS local perf numbers
- VPP cloud image in CSIT VS native built VPP in local env
- One DPDK patch introduced perf degradation on Arm platform
- Configuration difference between CSIT env and local env(Hugepage size, startup.conf parameters and etc)
- Juraj will write down the procedures on setting up Ampere Altra setup in FD.io lab
- And the procedures of developing/developing test cases in CSIT (performance & device testing)
- Juraj should have already sent to Jieqiang previously.
- 22.06 release testing will happen soon
- NDR/PDR data difference - deep dive needed, waiting ampere folks engagement
- DPDK testpmd XL710 interface not up failure(2 ampere back to back) - VPP and other apps, the same interface works fine.
- Local setup not reproduced - need to schedule a debug session to reproduce the issue in FD.io lab.
- Tried to wait more time, interface still not up, restart is not enough either - Need to figure out reliable workaround.
- Replace XL710 NIC? - try asking tomorrow.
- Tried old version of DPDK - 21.08 does not work. May need to try older version.
- Will it related to NIC's Speed, Duplex and Auto negotiation configuration?
- May try to upgrade the NIC's firmware. - check local xl710 firmware version
- CSIT perf numbers VS local perf numbers
- New links for VPP perf trending/report pages
- Daily trending: https://s3-docs.fd.io/csit/master/trending/
- Release report: https://s3-docs.fd.io/csit/master/report/
-
https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-3n-alt/
- VPP Performance Test
-
- VPP Path
- Voting and working fine.
- CentOS-8 jobs have been removed.
- VPP Device
- Device Testing on ThunderX2 servers
- Juraj will commit the patch to disable the failling test cases
-
https://jenkins.fd.io/view/vpp/job/vpp-csit-verify-device-master-ubuntu2004-aarch64-1n-tx2/
- Good news, No more slow down after 200 rounds of testing.
- Thunderx2 servers need to upgrade from 20.04 to 22.04 - Peter - ongoing, ETA a few days to 2 weeks
- Suggest to rerun test after upgrade to 22.04
- Re-enable voting after not more issue with 22.04 device testing
- Good news, No more slow down after 200 rounds of testing.
- Device Testing on ThunderX2 servers
- VPP Path
-
-
- VM cases failed only on Arm
- Tried to increase the timeout to see it will fix the issue
- VM cases failed on 1 min timeout on creating VM (qemu cli)- tx2 node
- Reboot server recover and monitoring
- Need to look into it, try manually
- May need to upgrade iavf driver
- Server in-accessiable
- Reboot server recover the service
- AVF interface creation issue:
- Can't create AVF interface on VFs with configured VLAN - happens with latest i40e driver on tx2
- Related to PF i40e driver, vlan stripping configured on VF, PF driver return not allowd.
- Not related to iavf driver, AVF interface - vpp native driver have this issue
- dpdk iavf ignore the error and continue initialization, while vpp abort the init process
- Intel will fix the issue from PF driver - workaround: use old i40e driver (from ubuntu 20.04)
- Race condition occur on /dev/vfio mounting
-
- Run six rounds of tests with patch https://gerrit.fd.io/r/c/csit/+/34045 and result looks good right now
- Patch has been merged
- By not mounting whole /dev/vfio, mount only /dev/vfio/xxx used
- Addressed comments, waiting Peter's review - Peter approved, patch merged and monitoring.
- Met tx2 server reboot issue when monitoring - RAS CONTROLLER: Fatal unrecoverable error detected ** NBU Error **
- Reinstall the tx2 server with older kernel version and VPP device testing works fine with Juraj's patch
- VPP Device configuration align with VPP Performance configuration - no issue yet
- Run six rounds of tests with patch https://gerrit.fd.io/r/c/csit/+/34045 and result looks good right now
- Enable VPP device testing per patch
- Monitor for a week and enable vote right then
- Ping Dave about enabling VPP device testing per patch
- Dave gave minor comments https://gerrit.fd.io/r/c/ci-management/+/34679 - comment addressed and will be merged soon
- Need to figure out how to reproduce the error - Juraj
- Periodic job will stop when per patch job enabled
-
- Can't create AVF interface on VFs with configured VLAN - happens with latest i40e driver on tx2
- VM cases failed only on Arm
- QAT cards
- Govind will ship another 2x QAT from Austin to FD.io lab
- Will procure 2x QAT cards and verify them internally firstly.
- The existing QAT cards will be removed and returned to Vexxhost/FD.io lab
- QAT enabled Kernel patch release about October, upgrade kernel required.
- QAT test cases are developed based on Python APIs / CLIs
- VPP build servers - 2 new ampere altra server, 2 old thunder x1 servers decommission or not - need to confirm
-
-
VPP
- VPP SVE implementation - Lijian
- SVE validation on FPGA platform - Confluence page ready
- Run L3 traffic testing with NEON/SVE-VLA/SVE-VLS VPP version
- Performance number: NEON>SVE-VLA>SVE-VLS on arm internal platform
- Investigate SVE vs NEON packet checksum comparison
- Investigate VPP cross compilation with buildroot - for running VPP on FVP - distro like ubuntu is slower than buildroot - Lijian
- Depends on some libraries, dpdk, ipsec_mb, rdma-core and nasm etc - optional
- Investigate One Terabit throughput test on Arm platform
- Investigate automate rfc2544 no drop rate throughput test with Ixia on N1 platform - Tianyu
- Kernel cmdline may impact on NDR PDR results - Jieqiang
- Intern help to benchmark VPP using 100G MLX5 NIC - scalabilty test
- VPP IPsec on Arm - Govind
- Work on IPsec input - Zach
- IPSec input node/decryption flow cache implemented in a separate patch - Zach
- Upstreamed https://gerrit.fd.io/r/c/vpp/+/32903
- Review SPD flow cache patch from Intel folks - Zach
-
- Depends on kernel patch
- Kernel with aarch64 patch is expected to release soon
- Depends on kernel patch
- Building Intel QAT driver on arm to test IPsec crypt - Picchi
- Run IPsec with QAT on skylake this week, run the same setup on Ampere is next step
- Patch to resolve iommu issue for mlx NIC when using with QAT card
- Benchmark IPSec test case with QAT card/mlx NIC(single-core/multi-core)
- Needs a kernel patch to resolve crash issue for QAT card
- Patch made by Yoan is upstream and waits for review
- Try patched VPP to verify QAT card usage
- Investigating crash issue with 90% linerate IPSec traffic with QAT card
- VPP rdma native driver only - vpp meta data corrupt - may related to memory barrier
- QAT single core test done - investigate multiple core QAT case
- IPSec two core test with QAT offload, performance is poor on Ampere - need to investigate
-
- VPP SVE implementation - Lijian
9/6/2022
-
Attendees
- Govindarajan Mohandoss
- Juraj Linkes
- Tianyu Li
- Lijian Zhang
-
CSIT
- VPP Performance Test
- https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-3n-tsh/
-
https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-report-iterative-2106-3n-tsh/
-
https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-3n-alt/
- CSIT perf numbers VS local perf numbers
- VPP cloud image in CSIT VS native built VPP in local env
- One DPDK patch introduced perf degradation on Arm platform
- Configuration difference between CSIT env and local env(Hugepage size, startup.conf parameters and etc)
- Juraj will write down the procedures on setting up Ampere Altra setup in FD.io lab
- And the procedures of developing/developing test cases in CSIT (performance & device testing)
- Juraj should have already sent to Jieqiang previously.
- 22.06 release testing will happen soon
- NDR/PDR data difference - deep dive needed, waiting ampere folks engagement
- DPDK testpmd XL710 interface not up failure(2 ampere back to back) - VPP and other apps, the same interface works fine.
- Local setup not reproduced - need to schedule a debug session to reproduce the issue in FD.io lab.
- Tried to wait more time, interface still not up, restart is not enough either - Need to figure out reliable workaround.
- Replace XL710 NIC? - try asking tomorrow.
- Tried old version of DPDK - 21.08 does not work. May need to try older version.
- Will it related to NIC's Speed, Duplex and Auto negotiation configuration?
- May try to upgrade the NIC's firmware. - check local xl710 firmware version
- CSIT perf numbers VS local perf numbers
- New links for VPP perf trending/report pages
- Daily trending: https://s3-docs.fd.io/csit/master/trending/
- Release report: https://s3-docs.fd.io/csit/master/report/
-
https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-3n-alt/
- VPP Performance Test
-
- VPP Path
- Voting and working fine.
- CentOS-8 jobs have been removed.
- VPP Device
- Device Testing on ThunderX2 servers
- Juraj will commit the patch to disable the failling test cases
-
https://jenkins.fd.io/view/vpp/job/vpp-csit-verify-device-master-ubuntu2004-aarch64-1n-tx2/
- Good news, No more slow down after 200 rounds of testing.
- Thunderx2 servers need to upgrade from 20.04 to 22.04 - Peter - ongoing, ETA a few days to 2 weeks
- Suggest to rerun test after upgrade to 22.04
- Re-enable voting after not more issue with 22.04 device testing
- Good news, No more slow down after 200 rounds of testing.
- Device Testing on ThunderX2 servers
- VPP Path
-
-
- VM cases failed only on Arm
- Tried to increase the timeout to see it will fix the issue
- VM cases failed on 1 min timeout on creating VM (qemu cli)- tx2 node
- Reboot server recover and monitoring
- Need to look into it, try manually
- May need to upgrade iavf driver
- Server in-accessiable
- Reboot server recover the service
- AVF interface creation issue:
- Can't create AVF interface on VFs with configured VLAN - happens with latest i40e driver on tx2
- Related to PF i40e driver, vlan stripping configured on VF, PF driver return not allowd.
- Not related to iavf driver, AVF interface - vpp native driver have this issue
- dpdk iavf ignore the error and continue initialization, while vpp abort the init process
- Intel will fix the issue from PF driver - workaround: use old i40e driver (from ubuntu 20.04)
- Race condition occur on /dev/vfio mounting
-
- Run six rounds of tests with patch https://gerrit.fd.io/r/c/csit/+/34045 and result looks good right now
- Patch has been merged
- By not mounting whole /dev/vfio, mount only /dev/vfio/xxx used
- Addressed comments, waiting Peter's review - Peter approved, patch merged and monitoring.
- Met tx2 server reboot issue when monitoring - RAS CONTROLLER: Fatal unrecoverable error detected ** NBU Error **
- Reinstall the tx2 server with older kernel version and VPP device testing works fine with Juraj's patch
- VPP Device configuration align with VPP Performance configuration - no issue yet
- Run six rounds of tests with patch https://gerrit.fd.io/r/c/csit/+/34045 and result looks good right now
- Enable VPP device testing per patch
- Monitor for a week and enable vote right then
- Ping Dave about enabling VPP device testing per patch
- Dave gave minor comments https://gerrit.fd.io/r/c/ci-management/+/34679 - comment addressed and will be merged soon
- Need to figure out how to reproduce the error - Juraj
- Periodic job will stop when per patch job enabled
-
- Can't create AVF interface on VFs with configured VLAN - happens with latest i40e driver on tx2
- VM cases failed only on Arm
- QAT cards
- Govind will ship another 2x QAT from Austin to FD.io lab
- Will procure 2x QAT cards and verify them internally firstly.
- The existing QAT cards will be removed and returned to Vexxhost/FD.io lab
- QAT enabled Kernel patch release about October, upgrade kernel required.
- QAT test cases are developed based on Python APIs / CLIs
- VPP build servers - 2 new ampere altra server, 2 old thunder x1 servers decommission or not - need to confirm
-
-
VPP
- VPP SVE implementation - Lijian
- SVE validation on FPGA platform - Confluence page ready
- Run L3 traffic testing with NEON/SVE-VLA/SVE-VLS VPP version
- Performance number: NEON>SVE-VLA>SVE-VLS on arm internal platform
- Investigate SVE vs NEON packet checksum comparison
- Investigate VPP cross compilation with buildroot - for running VPP on FVP - distro like ubuntu is slower than buildroot - Lijian
- Depends on some libraries, dpdk, ipsec_mb, rdma-core and nasm etc - optional
- Investigate One Terabit throughput test on Arm platform
- Investigate automate rfc2544 no drop rate throughput test with Ixia on N1 platform - Tianyu
- Kernel cmdline may impact on NDR PDR results - Jieqiang
- Intern help to benchmark VPP using 100G MLX5 NIC - scalabilty test
- VPP IPsec on Arm - Govind
- Work on IPsec input - Zach
- IPSec input node/decryption flow cache implemented in a separate patch - Zach
- Upstreamed https://gerrit.fd.io/r/c/vpp/+/32903
- Review SPD flow cache patch from Intel folks - Zach
-
- Depends on kernel patch
- Kernel with aarch64 patch is expected to release soon
- Depends on kernel patch
- Building Intel QAT driver on arm to test IPsec crypt - Picchi
- Run IPsec with QAT on skylake this week, run the same setup on Ampere is next step
- Patch to resolve iommu issue for mlx NIC when using with QAT card
- Benchmark IPSec test case with QAT card/mlx NIC(single-core/multi-core)
- Needs a kernel patch to resolve crash issue for QAT card
- Patch made by Yoan is upstream and waits for review
- Try patched VPP to verify QAT card usage
- Investigating crash issue with 90% linerate IPSec traffic with QAT card
- VPP rdma native driver only - vpp meta data corrupt - may related to memory barrier
- QAT single core test done - investigate multiple core QAT case
- IPSec two core test with QAT offload, performance is poor on Ampere - need to investigate
-
- VPP SVE implementation - Lijian
8/16/2022
-
Attendees
- Govindarajan Mohandoss
- Juraj Linkes
- Masksym Vynnvk
- Jieqiang Wang
- Tianyu Li
- Lijian Zhang
-
CSIT
- VPP Performance Test
- https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-3n-tsh/
-
https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-report-iterative-2106-3n-tsh/
-
https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-3n-alt/
- CSIT perf numbers VS local perf numbers
- VPP cloud image in CSIT VS native built VPP in local env
- One DPDK patch introduced perf degradation on Arm platform
- Configuration difference between CSIT env and local env(Hugepage size, startup.conf parameters and etc)
- Juraj will write down the procedures on setting up Ampere Altra setup in FD.io lab
- And the procedures of developing/developing test cases in CSIT (performance & device testing)
- Juraj should have already sent to Jieqiang previously.
- 22.06 release testing will happen soon
- NDR/PDR data difference - deep dive needed
- CSIT perf numbers VS local perf numbers
- New links for VPP perf trending/report pages
- Daily trending: https://s3-docs.fd.io/csit/master/trending/
- Release report: https://s3-docs.fd.io/csit/master/report/
-
https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-3n-alt/
- VPP Performance Test
-
- VPP Path
- Voting and working fine.
- CentOS-8 jobs have been removed.
- VPP Device
- Device Testing on ThunderX2 servers
- Juraj will commit the patch to disable the failling test cases
- https://jenkins.fd.io/view/vpp/job/vpp-csit-verify-device-master-ubuntu2004-aarch64-1n-tx2/
- VM cases failed only on Arm
- Tried to increase the timeout to see it will fix the issue
- VM cases failed on 1 min timeout on creating VM (qemu cli)- tx2 node
- Reboot server recover and monitoring
- Need to look into it, try manually
- May need to upgrade iavf driver
- Server in-accessiable
- Reboot server recover the service
- AVF interface creation issue:
- Can't create AVF interface on VFs with configured VLAN - happens with latest i40e driver on tx2
- Related to PF i40e driver, vlan stripping configured on VF, PF driver return not allowd.
- Not related to iavf driver, AVF interface - vpp native driver have this issue
- dpdk iavf ignore the error and continue initialization, while vpp abort the init process
- Intel will fix the issue from PF driver - workaround: use old i40e driver (from ubuntu 20.04)
- Race condition occur on /dev/vfio mounting
-
- Run six rounds of tests with patch https://gerrit.fd.io/r/c/csit/+/34045 and result looks good right now
- Patch has been merged
- By not mounting whole /dev/vfio, mount only /dev/vfio/xxx used
- Addressed comments, waiting Peter's review - Peter approved, patch merged and monitoring.
- Met tx2 server reboot issue when monitoring - RAS CONTROLLER: Fatal unrecoverable error detected ** NBU Error **
- Reinstall the tx2 server with older kernel version and VPP device testing works fine with Juraj's patch
- VPP Device configuration align with VPP Performance configuration - no issue yet
- Run six rounds of tests with patch https://gerrit.fd.io/r/c/csit/+/34045 and result looks good right now
- Enable VPP device testing per patch
- Monitor for a week and enable vote right then
- Ping Dave about enabling VPP device testing per patch
- Dave gave minor comments https://gerrit.fd.io/r/c/ci-management/+/34679 - comment addressed and will be merged soon
- Need to figure out how to reproduce the error - Juraj
- Periodic job will stop when per patch job enabled
-
- Can't create AVF interface on VFs with configured VLAN - happens with latest i40e driver on tx2
- Device Testing on ThunderX2 servers
- QAT cards
- Govind will ship another 2x QAT from Austin to FD.io lab
- Will procure 2x QAT cards and verify them internally firstly.
- The existing QAT cards will be removed and returned to Vexxhost/FD.io lab
- QAT test cases are developed based on Python APIs / CLIs
- VPP build servers - 2 new ampere altra server, 2 old thunder x1 servers decommission or not - need to confirm
- VPP Path
-
VPP
- VPP SVE implementation - Lijian
- SVE validation on FPGA platform - Confluence page ready
- Run L3 traffic testing with NEON/SVE-VLA/SVE-VLS VPP version
- Performance number: NEON>SVE-VLA>SVE-VLS on arm internal platform
- Investigate SVE vs NEON packet checksum comparison
- Investigate VPP cross compilation with buildroot - for running VPP on FVP - distro like ubuntu is slower than buildroot - Lijian
- Depends on some libraries, dpdk, ipsec_mb, rdma-core and nasm etc - optional
- Investigate One Terabit throughput test on Arm platform
- Investigate automate rfc2544 no drop rate throughput test with Ixia on N1 platform - Tianyu
- Kernel cmdline may impact on NDR PDR results - Jieqiang
- Intern help to benchmark VPP using 100G MLX5 NIC - scalabilty test
- VPP IPsec on Arm - Govind
- Work on IPsec input - Zach
- IPSec input node/decryption flow cache implemented in a separate patch - Zach
- Upstreamed https://gerrit.fd.io/r/c/vpp/+/32903
- Review SPD flow cache patch from Intel folks - Zach
-
- Depends on kernel patch
- Kernel with aarch64 patch is expected to release soon
- Depends on kernel patch
- Building Intel QAT driver on arm to test IPsec crypt - Picchi
- Run IPsec with QAT on skylake this week, run the same setup on Ampere is next step
- Patch to resolve iommu issue for mlx NIC when using with QAT card
- Benchmark IPSec test case with QAT card/mlx NIC(single-core/multi-core)
- Needs a kernel patch to resolve crash issue for QAT card
- Patch made by Yoan is upstream and waits for review
- Try patched VPP to verify QAT card usage
- Investigating crash issue with 90% linerate IPSec traffic with QAT card
- VPP rdma native driver only - vpp meta data corrupt - may related to memory barrier
- QAT single core test done - investigate multiple core QAT case
- IPSec two core test with QAT offload, performance is poor on Ampere - need to investigate
-
- VPP SVE implementation - Lijian
8/2/2022
-
Attendees
- Govindarajan Mohandoss
- Juraj Linkes
- Masksym Vynnvk
- Jieqiang Wang
- Tianyu Li
-
CSIT
- VPP Performance Test
- https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-3n-tsh/
-
https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-report-iterative-2106-3n-tsh/
-
https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-3n-alt/
- CSIT perf numbers VS local perf numbers
- VPP cloud image in CSIT VS native built VPP in local env
- One DPDK patch introduced perf degradation on Arm platform
- Configuration difference between CSIT env and local env(Hugepage size, startup.conf parameters and etc)
- Check if there is customer support can help with the PEX installation issue - Juraj
- Juraj will write down the procedures on setting up Ampere Altra setup in FD.io lab
- And the procedures of developing/developing test cases in CSIT (performance & device testing)
- Juraj should have already sent to Jieqiang previously.
- 22.06 release testing will happen soon
- NDR PDR data difference - deep dive needed, MRR is
- CSIT perf numbers VS local perf numbers
- Add new IPSec NULL encryption & decryption test cases - Juraj
- IPsec SPD input/output test case ongoing - Juraj
- Enable the SPD outbound tests
- Patches ready, waiting release testing done - ETA 1 week or 2
- Outbound SPD test patch has been enabled
- Investigate Inbound SPD test cases - Juraj
- Juraj will commit the patch and get it confirmed with Zachary
- Release testing for 21.10 is done
- Enable the SPD outbound tests
- New links for VPP perf trending/report pages
- Daily trending: https://s3-docs.fd.io/csit/master/trending/
- Release report: https://s3-docs.fd.io/csit/master/report/
- Release testing for 22.02 is done
- 3 nodes Taishan crypto test case failed - related to CSIT change
- Patch ready for review and to be merged https://gerrit.fd.io/r/c/csit/+/34591 -- Merged
-
https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-3n-alt/
- VPP Path
- Voting and working fine.
- CentOS-8 jobs have been removed.
- VPP Device
- Device Testing on ThunderX2 servers
- Juraj will commit the patch to disable the failling test cases
- https://jenkins.fd.io/view/vpp/job/vpp-csit-verify-device-master-ubuntu2004-aarch64-1n-tx2/
- VM cases failed only on Arm
- Tried to increase the timeout to see it will fix the issue
- VM cases failed on 1 min timeout on creating VM (qemu cli)- tx2 node
- Reboot server recover and monitoring
- Need to look into it, try manually
- May need to upgrade iavf driver
- Server in-accessiable
- Reboot server recover the service
- AVF interface creation issue:
- Can't create AVF interface on VFs with configured VLAN - happens with latest i40e driver on tx2
- Related to PF i40e driver, vlan stripping configured on VF, PF driver return not allowd.
- Not related to iavf driver, AVF interface - vpp native driver have this issue
- dpdk iavf ignore the error and continue initialization, while vpp abort the init process
- Intel will fix the issue from PF driver - workaround: use old i40e driver (from ubuntu 20.04)
- Race condition occur on /dev/vfio mounting
-
- Run six rounds of tests with patch https://gerrit.fd.io/r/c/csit/+/34045 and result looks good right now
- Patch has been merged
- By not mounting whole /dev/vfio, mount only /dev/vfio/xxx used
- Addressed comments, waiting Peter's review - Peter approved, patch merged and monitoring.
- Met tx2 server reboot issue when monitoring - RAS CONTROLLER: Fatal unrecoverable error detected ** NBU Error **
- Reinstall the tx2 server with older kernel version and VPP device testing works fine with Juraj's patch
- VPP Device configuration align with VPP Performance configuration - no issue yet
- Run six rounds of tests with patch https://gerrit.fd.io/r/c/csit/+/34045 and result looks good right now
- Enable VPP device testing per patch
- Monitor for a week and enable vote right then
- Ping Dave about enabling VPP device testing per patch
- Dave gave minor comments https://gerrit.fd.io/r/c/ci-management/+/34679 - comment addressed and will be merged soon
- Need to figure out how to reproduce the error - Juraj
- Periodic job will stop when per patch job enabled
-
- Can't create AVF interface on VFs with configured VLAN - happens with latest i40e driver on tx2
- Device Testing on ThunderX2 servers
- New Arm servers shipment to the FD.io lab - about Jan 2022
- 2 build servers ready for shipment - 1 RU, no pcie slot for NICs
- 2 performance servers waiting for Intel NICs
- New servers are in the procurement process
- Plan to replace old thunderx1 build servers with more advanced Arm servers
- Need to confirm with RAM/disk size for the new build servers
- 500G disk/256G RAM
- Each job will consume about 16G memory
- Intel NIC firmware upgrade on Arm - not supported
- Performance servers have arrived at FD.io lab
- Servers are in the processing of wiring, expected to be operational soon
- Will follow the trend for Arm servers if more mlx NICs are installed on X86
- Plan to install QAT cards on performance servers
- Juraj to get QAT card avalibility from CSIT community
- QAT cards
- Govind will ship another 2x QAT from Austin to FD.io lab
- Will procure 2x QAT cards and verify them internally firstly.
- The existing QAT cards will be removed and returned to Vexxhost/FD.io lab
- QAT test cases are developed based on Python APIs / CLIs
- VPP build servers - 2 new ampere altra server, 2 old thunder x1 servers decommission or not - need to confirm
- VPP Performance Test
-
VPP
- VPP SVE implementation - Lijian
- SVE validation on FPGA platform - Confluence page ready
- Run L3 traffic testing with NEON/SVE-VLA/SVE-VLS VPP version
- Performance number: NEON>SVE-VLA>SVE-VLS on arm internal platform
- Investigate SVE vs NEON packet checksum comparison
- Investigate VPP cross compilation with buildroot - for running VPP on FVP - distro like ubuntu is slower than buildroot - Lijian
- Depends on some libraries, dpdk, ipsec_mb, rdma-core and nasm etc - optional
- Investigate One Terabit throughput test on Arm platform
- Investigate automate rfc2544 no drop rate throughput test with Ixia on N1 platform - Tianyu
- Kernel cmdline may impact on NDR PDR results - Jieqiang
- Intern help to benchmark VPP using 100G MLX5 NIC - scalabilty test
- CNF PoC proposal preparation- Tianyu
- Add support for VPP aarch64 docker image build
- Calico use cases exploration on VPP
- Improve ansible scripts to deploy VPP&snort on K8S pods automatically
- VPP IPsec on Arm - Govind
- Work on IPsec input - Zach
- IPSec input node/decryption flow cache implemented in a separate patch - Zach
- Upstreamed https://gerrit.fd.io/r/c/vpp/+/32903
- Review SPD flow cache patch from Intel folks - Zach
-
- Depends on kernel patch
- Kernel with aarch64 patch is expected to release soon
- Depends on kernel patch
- Building Intel QAT driver on arm to test IPsec crypt - Picchi
- Run IPsec with QAT on skylake this week, run the same setup on Ampere is next step
- Patch to resolve iommu issue for mlx NIC when using with QAT card
- Benchmark IPSec test case with QAT card/mlx NIC(single-core/multi-core)
- Needs a kernel patch to resolve crash issue for QAT card
- Patch made by Yoan is upstream and waits for review
- Try patched VPP to verify QAT card usage
- Investigating crash issue with 90% linerate IPSec traffic with QAT card
- VPP rdma native driver only - vpp meta data corrupt - may related to memory barrier
- QAT single core test done - investigate multiple core QAT case
- IPSec two core test with QAT offload, performance is poor on Ampere - need to investigate
-
- VPP SVE implementation - Lijian
7/19/2022
-
Attendees
- Govindarajan Mohandoss
- Lijian Zhang
- Jieqiang Wang
- Tianyu Li
-
CSIT
- VPP Performance Test
- https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-3n-tsh/
-
https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-report-iterative-2106-3n-tsh/
-
https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-3n-alt/
- CSIT perf numbers VS local perf numbers
- VPP cloud image in CSIT VS native built VPP in local env
- One DPDK patch introduced perf degradation on Arm platform
- Configuration difference between CSIT env and local env(Hugepage size, startup.conf parameters and etc)
- Check if there is customer support can help with the PEX installation issue - Juraj
- Juraj will write down the procedures on setting up Ampere Altra setup in FD.io lab
- And the procedures of developing/developing test cases in CSIT (performance & device testing)
- Juraj should have already sent to Jieqiang previously.
- 22.06 release testing will happen soon
- CSIT perf numbers VS local perf numbers
-
https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-3n-alt/
- VPP Performance Test
-
-
-
- Add new IPSec NULL encryption & decryption test cases - Juraj
- IPsec SPD input/output test case ongoing - Juraj
- Enable the SPD outbound tests
- Patches ready, waiting release testing done - ETA 1 week or 2
- Outbound SPD test patch has been enabled
- Investigate Inbound SPD test cases - Juraj
- Juraj will commit the patch and get it confirmed with Zachary
- Release testing for 21.10 is done
- Enable the SPD outbound tests
- New links for VPP perf trending/report pages
- Daily trending: https://s3-docs.fd.io/csit/master/trending/
- Release report: https://s3-docs.fd.io/csit/master/report/
- Release testing for 22.02 is done
- 3 nodes Taishan crypto test case failed - related to CSIT change
- Patch ready for review and to be merged https://gerrit.fd.io/r/c/csit/+/34591 -- Merged
-
- VPP Path
- Voting and working fine.
- CentOS-8 jobs have been removed.
- VPP Device
- Device Testing on ThunderX2 servers
- Juraj will commit the patch to disable the failling test cases
- https://jenkins.fd.io/view/vpp/job/vpp-csit-verify-device-master-ubuntu2004-aarch64-1n-tx2/
- VM cases failed only on Arm
- Tried to increase the timeout to see it will fix the issue
- VM cases failed on 1 min timeout on creating VM (qemu cli)- tx2 node
- Reboot server recover and monitoring
- Need to look into it, try manually
- May need to upgrade iavf driver
- Server in-accessiable
- Reboot server recover the service
- AVF interface creation issue:
- Can't create AVF interface on VFs with configured VLAN - happens with latest i40e driver on tx2
- Related to PF i40e driver, vlan stripping configured on VF, PF driver return not allowd.
- Not related to iavf driver, AVF interface - vpp native driver have this issue
- dpdk iavf ignore the error and continue initialization, while vpp abort the init process
- Intel will fix the issue from PF driver - workaround: use old i40e driver (from ubuntu 20.04)
- Race condition occur on /dev/vfio mounting
-
- Run six rounds of tests with patch https://gerrit.fd.io/r/c/csit/+/34045 and result looks good right now
- Patch has been merged
- By not mounting whole /dev/vfio, mount only /dev/vfio/xxx used
- Addressed comments, waiting Peter's review - Peter approved, patch merged and monitoring.
- Met tx2 server reboot issue when monitoring - RAS CONTROLLER: Fatal unrecoverable error detected ** NBU Error **
- Reinstall the tx2 server with older kernel version and VPP device testing works fine with Juraj's patch
- VPP Device configuration align with VPP Performance configuration - no issue yet
- Run six rounds of tests with patch https://gerrit.fd.io/r/c/csit/+/34045 and result looks good right now
- Enable VPP device testing per patch
- Monitor for a week and enable vote right then
- Ping Dave about enabling VPP device testing per patch
- Dave gave minor comments https://gerrit.fd.io/r/c/ci-management/+/34679 - comment addressed and will be merged soon
- Need to figure out how to reproduce the error - Juraj
- Periodic job will stop when per patch job enabled
-
- Can't create AVF interface on VFs with configured VLAN - happens with latest i40e driver on tx2
- Device Testing on ThunderX2 servers
- New Arm servers shipment to the FD.io lab - about Jan 2022
- 2 build servers ready for shipment - 1 RU, no pcie slot for NICs
- 2 performance servers waiting for Intel NICs
- New servers are in the procurement process
- Plan to replace old thunderx1 build servers with more advanced Arm servers
- Need to confirm with RAM/disk size for the new build servers
- 500G disk/256G RAM
- Each job will consume about 16G memory
- Intel NIC firmware upgrade on Arm - not supported
- Performance servers have arrived at FD.io lab
- Servers are in the processing of wiring, expected to be operational soon
- Will follow the trend for Arm servers if more mlx NICs are installed on X86
- Plan to install QAT cards on performance servers
- Juraj to get QAT card avalibility from CSIT community
- QAT cards
- Govind will ship another 2x QAT from Austin to FD.io lab
- Will procure 2x QAT cards and verify them internally firstly.
- The existing QAT cards will be removed and returned to Vexxhost/FD.io lab
- QAT test cases are developed based on Python APIs / CLIs
-
-
VPP
- VPP SVE implementation - Lijian
- SVE validation on FPGA platform - Confluence page ready
- Run L3 traffic testing with NEON/SVE-VLA/SVE-VLS VPP version
- Performance number: NEON>SVE-VLA>SVE-VLS on arm internal platform
- Investigate SVE vs NEON packet checksum comparison
- Investigate VPP cross compilation with buildroot - for running VPP on FVP - distro like ubuntu is slower than buildroot - Lijian
- Depends on some libraries, dpdk, ipsec_mb, rdma-core and nasm etc - optional
- Investigate One Terabit throughput test on Arm platform
- Investigate automate rfc2544 no drop rate throughput test with Ixia on N1 platform - Tianyu
- Kernel cmdline may impact on NDR PDR results - Jieqiang
- Intern help to benchmark VPP using 100G MLX NIC
- CNF PoC proposal preparation- Tianyu
- Add support for VPP aarch64 docker image build
- Calico use cases exploration on VPP
- Improve ansible scripts to deploy VPP&snort on K8S pods automatically
- VPP IPsec on Arm - Govind
- Work on IPsec input - Zach
- IPSec input node/decryption flow cache implemented in a separate patch - Zach
- Upstreamed https://gerrit.fd.io/r/c/vpp/+/32903
- Review SPD flow cache patch from Intel folks - Zach
-
- Depends on kernel patch
- Kernel with aarch64 patch is expected to release soon
- Depends on kernel patch
- Building Intel QAT driver on arm to test IPsec crypt - Picchi
- Run IPsec with QAT on skylake this week, run the same setup on Ampere is next step
- Patch to resolve iommu issue for mlx NIC when using with QAT card
- Benchmark IPSec test case with QAT card/mlx NIC(single-core/multi-core)
- Needs a kernel patch to resolve crash issue for QAT card
- Patch made by Yoan is upstream and waits for review
- Try patched VPP to verify QAT card usage
- Investigating crash issue with 90% linerate IPSec traffic with QAT card
- VPP rdma native driver only - vpp meta data corrupt - may related to memory barrier
- QAT single core test done - investigate multiple core QAT case
-
- VPP SVE implementation - Lijian
7/5/2022
-
Attendees
- Govindarajan Mohandoss
- Lijian Zhang
- Jieqiang Wang
- Tianyu Li
-
CSIT
- VPP Performance Test
- https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-3n-tsh/
-
https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-report-iterative-2106-3n-tsh/
-
https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-3n-alt/
- CSIT perf numbers VS local perf numbers
- VPP cloud image in CSIT VS native built VPP in local env
- One DPDK patch introduced perf degradation on Arm platform
- Configuration difference between CSIT env and local env(Hugepage size, startup.conf parameters and etc)
- Check if there is customer support can help with the PEX installation issue - Juraj
- Juraj will write down the procedures on setting up Ampere Altra setup in FD.io lab
- And the procedures of developing/developing test cases in CSIT (performance & device testing)
- Juraj should have already sent to Jieqiang previously.
- 22.06 release testing will happen soon
- CSIT perf numbers VS local perf numbers
-
https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-3n-alt/
- VPP Performance Test
-
-
-
- Add new IPSec NULL encryption & decryption test cases - Juraj
- IPsec SPD input/output test case ongoing - Juraj
- Enable the SPD outbound tests
- Patches ready, waiting release testing done - ETA 1 week or 2
- Outbound SPD test patch has been enabled
- Investigate Inbound SPD test cases - Juraj
- Juraj will commit the patch and get it confirmed with Zachary
- Release testing for 21.10 is done
- Enable the SPD outbound tests
- New links for VPP perf trending/report pages
- Daily trending: https://s3-docs.fd.io/csit/master/trending/
- Release report: https://s3-docs.fd.io/csit/master/report/
- Release testing for 22.02 is done
- 3 nodes Taishan crypto test case failed - related to CSIT change
- Patch ready for review and to be merged https://gerrit.fd.io/r/c/csit/+/34591 -- Merged
-
- VPP Path
- Voting and working fine.
- CentOS-8 jobs have been removed.
- VPP Device
- Device Testing on ThunderX2 servers
- Juraj will commit the patch to disable the failling test cases
- https://jenkins.fd.io/view/vpp/job/vpp-csit-verify-device-master-ubuntu2004-aarch64-1n-tx2/
- VM cases failed only on Arm
- Tried to increase the timeout to see it will fix the issue
- VM cases failed on 1 min timeout on creating VM (qemu cli)- tx2 node
- Reboot server recover and monitoring
- Need to look into it, try manually
- May need to upgrade iavf driver
- Server in-accessiable
- Reboot server recover the service
- AVF interface creation issue:
- Can't create AVF interface on VFs with configured VLAN - happens with latest i40e driver on tx2
- Related to PF i40e driver, vlan stripping configured on VF, PF driver return not allowd.
- Not related to iavf driver, AVF interface - vpp native driver have this issue
- dpdk iavf ignore the error and continue initialization, while vpp abort the init process
- Intel will fix the issue from PF driver - workaround: use old i40e driver (from ubuntu 20.04)
- Race condition occur on /dev/vfio mounting
-
- Run six rounds of tests with patch https://gerrit.fd.io/r/c/csit/+/34045 and result looks good right now
- Patch has been merged
- By not mounting whole /dev/vfio, mount only /dev/vfio/xxx used
- Addressed comments, waiting Peter's review - Peter approved, patch merged and monitoring.
- Met tx2 server reboot issue when monitoring - RAS CONTROLLER: Fatal unrecoverable error detected ** NBU Error **
- Reinstall the tx2 server with older kernel version and VPP device testing works fine with Juraj's patch
- VPP Device configuration align with VPP Performance configuration - no issue yet
- Run six rounds of tests with patch https://gerrit.fd.io/r/c/csit/+/34045 and result looks good right now
- Enable VPP device testing per patch
- Monitor for a week and enable vote right then
- Ping Dave about enabling VPP device testing per patch
- Dave gave minor comments https://gerrit.fd.io/r/c/ci-management/+/34679 - comment addressed and will be merged soon
- Need to figure out how to reproduce the error - Juraj
- Periodic job will stop when per patch job enabled
-
- Can't create AVF interface on VFs with configured VLAN - happens with latest i40e driver on tx2
- Device Testing on ThunderX2 servers
- New Arm servers shipment to the FD.io lab - about Jan 2022
- 2 build servers ready for shipment - 1 RU, no pcie slot for NICs
- 2 performance servers waiting for Intel NICs
- New servers are in the procurement process
- Plan to replace old thunderx1 build servers with more advanced Arm servers
- Need to confirm with RAM/disk size for the new build servers
- 500G disk/256G RAM
- Each job will consume about 16G memory
- Intel NIC firmware upgrade on Arm - not supported
- Performance servers have arrived at FD.io lab
- Servers are in the processing of wiring, expected to be operational soon
- Will follow the trend for Arm servers if more mlx NICs are installed on X86
- Plan to install QAT cards on performance servers
- Juraj to get QAT card avalibility from CSIT community
- QAT cards
- Govind will ship another 2x QAT from Austin to FD.io lab
- Will procure 2x QAT cards and verify them internally firstly.
- The existing QAT cards will be removed and returned to Vexxhost/FD.io lab
- QAT test cases are developed based on Python APIs / CLIs
-
-
VPP
- VPP SVE implementation - Lijian
- SVE validation on FPGA platform - Confluence page ready
- Run L3 traffic testing with NEON/SVE-VLA/SVE-VLS VPP version
- Performance number: NEON>SVE-VLA>SVE-VLS on arm internal platform
- Investigate SVE vs NEON packet checksum comparison
- Investigate VPP cross compilation with buildroot - for running VPP on FVP - distro like ubuntu is slower than buildroot - Lijian
- Depends on some libraries, dpdk, ipsec_mb, rdma-core and nasm etc - optional
- Investigate One Terabit throughput test on Arm platform
- Investigate automate rfc2544 no drop rate throughput test with Ixia on N1 platform - Tianyu
- Kernel cmdline may impact on NDR PDR results - Jieqiang
- Intern help to benchmark VPP on N1 platforms
- VPP IPv4 fragmentation & reassembly - Tianyu & Jieqiang
- reassembly node opt by adding prefetch
- Add multi-arch support for ip4-frag node but see no perf bump
- Apply loop unrolling on ip4-frag node
- Benchmark IPv4 fragmentation node using rdma plugin
- Rebase the patch and final round of benchmarking for frag/reassembly nodes
- VPP IPv6 fragmentation
- Multi-arch node and batch memcpy - src, dst, bytes.
- VPP performance drop seen in CSIT after bump dpdk version to 21.11
- CNF PoC proposal preparation- Tianyu
- Add support for VPP aarch64 docker image build
- Calico use cases exploration on VPP
- Improve ansible scripts to deploy VPP&snort on K8S pods automatically
- VPP IPsec on Arm - Govind
- Work on IPsec input - Zach
- IPSec input node/decryption flow cache implemented in a separate patch - Zach
- Upstreamed https://gerrit.fd.io/r/c/vpp/+/32903
- VPP Perfmon plugin enablement on Arm - Zach
- Rely on kernel patch https://lore.kernel.org/linux-arm-kernel/[email protected]/ to enable the feature
- v10 kernel patch is ready, which fixes intermittent large statistic number for events
- Modify the commit message and upstream the perfmon patch - Zach
- Tested perfmon patch - Jieqiang
- Modify the commit message and upstream the perfmon patch - Zach
- Review SPD flow cache patch from Intel folks - Zach
-
- Depends on kernel patch
- Kernel with aarch64 patch is expected to release soon
- Depends on kernel patch
- Building Intel QAT driver on arm to test IPsec crypt - Picchi
- Run IPsec with QAT on skylake this week, run the same setup on Ampere is next step
- Patch to resolve iommu issue for mlx NIC when using with QAT card
- Benchmark IPSec test case with QAT card/mlx NIC(single-core/multi-core)
- Needs a kernel patch to resolve crash issue for QAT card
- Patch made by Yoan is upstream and waits for review
- Try patched VPP to verify QAT card usage
- Investigating crash issue with 90% linerate IPSec traffic with QAT card
- VPP rdma native driver only - vpp meta data corrupt - may related to memory barrier
- QAT single core test done - investigate multiple core QAT case
-
- VPP SVE implementation - Lijian
6/21/2022
-
Attendees
- Govindarajan Mohandoss
- Juraj Linkes
- Lijian Zhang
- Jieqiang Wang
- Tianyu Li
- Tina Tsou
-
CSIT
- VPP Performance Test
- https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-3n-tsh/
-
https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-report-iterative-2106-3n-tsh/
-
https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-3n-alt/
- CSIT perf numbers VS local perf numbers
- VPP cloud image in CSIT VS native built VPP in local env
- One DPDK patch introduced perf degradation on Arm platform
- Configuration difference between CSIT env and local env(Hugepage size, startup.conf parameters and etc)
- Check if there is customer support can help with the PEX installation issue - Juraj
- Juraj will write down the procedures on setting up Ampere Altra setup in FD.io lab
- And the procedures of developing/developing test cases in CSIT (performance & device testing)
- Juraj should have already sent to Jieqiang previously.
- 22.06 release testing will happen soon
- CSIT perf numbers VS local perf numbers
-
https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-3n-alt/
- VPP Performance Test
-
-
-
- Add new IPSec NULL encryption & decryption test cases - Juraj
- IPsec SPD input/output test case ongoing - Juraj
- Enable the SPD outbound tests
- Patches ready, waiting release testing done - ETA 1 week or 2
- Outbound SPD test patch has been enabled
- Investigate Inbound SPD test cases - Juraj
- Juraj will commit the patch and get it confirmed with Zachary
- Release testing for 21.10 is done
- Enable the SPD outbound tests
- New links for VPP perf trending/report pages
- Daily trending: https://s3-docs.fd.io/csit/master/trending/
- Release report: https://s3-docs.fd.io/csit/master/report/
- Release testing for 22.02 is done
- 3 nodes Taishan crypto test case failed - related to CSIT change
- Patch ready for review and to be merged https://gerrit.fd.io/r/c/csit/+/34591 -- Merged
-
- VPP Path
- Voting and working fine.
- CentOS-8 jobs have been removed.
- VPP Device
- Device Testing on ThunderX2 servers
- Juraj will commit the patch to disable the failling test cases
- https://jenkins.fd.io/view/vpp/job/vpp-csit-verify-device-master-ubuntu2004-aarch64-1n-tx2/
- VM cases failed only on Arm
- Tried to increase the timeout to see it will fix the issue
- VM cases failed on 1 min timeout on creating VM (qemu cli)- tx2 node
- Reboot server recover and monitoring
- Need to look into it, try manually
- May need to upgrade iavf driver
- Server in-accessiable
- Reboot server recover the service
- AVF interface creation issue:
- Can't create AVF interface on VFs with configured VLAN - happens with latest i40e driver on tx2
- Related to PF i40e driver, vlan stripping configured on VF, PF driver return not allowd.
- Not related to iavf driver, AVF interface - vpp native driver have this issue
- dpdk iavf ignore the error and continue initialization, while vpp abort the init process
- Intel will fix the issue from PF driver - workaround: use old i40e driver (from ubuntu 20.04)
- Race condition occur on /dev/vfio mounting
-
- Run six rounds of tests with patch https://gerrit.fd.io/r/c/csit/+/34045 and result looks good right now
- Patch has been merged
- By not mounting whole /dev/vfio, mount only /dev/vfio/xxx used
- Addressed comments, waiting Peter's review - Peter approved, patch merged and monitoring.
- Met tx2 server reboot issue when monitoring - RAS CONTROLLER: Fatal unrecoverable error detected ** NBU Error **
- Reinstall the tx2 server with older kernel version and VPP device testing works fine with Juraj's patch
- VPP Device configuration align with VPP Performance configuration - no issue yet
- Run six rounds of tests with patch https://gerrit.fd.io/r/c/csit/+/34045 and result looks good right now
- Enable VPP device testing per patch
- Monitor for a week and enable vote right then
- Ping Dave about enabling VPP device testing per patch
- Dave gave minor comments https://gerrit.fd.io/r/c/ci-management/+/34679 - comment addressed and will be merged soon
- Need to figure out how to reproduce the error - Juraj
- Periodic job will stop when per patch job enabled
-
- Can't create AVF interface on VFs with configured VLAN - happens with latest i40e driver on tx2
- Device Testing on ThunderX2 servers
- New Arm servers shipment to the FD.io lab - about Jan 2022
- 2 build servers ready for shipment - 1 RU, no pcie slot for NICs
- 2 performance servers waiting for Intel NICs
- New servers are in the procurement process
- Plan to replace old thunderx1 build servers with more advanced Arm servers
- Need to confirm with RAM/disk size for the new build servers
- 500G disk/256G RAM
- Each job will consume about 16G memory
- Intel NIC firmware upgrade on Arm - not supported
- Performance servers have arrived at FD.io lab
- Servers are in the processing of wiring, expected to be operational soon
- Will follow the trend for Arm servers if more mlx NICs are installed on X86
- Plan to install QAT cards on performance servers
- Juraj to get QAT card avalibility from CSIT community
- QAT cards
- Govind will ship another 2x QAT from Austin to FD.io lab
- Will procure 2x QAT cards and verify them internally firstly.
- The existing QAT cards will be removed and returned to Vexxhost/FD.io lab
- QAT test cases are developed based on Python APIs / CLIs
-
-
VPP
- VPP SVE implementation - Lijian
- SVE validation on FPGA platform - Confluence page ready
- Run L3 traffic testing with NEON/SVE-VLA/SVE-VLS VPP version
- Performance number: NEON>SVE-VLA>SVE-VLS on arm internal platform
- Investigate SVE vs NEON packet checksum comparison
- VPP IPv4 fragmentation & reassembly - Tianyu & Jieqiang
- reassembly node opt by adding prefetch
- Add multi-arch support for ip4-frag node but see no perf bump
- Apply loop unrolling on ip4-frag node
- Benchmark IPv4 fragmentation node using rdma plugin
- Rebase the patch and final round of benchmarking for frag/reassembly nodes
- VPP IPv6 fragmentation
- Multi-arch node and batch memcpy - src, dst, bytes.
- VPP performance drop seen in CSIT after bump dpdk version to 21.11
- CNF PoC proposal preparation- Tianyu
- Add support for VPP aarch64 docker image build
- Calico use cases exploration on VPP
- Improve ansible scripts to deploy VPP&snort on K8S pods automatically
- VPP IPsec on Arm - Govind
- Work on IPsec input - Zach
- IPSec input node/decryption flow cache implemented in a separate patch - Zach
- Upstreamed https://gerrit.fd.io/r/c/vpp/+/32903
- VPP Perfmon plugin enablement on Arm - Zach
- Rely on kernel patch https://lore.kernel.org/linux-arm-kernel/[email protected]/ to enable the feature
- v10 kernel patch is ready, which fixes intermittent large statistic number for events
- Modify the commit message and upstream the perfmon patch - Zach
- Tested perfmon patch - Jieqiang
- Modify the commit message and upstream the perfmon patch - Zach
- Review SPD flow cache patch from Intel folks - Zach
-
- Depends on kernel patch
- Kernel with aarch64 patch is expected to release soon
- Depends on kernel patch
- Building Intel QAT driver on arm to test IPsec crypt - Picchi
- Run IPsec with QAT on skylake this week, run the same setup on Ampere is next step
- Patch to resolve iommu issue for mlx NIC when using with QAT card
- Benchmark IPSec test case with QAT card/mlx NIC(single-core/multi-core)
- Needs a kernel patch to resolve crash issue for QAT card
- Patch made by Yoan is upstream and waits for review
- Try patched VPP to verify QAT card usage
- Investigating crash issue with 90% linerate IPSec traffic with QAT card
-
- VPP SVE implementation - Lijian
6/7/2022
-
Attendees
- Govindarajan Mohandoss
- Juraj Linkes
- Lijian Zhang
- Jieqiang Wang
- Tianyu Li
- Tina Tsou
-
CSIT
- VPP Performance Test
- https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-3n-tsh/
-
https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-report-iterative-2106-3n-tsh/
-
https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-3n-alt/
- CSIT perf numbers VS local perf numbers
- VPP cloud image in CSIT VS native built VPP in local env
- One DPDK patch introduced perf degradation on Arm platform
- Configuration difference between CSIT env and local env(Hugepage size, startup.conf parameters and etc)
- Check if there is customer support can help with the PEX installation issue - Juraj
- Juraj will write down the procedures on setting up Ampere Altra setup in FD.io lab
- And the procedures of developing/developing test cases in CSIT (performance & device testing)
- Juraj should have already sent to Jieqiang previously.
- CSIT perf numbers VS local perf numbers
-
https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-3n-alt/
- VPP Performance Test
-
-
-
- Add new IPSec NULL encryption & decryption test cases - Juraj
- IPsec SPD input/output test case ongoing - Juraj
- Enable the SPD outbound tests
- Patches ready, waiting release testing done - ETA 1 week or 2
- Outbound SPD test patch has been enabled
- Investigate Inbound SPD test cases - Juraj
- Juraj will commit the patch and get it confirmed with Zachary
- Release testing for 21.10 is done
- Enable the SPD outbound tests
- New links for VPP perf trending/report pages
- Daily trending: https://s3-docs.fd.io/csit/master/trending/
- Release report: https://s3-docs.fd.io/csit/master/report/
- Release testing for 22.02 is done
- 3 nodes Taishan crypto test case failed - related to CSIT change
- Patch ready for review and to be merged https://gerrit.fd.io/r/c/csit/+/34591 -- Merged
-
- VPP Path
- Voting and working fine.
- CentOS-8 jobs have been removed.
- VPP Device
- Device Testing on ThunderX2 servers
- Juraj will commit the patch to disable the failling test cases
- https://jenkins.fd.io/view/vpp/job/vpp-csit-verify-device-master-ubuntu2004-aarch64-1n-tx2/
- VM cases failed only on Arm
- Tried to increase the timeout to see it will fix the issue
- VM cases failed on 1 min timeout on creating VM (qemu cli)- tx2 node
- Reboot server recover and monitoring
- Need to look into it, try manually
- May need to upgrade iavf driver
- Server in-accessiable
- Reboot server recover the service
- AVF interface creation issue:
- Can't create AVF interface on VFs with configured VLAN - happens with latest i40e driver on tx2
- Related to PF i40e driver, vlan stripping configured on VF, PF driver return not allowd.
- Not related to iavf driver, AVF interface - vpp native driver have this issue
- dpdk iavf ignore the error and continue initialization, while vpp abort the init process
- Intel will fix the issue from PF driver - workaround: use old i40e driver (from ubuntu 20.04)
- Race condition occur on /dev/vfio mounting
-
- Run six rounds of tests with patch https://gerrit.fd.io/r/c/csit/+/34045 and result looks good right now
- Patch has been merged
- By not mounting whole /dev/vfio, mount only /dev/vfio/xxx used
- Addressed comments, waiting Peter's review - Peter approved, patch merged and monitoring.
- Met tx2 server reboot issue when monitoring - RAS CONTROLLER: Fatal unrecoverable error detected ** NBU Error **
- Reinstall the tx2 server with older kernel version and VPP device testing works fine with Juraj's patch
- VPP Device configuration align with VPP Performance configuration - no issue yet
- Run six rounds of tests with patch https://gerrit.fd.io/r/c/csit/+/34045 and result looks good right now
- Enable VPP device testing per patch
- Monitor for a week and enable vote right then
- Ping Dave about enabling VPP device testing per patch
- Dave gave minor comments https://gerrit.fd.io/r/c/ci-management/+/34679 - comment addressed and will be merged soon
- Need to figure out how to reproduce the error - Juraj
- Periodic job will stop when per patch job enabled
-
- Can't create AVF interface on VFs with configured VLAN - happens with latest i40e driver on tx2
- Device Testing on ThunderX2 servers
- New Arm servers shipment to the FD.io lab - about Jan 2022
- 2 build servers ready for shipment - 1 RU, no pcie slot for NICs
- 2 performance servers waiting for Intel NICs
- New servers are in the procurement process
- Plan to replace old thunderx1 build servers with more advanced Arm servers
- Need to confirm with RAM/disk size for the new build servers
- 500G disk/256G RAM
- Each job will consume about 16G memory
- Intel NIC firmware upgrade on Arm - not supported
- Performance servers have arrived at FD.io lab
- Servers are in the processing of wiring, expected to be operational soon
- Will follow the trend for Arm servers if more mlx NICs are installed on X86
- Plan to install QAT cards on performance servers
- Juraj to get QAT card avalibility from CSIT community
- QAT cards
- Govind will ship another 2x QAT from Austin to FD.io lab
- Will procure 2x QAT cards and verify them internally firstly.
- The existing QAT cards will be removed and returned to Vexxhost/FD.io lab
- QAT test cases are developed based on Python APIs / CLIs
-
-
VPP
- VPP SVE implementation - Lijian
- SVE validation on FPGA platform - Confluence page ready
- Run L3 traffic testing with NEON/SVE-VLA/SVE-VLS VPP version
- Performance number: NEON>SVE-VLA>SVE-VLS on arm internal platform
- Investigate SVE vs NEON packet checksum comparison
- VPP IPv4 fragmentation & reassembly - Tianyu & Jieqiang
- reassembly node opt by adding prefetch
- Add multi-arch support for ip4-frag node but see no perf bump
- Apply loop unrolling on ip4-frag node
- Benchmark IPv4 fragmentation node using rdma plugin
- Rebase the patch and final round of benchmarking for frag/reassembly nodes
- VPP IPv6 fragmentation
- Multi-arch node and batch memcpy - src, dst, bytes.
- VPP performance drop seen in CSIT after bump dpdk version to 21.11
- CNF PoC proposal preparation- Tianyu
- Add support for VPP aarch64 docker image build
- Calico use cases exploration on VPP
- Improve ansible scripts to deploy VPP&snort on K8S pods automatically
- VPP IPsec on Arm - Govind
- Work on IPsec input - Zach
- IPSec input node/decryption flow cache implemented in a separate patch - Zach
- Upstreamed https://gerrit.fd.io/r/c/vpp/+/32903
- VPP Perfmon plugin enablement on Arm - Zach
- Rely on kernel patch https://lore.kernel.org/linux-arm-kernel/[email protected]/ to enable the feature
- v10 kernel patch is ready, which fixes intermittent large statistic number for events
- Modify the commit message and upstream the perfmon patch - Zach
- Tested perfmon patch - Jieqiang
- Modify the commit message and upstream the perfmon patch - Zach
- Review SPD flow cache patch from Intel folks - Zach
-
- Depends on kernel patch
- Kernel with aarch64 patch is expected to release soon
- Depends on kernel patch
- Building Intel QAT driver on arm to test IPsec crypt - Picchi
- Run IPsec with QAT on skylake this week, run the same setup on Ampere is next step
- Patch to resolve iommu issue for mlx NIC when using with QAT card
- Benchmark IPSec test case with QAT card/mlx NIC(single-core/multi-core)
- Needs a kernel patch to resolve crash issue for QAT card
- Patch made by Yoan is upstream and waits for review
- Try patched VPP to verify QAT card usage
- Investigating crash issue with 90% linerate IPSec traffic with QAT card
-
- VPP SVE implementation - Lijian
5/17/2022
-
Attendees
- Govindarajan Mohandoss
- Juraj Linkes
- Lijian Zhang
- Tina Tsou
-
CSIT
- VPP Performance Test
- https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-3n-tsh/
-
https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-report-iterative-2106-3n-tsh/
-
https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-3n-alt/
- CSIT perf numbers VS local perf numbers
- VPP cloud image in CSIT VS native built VPP in local env
- One DPDK patch introduced perf degradation on Arm platform
- Configuration difference between CSIT env and local env(Hugepage size, startup.conf parameters and etc)
- Check if there is customer support can help with the PEX installation issue - Juraj
- Juraj will write down the procedures on setting up Ampere Altra setup in FD.io lab
- And the procedures of developing/developing test cases in CSIT (performance & device testing)
- Juraj should have already sent to Jieqiang previously.
- CSIT perf numbers VS local perf numbers
-
https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-3n-alt/
- VPP Performance Test
-
-
-
- Add new IPSec NULL encryption & decryption test cases - Juraj
- IPsec SPD input/output test case ongoing - Juraj
- Enable the SPD outbound tests
- Patches ready, waiting release testing done - ETA 1 week or 2
- Outbound SPD test patch has been enabled
- Investigate Inbound SPD test cases - Juraj
- Juraj will commit the patch and get it confirmed with Zachary
- Release testing for 21.10 is done
- Enable the SPD outbound tests
- New links for VPP perf trending/report pages
- Daily trending: https://s3-docs.fd.io/csit/master/trending/
- Release report: https://s3-docs.fd.io/csit/master/report/
- Release testing for 22.02 is done
- 3 nodes Taishan crypto test case failed - related to CSIT change
- Patch ready for review and to be merged https://gerrit.fd.io/r/c/csit/+/34591 -- Merged
-
- VPP Path
- Voting and working fine.
- CentOS-8 jobs have been removed.
- VPP Device
- Device Testing on ThunderX2 servers
- Juraj will commit the patch to disable the failling test cases
- https://jenkins.fd.io/view/vpp/job/vpp-csit-verify-device-master-ubuntu2004-aarch64-1n-tx2/
- VM cases failed only on Arm
- Tried to increase the timeout to see it will fix the issue
- VM cases failed on 1 min timeout on creating VM (qemu cli)- tx2 node
- Reboot server recover and monitoring
- Need to look into it, try manually
- May need to upgrade iavf driver
- Server in-accessiable
- Reboot server recover the service
- AVF interface creation issue:
- Can't create AVF interface on VFs with configured VLAN - happens with latest i40e driver on tx2
- Related to PF i40e driver, vlan stripping configured on VF, PF driver return not allowd.
- Not related to iavf driver, AVF interface - vpp native driver have this issue
- dpdk iavf ignore the error and continue initialization, while vpp abort the init process
- Intel will fix the issue from PF driver - workaround: use old i40e driver (from ubuntu 20.04)
- Race condition occur on /dev/vfio mounting
-
- Run six rounds of tests with patch https://gerrit.fd.io/r/c/csit/+/34045 and result looks good right now
- Patch has been merged
- By not mounting whole /dev/vfio, mount only /dev/vfio/xxx used
- Addressed comments, waiting Peter's review - Peter approved, patch merged and monitoring.
- Met tx2 server reboot issue when monitoring - RAS CONTROLLER: Fatal unrecoverable error detected ** NBU Error **
- Reinstall the tx2 server with older kernel version and VPP device testing works fine with Juraj's patch
- VPP Device configuration align with VPP Performance configuration - no issue yet
- Run six rounds of tests with patch https://gerrit.fd.io/r/c/csit/+/34045 and result looks good right now
- Enable VPP device testing per patch
- Monitor for a week and enable vote right then
- Ping Dave about enabling VPP device testing per patch
- Dave gave minor comments https://gerrit.fd.io/r/c/ci-management/+/34679 - comment addressed and will be merged soon
- Need to figure out how to reproduce the error - Juraj
- Periodic job will stop when per patch job enabled
-
- Can't create AVF interface on VFs with configured VLAN - happens with latest i40e driver on tx2
- Device Testing on ThunderX2 servers
- New Arm servers shipment to the FD.io lab - about Jan 2022
- 2 build servers ready for shipment - 1 RU, no pcie slot for NICs
- 2 performance servers waiting for Intel NICs
- New servers are in the procurement process
- Plan to replace old thunderx1 build servers with more advanced Arm servers
- Need to confirm with RAM/disk size for the new build servers
- 500G disk/256G RAM
- Each job will consume about 16G memory
- Intel NIC firmware upgrade on Arm - not supported
- Performance servers have arrived at FD.io lab
- Servers are in the processing of wiring, expected to be operational soon
- Will follow the trend for Arm servers if more mlx NICs are installed on X86
- Plan to install QAT cards on performance servers
- Juraj to get QAT card avalibility from CSIT community
- QAT cards
- Govind will ship another 2x QAT from Austin to FD.io lab
- Will procure 2x QAT cards and verify them internally firstly.
- The existing QAT cards will be removed and returned to Vexxhost/FD.io lab
- QAT test cases are developed based on Python APIs / CLIs
-
-
VPP
- VPP SVE implementation - Lijian
- SVE validation on FPGA platform - Confluence page ready
- Run L3 traffic testing with NEON/SVE-VLA/SVE-VLS VPP version
- Performance number: NEON>SVE-VLA>SVE-VLS on arm internal platform
- VPP IPv4 fragmentation & reassembly - Tianyu & Jieqiang
- reassembly node opt by adding prefetch
- Add multi-arch support for ip4-frag node but see no perf bump
- Apply loop unrolling on ip4-frag node
- Benchmark IPv4 fragmentation node using rdma plugin
- Rebase the patch and final round of benchmarking for frag/reassembly nodes
- VPP IPv6 fragmentation
- Multi-arch node and batch memcpy - src, dst, bytes.
- VPP performance drop seen in CSIT after bump dpdk version to 21.11
- CNF PoC proposal preparation- Tianyu
- Add support for VPP aarch64 docker image build
- Calico use cases exploration on VPP
- Improve ansible scripts to deploy VPP&snort on K8S pods automatically
- VPP IPsec on Arm - Govind
- Work on IPsec input - Zach
- IPSec input node/decryption flow cache implemented in a separate patch - Zach
- Upstreamed https://gerrit.fd.io/r/c/vpp/+/32903
- VPP Perfmon plugin enablement on Arm - Zach
- Rely on kernel patch https://lore.kernel.org/linux-arm-kernel/[email protected]/ to enable the feature
- v10 kernel patch is ready, which fixes intermittent large statistic number for events
- Modify the commit message and upstream the perfmon patch - Zach
- Depends on kernel patch
- Kernel with aarch64 patch is expected to release soon
- Depends on kernel patch
- Building Intel QAT driver on arm to test IPsec crypt - Picchi
- Run IPsec with QAT on skylake this week, run the same setup on Ampere is next step
- Patch to resolve iommu issue for mlx NIC when using with QAT card
- Benchmark IPSec test case with QAT card/mlx NIC(single-core/multi-core)
- Needs a kernel patch to resolve crash issue for QAT card
- Patch made by Yoan is upstream and waits for review
- Try patched VPP to verify QAT card usage
- Modify the commit message and upstream the perfmon patch - Zach
- VPP SVE implementation - Lijian
4/5/2022
-
Attendees
- Govindarajan Mohandoss
- Juraj Linkes
- Lijian Zhang
- Tina Tsou
-
CSIT
- VPP Performance Test
- https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-3n-tsh/
-
https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-report-iterative-2106-3n-tsh/
-
https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-3n-alt/
- Check if there is customer support can help with the PEX installation issue - Juraj
- Juraj will write down the procedures on setting up Ampere Altra setup in FD.io lab
- And the procedures of developing/developing test cases in CSIT (performance & device testing)
- Juraj should have alrady sent to Jieqiang previously.
-
https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-3n-alt/
- VPP Performance Test
-
-
-
- Add new IPSec NULL encryption & decryption test cases - Juraj
- IPsec SPD input/output test case ongoing - Juraj
- Enable the SPD outbound tests
- Patches ready, waiting release testing done - ETA 1 week or 2
- Outbound SPD test patch has been enabled
- Investigate Inbound SPD test cases - Juraj
- Juraj will commit the patch and get it confirmed with Zachary
- Release testing for 21.10 is done
- Enable the SPD outbound tests
- New links for VPP perf trending/report pages
- Daily trending: https://s3-docs.fd.io/csit/master/trending/
- Release report: https://s3-docs.fd.io/csit/master/report/
- Release testing for 22.02 is done
- 3 nodes Taishan crypto test case failed - related to CSIT change
- Patch ready for review and to be merged https://gerrit.fd.io/r/c/csit/+/34591 -- Merged
-
- VPP Path
- Voting and working fine.
- CentOS-8 jobs have been removed.
- VPP Device
- Device Testing on ThunderX2 servers
- Juraj will commit the patch to disable the failling test cases
- https://jenkins.fd.io/view/vpp/job/vpp-csit-verify-device-master-ubuntu2004-aarch64-1n-tx2/
- VM cases failed only on Arm
- Tried to increase the timeout to see it will fix the issue
- VM cases failed on 1 min timeout on creating VM (qemu cli)- tx2 node
- Reboot server recover and monitoring
- Need to look into it, try manually
- May need to upgrade iavf driver
- Server in-accessiable
- Reboot server recover the service
- AVF interface creation issue:
- Can't create AVF interface on VFs with configured VLAN - happens with latest i40e driver on tx2
- Related to PF i40e driver, vlan stripping configured on VF, PF driver return not allowd.
- Not related to iavf driver, AVF interface - vpp native driver have this issue
- dpdk iavf ignore the error and continue initialization, while vpp abort the init process
- Intel will fix the issue from PF driver - workaround: use old i40e driver (from ubuntu 20.04)
- Race condition occur on /dev/vfio mounting
-
- Run six rounds of tests with patch https://gerrit.fd.io/r/c/csit/+/34045 and result looks good right now
- Patch has been merged
- By not mounting whole /dev/vfio, mount only /dev/vfio/xxx used
- Addressed comments, waiting Peter's review - Peter approved, patch merged and monitoring.
- Met tx2 server reboot issue when monitoring - RAS CONTROLLER: Fatal unrecoverable error detected ** NBU Error **
- Reinstall the tx2 server with older kernel version and VPP device testing works fine with Juraj's patch
- VPP Device configuration align with VPP Performance configuration - no issue yet
- Run six rounds of tests with patch https://gerrit.fd.io/r/c/csit/+/34045 and result looks good right now
- Enable VPP device testing per patch
- Monitor for a week and enable vote right then
- Ping Dave about enabling VPP device testing per patch
- Dave gave minor comments https://gerrit.fd.io/r/c/ci-management/+/34679 - comment addressed and will be merged soon
- Need to figure out how to reproduce the error - Juraj
- Periodic job will stop when per patch job enabled
-
- Can't create AVF interface on VFs with configured VLAN - happens with latest i40e driver on tx2
- Device Testing on ThunderX2 servers
- New Arm servers shipment to the FD.io lab - about Jan 2022
- 2 build servers ready for shipment - 1 RU, no pcie slot for NICs
- 2 performance servers waiting for Intel NICs
- New servers are in the procurement process
- Plan to replace old thunderx1 build servers with more advanced Arm servers
- Need to confirm with RAM/disk size for the new build servers
- 500G disk/256G RAM
- Each job will consume about 16G memory
- Intel NIC firmware upgrade on Arm - not supported
- Performance servers have arrived at FD.io lab
- Servers are in the processing of wiring, expected to be operational soon
- Will follow the trend for Arm servers if more mlx NICs are installed on X86
- Plan to install QAT cards on performance servers
- Juraj to get QAT card avalibility from CSIT community
- QAT cards
- Govind will ship another 2x QAT from Austin to FD.io lab
- Will procure 2x QAT cards and verify them internally firstly.
- The existing QAT cards will be removed and returned to Vexxhost/FD.io lab
- QAT test cases are developed based on Python APIs / CLIs
-
-
VPP
- VPP SVE implementation - Lijian
- SVE validation on FPGA platform - Confluence page ready
- Run L3 traffic testing with NEON/SVE-VLA/SVE-VLS VPP version
- Performance number: NEON>SVE-VLA>SVE-VLS on arm internal platform
- VPP IPv4 fragmentation & reassembly - Tianyu & Jieqiang
- reassembly node opt by adding prefetch
- Add multi-arch support for ip4-frag node but see no perf bump
- Apply loop unrolling on ip4-frag node
- Benchmark IPv4 fragmentation node using rdma plugin
- Rebase the patch and final round of benchmarking for frag/reassembly nodes
- VPP IPv6 fragmentation
- Multi-arch node and batch memcpy - src, dst, bytes.
- VPP performance drop seen in CSIT after bump dpdk version to 21.11
- CNF PoC proposal preparation- Tianyu
- Add support for VPP aarch64 docker image build
- Calico use cases exploration on VPP
- Improve ansible scripts to deploy VPP&snort on K8S pods automatically
- VPP IPsec on Arm - Govind
- Work on IPsec input - Zach
- IPSec input node/decryption flow cache implemented in a separate patch - Zach
- Upstreamed https://gerrit.fd.io/r/c/vpp/+/32903
- VPP Perfmon plugin enablement on Arm - Zach
- Rely on kernel patch https://lore.kernel.org/linux-arm-kernel/[email protected]/ to enable the feature
- v10 kernel patch is ready, which fixes intermittent large statistic number for events
- Modify the commit message and upstream the perfmon patch - Zach
- Depends on kernel patch
- Kernel with aarch64 patch is expected to release soon
- Depends on kernel patch
- Building Intel QAT driver on arm to test IPsec crypt - Picchi
- Run IPsec with QAT on skylake this week, run the same setup on Ampere is next step
- Patch to resolve iommu issue for mlx NIC when using with QAT card
- Benchmark IPSec test case with QAT card/mlx NIC(single-core/multi-core)
- Modify the commit message and upstream the perfmon patch - Zach
- VPP SVE implementation - Lijian
3/15/2022
-
Attendees
- Govindarajan Mohandoss
- Juraj Linkes
- Lijian Zhang
- Jieqiang Wang
- Tianyu Li
- Tina Tsou
-
CSIT
- VPP Performance Test
- https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-3n-tsh/
-
https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-report-iterative-2106-3n-tsh/
- Add new IPSec NULL encryption & decryption test cases - Juraj
- IPsec SPD input/output test case ongoing - Juraj
- Enable the SPD outbound tests
- Patches ready, waiting release testing done - ETA 1 week or 2
- Outbound SPD test patch has been enabled
- Investigate Inbound SPD test cases - Juraj
- Release testing for 21.10 is done
- Enable the SPD outbound tests
- New links for VPP perf trending/report pages
- Daily trending: https://s3-docs.fd.io/csit/master/trending/
- Release report: https://s3-docs.fd.io/csit/master/report/
- Release testing for 22.02 is done
- 3 nodes Taishan crypto test case failed - related to CSIT change
- Patch ready for review and to be merged https://gerrit.fd.io/r/c/csit/+/34591 -- Merged
- VPP Path
- Voting and working fine.
- CentOS-8 jobs have been removed.
- VPP Device
- https://jenkins.fd.io/view/vpp/job/vpp-csit-verify-device-master-ubuntu2004-aarch64-1n-tx2/
- VM cases failed only on Arm
- Tried to increase the timeout to see it will fix the issue
- VM cases failed on 1 min timeout on creating VM (qemu cli)- tx2 node
- Reboot server recover and monitoring
- Need to look into it, try manually
- May need to upgrade iavf driver
- Server in-accessiable
- Reboot server recover the service
- AVF interface creation issue:
- Can't create AVF interface on VFs with configured VLAN - happens with latest i40e driver on tx2
- Related to PF i40e driver, vlan stripping configured on VF, PF driver return not allowd.
- Not related to iavf driver, AVF interface - vpp native driver have this issue
- dpdk iavf ignore the error and continue initialization, while vpp abort the init process
- Intel will fix the issue from PF driver - workaround: use old i40e driver (from ubuntu 20.04)
- Race condition occur on /dev/vfio mounting
-
- Run six rounds of tests with patch https://gerrit.fd.io/r/c/csit/+/34045 and result looks good right now
- Patch has been merged
- By not mounting whole /dev/vfio, mount only /dev/vfio/xxx used
- Addressed comments, waiting Peter's review - Peter approved, patch merged and monitoring.
- Met tx2 server reboot issue when monitoring - RAS CONTROLLER: Fatal unrecoverable error detected ** NBU Error **
- Reinstall the tx2 server with older kernel version and VPP device testing works fine with Juraj's patch
- VPP Device configuration align with VPP Performance configuration - no issue yet
- Run six rounds of tests with patch https://gerrit.fd.io/r/c/csit/+/34045 and result looks good right now
- Enable VPP device testing per patch
- Monitor for a week and enable vote right then
- Ping Dave about enabling VPP device testing per patch
- Dave gave minor comments https://gerrit.fd.io/r/c/ci-management/+/34679 - comment addressed and will be merged soon
- Need to figure out how to reproduce the error - Juraj
- Periodic job will stop when per patch job enabled
-
- Can't create AVF interface on VFs with configured VLAN - happens with latest i40e driver on tx2
- New Arm servers shipment to the FD.io lab - about Jan 2022
- 2 build servers ready for shipment - 1 RU, no pcie slot for NICs
- 2 performance servers waiting for Intel NICs
- New servers are in the procurement process
- Plan to replace old thunderx1 build servers with more advanced Arm servers
- Need to confirm with RAM/disk size for the new build servers
- 500G disk/256G RAM
- Each job will consume about 16G memory
- Intel NIC firmware upgrade on Arm - not supported
- Performance servers have arrived at FD.io lab
- Servers are in the processing of wiring, expected to be operational soon
- Will follow the trend for Arm servers if more mlx NICs are installed on X86
- Plan to install QAT cards on performance servers
- Juraj to get QAT card avalibility from CSIT community
- VPP Performance Test
-
VPP
- VPP SVE implementation - Lijian
- SVE validation on FPGA platform - Confluence page ready
- Run L3 traffic testing with NEON/SVE-VLA/SVE-VLS VPP version
- Performance number: NEON>SVE-VLA>SVE-VLS on arm internal platform
- VPP IPv4 fragmentation & reassembly - Tianyu & Jieqiang
- reassembly node opt by adding prefetch
- Add multi-arch support for ip4-frag node but see no perf bump
- Apply loop unrolling on ip4-frag node
- Benchmark IPv4 fragmentation node using rdma plugin
- Rebase the patch and final round of benchmarking for frag/reassembly nodes
- VPP IPv6 fragmentation
- Multi-arch node and batch memcpy - src, dst, bytes.
- VPP performance drop seen in CSIT after bump dpdk version to 21.11
- CNF PoC proposal preparation- Tianyu
- Add support for VPP aarch64 docker image build
- Calico use cases exploration on VPP
- Improve ansible scripts to deploy VPP&snort on K8S pods automatically
- VPP IPsec on Arm - Govind
- Work on IPsec input - Zach
- IPSec input node/decryption flow cache implemented in a separate patch - Zach
- Upstreamed https://gerrit.fd.io/r/c/vpp/+/32903
- VPP Perfmon plugin enablement on Arm - Zach
- Rely on kernel patch https://lore.kernel.org/linux-arm-kernel/[email protected]/ to enable the feature
- v10 kernel patch is ready, which fixes intermittent large statistic number for events
- Modify the commit message and upstream the perfmon patch - Zach
- Depends on kernel patch
- Kernel with aarch64 patch is expected to release soon
- Depends on kernel patch
- Building Intel QAT driver on arm to test IPsec crypt - Picchi
- Run IPsec with QAT on skylake this week, run the same setup on Ampere is next step
- Patch to resolve iommu issue for mlx NIC when using with QAT card
- Benchmark IPSec test case with QAT card/mlx NIC(single-core/multi-core)
- Modify the commit message and upstream the perfmon patch - Zach
- VPP SVE implementation - Lijian
3/1/2022
-
Attendees
- Govindarajan Mohandoss
- Jieqiang Wang
- Tianyu Li
- Tina Tsou
-
CSIT
- VPP Performance Test
- https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-3n-tsh/
-
https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-report-iterative-2106-3n-tsh/
- Add new IPSec NULL encryption & decryption test cases - Juraj
- IPsec SPD input/output test case ongoing - Juraj
- Enable the SPD outbound tests
- Patches ready, waiting release testing done - ETA 1 week or 2
- Outbound SPD test patch has been enabled
- Release testing for 21.10 is done
- Enable the SPD outbound tests
- New links for VPP perf trending/report pages
- Daily trending: https://s3-docs.fd.io/csit/master/trending/
- Release report: https://s3-docs.fd.io/csit/master/report/
- 3 nodes Taishan crypto test case failed - related to CSIT change
- Patch ready for review and to be merged https://gerrit.fd.io/r/c/csit/+/34591 -- Merged
- VPP Path
- Voting and working fine.
- CentOS-8 jobs have been removed.
- VPP Device
- https://jenkins.fd.io/view/vpp/job/vpp-csit-verify-device-master-ubuntu2004-aarch64-1n-tx2/
- VM cases failed only on Arm
- Tried to increase the timeout to see it will fix the issue
- VM cases failed on 1 min timeout on creating VM (qemu cli)- tx2 node
- Reboot server recover and monitoring
- Need to look into it, try manually
- May need to upgrade iavf driver
- Server in-accessiable
- Reboot server recover the service
- AVF interface creation issue:
- Can't create AVF interface on VFs with configured VLAN - happens with latest i40e driver on tx2
- Related to PF i40e driver, vlan stripping configured on VF, PF driver return not allowd.
- Not related to iavf driver, AVF interface - vpp native driver have this issue
- dpdk iavf ignore the error and continue initialization, while vpp abort the init process
- Intel will fix the issue from PF driver - workaround: use old i40e driver (from ubuntu 20.04)
- Race condition occur on /dev/vfio mounting
-
- Run six rounds of tests with patch https://gerrit.fd.io/r/c/csit/+/34045 and result looks good right now
- Patch has been merged
- By not mounting whole /dev/vfio, mount only /dev/vfio/xxx used
- Addressed comments, waiting Peter's review - Peter approved, patch merged and monitoring.
- Met tx2 server reboot issue when monitoring - RAS CONTROLLER: Fatal unrecoverable error detected ** NBU Error **
- Reinstall the tx2 server with older kernel version and VPP device testing works fine with Juraj's patch
- VPP Device configuration align with VPP Performance configuration - no issue yet
- Run six rounds of tests with patch https://gerrit.fd.io/r/c/csit/+/34045 and result looks good right now
- Enable VPP device testing per patch
- Monitor for a week and enable vote right then
- Ping Dave about enabling VPP device testing per patch
- Dave gave minor comments https://gerrit.fd.io/r/c/ci-management/+/34679 - comment addressed and will be merged soon
- Need to figure out how to reproduce the error - Juraj
- Periodic job will stop when per patch job enabled
-
- Can't create AVF interface on VFs with configured VLAN - happens with latest i40e driver on tx2
- New Arm servers shipment to the FD.io lab - about Jan 2022
- 2 build servers ready for shipment - 1 RU, no pcie slot for NICs
- 2 performance servers waiting for Intel NICs
- New servers are in the procurement process
- Plan to replace old thunderx1 build servers with more advanced Arm servers
- Need to confirm with RAM/disk size for the new build servers
- 500G disk/256G RAM
- Each job will consume about 16G memory
- Intel NIC firmware upgrade on Arm - not supported
- Paper work for shipment is done
- Build servers will arrive at end of Jan
- Performance servers will arrive in Feb
- VPP Performance Test
-
VPP
- VPP SVE implementation - Lijian
- SVE validation on FPGA platform - Confluence page ready
- Run L3 traffic testing with NEON/SVE-VLA/SVE-VLS VPP version
- Performance number: NEON>SVE-VLA>SVE-VLS on arm internal platform
- VPP IPv4 fragmentation & reassembly - Tianyu & Jieqiang
- reassembly node opt by adding prefetch
- Add multi-arch support for ip4-frag node but see no perf bump
- Apply loop unrolling on ip4-frag node
- Benchmark IPv4 fragmentation node using rdma plugin
- VPP IPv6 fragmentation
- Multi-arch node and batch memcpy - src, dst, bytes.
- VPP performance drop seen in CSIT after bump dpdk version to 21.11
- CNF PoC proposal preparation- Tianyu
- Add support for VPP aarch64 docker image build
- Calico use cases exploration on VPP
- Improve ansible scripts to deploy VPP&snort on K8S pods automatically
- VPP IPsec on Arm - Govind
- Work on IPsec input - Zach
- IPSec input node/decryption flow cache implemented in a separate patch - Zach
- Upstreamed https://gerrit.fd.io/r/c/vpp/+/32903
- VPP Perfmon plugin enablement on Arm - Zach
- Rely on kernel patch https://lore.kernel.org/linux-arm-kernel/[email protected]/ to enable the feature
- v10 kernel patch is ready, which fixes intermittent large statistic number for events
- Modify the commit message and upstream the perfmon patch - Zach
- Depends on kernel patch
- Building Intel QAT driver on arm to test IPsec crypt - Picchi
- Run IPsec with QAT on skylake this week, run the same setup on Ampere is next step
- Modify the commit message and upstream the perfmon patch - Zach
- VPP SVE implementation - Lijian
1/25/2022
-
Attendees
- Govindarajan Mohandoss
- Lijian Zhang
- Jieqiang Wang
- Juraj Linkes
- Tianyu Li
- Tina Tsou
-
CSIT
- VPP Performance Test
- https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-3n-tsh/
-
https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-report-iterative-2106-3n-tsh/
- Add new IPSec NULL encryption & decryption test cases - Juraj
- IPsec SPD input/output test case ongoing - Juraj
- Enable the SPD outbound tests
- Patches ready, waiting release testing done - ETA 1 week or 2
- Outbound SPD test patch has been enabled
- Release testing for 21.10 is done
- Enable the SPD outbound tests
- New links for VPP perf trending/report pages
- Daily trending: https://s3-docs.fd.io/csit/master/trending/
- Release report: https://s3-docs.fd.io/csit/master/report/
- 3 nodes Taishan crypto test case failed - related to CSIT change
- Patch ready for review and to be merged https://gerrit.fd.io/r/c/csit/+/34591 -- Merged
- VPP Path
- Voting and working fine.
- CentOS-8 jobs have been removed.
- VPP Device
- https://jenkins.fd.io/view/vpp/job/vpp-csit-verify-device-master-ubuntu2004-aarch64-1n-tx2/
- VM cases failed only on Arm
- Tried to increase the timeout to see it will fix the issue
- VM cases failed on 1 min timeout on creating VM (qemu cli)- tx2 node
- Reboot server recover and monitoring
- Need to look into it, try manually
- May need to upgrade iavf driver
- Server in-accessiable
- Reboot server recover the service
- AVF interface creation issue:
- Can't create AVF interface on VFs with configured VLAN - happens with latest i40e driver on tx2
- Related to PF i40e driver, vlan stripping configured on VF, PF driver return not allowd.
- Not related to iavf driver, AVF interface - vpp native driver have this issue
- dpdk iavf ignore the error and continue initialization, while vpp abort the init process
- Intel will fix the issue from PF driver - workaround: use old i40e driver (from ubuntu 20.04)
- Race condition occur on /dev/vfio mounting
-
- Run six rounds of tests with patch https://gerrit.fd.io/r/c/csit/+/34045 and result looks good right now
- Patch has been merged
- By not mounting whole /dev/vfio, mount only /dev/vfio/xxx used
- Addressed comments, waiting Peter's review - Peter approved, patch merged and monitoring.
- Met tx2 server reboot issue when monitoring - RAS CONTROLLER: Fatal unrecoverable error detected ** NBU Error **
- Reinstall the tx2 server with older kernel version and VPP device testing works fine with Juraj's patch
- VPP Device configuration align with VPP Performance configuration - no issue yet
- Run six rounds of tests with patch https://gerrit.fd.io/r/c/csit/+/34045 and result looks good right now
- Enable VPP device testing per patch
- Monitor for a week and enable vote right then
- Ping Dave about enabling VPP device testing per patch
- Dave gave minor comments https://gerrit.fd.io/r/c/ci-management/+/34679 - comment addressed and will be merged soon
- Need to figure out how to reproduce the error - Juraj
- Periodic job will stop when per patch job enabled
-
- Can't create AVF interface on VFs with configured VLAN - happens with latest i40e driver on tx2
- New Arm servers shipment to the FD.io lab - about Jan 2022
- 2 build servers ready for shipment - 1 RU, no pcie slot for NICs
- 2 performance servers waiting for Intel NICs
- New servers are in the procurement process
- Plan to replace old thunderx1 build servers with more advanced Arm servers
- Need to confirm with RAM/disk size for the new build servers
- 500G disk/256G RAM
- Each job will consume about 16G memory
- Intel NIC firmware upgrade on Arm - not supported
- Paper work for shipment is done
- Build servers will arrive at end of Jan
- Performance servers will arrive in Feb
- VPP Performance Test
-
VPP
- VPP SVE implementation - Lijian
- SVE validation on FPGA platform - Confluence page ready
- Run L3 traffic testing with NEON/SVE-VLA/SVE-VLS VPP version
- Performance number: NEON>SVE-VLA>SVE-VLS on arm internal platform
- VPP IPv4 fragmentation & reassembly - Tianyu & Jieqiang
- Add multi-arch support for ip4-frag node but see no perf bump
- Apply loop unrolling on ip4-frag node
- Benchmark IPv4 fragmentation node using rdma plugin
- CNF PoC proposal preparation- Tianyu
- Add support for VPP aarch64 docker image build
- Calico use cases exploration on VPP
- Improve ansible scripts to deploy VPP&snort on K8S pods automatically
- VPP IPsec on Arm - Govind
- Work on IPsec input - Zach
- IPSec input node/decryption flow cache implemented in a separate patch - Zach
- Upstreamed https://gerrit.fd.io/r/c/vpp/+/32903
- VPP Perfmon plugin enablement on Arm - Zach
- Rely on kernel patch https://lore.kernel.org/linux-arm-kernel/[email protected]/ to enable the feature
- v10 kernel patch is ready, which fixes intermittent large statistic number for events
- Modify the commit message and upstream the perfmon patch - Zach
- Depends on kernel patch
- Building Intel QAT driver on arm to test IPsec crypt - Picchi
- Run IPsec with QAT on skylake this week, run the same setup on Ampere is next step
- Modify the commit message and upstream the perfmon patch - Zach
- VPP SVE implementation - Lijian
1/18/2022
-
Attendees
- Govindarajan Mohandoss
- Lijian Zhang
- Jieqiang Wang
- Juraj Linkes
- Tianyu Li
- Tina Tsou
-
CSIT
- VPP Performance Test
- https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-3n-tsh/
-
https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-report-iterative-2106-3n-tsh/
- Add new IPSec NULL encryption & decryption test cases - Juraj
- IPsec SPD input/output test case ongoing - Juraj
- Enable the SPD outbound tests
- Patches ready, waiting release testing done - ETA 1 week or 2
- Outbound SPD test patch has been enabled
- Release testing for 21.10 is done
- Enable the SPD outbound tests
- New links for VPP perf trending/report pages
- Daily trending: https://s3-docs.fd.io/csit/master/trending/
- Release report: https://s3-docs.fd.io/csit/master/report/
- 3 nodes Taishan crypto test case failed - related to CSIT change
- Patch ready for review and to be merged https://gerrit.fd.io/r/c/csit/+/34591 -- Merged
- VPP Path
- Voting and working fine.
- CentOS-8 jobs have been removed.
- VPP Device
- https://jenkins.fd.io/view/vpp/job/vpp-csit-verify-device-master-ubuntu2004-aarch64-1n-tx2/
- VM cases failed only on Arm
- Tried to increase the timeout to see it will fix the issue
- VM cases failed on 1 min timeout on creating VM (qemu cli)- tx2 node
- Reboot server recover and monitoring
- Need to look into it, try manually
- May need to upgrade iavf driver
- Server in-accessiable
- Reboot server recover the service
- AVF interface creation issue:
- Can't create AVF interface on VFs with configured VLAN - happens with latest i40e driver on tx2
- Related to PF i40e driver, vlan stripping configured on VF, PF driver return not allowd.
- Not related to iavf driver, AVF interface - vpp native driver have this issue
- dpdk iavf ignore the error and continue initialization, while vpp abort the init process
- Intel will fix the issue from PF driver - workaround: use old i40e driver (from ubuntu 20.04)
- Race condition occur on /dev/vfio mounting
-
- Run six rounds of tests with patch https://gerrit.fd.io/r/c/csit/+/34045 and result looks good right now
- Patch has been merged
- By not mounting whole /dev/vfio, mount only /dev/vfio/xxx used
- Addressed comments, waiting Peter's review - Peter approved, patch merged and monitoring.
- Met tx2 server reboot issue when monitoring - RAS CONTROLLER: Fatal unrecoverable error detected ** NBU Error **
- Reinstall the tx2 server with older kernel version and VPP device testing works fine with Juraj's patch
- VPP Device configuration align with VPP Performance configuration - no issue yet
- Run six rounds of tests with patch https://gerrit.fd.io/r/c/csit/+/34045 and result looks good right now
- Enable VPP device testing per patch
- Monitor for a week and enable vote right then
- Ping Dave about enabling VPP device testing per patch
- Dave gave minor comments https://gerrit.fd.io/r/c/ci-management/+/34679 - comment addressed and will be merged soon
- Periodic job will stop when per patch job enabled
-
- Can't create AVF interface on VFs with configured VLAN - happens with latest i40e driver on tx2
- New Arm servers shipment to the FD.io lab - about Jan 2022
- 2 build servers ready for shipment - 1 RU, no pcie slot for NICs
- 2 performance servers waiting for Intel NICs
- New servers are in the procurement process
- Plan to replace old thunderx1 build servers with more advanced Arm servers
- Need to confirm with RAM/disk size for the new build servers
- 500G disk/256G RAM
- Each job will consume about 16G memory
- Intel NIC firmware upgrade on Arm - not supported
- VPP Performance Test
-
VPP
- VPP SVE implementation - Lijian
- SVE validation on FPGA platform - Conflunence page ready
- Run L3 traffic testing with NEON/SVE-VLA/SVE-VLS VPP version
- Performance number: NEON>SVE-VLA>SVE-VLS on FPGA without DMC-620
- FPGA team promises to provide FPGA image with DMC-620
- Liangxing will help to benchmark VPP with FPGA enabled with DMC-620
- VPP IPv4 fragmentation - Tianyu & Jieqiang
- Add multi-arch support for ip4-frag node but see no perf bump
- Apply loop unrolling on ip4-frag node
- Benchmark IPv4 fragmentation node using rdma plugin
- CNF PoC proposal preparation- Tianyu
- Add support for VPP aarch64 docker image build
- Calico use cases exploration on VPP
- Improve ansible scripts to deploy VPP&snort on K8S pods automatically
- VPP IPsec on Arm - Govind
- Work on IPsec input - Zach
- IPSec input node/decryption flow cache implemented in a separate patch - Zach
- Upstreamed https://gerrit.fd.io/r/c/vpp/+/32903
- VPP Perfmon plugin enablement on Arm - Zach
- Rely on kernel patch https://lore.kernel.org/linux-arm-kernel/[email protected]/ to enable the feature
- v10 kernel patch is ready, which fixes intermittent large statistic number for events
- Modify the commit message and upstream the perfmon patch - Zach
- Depends on kernel patch
- Building Intel QAT driver on arm to test IPsec crypt - Picchi
- Run IPsec with QAT on skylake this week, run the same setup on Ampere is next step
- Modify the commit message and upstream the perfmon patch - Zach
- VPP SVE implementation - Lijian
1/11/2022
-
Attendees
- Govindarajan Mohandoss
- Lijian Zhang
- Jieqiang Wang
- Juraj Linkes
- Tina Tsou
-
CSIT
- VPP Performance Test
- https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-3n-tsh/
-
https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-report-iterative-2106-3n-tsh/
- Add new IPSec NULL encryption & decryption test cases - Juraj
- IPsec SPD input/output test case ongoing - Juraj
- Enable the SPD outbound tests
- Patches ready, waiting release testing done - ETA 1 week or 2
- Outbound SPD test patch has been enabled
- Release testing for 21.10 is done
- Enable the SPD outbound tests
- New links for VPP perf trending/report pages
- Daily trending: https://s3-docs.fd.io/csit/master/trending/
- Release report: https://s3-docs.fd.io/csit/master/report/
- 3 nodes Taishan crypto test case failed - related to CSIT change
- Patch ready for review and to be merged https://gerrit.fd.io/r/c/csit/+/34591
- VPP Path
- Voting and working fine.
- CentOS-8 jobs have been removed.
- VPP Device
- https://jenkins.fd.io/view/vpp/job/vpp-csit-verify-device-master-ubuntu2004-aarch64-1n-tx2/
- VM cases failed only on Arm
- Tried to increase the timeout to see it will fix the issue
- AVF interface creation issue:
- Can't create AVF interface on VFs with configured VLAN - happens with latest i40e driver on tx2
- Related to PF i40e driver, vlan stripping configured on VF, PF driver return not allowd.
- Not related to iavf driver, AVF interface - vpp native driver have this issue
- dpdk iavf ignore the error and continue initialization, while vpp abort the init process
- Intel will fix the issue from PF driver - workaround: use old i40e driver (from ubuntu 20.04)
- Race condition occur on /dev/vfio mounting
-
- Run six rounds of tests with patch https://gerrit.fd.io/r/c/csit/+/34045 and result looks good right now
- Patch has been merged
- By not mounting whole /dev/vfio, mount only /dev/vfio/xxx used
- Addressed comments, waiting Peter's review - Peter approved, patch merged and monitoring.
- Met tx2 server reboot issue when monitoring - RAS CONTROLLER: Fatal unrecoverable error detected ** NBU Error **
- Reinstall the tx2 server with older kernel version and VPP device testing works fine with Juraj's patch
- VPP Device configuration align with VPP Performance configuration - no issue yet
- Run six rounds of tests with patch https://gerrit.fd.io/r/c/csit/+/34045 and result looks good right now
- Enable VPP device testing per patch
- Monitor for a week and enable vote right then
- Ping Dave about enabling VPP device testing per patch
- Dave gave minor comments https://gerrit.fd.io/r/c/ci-management/+/34679 - comment addressed and will be merged soon
- Periodic job will stop when per patch job enabled
-
- Can't create AVF interface on VFs with configured VLAN - happens with latest i40e driver on tx2
- New Arm servers shipment to the FD.io lab - about Jan 2022
- New servers are in the procurement process
- Plan to replace old thunderx1 build servers with more advanced Arm servers
- Need to confirm with RAM/disk size for the new build servers
- 500G disk/256G RAM
- Each job will consume about 16G memory
- Intel NIC firmware upgrade on Arm - not supported
- VPP Performance Test
-
VPP
- VPP SVE implementation - Lijian
- SVE validation on FPGA platform - Conflunence page ready
- Run L3 traffic testing with NEON/SVE-VLA/SVE-VLS VPP version
- Performance number: NEON>SVE-VLA>SVE-VLS on FPGA without DMC-620
- FPGA team promises to provide FPGA image with DMC-620
- Liangxing will help to benchmark VPP with FPGA enabled with DMC-620
- VPP IPv4 fragmentation - Tianyu & Jieqiang
- Add multi-arch support for ip4-frag node but see no perf bump
- Apply loop unrolling on ip4-frag node
- Benchmark IPv4 fragmentation node using rdma plugin
- CNF PoC proposal preparation- Tianyu
- Add support for VPP aarch64 docker image build
- Calico use cases exploration on VPP
- Improve ansible scripts to deploy VPP&snort on K8S pods automatically
- VPP IPsec on Arm - Govind
- Work on IPsec input - Zach
- IPSec input node/decryption flow cache implemented in a separate patch - Zach
- Upstreamed https://gerrit.fd.io/r/c/vpp/+/32903
- VPP Perfmon plugin enablement on Arm - Zach
- Rely on kernel patch https://lore.kernel.org/linux-arm-kernel/[email protected]/ to enable the feature
- v10 kernel patch is ready, which fixes intermittent large statistic number for events
- Modify the commit message and upstream the perfmon patch - Zach
- Depends on kernel patch
- Building Intel QAT driver on arm to test IPsec crypt - Picchi
- Run IPsec with QAT on skylake this week, run the same setup on Ampere is next step
- Modify the commit message and upstream the perfmon patch - Zach
- VPP SVE implementation - Lijian
12/14/2021
-
Attendees
- Tianyu Li
- Govindarajan Mohandoss
- Jieqiang Wang
- Juraj Linkes
- Tina Tsou
-
CSIT
- VPP Performance Test
- https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-3n-tsh/
-
https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-report-iterative-2106-3n-tsh/
- Add new IPSec NULL encryption & decryption test cases - Juraj
- IPsec SPD input/output test case ongoing - Juraj
- Enable the SPD outbound tests
- Patches ready, waiting release testing done - ETA 1 week or 2
- Outbound SPD test patch has been enabled
- Release testing for 21.10 is done
- Enable the SPD outbound tests
- New links for VPP perf trending/report pages
- Daily trending: https://s3-docs.fd.io/csit/master/trending/
- Release report: https://s3-docs.fd.io/csit/master/report/
- 3 nodes Taishan crypto test case failed - related to CSIT change
- Patch ready for review and to be merged https://gerrit.fd.io/r/c/csit/+/34591
- VPP Path
- Voting and working fine.
- CentOS-8 jobs have been removed.
- VPP Device
- https://jenkins.fd.io/view/vpp/job/vpp-csit-verify-device-master-ubuntu2004-aarch64-1n-tx2/
- VM cases failed only on Arm
- Tried to increase the timeout to see it will fix the issue
- AVF interface creation issue:
- Can't create AVF interface on VFs with configured VLAN - happens with latest i40e driver on tx2
- Related to PF i40e driver, vlan stripping configured on VF, PF driver return not allowd.
- Not related to iavf driver, AVF interface - vpp native driver have this issue
- dpdk iavf ignore the error and continue initialization, while vpp abort the init process
- Intel will fix the issue from PF driver - workaround: use old i40e driver (from ubuntu 20.04)
- Race condition occur on /dev/vfio mounting
-
- Run six rounds of tests with patch https://gerrit.fd.io/r/c/csit/+/34045 and result looks good right now
- Patch has been merged
- By not mounting whole /dev/vfio, mount only /dev/vfio/xxx used
- Addressed comments, waiting Peter's review - Peter approved, patch merged and monitoring.
- Met tx2 server reboot issue when monitoring - RAS CONTROLLER: Fatal unrecoverable error detected ** NBU Error **
- Reinstall the tx2 server with older kernel version and VPP device testing works fine with Juraj's patch
- VPP Device configuration align with VPP Performance configuration - no issue yet
- Run six rounds of tests with patch https://gerrit.fd.io/r/c/csit/+/34045 and result looks good right now
- Enable VPP device testing per patch
- Monitor for a week and enable vote right then
- Ping Dave about enabling VPP device testing per patch
- Dave gave minor comments https://gerrit.fd.io/r/c/ci-management/+/34679 - comment addressed and will be merged soon
- Periodic job will stop when per patch job enabled
-
- Can't create AVF interface on VFs with configured VLAN - happens with latest i40e driver on tx2
- New Arm servers shipment to the FD.io lab - about Jan 2022
- New servers are in the procurement process
- Plan to replace old thunderx1 build servers with more advanced Arm servers
- Need to confirm with RAM/disk size for the new build servers
- 500G disk/256G RAM
- Each job will consume about 16G memory
- Intel NIC firmware upgrade on Arm - not supported
- VPP Performance Test
-
VPP
- VPP SVE implementation - Lijian
- SVE validation on FPGA platform - Conflunence page ready
- Run L3 traffic testing with NEON/SVE-VLA/SVE-VLS VPP version
- Performance number: NEON>SVE-VLA>SVE-VLS on FPGA without DMC-620
- FPGA team promises to provide FPGA image with DMC-620
- Liangxing will help to benchmark VPP with FPGA enabled with DMC-620
- VPP IPv4 fragmentation - Tianyu & Jieqiang
- Add multi-arch support for ip4-frag node but see no perf bump
- Apply loop unrolling on ip4-frag node
- VPP IPv6 Benchmarking and Profiling - Jieqiang
- IPv6 profiling
- No perf bump for lookup_x2 function in Fd.io gerrit
- Try Mellaonx nics for IPv6 routing tests
- IPv6 profiling
- CNF PoC proposal preparation- Tianyu
- Add support for VPP aarch64 docker image build
- Calico use cases exploration on VPP
- Improve ansible scripts to deploy VPP&snort on K8S pods automatically
- VPP IPsec on Arm - Govind
- Work on IPsec input - Zach
- IPSec input node/decryption flow cache implemented in a separate patch - Zach
- Upstreamed https://gerrit.fd.io/r/c/vpp/+/32903
- VPP Perfmon plugin enablement on Arm - Zach
- Rely on kernel patch https://lore.kernel.org/linux-arm-kernel/[email protected]/ to enable the feature
- v10 kernel patch is ready, which fixes intermittent large statistic number for events
- Modify the commit message and upstream the perfmon patch - Zach
- Depends on kernel patch
- Building Intel QAT driver on arm to test IPsec crypt - Picchi
- Run IPsec with QAT on skylake this week, run the same setup on Ampere is next step
- Modify the commit message and upstream the perfmon patch - Zach
- VPP SVE implementation - Lijian
12/07/2021
-
Attendees
- Tianyu Li
- Govindarajan Mohandoss
- Lijian Zhang
- Jieqiang Wang
- Juraj Linkes
- Tina Tsou
-
CSIT
- VPP Performance Test
- https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-3n-tsh/
-
https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-report-iterative-2106-3n-tsh/
- Add new IPSec NULL encryption & decryption test cases - Juraj
- IPsec SPD input/output test case ongoing - Juraj
- Enable the SPD outbound tests
- Patches ready, waiting release testing done - ETA 1 week or 2
- Outbound SPD test patch has been enabled
- Release testing for 21.10 is done
- Enable the SPD outbound tests
- New links for VPP perf trending/report pages
- Daily trending: https://s3-docs.fd.io/csit/master/trending/
- Release report: https://s3-docs.fd.io/csit/master/report/
- 3 nodes Taishan crypto test case failed - related to CSIT change
- Patch ready for review and to be merged https://gerrit.fd.io/r/c/csit/+/34591
- VPP Path
- Voting and working fine.
- CentOS-8 jobs have been removed.
- VPP Device
- https://jenkins.fd.io/view/vpp/job/vpp-csit-verify-device-master-ubuntu2004-aarch64-1n-tx2/
- VM cases failed only on Arm
- Tried to increase the timeout to see it will fix the issue
- AVF interface creation issue:
- Can't create AVF interface on VFs with configured VLAN - happens with latest i40e driver on tx2
- Related to PF i40e driver, vlan stripping configured on VF, PF driver return not allowd.
- Not related to iavf driver, AVF interface - vpp native driver have this issue
- dpdk iavf ignore the error and continue initialization, while vpp abort the init process
- Intel will fix the issue from PF driver - workaround: use old i40e driver (from ubuntu 20.04)
- Race condition occur on /dev/vfio mounting
-
- Run six rounds of tests with patch https://gerrit.fd.io/r/c/csit/+/34045 and result looks good right now
- Patch has been merged
- By not mounting whole /dev/vfio, mount only /dev/vfio/xxx used
- Addressed comments, waiting Peter's review - Peter approved, patch merged and monitoring.
- Met tx2 server reboot issue when monitoring - RAS CONTROLLER: Fatal unrecoverable error detected ** NBU Error **
- Reinstall the tx2 server with older kernel version and VPP device testing works fine with Juraj's patch
- VPP Device configuration align with VPP Performance configuration - no issue yet
- Run six rounds of tests with patch https://gerrit.fd.io/r/c/csit/+/34045 and result looks good right now
- Enable VPP device testing per patch
- Monitor for a week and enable vote right then
- Ping Dave about enabling VPP device testing per patch
- Dave gave minor comments https://gerrit.fd.io/r/c/ci-management/+/34679 - comment addressed and will be merged soon
- Periodic job will stop when per patch job enabled
-
- Can't create AVF interface on VFs with configured VLAN - happens with latest i40e driver on tx2
- New Arm servers shipment to the FD.io lab - about Jan 2022
- New servers are in the procurement process
- Plan to replace old thunderx1 build servers with more advanced Arm servers
- Need to confirm with RAM/disk size for the new build servers
- 500G disk/256G RAM
- Each job will consume about 16G memory
- Intel NIC firmware upgrade on Arm - not supported
- VPP Performance Test
-
VPP
- VPP SVE implementation - Lijian
- SVE validation on FPGA platform - Conflunence page ready
- Run L3 traffic testing with NEON/SVE-VLA/SVE-VLS VPP version
- Performance number: NEON>SVE-VLA>SVE-VLS on FPGA without DMC-620
- FPGA team promises to provide FPGA image with DMC-620
- Liangxing will help to benchmark VPP with FPGA enabled with DMC-620
- VPP IPv4 fragmentation - Tianyu & Jieqiang
- Add multi-arch support for ip4-frag node but see no perf bump
- Apply loop unrolling on ip4-frag node
- VPP IPv6 Benchmarking and Profiling - Jieqiang
- IPv6 profiling
- No perf bump for lookup_x2 function in Fd.io gerrit
- Try Mellaonx nics for IPv6 routing tests
- IPv6 profiling
- CNF PoC proposal preparation- Tianyu
- Add support for VPP aarch64 docker image build
- Calico use cases exploration on VPP
- Improve ansible scripts to deploy VPP&snort on K8S pods automatically
- VPP IPsec on Arm - Govind
- Work on IPsec input - Zach
- IPSec input node/decryption flow cache implemented in a separate patch - Zach
- Upstreamed https://gerrit.fd.io/r/c/vpp/+/32903
- VPP Perfmon plugin enablement on Arm - Zach
- Rely on kernel patch https://lore.kernel.org/linux-arm-kernel/[email protected]/ to enable the feature
- v10 kernel patch is ready, which fixes intermittent large statistic number for events
- Modify the commit message and upstream the perfmon patch - Zach
- Depends on kernel patch
- Building Intel QAT driver on arm to test IPsec crypt - Picchi
- Run IPsec with QAT on skylake this week, run the same setup on Ampere is next step
- Modify the commit message and upstream the perfmon patch - Zach
- VPP SVE implementation - Lijian
11/30/2021
-
Attendees
- Tianyu Li
- Govindarajan Mohandoss
- Lijian Zhang
- Jieqiang Wang
- Juraj Linkes
- Tina Tsou
-
CSIT
- VPP Performance Test
- https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-3n-tsh/
-
https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-report-iterative-2106-3n-tsh/
- Add new IPSec NULL encryption & decryption test cases - Juraj
- IPsec SPD input/output test case ongoing - Juraj
- Enable the SPD outbound tests
- Patches ready, waiting release testing done - ETA 1 week or 2
- Outbound SPD test patch has been enabled
- Release testing for 21.10 is done
- Enable the SPD outbound tests
- New links for VPP perf trending/report pages
- Daily trending: https://s3-docs.fd.io/csit/master/trending/
- Release report: https://s3-docs.fd.io/csit/master/report/
- 3 nodes Taishan crypto test case failed - related to CSIT change
- VPP Path
- Voting and working fine.
- CentOS-8 jobs have been removed.
- VPP Device
- https://jenkins.fd.io/view/vpp/job/vpp-csit-verify-device-master-ubuntu2004-aarch64-1n-tx2/
- AVF interface creation issue:
- Can't create AVF interface on VFs with configured VLAN - happens with latest i40e driver on tx2
- Related to PF i40e driver, vlan stripping configured on VF, PF driver return not allowd.
- Not related to iavf driver, AVF interface - vpp native driver have this issue
- dpdk iavf ignore the error and continue initialization, while vpp abort the init process
- Intel will fix the issue from PF driver - workaround: use old i40e driver (from ubuntu 20.04)
- Race condition occur on /dev/vfio mounting
-
- Run six rounds of tests with patch https://gerrit.fd.io/r/c/csit/+/34045 and result looks good right now
- Patch has been merged
- By not mounting whole /dev/vfio, mount only /dev/vfio/xxx used
- Addressed comments, waiting Peter's review - Peter approved, patch merged and monitoring.
- Met tx2 server reboot issue when monitoring - RAS CONTROLLER: Fatal unrecoverable error detected ** NBU Error **
- Reinstall the tx2 server with older kernel version and VPP device testing works fine with Juraj's patch
- VPP Device configuration align with VPP Performance configuration - no issue yet
- Run six rounds of tests with patch https://gerrit.fd.io/r/c/csit/+/34045 and result looks good right now
- Enable VPP device testing per patch
- Monitor for a week and enable vote right then
- Ping Dave about enabling VPP device testing per patch
-
- Can't create AVF interface on VFs with configured VLAN - happens with latest i40e driver on tx2
- New Arm servers shipment to the FD.io lab
- New servers are in the procurement process
- Plan to replace old thunderx1 build servers with more advanced Arm servers
- Need to confirm with RAM/disk size for the new build servers
- 500G disk/256G RAM
- Each job will consume about 16G memory
- Intel NIC firmware upgrade on Arm - not supported
- VPP Performance Test
-
VPP
- VPP SVE implementation - Lijian
- SVE validation on FPGA platform - Conflunence page ready
- Run L3 traffic testing with NEON/SVE-VLA/SVE-VLS VPP version
- Performance number: NEON>SVE-VLA>SVE-VLS on FPGA without DMC-620
- FPGA team promises to provide FPGA image with DMC-620
- Liangxing will help to benchmark VPP with FPGA enabled with DMC-620
- VPP IPv4 fragmentation - Tianyu & Jieqiang
- Add multi-arch support for ip4-frag node but see no perf bump
- Apply loop unrolling on ip4-frag node
- VPP IPv6 Benchmarking and Profiling - Jieqiang
- IPv6 profiling
- No perf bump for lookup_x2 function in Fd.io gerrit
- Try Mellaonx nics for IPv6 routing tests
- IPv6 profiling
- CNF PoC proposal preparation- Tianyu
- Add support for VPP aarch64 docker image build
- Calico use cases exploration on VPP
- Improve ansible scripts to deploy VPP&snort on K8S pods automatically
- VPP IPsec on Arm - Govind
- Work on IPsec input - Zach
- IPSec input node/decryption flow cache implemented in a separate patch - Zach
- Upstreamed https://gerrit.fd.io/r/c/vpp/+/32903
- VPP Perfmon plugin enablement on Arm - Zach
- Rely on kernel patch https://lore.kernel.org/linux-arm-kernel/[email protected]/ to enable the feature
- v10 kernel patch is ready, which fixes intermittent large statistic number for events
- Modify the commit message and upstream the perfmon patch - Zach
- Building Intel QAT driver on arm to test IPsec crypt - Picchi
- VPP SVE implementation - Lijian
11/23/2021
-
Attendees
- Tianyu Li
- Lijian Zhang
- Jieqiang Wang
- Juraj Linkes
- Tina Tsou
-
CSIT
- VPP Performance Test
- https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-3n-tsh/
-
https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-report-iterative-2106-3n-tsh/
- Add new IPSec NULL encryption & decryption test cases - Juraj
- IPsec SPD input/output test case ongoing - Juraj
- Enable the SPD outbound tests
- Patches ready, waiting release testing done - ETA 1 week or 2
- Outbound SPD test patch merged and running, expected report shows next week.
- Inbound patch pending on merge, need maintainer's review
- https://gerrit.fd.io/r/c/csit/+/34256
- Release testing for 21.10 is done
- Enable the SPD outbound tests
- New links for VPP perf trending/report pages
- Daily trending: https://s3-docs.fd.io/csit/master/trending/
- Release report: https://s3-docs.fd.io/csit/master/report/
- 3 nodes Taishan crypto test case failed - related to CSIT change
- VPP Path
- Voting and working fine.
- CentOS-8 jobs have been removed.
- VPP Device
- https://jenkins.fd.io/view/vpp/job/vpp-csit-verify-device-master-ubuntu2004-aarch64-1n-tx2/
- AVF interface creation issue:
- Can't create AVF interface on VFs with configured VLAN - happens with latest i40e driver on tx2
- Related to PF i40e driver, vlan stripping configured on VF, PF driver return not allowd.
- Not related to iavf driver, AVF interface - vpp native driver have this issue
- dpdk iavf ignore the error and continue initialization, while vpp abort the init process
- Intel will fix the issue from PF driver - workaround: use old i40e driver (from ubuntu 20.04)
- Race condition occur on /dev/vfio mounting
-
- Run six rounds of tests with patch https://gerrit.fd.io/r/c/csit/+/34045 and result looks good right now
- Patch has been merged
- By not mounting whole /dev/vfio, mount only /dev/vfio/xxx used
- Addressed comments, waiting Peter's review - Peter approved, patch merged and monitoring.
- Met tx2 server reboot issue when monitoring - RAS CONTROLLER: Fatal unrecoverable error detected ** NBU Error **
- Reinstall the tx2 server with older kernel version and VPP device testing works fine with Juraj's patch
- VPP Device configuration align with VPP Performance configuration - no issue yet
- Run six rounds of tests with patch https://gerrit.fd.io/r/c/csit/+/34045 and result looks good right now
- Enable VPP device testing per patch
- Monitor for a week and enable vote right then
-
- Can't create AVF interface on VFs with configured VLAN - happens with latest i40e driver on tx2
- New Arm servers shipment to the FD.io lab
- New servers are in the procurement process
- Plan to replace old thunderx1 build servers with more advanced Arm servers
- Need to confirm with RAM/disk size for the new build servers
- Intel NIC firmware upgrade on Arm - not supported
- VPP Performance Test
-
VPP
- VPP SVE implementation - Lijian
- SVE validation on FPGA platform - Conflunence page ready
- Run L3 traffic testing with NEON/SVE-VLA/SVE-VLS VPP version
- Performance number: NEON>SVE-VLA>SVE-VLS on FPGA without DMC-620
- FPGA team promises to provide FPGA image with DMC-620
- Liangxing will help to benchmark VPP with FPGA enbaled with DMC-620
- VPP IPv4 fragmentation
- VPP IPv6 Benchmarking and Profiling
- IPv6 profiling
- Hotspot function - ip6_lookup_node/ip6_rewrite_node
- Performance degradation with quad loop unrolling applied on ip6_lookup_inline
- Patch the current kernel to enable perfmon plugin on VPP
- Need to check performance for IPv6 subnet routing
- Try Mellaonx nics for IPv6 routing tests
- IPv6 profiling
- VPP SVE implementation - Lijian
-
- VPP memif - Tianyu
- CNF PoC proposal preparation
-
- Add support for VPP aarch64 docker image build
- Calico use cases exploration on VPP
- Improve ansible scripts to deploy VPP&snort on K8S pods automatically
-
- CNF PoC proposal preparation
- VPP IPsec on Arm - Govind
- Work on IPsec input - Zach
- IPSec input node/decryption flow cache implemented in a separate patch - Zach
- Upstreamed https://gerrit.fd.io/r/c/vpp/+/32903
- VPP Perfmon plugin enablement on Arm - Zach
- Rely on kernel patch https://lore.kernel.org/linux-arm-kernel/[email protected]/ to enable the feature
- v10 kernel patch is ready, which fixes intermittent large statistic number for events
- Modify the commit message and upstream the perfmon patch - Zach
- Building Intel QAT driver on arm to test IPsec crypt - Picchi
- VPP memif - Tianyu
11/16/2021
-
Attendees
- Tianyu Li
- Govindarajan Mohandoss
- Lijian Zhang
- Jieqiang Wang
- Juraj Linkes
- Tina Tsou
-
CSIT
- VPP Performance Test
- https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-3n-tsh/
-
https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-report-iterative-2106-3n-tsh/
- Add new IPSec NULL encryption & decryption test cases - Juraj
- IPsec SPD input/output test case ongoing - Juraj
- Enable the SPD outbound tests
- Patches ready, waiting release testing done - ETA 1 week or 2
- Inbound patch pending on merge, need maintainer's review
- https://gerrit.fd.io/r/c/csit/+/34256
- Release testing for 21.10 is done
- Enable the SPD outbound tests
- New links for VPP perf trending/report pages
- Daily trending: https://s3-docs.fd.io/csit/master/trending/
- Release report: https://s3-docs.fd.io/csit/master/report/
- VPP Path
- Voting and working fine.
- CentOS-8 jobs have been removed.
- VPP Device
- https://jenkins.fd.io/view/vpp/job/vpp-csit-verify-device-master-ubuntu2004-aarch64-1n-tx2/
- AVF interface creation issue:
- Can't create AVF interface on VFs with configured VLAN - happens with latest i40e driver on tx2
- Related to PF i40e driver, vlan stripping configured on VF, PF driver return not allowd.
- Not related to iavf driver, AVF interface - vpp native driver have this issue
- dpdk iavf ignore the error and continue initialization, while vpp abort the init process
- Intel will fix the issue from PF driver - workaround: use old i40e driver (from ubuntu 20.04)
- Race condition occur on /dev/vfio mounting
-
- Run six rounds of tests with patch https://gerrit.fd.io/r/c/csit/+/34045 and result looks good right now
- Patch has been merged
- By not mounting whole /dev/vfio, mount only /dev/vfio/xxx used
- Addressed comments, waiting Peter's review - Peter approved, patch merged and monitoring.
- Met tx2 server reboot issue when monitoring - RAS CONTROLLER: Fatal unrecoverable error detected ** NBU Error **
- Reinstall the tx2 server with older kernel version and VPP device testing works fine with Juraj's patch
- Run six rounds of tests with patch https://gerrit.fd.io/r/c/csit/+/34045 and result looks good right now
- Enable VPP device testing per patch
-
- Can't create AVF interface on VFs with configured VLAN - happens with latest i40e driver on tx2
- New Arm servers shipment to the FD.io lab
- New servers are in the procurement process
- Plan to replace old thunderx1 build servers with more advanced Arm servers
- Intel NIC firmware upgrade on Arm - not supported
- VPP Performance Test
-
VPP
- VPP SVE implementation - Lijian
- SVE validation on FPGA platform - Conflunence page ready
- Run L3 traffic testing with NEON/SVE-VLA/SVE-VLS VPP version
- Performonce number: NEON>SVE-VLA>SVE-VLS on FPGA without DMC-620
- FPGA team promises to provide FPGA image with DMC-620
- Liangxing will help to benchmark VPP with FPGA enbaled with DMC-620
- VPP IPv4 fragmentation
- VPP IPv6 Benchmarking and Profiling
- IPv6 profiling
- Hotspot function - ip6_lookup_node/ip6_rewrite_node
- Performance degradation with quad loop unrolling applied on ip6_lookup_inline
- Patch the current kernel to enable perfmon plugin on VPP
- Need to check performance for IPv6 subnet routing
- Try Mellaonx nics for IPv6 routing tests
- IPv6 profiling
- VPP SVE implementation - Lijian
-
- VPP memif - Tianyu
- CNF PoC proposal preparation
-
- Add support for VPP aarch64 docker image build
- Calico use cases exploration on VPP
- Improve ansible scripts to deploy VPP&snort on K8S pods automatically
-
- CNF PoC proposal preparation
- VPP IPsec on Arm - Govind
- Work on IPsec input - Zach
- IPSec input node/decryption flow cache implemented in a separate patch - Zach
- Upstreamed https://gerrit.fd.io/r/c/vpp/+/32903
- VPP Perfmon plugin enablement on Arm - Zach
- Rely on kernel patch https://lore.kernel.org/linux-arm-kernel/[email protected]/ to enable the feature
- v10 kernel patch is ready, which fixes intermittent large statistic number for events
- Modify the commit message and upstream the perfmon patch - Zach
- Building Intel QAT driver on arm to test IPsec crypt - Picchi
- VPP memif - Tianyu
11/09/2021
-
Attendees
- Tianyu Li
- Govindarajan Mohandoss
- Lijian Zhang
- Juraj Linkes
- Tina Tsou
-
CSIT
- VPP Performance Test
- https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-3n-tsh/
-
https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-report-iterative-2106-3n-tsh/
- Add new IPSec NULL encryption & decryption test cases - Juraj
- IPsec SPD input/output test case ongoing - Juraj
- Enable the SPD outbound tests
- Patches ready, waiting release testing done - ETA 1 week or 2
- Inbound patch pending on merge, need maintainer's review
- https://gerrit.fd.io/r/c/csit/+/34256
- Enable the SPD outbound tests
- New links for VPP perf trending/report pages
- Daily trending: https://s3-docs.fd.io/csit/master/trending/
- Release report: https://s3-docs.fd.io/csit/master/report/
- VPP Path
- Voting and working fine.
- CentOS-8 jobs have been removed.
- VPP Device
- https://jenkins.fd.io/view/vpp/job/vpp-csit-verify-device-master-ubuntu2004-aarch64-1n-tx2/
- AVF interface creation issue:
- Can't create AVF interface on VFs with configured VLAN - happens with latest i40e driver on tx2
- Related to PF i40e driver, vlan stripping configured on VF, PF driver return not allowd.
- Not related to iavf driver, AVF interface - vpp native driver have this issue
- dpdk iavf ignore the error and continue initialization, while vpp abort the init process
- Intel will fix the issue from PF driver - workaround: use old i40e driver (from ubuntu 20.04)
- Race condition occur on /dev/vfio mounting
-
- Run six rounds of tests with patch https://gerrit.fd.io/r/c/csit/+/34045 and result looks good right now
- By not mounting whole /dev/vfio, mount only /dev/vfio/xxx used
- Addressed comments, waiting Peter's review - Peter approved, patch merged and monitoring.
- Met tx2 server reboot issue when monitoring - RAS CONTROLLER: Fatal unrecoverable error detected ** NBU Error **
- Will enable voting right soon after the patch gets merged
- Run six rounds of tests with patch https://gerrit.fd.io/r/c/csit/+/34045 and result looks good right now
-
- Can't create AVF interface on VFs with configured VLAN - happens with latest i40e driver on tx2
- New Arm servers shippment to the FD.io lab
- New servers are in the procurement process
- Plan to replace old thunderx1 build servers with more advanced Arm servers
- Intel NIC firmware upgrade on Arm - not supported
- VPP Performance Test
-
VPP
- VPP SVE implementation - Lijian
- SVE validation on FPGA platform - Conflunce page ready
- Run L3 traffic testing with NEON/SVE-VLA/SVE-VLS VPP version
- Perfmonce number: NEON>SVE-VLA>SVE-VLS on FPGA without DMC-620
- FPGA team promises to provide FPGA image with DMC-620
- VPP IPv4 fragmetation
- VPP IPv6 Benchmarking and Profiling
- IPv6 profiling
- Hotspot function - ip6_lookup_node/ip6_rewrite_node
- Performance degradation with quad loop unrolling applied on ip6_lookup_inline
- Patch the current kernel to enable perfmon plugin on VPP
- Need to check performance for IPv6 subnet routing
- IPv6 profiling
- VPP SVE implementation - Lijian
-
- VPP memif - Tianyu
- CNF PoC proposal preparation
-
- Add support for VPP aarch64 docker image build
- Calico use cases exploration on VPP
- Improve ansible scripts to deploy VPP&snort on K8S pods automatically
-
- CNF PoC proposal preparation
- VPP IPsec on Arm - Govind
- Work on IPsec input - Zach
- IPSec input node/decryption flow cache implemented in a separate patch - Zach
- Upstreamed https://gerrit.fd.io/r/c/vpp/+/32903
- VPP Perfmon plugin enablement on Arm - Zach
- Rely on kernel patch https://lore.kernel.org/linux-arm-kernel/[email protected]/ to enable the feature
- v10 kernel patch is ready, which fixes intermittent large statistic number for events
- Modify the commit message and upstream the perfmon patch - Zach
- Building Intel QAT driver on arm to test IPsec crypt - Picchi
- VPP memif - Tianyu
11/02/2021
-
Attendees
- Tianyu Li
- Govindarajan Mohandoss
- Jieqiang Wang
- Tina Tsou
-
CSIT
- VPP Performance Test
- https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-3n-tsh/
-
https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-report-iterative-2106-3n-tsh/
- Add new IPSec NULL encryption & decryption test cases - Juraj
- IPsec SPD input/output test case ongoing - Juraj
- Enable the SPD outbound tests
- New links for VPP perf trending/report pages
- Daily trending: https://s3-docs.fd.io/csit/master/trending/
- Release report: https://s3-docs.fd.io/csit/master/report/
- VPP Path
- Voting and working fine.
- CentOS-8 jobs have been removed.
- VPP Device
- https://jenkins.fd.io/view/vpp/job/vpp-csit-verify-device-master-ubuntu2004-aarch64-1n-tx2/
- AVF interface creation issue:
- Can't create AVF interface on VFs with configured VLAN - happens with latest i40e driver on tx2
- race condition occur
- Run six rounds of tests with patch https://gerrit.fd.io/r/c/csit/+/34045 and result looks good right now
- Addressed comments, waiting Peter's review.
- Will enable voting right soon after the patch gets merged
- Run six rounds of tests with patch https://gerrit.fd.io/r/c/csit/+/34045 and result looks good right now
- race condition occur
- Can't create AVF interface on VFs with configured VLAN - happens with latest i40e driver on tx2
- New Arm servers shippment to the FD.io lab
- New servers are in the procurement process
- Plan to replace old thunderx1 build servers with more advanced Arm servers
- Intel NIC firmware upgrade on Arm - not supported
- VPP Performance Test
-
VPP
- VPP SVE implementation - Lijian
- SVE validation on FPGA platform - Conflunce page ready
- Run L3 traffic testing with NEON/SVE-VLA/SVE-VLS VPP version
- Perfmonce number: NEON>SVE-VLA>SVE-VLS on FPGA without DMC-620
- FPGA team promises to provide FPGA image with DMC-620
- VPP IPv6 Benchmarking and Profiling
- IPv6 profiling
- Hotspot function - ip6_lookup_node/ip6_rewrite_node
- Performance degradation with quad loop unrolling applied on ip6_lookup_inline
- Patch the current kernel to enable perfmon plugin on VPP
- Need to check performance for IPv6 subnet routing
- IPv6 profiling
- VPP Prefetch
- Refactor prefetch implementation in VPP per CPU's actual cache line size - Tianyu
- Two patches unmerged
- l2: fix array-bounds error for prefetch on Arm https://gerrit.fd.io/r/c/vpp/+/33307
- ioam: fix prefetch out bound on Arm https://gerrit.fd.io/r/c/vpp/+/33506
- Two patches unmerged
- Refactor prefetch implementation in VPP per CPU's actual cache line size - Tianyu
- VPP SVE implementation - Lijian
-
- VPP memif - Tianyu
- CNF PoC proposal preparation
-
- Add support for VPP aarch64 docker image build
- Calico use cases exploration on VPP
- Improve ansible scripts to deploy VPP&snort on K8S pods automatically
-
- CNF PoC proposal preparation
- VPP IPsec on Arm - Govind
- Work on IPsec input - Zach
- IPSec input node/decryption flow cache implemented in a separate patch - Zach
- Upstreamed https://gerrit.fd.io/r/c/vpp/+/32903
- VPP Perfmon plugin enablement on Arm - Zach
- Rely on kernel patch https://lore.kernel.org/linux-arm-kernel/[email protected]/ to enable the feature
- v10 kernel patch is ready, which fixes intermittent large statistic number for events
- Modify the commit message and upstream the perfmon patch - Zach
- Building Intel QAT driver on arm to test IPsec crypt - Picchi
- VPP memif - Tianyu
10/26/2021
-
Attendees
- Juraj Linkes
- Tianyu Li
- Govindarajan Mohandoss
- Lijian Zhang
- Jieqiang Wang
- Tina Tsou
-
CSIT
- VPP Performance Test
- https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-3n-tsh/
-
https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-report-iterative-2106-3n-tsh/
- 2106 testing partial finished. 21.01.1 ongoing, should be done sometime next week. - closed
- Leverage current spare TX2 server as 2-node topology performance test-bed.
- Add new IPSec NULL encryption & decryption test cases - Juraj
- Inbound IPsec: reproduced and need to investigate - Juraj
- Flow cache with 1, 10 SPD entries slower, still investigating. Manual test local vs CSIT have different result on 1-10 SPD policies.
- IPsec SPD input/output case ongoing
- Adding IPsec SPD outbound test cases 64B 1, 100 and 1k SPD entries, 1, 2, 4 cores, on tx2 testbed - clarified
- Flow cache on and off cases need to be measured.
- L2 BD 20k test cases execute time too long, removed on taishan.
- Adding IPsec SPD outbound test cases 64B 1, 100 and 1k SPD entries, 1, 2, 4 cores, on tx2 testbed - clarified
- Steps to enable test case in CSIT https://gerrit.fd.io/r/c/csit/+/31863
- 3n-tsh testbed unreachable, investigating right now - Juraj
- TG firmware is under upgradation
- Server unreachable due to firmware & driver update - resolved - update all done
- Release testing for 21.10 starts
- Add new IPSec NULL encryption & decryption test cases - Juraj
- VPP Path
- Voting and working fine.
- CentOS-8 jobs have been removed.
- VPP Device
- https://jenkins.fd.io/view/vpp/job/vpp-csit-verify-device-master-ubuntu2004-aarch64-1n-tx2/
- VPP device testing issues may be caused by XL710 i40e fw or kernel module.
-
- Try to reproduce with another set of firmware and etc but issues still exist
-
https://doc.dpdk.org/guides/nics/i40e.html
- DPDK uses i40evf PMD and it is old and scheduled to be deprecated.
- This is fixed in DPDK 21.05 version by making iavf PMD as default.
- Use iavf PMD instead of i40evf on all VPP branches, waiting for the test results - not reproduced so far.
-
- New issue: different error related moving VF from host to container(not involving dpdk/VPP) - just started investigating
- /usr/bin/vpp[3789]: pci: 0000:91:02.0: open_vfio_iommu_group: open '/dev/vfio/141': Device or resource busy
- Resulting in the same failure as before, only happen on AArch64 platform
- Still see /dev/vfio resource busy error after linux kernel upgradation, but less frequently than before
- Dig into the log for more details - Juraj
- Narrow down whether some of those arguments are the reason behind this and that is indeed the case: --volume /dev/vfio:/dev/vfio causes the issue.
- AVF interface creation issue:
- Can't create AVF interface on VFs with configured VLAN - happens with latest i40e driver on tx2
- race condition occur
- try mounting a part of /dev/vfio to see if issue can be resolved
- Run six rounds of tests with patch https://gerrit.fd.io/r/c/csit/+/34045 and looks good right now
- Addressed comments, waiting Peter's review..
- Will enable voting right soon after the patch gets merged
- Can't create AVF interface on VFs with configured VLAN - happens with latest i40e driver on tx2
- /usr/bin/vpp[3789]: pci: 0000:91:02.0: open_vfio_iommu_group: open '/dev/vfio/141': Device or resource busy
- Shipment of new advanced server to the FD.io lab
- One link between TG and DUT, multiple link between DUT for testing LACP.
- Two advanced servers are in plan to ship
- Need vexxhost guys confirm with ethernet / power cable type info.
- Lijian has summarized feedbacks from Juraj and raised Jira ticket to DevOps team
- New servers are in the procurement process
- Plan to replace old thunderx1 build servers with more advanced Arm servers
-
- CPU not fully utilized on Arm, need further investigation
-
- Intel NIC firmware upgrade on Arm - not supported
- VPN access request to FD.io Arm servers
- Lijian/Jieqiang has got VPN access now
- mbuf-fast-free dpdk enablement with VPP (DEV_TX_OFFLOAD_MBUF_FAST_FREE)
- VPP Performance Test
-
VPP
- VPP SVE implementation - Lijian
-
- SVE validation on FPGA platform - Conflunce page ready
- Run unit tests from DPDK and VPP bihash on FPGA
- Try Lijian's SVE patch to see any cycle count improvement
- Able to access FPGA platform, investigating adding vpp to buildroot filesystem - Lijian
- Run standalone SVE test cases on FPGA
- Ask for DMC 620 images to run for FPGA
- Enable DMC 620 more close to real system, but performance will drop
- Build a system using VPP memif and pktgen
- SVE validation on FPGA platform - Conflunce page ready
-
- VPP IPv6 Benchmarking and Profiling
- VPP CLI 'ip route add ipv6_addr/mask' outputs wrong IPv6 routes with mask 123-128
- Patch upstreamed https://gerrit.fd.io/r/c/vpp/+/33422
- IPv6 profiling
- Hotspot function - ip6_lookup_node/ip6_rewrite_node
- Will try perfmon & understand two node functions
- VPP CLI configuration and hotspot function are recorded in Confluence page
- Plan to try quad loop unrolling for ip6_lookup_inline function
- VPP CLI 'ip route add ipv6_addr/mask' outputs wrong IPv6 routes with mask 123-128
- VPP Prefetch
- Refactor prefetch implementation in VPP per CPU's actual cache line size - Tianyu
- Two patches unmerged
- l2: fix array-bounds error for prefetch on Arm https://gerrit.fd.io/r/c/vpp/+/33307
- ioam: fix prefetch out bound on Arm https://gerrit.fd.io/r/c/vpp/+/33506
- Two patches unmerged
- Refactor prefetch implementation in VPP per CPU's actual cache line size - Tianyu
- VPP SVE implementation - Lijian
-
- VPP memif - Tianyu
- CNF PoC proposal preparation
-
- Add support for VPP aarch64 docker image build
- Calico use cases exploration on VPP
- Try to use ansible to deploy VPP automatically
-
- CNF PoC proposal preparation
- VPP IPsec on Arm - Govind
- Work on IPsec input - Zach
- IPSec input node/decryption flow cache implemented in a separate patch - Zach
- Upstreamed https://gerrit.fd.io/r/c/vpp/+/32903
- VPP Perfmon plugin enablement on Arm - Zach
- Review perfmon code - Lijian & Govind
- Implemented statistics from PMUv3 - done
- Patch upstream has dependency on kernel patch, waiting for this before upstream to VPP
- https://lore.kernel.org/linux-arm-kernel/[email protected]/
- Wait for v10 kernel patch to fix intermittent large statistic number for events
- Modify the commit message and upstream the perfmon patch - Zach
- Patch has been upstreamed and recieved comments from maintainer and Intel engineer
- Building Intel QAT driver on arm to test IPsec crypt - Zach
- Patch upstream has dependency on kernel patch, waiting for this before upstream to VPP
- VPP memif - Tianyu
10/19/2021
-
Attendees
- Juraj Linkes
- Tianyu Li
- Govindarajan Mohandoss
- Jieqiang Wang
- Tina Tsou
-
CSIT
- VPP Performance Test
- https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-3n-tsh/
-
https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-report-iterative-2106-3n-tsh/
- 2106 testing partial finished. 21.01.1 ongoing, should be done sometime next week. - closed
- Leverage current spare TX2 server as 2-node topology performance test-bed.
- Add new IPSec NULL encryption & decryption test cases - Juraj
- Inbound IPsec: reproduced and need to investigate - Juraj
- Flow cache with 1, 10 SPD entries slower, still investigating. Manual test local vs CSIT have different result on 1-10 SPD policies.
- IPsec SPD input/output case ongoing
- Steps to enable test case in CSIT https://gerrit.fd.io/r/c/csit/+/31863
- 3n-tsh testbed unreachable, investigating right now - Juraj
- TG firmware is under upgradation
- Server unreachable due to firmware & driver update - resolved - update all done
- Release testing for 21.10 starts
- Add new IPSec NULL encryption & decryption test cases - Juraj
- VPP Path
- Voting and working fine.
- CentOS-8 jobs have been removed.
- VPP Device
- https://jenkins.fd.io/view/vpp/job/vpp-csit-verify-device-master-ubuntu2004-aarch64-1n-tx2/
- VPP device testing issues may be caused by XL710 i40e fw or kernel module.
-
- Try to reproduce with another set of firmware and etc but issues still exist
-
https://doc.dpdk.org/guides/nics/i40e.html
- DPDK uses i40evf PMD and it is old and scheduled to be deprecated.
- This is fixed in DPDK 21.05 version by making iavf PMD as default.
- Use iavf PMD instead of i40evf on all VPP branches, waiting for the test results - not reproduced so far.
-
- New issue: different error related moving VF from host to container(not involving dpdk/VPP) - just started investigating
- /usr/bin/vpp[3789]: pci: 0000:91:02.0: open_vfio_iommu_group: open '/dev/vfio/141': Device or resource busy
- Resulting in the same failure as before, only happen on AArch64 platform
- Still see /dev/vfio resource busy error after linux kernel upgradation, but less frequently than before
- Dig into the log for more details - Juraj
- Narrow down whether some of those arguments are the reason behind this and that is indeed the case: --volume /dev/vfio:/dev/vfio causes the issue.
- AVF interface creation issue:
- Can't create AVF interface on VFs with configured VLAN - happens with latest i40e driver on tx2
- race condition occur
- try mounting a part of /dev/vfio to see if issue can be resolved
- Run six rounds of tests with patch https://gerrit.fd.io/r/c/csit/+/34045 and looks good right now
- Will enable voting right soon after the patch gets merged
- Can't create AVF interface on VFs with configured VLAN - happens with latest i40e driver on tx2
- /usr/bin/vpp[3789]: pci: 0000:91:02.0: open_vfio_iommu_group: open '/dev/vfio/141': Device or resource busy
- Shipment of new advanced server to the FD.io lab
- One link between TG and DUT, multiple link between DUT for testing LACP.
- Two advanced servers are in plan to ship
- Need vexxhost guys confirm with ethernet / power cable type info.
- Lijian has summarized feedbacks from Juraj and raised Jira ticket to DevOps team
- New servers are in the procurement process
- Plan to replace old thunderx1 build servers with more advanced Arm servers
-
- CPU not fully utilized on Arm, need further investigation
-
- Intel NIC firmware upgrade on Arm - not supported
- VPN access request to FD.io Arm servers
- Lijian/Jieqiang has got VPN access now
- mbuf-fast-free dpdk enablement with VPP (DEV_TX_OFFLOAD_MBUF_FAST_FREE)
- VPP Performance Test
-
VPP
- VPP SVE implementation - Lijian
-
- SVE validation on FPGA platform - Conflunce page ready
- Run unit tests from DPDK and VPP bihash on FPGA
- Try Lijian's SVE patch to see any cycle count improvement
- Able to access FPGA platform, investigating adding vpp to buildroot filesystem - Lijian
- Run standalone SVE test cases on FPGA
- Ask for DMC 620 images to run for FPGA
- Enable DMC 620 more close to real system, but performance will drop
- Build a system using VPP memif and pktgen
- SVE validation on FPGA platform - Conflunce page ready
-
- VPP IPv6 Benchmarking and Profiling
- VPP CLI 'ip route add ipv6_addr/mask' outputs wrong IPv6 routes with mask 123-128
- Patch upstreamed https://gerrit.fd.io/r/c/vpp/+/33422
- IPv6 profiling
- Hotspot function - ip6_lookup_node/ip6_rewrite_node
- Will try perfmon & understand two node functions
- VPP CLI configuration and hotspot function are recorded in Confluence page
- Plan to try quad loop unrolling for ip6_lookup_inline function
- VPP CLI 'ip route add ipv6_addr/mask' outputs wrong IPv6 routes with mask 123-128
- VPP Prefetch
- Refactor prefetch implementation in VPP per CPU's actual cache line size - Tianyu
- Two patches unmerged
- l2: fix array-bounds error for prefetch on Arm https://gerrit.fd.io/r/c/vpp/+/33307
- ioam: fix prefetch out bound on Arm https://gerrit.fd.io/r/c/vpp/+/33506
- Two patches unmerged
- Refactor prefetch implementation in VPP per CPU's actual cache line size - Tianyu
- VPP SVE implementation - Lijian
-
- VPP memif - Tianyu
- CNF PoC proposal preparation
-
- Add support for VPP aarch64 docker image build
- Calico use cases exploration on VPP
- Try to use ansible to deploy VPP automatically
-
- CNF PoC proposal preparation
- VPP IPsec on Arm - Govind
- Work on IPsec input - Zach
- IPSec input node/decryption flow cache implemented in a separate patch - Zach
- Upstreamed https://gerrit.fd.io/r/c/vpp/+/32903
- VPP Perfmon plugin enablement on Arm - Zach
- Review perfmon code - Lijian & Govind
- Implemented statistics from PMUv3 - done
- Patch upstream has dependency on kernel patch, waiting for this before upstream to VPP
- https://lore.kernel.org/linux-arm-kernel/[email protected]/
- Wait for v10 kernel patch to fix intermittent large statistic number for events
- Modify the commit message and upstream the perfmon patch - Zach
- Patch has been upstreamed and recieved comments from maintainer and Intel engineer
- Building Intel QAT driver on arm to test IPsec crypt - Zach
- Patch upstream has dependency on kernel patch, waiting for this before upstream to VPP
- VPP memif - Tianyu
10/12/2021
-
Attendees
- Lijian Zhang
- Juraj Linkes
- Tianyu Li
- Govindarajan Mohandoss
- Tina Tsou
-
CSIT
- VPP Performance Test
- https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-3n-tsh/
-
https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-report-iterative-2106-3n-tsh/
- 2106 testing partial finished. 21.01.1 ongoing, should be done sometime next week. - closed
- Leverage current spare TX2 server as 2-node topology performance test-bed.
- Add new IPSec NULL encryption & decryption test cases - Juraj
- Inbound IPsec: reproduced and need to investigate - Juraj
- Flow cache with 1, 10 SPD entries slower, still investigating. Manual test local vs CSIT have different result on 1-10 SPD policies.
- Release testing done.
- IPsec SPD input/output case ongoing
- Steps to enable test case in CSIT https://gerrit.fd.io/r/c/csit/+/31863
- 3n-tsh testbed unreachable, investigating right now - Juraj
- TG firmware is under ugradation
- Server unreachable due to firmware & driver update - resolved - update all done
- Add new IPSec NULL encryption & decryption test cases - Juraj
- VPP Path
- Voting and working fine.
- CentOS-8 jobs have been removed.
- VPP Device
- https://jenkins.fd.io/view/vpp/job/vpp-csit-verify-device-master-ubuntu2004-aarch64-1n-tx2/
- VPP device testing issues may be caused by XL710 i40e fw or kernel module.
-
- Try to reproduce with another set of firmware and etc but issues still exist
-
https://doc.dpdk.org/guides/nics/i40e.html
- DPDK uses i40evf PMD and it is old and scheduled to be deprecated.
- This is fixed in DPDK 21.05 version by making iavf PMD as default.
- Use iavf PMD instead of i40evf on all VPP branches, waiting for the test results - not reproduced so far.
-
- New issue: different error related moving VF from host to container(not involving dpdk/VPP) - just started investigating
- /usr/bin/vpp[3789]: pci: 0000:91:02.0: open_vfio_iommu_group: open '/dev/vfio/141': Device or resource busy
- Resulting in the same failure as before, only happen on AArch64 platform
- Still see /dev/vfio resource busy error after linux kernel upgradation, but less frequently than before
- Dig into the log for more details - Juraj
- Narrow down whether some of those arguments are the reason behind this and that is indeed the case: --volume /dev/vfio:/dev/vfio causes the issue.
- AVF interface creation issue:
- Can't create AVF interface on VFs with configured VLAN - happens with latest i40e driver on tx2
- race condition occur
- try mounting a part of /dev/vfio to see if issue can be resolved
- Talked with Peter, Juraj is working on prototype of mounting part of /dev/vfio
- x86 vpp device job is fine, duo to firmware & driver is old
- arm vpp device servers have drivers updated, vlan striping not allowed, vlan configuration cannot removed from lab view.
- only performance testbeds have NIC drivers updated
- maintainer doesn't want to a option from vpp config
- may need to check x86 have the same issue with the same version driver before reaching intel folks
- Can't create AVF interface on VFs with configured VLAN - happens with latest i40e driver on tx2
- /usr/bin/vpp[3789]: pci: 0000:91:02.0: open_vfio_iommu_group: open '/dev/vfio/141': Device or resource busy
- Shipment of new advanced server to the FD.io lab
- One link between TG and DUT, multiple link between DUT for testing LACP.
- Two advanced servers are in plan to ship
- Need vexxhost guys confirm with ethernet / power cable type info.
- Lijian has summarized feedbacks from Juraj and raised Jira ticket to DevOps team
- New servers are in the procurement process
- Plan to replace old thunderx1 build servers with more advanced Arm servers
-
- CPU not fully utilized on Arm, need further investigation
-
- Intel NIC firmware upgrade on Arm - not supported
- VPN access request to FD.io Arm servers
- Lijian/Jieqiang has got VPN access now
- mbuf-fast-free dpdk enablement with VPP (DEV_TX_OFFLOAD_MBUF_FAST_FREE)
- VPP Performance Test
-
VPP
- VPP SVE implementation - Lijian
-
- SVE validation on FPGA platform - Conflunce page ready
- Run unit tests from DPDK and VPP bihash on FPGA
- Try Lijian's SVE patch to see any cycle count improvement
- Able to access FPGA platform, investigating adding vpp to buildroot filesystem - Lijian
- Run standalone SVE test cases on FPGA
- Ask for DMC 620 images to run for FPGA
- Enable DMC 620 more close to real system, but performance will drop
- SVE validation on FPGA platform - Conflunce page ready
-
- VPP IPv6 Benchmarking and Profiling
- VPP CLI 'ip route add ipv6_addr/mask' outputs wrong IPv6 routes with mask 123-128
- Patch upstreamed https://gerrit.fd.io/r/c/vpp/+/33422
- IPv6 profiling
- Hotspot function - ip6_lookup_node/ip6_rewrite_node
- Will try perfmon & understand two node functions
- VPP CLI configuration and hotspot function are recorded in Confluence page
- VPP CLI 'ip route add ipv6_addr/mask' outputs wrong IPv6 routes with mask 123-128
- VPP Prefetch
- Refactor prefetch implementation in VPP per CPU's actual cache line size - Tianyu
- Two patches unmerged
- l2: fix array-bounds error for prefetch on Arm https://gerrit.fd.io/r/c/vpp/+/33307
- ioam: fix prefetch out bound on Arm https://gerrit.fd.io/r/c/vpp/+/33506
- Two patches unmerged
- Refactor prefetch implementation in VPP per CPU's actual cache line size - Tianyu
- VPP SVE implementation - Lijian
-
- VPP memif - Tianyu
- CNF PoC proposal preparation
-
- Add support for VPP aarch64 docker image build
- Calico use cases exploration on VPP
-
- CNF PoC proposal preparation
- VPP IPsec on Arm - Govind
- Work on IPsec input/output nodes - Govind & Zach - waiting for maintainer review
- SPD prototype change on ipsec_output/encryption node - Govind
- Discussion on the default action on the IPsec inbound interface which does not match
- https://gerrit.fd.io/r/c/vpp/+/31694
- IPSec unit tests done - 'make test' cases implemented & included in patch - Zach
- Testing of flow cache functionality, including hash collisions and stale entry overwrites
- IPSec input node/decryption flow cache implemented in a separate patch - Zach
- Upstreamed https://gerrit.fd.io/r/c/vpp/+/32903
- Internal review for IPsec input node flow cache implementation - Zach & Govind
- Discovered issue with SPD policy counter/statistics on input side, to be fixed in additional standalone patch - merged
- VPP Perfmon plugin enablement on Arm - Zach
- Review perfmon code - Lijian & Govind
- Implemented statistics from PMUv3 - done
- Patch upstream has dependency on kernel patch, waiting for this before upstream to VPP
- https://lore.kernel.org/linux-arm-kernel/[email protected]/
- Wait for v10 kernel patch to fix intermittent large statistic number for events
- Modify the commit message and upstream the perfmon patch - Zach
- Patch has been upstreamed and recieved comments from maintainer and Intel engineer
- Building Intel QAT driver on arm to test IPsec crypt - Zach
- Patch upstream has dependency on kernel patch, waiting for this before upstream to VPP
- VPP memif - Tianyu
09/28/2021
-
Attendees
- Lijian Zhang
- Juraj Linkes
- Tianyu Li
- Jieqiang Wang
-
CSIT
- VPP Performance Test
- https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-3n-tsh/
-
https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-report-iterative-2106-3n-tsh/
- 2106 testing partial finished. 21.01.1 ongoing, should be done sometime next week.
- Leverage current spare TX2 server as 2-node topology performance test-bed.
- Add new IPSec NULL encryption & decryption test cases - Juraj
- Inbound IPsec: reproduced and need to investigate - Juraj
- Flow cache with 1, 10 SPD entries slower, still investigating. Manual test local vs CSIT have different result on 1-10 SPD policies.
- Release testing done.
- IPsec SPD input/output case ongoing
- Steps to enable test case in CSIT https://gerrit.fd.io/r/c/csit/+/31863
- 3n-tsh testbed unreachable, investigating right now - Juraj
- TG firmware is under ugradation
- Add new IPSec NULL encryption & decryption test cases - Juraj
- VPP Path
- Voting and working fine.
- CentOS-8 jobs have been removed.
- VPP Device
- https://jenkins.fd.io/view/vpp/job/vpp-csit-verify-device-master-ubuntu2004-aarch64-1n-tx2/
- VPP device testing issues may be caused by XL710 i40e fw or kernel module.
-
- Try to reproduce with another set of firmware and etc but issues still exist
-
https://doc.dpdk.org/guides/nics/i40e.html
- DPDK uses i40evf PMD and it is old and scheduled to be deprecated.
- This is fixed in DPDK 21.05 version by making iavf PMD as default.
- Use iavf PMD instead of i40evf on all VPP branches, waiting for the test results - not reproduced so far.
-
- New issue: different error related moving VF from host to container(not involving dpdk/VPP) - just started investigating
- /usr/bin/vpp[3789]: pci: 0000:91:02.0: open_vfio_iommu_group: open '/dev/vfio/141': Device or resource busy
- Resulting in the same failure as before, only happen on AArch64 platform
- Still see /dev/vfio resource busy error after linux kernel upgradation, but less frequently than before
- Dig into the log for more details - Juraj
- Narrow down whether some of those arguments are the reason behind this and that is indeed the case: --volume /dev/vfio:/dev/vfio causes the issue.
- AVF interface creation issue:
- Can't create AVF interface on VFs with configured VLAN - happens with latest i40e driver on tx2
- race condition occur
- try mounting a part of /dev/vfio to see if issue can be resolved
- Can't create AVF interface on VFs with configured VLAN - happens with latest i40e driver on tx2
- /usr/bin/vpp[3789]: pci: 0000:91:02.0: open_vfio_iommu_group: open '/dev/vfio/141': Device or resource busy
- Shipment of new advanced server to the FD.io lab
- One link between TG and DUT, multiple link between DUT for testing LACP.
- Two advanced servers are in plan to ship
- Need vexxhost guys confirm with ethernet / power cable type info.
- Lijian has summarized feedbacks from Juraj and raised Jira ticket to DevOps team
- New servers are in the procurement process
- Plan to replace old thunderx1 build servers with more advanced Arm servers
- VPN access request to FD.io Arm servers
- Lijian/Jieqiang has got VPN access now
- mbuf-fast-free dpdk enablement with VPP (DEV_TX_OFFLOAD_MBUF_FAST_FREE)
- VPP Performance Test
-
VPP
- VPP SVE implementation - Lijian
-
- SVE validation on FPGA platform - Conflunce page ready
- Run unit tests from DPDK and VPP bihash on FPGA
- Try Lijian's SVE patch to see any cycle count improvement
- Able to access FPGA platform, investigating adding vpp to buildroot filesystem - Lijian
- Run standalone SVE test cases on FPGA
- Ask for DMC 620 images to run for FPGA
- SVE validation on FPGA platform - Conflunce page ready
-
- VPP IPv6 Benchmarking and Profiling
- VPP CLI 'ip route add ipv6_addr/mask' outputs wrong IPv6 routes with mask 123-128
- Patch upstreamed https://gerrit.fd.io/r/c/vpp/+/33422
- IPv6 profiling
- Hotspot function - ip6_lookup_node/ip6_rewrite_node
- Will try perfmon & understand two node functions
- VPP CLI configuration and hotspot function are recorded in Confluence page
- VPP CLI 'ip route add ipv6_addr/mask' outputs wrong IPv6 routes with mask 123-128
- VPP Prefetch
- Refactor prefetch implementation in VPP per CPU's actual cache line size - Tianyu
- Two patches unmerged
- l2: fix array-bounds error for prefetch on Arm https://gerrit.fd.io/r/c/vpp/+/33307
- ioam: fix prefetch out bound on Arm https://gerrit.fd.io/r/c/vpp/+/33506
- Two patches unmerged
- Refactor prefetch implementation in VPP per CPU's actual cache line size - Tianyu
- VPP SVE implementation - Lijian
-
- VPP memif - Tianyu
- CNF PoC proposal preparation
-
- Add support for VPP aarch64 docker image build
- Calico use cases exploration on VPP
-
- CNF PoC proposal preparation
- VPP IPsec on Arm - Govind
- Work on IPsec input/output nodes - Govind & Zach - waiting for maintainer review
- SPD prototype change on ipsec_output/encryption node - Govind
- Discussion on the default action on the IPsec inbound interface which does not match
- https://gerrit.fd.io/r/c/vpp/+/31694
- IPSec unit tests done - 'make test' cases implemented & included in patch - Zach
- Testing of flow cache functionality, including hash collisions and stale entry overwrites
- IPSec input node/decryption flow cache implemented in a separate patch - Zach
- Upstreamed https://gerrit.fd.io/r/c/vpp/+/32903
- Internal review for IPsec input node flow cache implementation - Zach & Govind
- Discovered issue with SPD policy counter/statistics on input side, to be fixed in additional standalone patch - merged
- VPP Perfmon plugin enablement on Arm - Zach
- Review perfmon code - Lijian & Govind
- Implemented statistics from PMUv3 - done
- Patch upstream has dependency on kernel patch, waiting for this before upstream to VPP
- https://lore.kernel.org/linux-arm-kernel/[email protected]/
- Wait for v10 kernel patch to fix intermittent large statistic number for events
- Modify the commit message and upstream the perfmon patch - Zach
- Patch has been upstreamed and recieved comments from maintainer and Intel engineer
- Building Intel QAT driver on arm to test IPsec crypt - Zach
- Patch upstream has dependency on kernel patch, waiting for this before upstream to VPP
- VPP memif - Tianyu
09/14/2021
-
Attendees
- Lijian Zhang
- Govindarajan Mohandoss
- Juraj Linkes
- Tianyu Li
- Jieqiang Wang
- Tina Tsou
-
CSIT
- VPP Performance Test
- https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-3n-tsh/
-
https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-report-iterative-2106-3n-tsh/
- 2106 testing partial finished. 21.01.1 ongoing, should be done sometime next week.
- Leverage current spare TX2 server as 2-node topology performance test-bed.
- Add new IPSec NULL encryption & decryption test cases - Juraj
- New IPSec SPD test cases will not have NULL encrypt/decrypt config.
- Try VPP IPsec test cases with a fix on outbound interface - Govind & Juraj
- Outbound IPsec: 10 entries still slower with latest change, related to traffic pattern
- Outbound IPsec finished.
- Waiting for new version of patchset to verify test cases
- Will try the fix on inbound IPsec tests when the Jenkins is back to normal - Juraj
- Inbound IPsec: reproduced and need to investigate - Juraj
- Learn more about RFC and need time to understand more - on hold - waiting Neale's response
- Inbound IPsec: reproduced and need to investigate - Juraj
- Try VPP IPsec test cases with a fix on outbound interface - Govind & Juraj
- Flow cache with 1, 10 SPD entries slower, still investigating. Manual test local vs CSIT have different result on 1-10 SPD policies.
- New IPSec SPD test cases will not have NULL encrypt/decrypt config.
- Release testing done.
- A few more jobs run for release 21.06 and will be finished soon
- Comparison between 21.06 and 21.01.1 is ongoing.
- IPsec SPD input/output case ongoing
- Steps to enable test case in CSIT https://gerrit.fd.io/r/c/csit/+/31863
- Add new IPSec NULL encryption & decryption test cases - Juraj
- VPP Path
- Voting and working fine.
- CentOS-8 jobs have been removed.
- VPP Device
- https://jenkins.fd.io/view/vpp/job/vpp-csit-verify-device-master-ubuntu2004-aarch64-1n-tx2/
- VPP device testing issues may be caused by XL710 i40e fw or kernel module.
-
- Try to reproduce with another set of firmware and etc but issues still exist
-
https://doc.dpdk.org/guides/nics/i40e.html
- DPDK uses i40evf PMD and it is old and scheduled to be deprecated.
- This is fixed in DPDK 21.05 version by making iavf PMD as default.
- Use iavf PMD instead of i40evf on all VPP branches, waiting for the test results - not reproduced so far.
-
- New issue: different error related moving VF from host to container(not involving dpdk/VPP) - just started investigating
- /usr/bin/vpp[3789]: pci: 0000:91:02.0: open_vfio_iommu_group: open '/dev/vfio/141': Device or resource busy
- Resulting in the same failure as before, only happen on AArch64 platform
- Still see /dev/vfio resource busy error after linux kernel upgradation, but less frequently than before
- Dig into the log for more details - juraj
- Narrow down whether some of those arguments are the reason behind this and that is indeed the case: --volume /dev/vfio:/dev/vfio causes the issue.
- AVF interface creation issue:
- Can't create AVF interface on VFs with configured VLAN - happens with latest i40e driver on tx2
- /usr/bin/vpp[3789]: pci: 0000:91:02.0: open_vfio_iommu_group: open '/dev/vfio/141': Device or resource busy
- Shipment of new advanced server to the FD.io lab
- One link between TG and DUT, multiple link between DUT for testing LACP.
- Two advanced servers are in plan to ship
- Need vexxhost guys confirm with ethernet / power cable type info.
- Lijian has summarized feedbacks from Juraj and raised Jira ticket to DevOps team
- VPN access request to FD.io Arm servers
- Lijian/Jieqiang has got VPN access now
- mbuf-fast-free dpdk enablement with VPP (DEV_TX_OFFLOAD_MBUF_FAST_FREE)
- VPP Performance Test
-
VPP
- VPP SVE implementation - Lijian
- Vector length specific patch is ready
- SVE patch ready and upstreamed, under review - Lijian
- SVE patch sent to Nitin, Nitin will review the patch when back to work.
- Rebased SVE patch per Nitin's request, waiting for Nitin's feedback on running these patches
- Juan met and fixing some issue running SVE in qemu VM
- SVE validation on FPGA platform - Conflunce page ready
- Run unit tests from DPDK and VPP bihash on FPGA
- Try Lijian's SVE patch to see any cycle count improvement
- Able to access FPGA platform, investigating adding vpp to buildroot filesystem - Lijian
- Run standalone SVE test cases on FPGA
- Ask for DMC 620 images to run for FPGA
- SVE patch sent to Nitin, Nitin will review the patch when back to work.
- VPP IPv6 Benchmarking and Profiling
- VPP CLI 'ip route add ipv6_addr/mask' outputs wrong IPv6 routes with mask 123-128
- Patch upstreamed https://gerrit.fd.io/r/c/vpp/+/33422
- IPv6 profiling
- Hotspot function - ip6_lookup_node/ip6_rewrite_node
- Will try perfmon & understand two node functions
- VPP CLI configuration and hotspot function are recorded in Confluence page
- VPP CLI 'ip route add ipv6_addr/mask' outputs wrong IPv6 routes with mask 123-128
- VPP mbuf-fast-free tx offload
- Vector path shows performance improvement, still need to investigate scalar path
- Performance improvement for IPv4 routing test cases using vector path
- Try IPv4 multicast testing to verify the scenario when refcnt > 1
- GDB shows that mbufs are copied instead of reference from src port to all dst ports
- Direct/Indirect mbuf for VPP multicast testing
- Try IPv4 multicasting & L2 flood testing which works fine
- ip4-replicate node in IPv4 multicasting/l2-flood node in L2 flood testing
- show mbuf is copied so that ref_cnt will always be one
- dpdk 21.08 have the patches, need to verify on vpp
- show mbuf is copied so that ref_cnt will always be one
- VPP Prefetch
- Refactor prefetch implementation in VPP per CPU's actual cache line size - Tianyu & Jieqiang
- Damjan has merged 2 patches, waiting for the last patch, for generic 128B cacheline size.
- For 64B cacheline size native build on Arm, may need to change code.
- Current VPP does not support 64B cacheline size compilation for Arm images.
- Patch split into 3 components
- acl: fix prefetch out of struct bound on Arm https://gerrit.fd.io/r/c/vpp/+/33597 (Merged)
- dpdk: fix prefetch assert on Arm https://gerrit.fd.io/r/c/vpp/+/33598 (Merged)
- session: fix prefetch out of struct bound on Arm https://gerrit.fd.io/r/c/vpp/+/33599 (Merged)
- Refactor prefetch implementation in VPP per CPU's actual cache line size - Tianyu & Jieqiang
- VPP memif - Tianyu
- CNF PoC proposal preparation
-
- Add support for VPP aarch64 docker image build
- Calico use cases exploration on VPP
-
- CNF PoC proposal preparation
- VPP IPsec on Arm - Govind
- Work on IPsec input/output nodes - Govind & Zach - waiting for maintainer review
- SPD prototype change on ipsec_output/encryption node - Govind
- Discussion on the default action on the IPsec inbound interface which does not match
- https://gerrit.fd.io/r/c/vpp/+/31694
- IPSec unit tests done - 'make test' cases implemented & included in patch - Zach
- Testing of flow cache functionality, including hash collisions and stale entry overwrites
- IPSec input node/decryption flow cache implemented in a separate patch - Zach
- Upstreamed https://gerrit.fd.io/r/c/vpp/+/32903
- Internal review for IPsec input node flow cache implementation - Zach & Govind
- Discovered issue with SPD policy counter/statistics on input side, to be fixed in additional standalone patch - merged
- VPP Perfmon plugin enablement on Arm - Zach
- Review perfmon code - Lijian & Govind
- Implemented statistics from PMUv3 - done
- Patch upstream has dependency on kernel patch, waiting for this before upstream to VPP
- https://lore.kernel.org/linux-arm-kernel/[email protected]/
- Wait for v10 kernel patch to fix intermittent large statistic number for events
- Modify the commit message and upstream the perfmon patch - Zach
- Patch has been upstreamed and recieved comments from maintainer and Intel engineer
- Building Intel QAT driver on arm to test IPsec crypt - Zach
- Patch upstream has dependency on kernel patch, waiting for this before upstream to VPP
- VPP SVE implementation - Lijian
09/07/2021
-
Attendees
- Lijian Zhang
- Govindarajan Mohandoss
- Tianyu Li
- Jieqiang Wang
- Tina Tsou
-
CSIT
- VPP Performance Test
- https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-3n-tsh/
-
https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-report-iterative-2106-3n-tsh/
- 2106 testing partial finished. 21.01.1 ongoing, should be done sometime next week.
- Leverage current spare TX2 server as 2-node topology performance test-bed.
- Add new IPSec NULL encryption & decryption test cases - Juraj
- New IPSec SPD test cases will not have NULL encrypt/decrypt config.
- Try VPP IPsec test cases with a fix on outbound interface - Govind & Juraj
- Outbound IPsec: 10 entries still slower with latest change, related to traffic pattern
- Outbound IPsec finished.
- Waiting for new version of patchset to verify test cases
- Will try the fix on inbound IPsec tests when the Jenkins is back to normal - Juraj
- Inbound IPsec: reproduced and need to investigate - Juraj
- Learn more about RFC and need time to understand more - on hold - waiting Neale's response
- Inbound IPsec: reproduced and need to investigate - Juraj
- Try VPP IPsec test cases with a fix on outbound interface - Govind & Juraj
- Flow cache with 1, 10 SPD entries slower, still investigating. Manual test local vs CSIT have different result on 1-10 SPD policies.
- New IPSec SPD test cases will not have NULL encrypt/decrypt config.
- Release testing done.
- A few more jobs run for release 21.06 and will be finished soon
- Comparison between 21.06 and 21.01.1 is ongoing.
- IPsec SPD input/output case ongoing
- Steps to enable test case in CSIT https://gerrit.fd.io/r/c/csit/+/31863
- Add new IPSec NULL encryption & decryption test cases - Juraj
- VPP Path
- Voting and working fine.
- CentOS-8 jobs have been removed.
- VPP Device
- https://jenkins.fd.io/view/vpp/job/vpp-csit-verify-device-master-ubuntu2004-aarch64-1n-tx2/
- VPP device testing issues may be caused by XL710 i40e fw or kernel module.
-
- Try to reproduce with another set of firmware and etc but issues still exist
-
https://doc.dpdk.org/guides/nics/i40e.html
- DPDK uses i40evf PMD and it is old and scheduled to be deprecated.
- This is fixed in DPDK 21.05 version by making iavf PMD as default.
- Use iavf PMD instead of i40evf on all VPP branches, waiting for the test results - not reproduced so far.
-
- New issue: different error related moving VF from host to container(not involving dpdk/VPP) - just started investigating
- /usr/bin/vpp[3789]: pci: 0000:91:02.0: open_vfio_iommu_group: open '/dev/vfio/141': Device or resource busy
- Resulting in the same failure as before, only happen on AArch64 platform
- Still see /dev/vfio resource busy error after linux kernel upgradation, but less frequently than before
- Dig into the log for more details - juraj
- /usr/bin/vpp[3789]: pci: 0000:91:02.0: open_vfio_iommu_group: open '/dev/vfio/141': Device or resource busy
- Shipment of new advanced server to the FD.io lab
- One link between TG and DUT, multiple link between DUT for testing LACP.
- Two advanced servers are in plan to ship
- Need vexxhost guys confirm with ethernet / power cable type info.
- Lijian has summarized feedbacks from Juraj and raised Jira ticket to DevOps team
- VPN access request to FD.io Arm servers
- Lijian/Jieqiang has got VPN access now
- mbuf-fast-free dpdk enablement with VPP (DEV_TX_OFFLOAD_MBUF_FAST_FREE)
- VPP Performance Test
-
VPP
- VPP SVE implementation - Lijian
- Vector length specific patch is ready
- SVE patch ready and upstreamed, under review - Lijian
- SVE patch sent to Nitin, Nitin will review the patch when back to work.
- Rebased SVE patch per Nitin's request, waiting for Nitin's feedback on running these patches
- Juan met and fixing some issue running SVE in qemu VM
- SVE validation on FPGA platform - Conflunce page ready
- Run unit tests from DPDK and VPP bihash on FPGA
- Try Lijian's SVE patch to see any cycle count improvement
- Able to access FPGA platform, investigating adding vpp to buildroot filesystem - Lijian
- Run standalone SVE test cases on FPGA
- Ask for DMC 620 images to run for FPGA
- SVE patch sent to Nitin, Nitin will review the patch when back to work.
- VPP IPv6 Benchmarking and Profiling
- VPP CLI 'ip route add ipv6_addr/mask' outputs wrong IPv6 routes with mask 123-128
- Patch upstreamed https://gerrit.fd.io/r/c/vpp/+/33422
- IPv6 profiling
- Hotspot function - ip6_lookup_node/ip6_rewrite_node
- Will try perfmon & understand two node functions
- VPP CLI configuration and hotspot function are recorded in Confluence page
- VPP CLI 'ip route add ipv6_addr/mask' outputs wrong IPv6 routes with mask 123-128
- VPP mbuf-fast-free tx offload
- Vector path shows performance improvement, still need to investigate scalar path
- Performance improvement for IPv4 routing test cases using vector path
- Try IPv4 multicast testing to verify the scenario when refcnt > 1
- GDB shows that mbufs are copied instead of reference from src port to all dst ports
- Direct/Indirect mbuf for VPP multicast testing
- Try IPv4 multicasting & L2 flood testing which works fine
- ip4-replicate node in IPv4 multicasting/l2-flood node in L2 flood testing
- show mbuf is copied so that ref_cnt will always be one
- VPP Prefetch
- Refactor prefetch implementation in VPP per CPU's actual cache line size - Tianyu & Jieqiang
- Damjan has merged 2 patches, waiting for the last patch, for generic 128B cacheline size.
- For 64B cacheline size native build on Arm, may need to change code.
- Current VPP does not support 64B cacheline size compilation for Arm images.
- Patch split into 3 components
- acl: fix prefetch out of struct bound on Arm https://gerrit.fd.io/r/c/vpp/+/33597 (Merged)
- dpdk: fix prefetch assert on Arm https://gerrit.fd.io/r/c/vpp/+/33598 (Under review)
- session: fix prefetch out of struct bound on Arm https://gerrit.fd.io/r/c/vpp/+/33599 (Merged)
- Refactor prefetch implementation in VPP per CPU's actual cache line size - Tianyu & Jieqiang
- VPP memif - Tianyu
- CNF PoC proposal preparation
-
- Add support for VPP aarch64 docker image build
- Calico use cases exploration on VPP
-
- CNF PoC proposal preparation
- VPP IPsec on Arm - Govind
- Work on IPsec input/output nodes - Govind & Zach - waiting for maintainer review
- SPD prototype change on ipsec_output/encryption node - Govind
- Discussion on the default action on the IPsec inbound interface which does not match
- https://gerrit.fd.io/r/c/vpp/+/31694
- IPSec unit tests done - 'make test' cases implemented & included in patch - Zach
- Testing of flow cache functionality, including hash collisions and stale entry overwrites
- IPSec input node/decryption flow cache implemented in a separate patch - Zach
- Upstreamed https://gerrit.fd.io/r/c/vpp/+/32903
- Internal review for IPsec input node flow cache implementation - Zach & Govind
- Discovered issue with SPD policy counter/statistics on input side, to be fixed in additional standalone patch - merged
- VPP Perfmon plugin enablement on Arm - Zach
- Review perfmon code - Lijian & Govind
- Implemented statistics from PMUv3 - done
- Patch upstream has dependency on kernel patch, waiting for this before upstream to VPP
- https://lore.kernel.org/linux-arm-kernel/[email protected]/
- Wait for v10 kernel patch to fix intermittent large statistic number for events
- Modify the commit message and upstream the perfmon patch - Zach
- Patch has been upstreamed and recieved comments from one Intel engineer
- Building Intel QAT driver on arm to test IPsec crypt - Zach
- Patch upstream has dependency on kernel patch, waiting for this before upstream to VPP
- VPP SVE implementation - Lijian
08/31/2021
-
Attendees
- Lijian Zhang
- Govindarajan Mohandoss
- Tianyu Li
- Jieqiang Wang
- Zachary Leaf
- Tina Tsou
-
CSIT
- VPP Performance Test
- https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-3n-tsh/
-
https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-report-iterative-2106-3n-tsh/
- 2106 testing partial finished. 21.01.1 ongoing, should be done sometime next week.
- Leverage current spare TX2 server as 2-node topology performance test-bed.
- Add new IPSec NULL encryption & decryption test cases - Juraj
- New IPSec SPD test cases will not have NULL encrypt/decrypt config.
- Try VPP IPsec test cases with a fix on outbound interface - Govind & Juraj
- Outbound IPsec: 10 entries still slower with latest change, related to traffic pattern
- Outbound IPsec finished.
- Waiting for new version of patchset to verify test cases
- Will try the fix on inbound IPsec tests when the Jenkins is back to normal - Juraj
- Inbound IPsec: reproduced and need to investigate - Juraj
- Learn more about RFC and need time to understand more - on hold - waiting Neale's response
- Inbound IPsec: reproduced and need to investigate - Juraj
- Try VPP IPsec test cases with a fix on outbound interface - Govind & Juraj
- Flow cache with 1, 10 SPD entries slower, still investigating. Manual test local vs CSIT have different result on 1-10 SPD policies.
- New IPSec SPD test cases will not have NULL encrypt/decrypt config.
- Release testing done.
- A few more jobs run for release 21.06 and will be finished soon
- Comparison between 21.06 and 21.01.1 is ongoing.
- IPsec SPD input/output case ongoing
- Steps to enable test case in CSIT https://gerrit.fd.io/r/c/csit/+/31863
- Add new IPSec NULL encryption & decryption test cases - Juraj
- VPP Path
- Voting and working fine.
- CentOS-8 jobs have been removed.
- VPP Device
- https://jenkins.fd.io/view/vpp/job/vpp-csit-verify-device-master-ubuntu2004-aarch64-1n-tx2/
- VPP device testing issues may be caused by XL710 i40e fw or kernel module.
-
- Try to reproduce with another set of firmware and etc but issues still exist
-
https://doc.dpdk.org/guides/nics/i40e.html
- DPDK uses i40evf PMD and it is old and scheduled to be deprecated.
- This is fixed in DPDK 21.05 version by making iavf PMD as default.
- Use iavf PMD instead of i40evf on all VPP branches, waiting for the test results - not reproduced so far.
-
- New issue: different error related moving VF from host to container(not involving dpdk/VPP) - just started investigating
- /usr/bin/vpp[3789]: pci: 0000:91:02.0: open_vfio_iommu_group: open '/dev/vfio/141': Device or resource busy
- Resulting in the same failure as before, only happen on AArch64 platform
- /sys/module/vfio/parameters/enable_unsafe_noiommu_mode affect the behavior in some way
- config option set to N, /dev/vfio device or resource busy error
- config option set to Y & iommu_passthrough = 1, IP packets Rx timeout
- The longer the server runs, more test cases fails
- Next to do
- Need to figure out what arm-smmu-v3.4.auto: event 0x10 means
- Juraj modified script to reproduce the issue - Lijian will try it locally
- Also seen in Intel QAT card from Zach
- Will try to reproduce this issue on local thx2 with 20.04 distro - Lijian
- Lijian have slight different firmware version, driver version
- Tried Mellanox card (rdma driver) multiple times - not see the same issue (on XL710 NIC) happens - Juraj
- Lijian can use Juraj's script to reproduce the issue on local tx2 server
- Reducing the numa buffer allocation size resolves this issue
- Observed from the error log of numa buffer allocation
- Need to figure out what arm-smmu-v3.4.auto: event 0x10 means
- /usr/bin/vpp[3789]: pci: 0000:91:02.0: open_vfio_iommu_group: open '/dev/vfio/141': Device or resource busy
- Shipment of new advanced server to the FD.io lab
- One link between TG and DUT, multiple link between DUT for testing LACP.
- Two advanced servers are in plan to ship
- Need vexxhost guys confirm with ethernet / power cable type info.
- VPN access request to FD.io Arm servers
- Lijian/Jieqiang has got VPN access now
- mbuf-fast-free dpdk enablement with VPP (DEV_TX_OFFLOAD_MBUF_FAST_FREE)
- VPP Performance Test
-
VPP
- VPP SVE implementation - Lijian
- Vector length specific patch is ready
- SVE patch ready and upstreamed, under review - Lijian
- SVE patch sent to Nitin, Nitin will review the patch when back to work.
- Rebased SVE patch per Nitin's request, waiting for Nitin's feedback on running these patches
- Juan met and fixing some issue running SVE in qemu VM
- SVE validation on FPGA platform - Conflunce page ready
- Run unit tests from DPDK and VPP bihash on FPGA
- Try Lijian's SVE patch to see any cycle count improvement
- Able to access FPGA platform, investigating adding vpp to buildroot filesystem - Lijian
- Run standalone SVE test cases on FPGA
- Ask for DMC 620 images to run for FPGA
- SVE patch sent to Nitin, Nitin will review the patch when back to work.
- VPP IPv6 Benchmarking and Profiling
- VPP CLI 'ip route add ipv6_addr/mask' outputs wrong IPv6 routes with mask 123-128
- Patch upstreamed https://gerrit.fd.io/r/c/vpp/+/33422
- IPv6 profiling
- Hotspot function - ip6_lookup_node/ip6_rewrite_node
- Will try perfmon & understand two node functions
- VPP CLI configuration and hotspot function are recorded in Confluence page
- VPP CLI 'ip route add ipv6_addr/mask' outputs wrong IPv6 routes with mask 123-128
- VPP mbuf-fast-free tx offload
- Vector path shows performance improvement, still need to investigate scalar path
- Performance improvement for IPv4 routing test cases using vector path
- Try IPv4 multicast testing to verify the scenario when refcnt > 1
- GDB shows that mbufs are copied instead of reference from src port to all dst ports
- Direct/Indirect mbuf for VPP multicast testing
- Try IPv4 multicasting & L2 flood testing which works fine
- ip4-replicate node in IPv4 multicasting/l2-flood node in L2 flood testing
- show mbuf is copied so that ref_cnt will always be one
- VPP Prefetch
- Refactor prefetch implementation in VPP per CPU's actual cache line size - Tianyu & Jieqiang
- Damjan has merged 2 patches, waiting for the last patch, for generic 128B cacheline size.
- For 64B cacheline size native build on Arm, may need to change code.
- Current VPP does not support 64B cacheline size compilation for Arm images.
- Patch split into 3 components
- acl: fix prefetch out of struct bound on Arm https://gerrit.fd.io/r/c/vpp/+/33597 (Merged)
- dpdk: fix prefetch assert on Arm https://gerrit.fd.io/r/c/vpp/+/33598 (Under review)
- session: fix prefetch out of struct bound on Arm https://gerrit.fd.io/r/c/vpp/+/33599 (Merged)
- Refactor prefetch implementation in VPP per CPU's actual cache line size - Tianyu & Jieqiang
- VPP memif - Tianyu
- CNF PoC proposal preparation
-
- Add support for VPP aarch64 docker image build
- Calico use cases exploration on VPP
-
- CNF PoC proposal preparation
- VPP IPsec on Arm - Govind
- Work on IPsec input/output nodes - Govind & Zach - waiting for maintainer review
- SPD prototype change on ipsec_output/encryption node - Govind
- Discussion on the default action on the IPsec inbound interface which does not match
- https://gerrit.fd.io/r/c/vpp/+/31694
- IPSec unit tests done - 'make test' cases implemented & included in patch - Zach
- Testing of flow cache functionality, including hash collisions and stale entry overwrites
- IPSec input node/decryption flow cache implemented in a separate patch - Zach
- Upstreamed https://gerrit.fd.io/r/c/vpp/+/32903
- Internal review for IPsec input node flow cache implementation - Zach & Govind
- Discovered issue with SPD policy counter/statistics on input side, to be fixed in additional standalone patch - merged
- VPP Perfmon plugin enablement on Arm - Zach
- Review perfmon code - Lijian & Govind
- Implemented statistics from PMUv3 - done
- Patch upstream has dependency on kernel patch, waiting for this before upstream to VPP
- https://lore.kernel.org/linux-arm-kernel/[email protected]/
- Wait for v10 kernel patch to fix intermittent large statistic number for events
- Modify the commit message and upstream the perfmon patch - Zach
- Building Intel QAT driver on arm to test IPsec crypt - Zach
- Patch upstream has dependency on kernel patch, waiting for this before upstream to VPP
- VPP SVE implementation - Lijian
08/24/2021
-
Attendees
- Lijian Zhang
- Govindarajan Mohandoss
- Juraj Linkes
- Tianyu Li
- Jieqiang Wang
- Zachary Leaf
- Tina Tsou
-
CSIT
- VPP Performance Test
- https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-3n-tsh/
-
https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-report-iterative-2106-3n-tsh/
- 2106 testing partial finished. 21.01.1 ongoing, should be done sometime next week.
- Leverage current spare TX2 server as 2-node topology performance test-bed.
- Add new IPSec NULL encryption & decryption test cases - Juraj
- New IPSec SPD test cases will not have NULL encrypt/decrypt config.
- Try VPP IPsec test cases with a fix on outbound interface - Govind & Juraj
- Outbound IPsec: 10 entries still slower with latest change, related to traffic pattern
- Outbound IPsec finished.
- Waiting for new version of patchset to verify test cases
- Will try the fix on inbound IPsec tests when the Jenkins is back to normal - Juraj
- Inbound IPsec: reproduced and need to investigate - Juraj
- Learn more about RFC and need time to understand more - on hold - waiting Neale's response
- Inbound IPsec: reproduced and need to investigate - Juraj
- Try VPP IPsec test cases with a fix on outbound interface - Govind & Juraj
- Flow cache with 1, 10 SPD entries slower, still investigating. Manual test local vs CSIT have different result on 1-10 SPD policies.
- New IPSec SPD test cases will not have NULL encrypt/decrypt config.
- Release testing done.
- A few more jobs run for release 21.06 and will be finished soon
- Comparison between 21.06 and 21.01.1 is ongoing.
- IPsec SPD input/output case ongoing
- Steps to enable test case in CSIT https://gerrit.fd.io/r/c/csit/+/31863
- Add new IPSec NULL encryption & decryption test cases - Juraj
- VPP Path
- Voting and working fine.
- CentOS-8 jobs have been removed.
- VPP Device
- https://jenkins.fd.io/view/vpp/job/vpp-csit-verify-device-master-ubuntu2004-aarch64-1n-tx2/
- VPP device testing issues may be caused by XL710 i40e fw or kernel module.
-
- Try to reproduce with another set of firmware and etc but issues still exist
-
https://doc.dpdk.org/guides/nics/i40e.html
- DPDK uses i40evf PMD and it is old and scheduled to be deprecated.
- This is fixed in DPDK 21.05 version by making iavf PMD as default.
- Use iavf PMD instead of i40evf on all VPP branches, waiting for the test results - not reproduced so far.
-
- New issue: different error related moving VF from host to container(not involving dpdk/VPP) - just started investigating
- /usr/bin/vpp[3789]: pci: 0000:91:02.0: open_vfio_iommu_group: open '/dev/vfio/141': Device or resource busy
- Resulting in the same failure as before, only happen on AArch64 platform
- /sys/module/vfio/parameters/enable_unsafe_noiommu_mode affect the behavior in some way
- config option set to N, /dev/vfio device or resource busy error
- config option set to Y & iommu_passthrough = 1, IP packets Rx timeout
- The longer the server runs, more test cases fails
- Next to do
- Need to figure out what arm-smmu-v3.4.auto: event 0x10 means
- Juraj modified script to reproduce the issue - Lijian will try it locally
- Also seen in Intel QAT card from Zach
- Will try to reproduce this issue on local thx2 with 20.04 distro - Lijian
- Lijian have slight different firmware version, driver version
- Tried Mellanox card (rdma driver) multiple times - not see the same issue (on XL710 NIC) happens - Juraj
- Lijian can use Juraj's script to reproduce the issue on local tx2 server
- Reducing the numa buffer allocation size resolves this issue
- Observed from the error log of numa buffer allocation
- Need to figure out what arm-smmu-v3.4.auto: event 0x10 means
- /usr/bin/vpp[3789]: pci: 0000:91:02.0: open_vfio_iommu_group: open '/dev/vfio/141': Device or resource busy
- Shipment of new advanced server to the FD.io lab
- One link between TG and DUT, multiple link between DUT for testing LACP.
- Two advanced servers are in plan to ship
- Need vexxhost guys confirm with ethernet / power cable type info.
- VPN access request to FD.io Arm servers
- Lijian/Jieqiang has got VPN access now
- mbuf-fast-free dpdk enablement with VPP (DEV_TX_OFFLOAD_MBUF_FAST_FREE)
- VPP Performance Test
-
VPP
- VPP SVE implementation - Lijian
- Vector length specific patch is ready
- SVE patch ready and upstreamed, under review - Lijian
- SVE patch sent to Nitin, Nitin will review the patch when back to work.
- Rebased SVE patch per Nitin's request, waiting for Nitin's feedback on running these patches
- Juan met and fixing some issue running SVE in qemu VM
- SVE validation on FPGA platform - Conflunce page ready
- Run unit tests from DPDK and VPP bihash on FPGA
- Try Lijian's SVE patch to see any cycle count improvement
- Able to access FPGA platform, investigating adding vpp to buildroot filesystem - Lijian
- SVE patch sent to Nitin, Nitin will review the patch when back to work.
- VPP IPv6 Benchmarking and Profiling
- VPP CLI 'ip route add ipv6_addr/mask' outputs wrong IPv6 routes with mask 123-128
- Patch upstreamed https://gerrit.fd.io/r/c/vpp/+/33422
- IPv6 profiling
- Hotspot function - ip6_lookup_node/ip6_rewrite_node
- Will try perfmon & understand two node functions
- VPP CLI 'ip route add ipv6_addr/mask' outputs wrong IPv6 routes with mask 123-128
- VPP mbuf-fast-free tx offload
- Vector path shows performance improvement, still need to investigate scalar path
- Performance improvement for IPv4 routing test cases using vector path
- Try IPv4 multicast testing to verify the scenario when refcnt > 1
- GDB shows that mbufs are copied instead of reference from src port to all dst ports
- Will try L2 flood test case & understand VPP/multicast code
- Direct/Indirect mbuf for VPP multicast testing
- VPP Prefetch
- Refactor prefetch implementation in VPP per CPU's actual cache line size - Tianyu & Jieqiang
- Damjan has merged 2 patches, waiting for the last patch, for generic 128B cacheline size.
- For 64B cacheline size native build on Arm, may need to change code.
- Current VPP does not support 64B cacheline size compilation for Arm images.
- Issues about prefetch on current VPP code base
- Issue 1 support 128B/64B cache-line size in Arm image
- Issue 2 prefetch 'overflow' for native build
- Refactor prefetch implementation in VPP per CPU's actual cache line size - Tianyu & Jieqiang
- VPP memif - Tianyu
- CNF PoC proposal preparation
-
- Add support for VPP aarch64 docker image build
- Calico use cases exploration on VPP
-
- CNF PoC proposal preparation
- VPP IPsec on Arm - Govind
- Work on IPsec input/output nodes - Govind & Zach - waiting for maintainer review
- SPD prototype change on ipsec_output/encryption node - Govind
- Discussion on the default action on the IPsec inbound interface which does not match
- https://gerrit.fd.io/r/c/vpp/+/31694
- IPSec unit tests done - 'make test' cases implemented & included in patch - Zach
- Testing of flow cache functionality, including hash collisions and stale entry overwrites
- IPSec input node/decryption flow cache implemented in a separate patch - Zach
- Upstreamed https://gerrit.fd.io/r/c/vpp/+/32903
- Internal review for IPsec input node flow cache implementation - Zach & Govind
- Discovered issue with SPD policy counter/statistics on input side, to be fixed in additional standalone patch - merged
- VPP Perfmon plugin enablement on Arm - Zach
- Review perfmon code - Lijian & Govind
- Implemented statistics from PMUv3 - done
- Patch upstream has dependency on kernel patch, waiting for this before upstream to VPP
- https://lore.kernel.org/linux-arm-kernel/[email protected]/
- Modify the commit message and upstream the perfmon patch - Zach
- Building Intel QAT driver on arm to test IPsec crypt - Zach
- Patch upstream has dependency on kernel patch, waiting for this before upstream to VPP
- VPP SVE implementation - Lijian
08/17/2021
-
Attendees
- Lijian Zhang
- Govindarajan Mohandoss
- Juraj Linkes
- Tianyu Li
- Jieqiang Wang
- Zachary Leaf
- Tina Tsou
-
CSIT
- VPP Performance Test
- https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-3n-tsh/
-
https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-report-iterative-2106-3n-tsh/
- 2106 testing partial finished. 21.01.1 ongoing, should be done sometime next week.
- Leverage current spare TX2 server as 2-node topology performance test-bed.
- Add new IPSec NULL encryption & decryption test cases - Juraj
- New IPSec SPD test cases will not have NULL encrypt/decrypt config.
- Try VPP IPsec test cases with a fix on outbound interface - Govind & Juraj
- Outbound IPsec: 10 entries still slower with latest change, related to traffic pattern
- Outbound IPsec finished.
- Waiting for new version of patchset to verify test cases
- Will try the fix on inbound IPsec tests when the Jenkins is back to normal - Juraj
- Inbound IPsec: reproduced and need to investigate - Juraj
- Learn more about RFC and need time to understand more
- Inbound IPsec: reproduced and need to investigate - Juraj
- Try VPP IPsec test cases with a fix on outbound interface - Govind & Juraj
- Flow cache with 1, 10 SPD entries slower, still investigating. Manual test local vs CSIT have different result on 1-10 SPD policies.
- New IPSec SPD test cases will not have NULL encrypt/decrypt config.
- Release testing ongoing
- A few more jobs run for release 21.06 and will be finished soon
- Comparison between 21.06 and 21.01.1 is ongoing.
- IPsec SPD input/output case ongoing
- Steps to enable test case in CSIT https://gerrit.fd.io/r/c/csit/+/31863
- Add new IPSec NULL encryption & decryption test cases - Juraj
- VPP Path
- Voting and working fine.
- CentOS-8 jobs have been removed.
- VPP Device
- https://jenkins.fd.io/view/vpp/job/vpp-csit-verify-device-master-ubuntu2004-aarch64-1n-tx2/
- VPP device testing issues may be caused by XL710 i40e fw or kernel module.
-
- Try to reproduce with another set of firmware and etc but issues still exist
-
https://doc.dpdk.org/guides/nics/i40e.html
- DPDK uses i40evf PMD and it is old and scheduled to be deprecated.
- This is fixed in DPDK 21.05 version by making iavf PMD as default.
- Use iavf PMD instead of i40evf on all VPP branches, waiting for the test results - not reproduced so far.
-
- New issue: different error related moving VF from host to container(not involving dpdk/VPP) - just started investigating
- /usr/bin/vpp[3789]: pci: 0000:91:02.0: open_vfio_iommu_group: open '/dev/vfio/141': Device or resource busy
- Resulting in the same failure as before, only happen on AArch64 platform
- /sys/module/vfio/parameters/enable_unsafe_noiommu_mode affect the behavior in some way
- config option set to N, /dev/vfio device or resource busy error
- config option set to Y & iommu_passthrough = 1, IP packets Rx timeout
- The longer the server runs, more test cases fails
- Next to do
- Need to figure out what arm-smmu-v3.4.auto: event 0x10 means
- Juraj modified script to reproduce the issue - Lijian will try it locally
- Also seen in Intel QAT card from Zach
- Will try to reproduce this issue on local thx2 with 20.04 distro - Lijian
- Lijian have slight different firmware version, driver version
- Tried Mellanox card (rdma driver) multiple times - not see the same issue (on XL710 NIC) happens - Juraj
- Lijian can use Juraj's script to reproduce the issue on local tx2 server
- Reducing the numa buffer allocation size resolves this issue
- Observed from the error log of numa buffer allocation
- Need to figure out what arm-smmu-v3.4.auto: event 0x10 means
- /usr/bin/vpp[3789]: pci: 0000:91:02.0: open_vfio_iommu_group: open '/dev/vfio/141': Device or resource busy
- Shipment of new advanced server to the FD.io lab
- One link between TG and DUT, multiple link between DUT for testing LACP.
- Two advanced servers are in plan to ship
- VPN access request to FD.io Arm servers
- Lijian/Jieqiang has got VPN access now
- mbuf-fast-free dpdk enablement with VPP (DEV_TX_OFFLOAD_MBUF_FAST_FREE)
- VPP Performance Test
-
VPP
- VPP SVE implementation - Lijian
- Vector length specific patch is ready
- SVE patch ready and upstreamed, under review - Lijian
- SVE patch sent to Nitin, Nitin will review the patch when back to work.
- Rebased SVE patch per Nitin's request, waiting for Nitin's feedback on running these patches
- Juan met and fixing some issue running SVE in qemu VM
- SVE validation on FPGA platform - Conflunce page ready
- Run unit tests from DPDK and VPP bihash on FPGA
- Try Lijian's SVE patch to see any cycle count improvement
- SVE patch sent to Nitin, Nitin will review the patch when back to work.
- VPP IPv6 Benchmarking and Profiling
- VPP CLI 'ip route add ipv6_addr/mask' outputs wrong IPv6 routes with mask 123-128
- Patch upstreamed https://gerrit.fd.io/r/c/vpp/+/33422
- IPv6 profiling
- Hotspot function - ip6_lookup_node/ip6_rewrite_node
- Will try perfmon & understand two node functions
- VPP CLI 'ip route add ipv6_addr/mask' outputs wrong IPv6 routes with mask 123-128
- VPP mbuf-fast-free tx offload
- Vector path shows performance improvement, still need to investigate scalar path
- Performance improvement for IPv4 routing test cases using vector path
- Try IPv4 multicast testing to verify the scenario when refcnt > 1
- GDB shows that mbufs are copied instead of reference from src port to all dst ports
- Will try L2 flood test case & understand VPP/multicast code
- Direct/Indirect mbuf for VPP multicast testing
- VPP Prefetch
- Refactor prefetch implementation in VPP per CPU's actual cache line size - Tianyu & Jieqiang
- Damjan has merged 2 patches, waiting for the last patch, for generic 128B cacheline size.
- For 64B cacheline size native build on Arm, may need to change code.
- Current VPP does not support 64B cacheline size compilation for Arm images.
- Issues about prefetch on current VPP code base
- Issue 1 support 128B/64B cache-line size in Arm image
- Issue 2 prefetch 'overflow' for native build
- Refactor prefetch implementation in VPP per CPU's actual cache line size - Tianyu & Jieqiang
- VPP memif - Tianyu
- CNF PoC proposal preparation
-
- Add support for VPP aarch64 docker image build
- Calico use cases exploration on VPP
-
- CNF PoC proposal preparation
- VPP IPsec on Arm - Govind
- Work on IPsec input/output nodes - Govind & Zach - waiting for maintainer review
- SPD prototype change on ipsec_output/encryption node - Govind
- Discussion on the default action on the IPsec inbound interface which does not match
- https://gerrit.fd.io/r/c/vpp/+/31694
- IPSec unit tests done - 'make test' cases implemented & included in patch - Zach
- Testing of flow cache functionality, including hash collisions and stale entry overwrites
- IPSec input node/decryption flow cache implemented in a separate patch - Zach
- Upstreamed https://gerrit.fd.io/r/c/vpp/+/32903
- Internal review for IPsec input node flow cache implementation - Zach & Govind
- Discovered issue with SPD policy counter/statistics on input side, to be fixed in additional standalone patch - merged
- VPP Perfmon plugin enablement on Arm - Zach
- Review perfmon code - Lijian & Govind
- Implemented statistics from PMUv3 - done
- Patch upstream has dependency on kernel patch, waiting for this before upstream to VPP
- https://lore.kernel.org/linux-arm-kernel/[email protected]/
- Modify the commit message and upstream the perfmon patch - Zach
- Building Intel QAT driver on arm to test IPsec crypt - Zach
- Patch upstream has dependency on kernel patch, waiting for this before upstream to VPP
- VPP SVE implementation - Lijian
08/10/2021
- Attendees
- Lijian Zhang
- Govindarajan Mohandoss
- Juraj Linkes
- Tianyu Li
- Jieqiang Wang
- Zachary Leaf
- Tina Tsou
- CSIT
- VPP Performance Test
- https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-3n-tsh/
-
https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-report-iterative-2106-3n-tsh/
- 2106 testing partial finished. 21.01.1 ongoing, should be done sometime next week.
- Leverage current spare TX2 server as 2-node topology performance test-bed.
- Add new IPSec NULL encryption & decryption test cases - Juraj
- New IPSec SPD test cases will not have NULL encrypt/decrypt config.
- CSIT Maintainers approved to add new IPSec Policy mode test cases with multiple SPD rules.
- New IPsec test cases - https://gerrit.fd.io/r/c/csit/+/32308 - patch fully tested, waiting for review - Juraj
- Test cases with 1, 10, 100, 1000 SPD entries - still under review
- Try VPP IPsec test cases with a fix on outbound interface - Govind & Juraj
- Outbound IPsec: 10 entries still slower with latest change, related to traffic pattern
- Outbound IPsec finished.
- Waiting for new version of patcheset to verify test cases
- Will try the fix on inbound IPsec tests when the Jenkins is back to normal - Juraj
- Inbound IPsec: reproduced and need to investigate
- Flow cache with 1, 10 SPD entries slower, still investigating. Manual test local vs CSIT have different result on 1-10 SPD policies.
- Release testing ongoing
- Comparison between 21.06 and 21.01.1 is ongoing.
- IPsec SPD input/output case ongoing
- Steps to enable test case in CSIT https://gerrit.fd.io/r/c/csit/+/31863
- Add new IPSec NULL encryption & decryption test cases - Juraj
- VPP Performance Test
`
-
- VPP Path
- Voting and working fine.
- CentOS-8 jobs have been removed.
- VPP Device
- https://jenkins.fd.io/view/vpp/job/vpp-csit-verify-device-master-ubuntu2004-aarch64-1n-tx2/
- VPP device testing issues may be caused by XL710 i40e fw or kernel module.
-
- Try to reproduce with another set of firmware and etc but issues still exist
-
https://doc.dpdk.org/guides/nics/i40e.html
- DPDK uses i40evf PMD and it is old and scheduled to be deprecated.
- This is fixed in DPDK 21.05 version by making iavf PMD as default.
- Use iavf PMD instead of i40evf on all VPP branches, waiting for the test results - not reproduced so far.
-
- New issue: different error related moving VF from host to container(not involving dpdk/VPP) - just started investigating
- /usr/bin/vpp[3789]: pci: 0000:91:02.0: open_vfio_iommu_group: open '/dev/vfio/141': Device or resource busy
- Resulting in the same failure as before, only happen on AArch64 platform
- /sys/module/vfio/parameters/enable_unsafe_noiommu_mode affect the behavior in some way
- config option set to N, /dev/vfio device or resource busy error
- config option set to Y & iommu_passthrough = 1, IP packets Rx timeout
- The longer the server runs, more test cases fails
- Next to do
- Need to figure out what arm-smmu-v3.4.auto: event 0x10 means
- Juraj modified script to reproduce the issue - Lijian will try it locally
- Also seen in Intel QAT card from Zach
- Will try to reproduce this issue on local thx2 with 20.04 distro - Lijian
- Lijian have slight different firmware version, driver version
- Tried Mellanox card (rdma driver) multiple times - not see the same issue (on XL710 NIC) happens - Juraj
- Need to figure out what arm-smmu-v3.4.auto: event 0x10 means
- /usr/bin/vpp[3789]: pci: 0000:91:02.0: open_vfio_iommu_group: open '/dev/vfio/141': Device or resource busy
- Shipment of new advanced server to the FD.io lab
- One link between TG and DUT, multiple link between DUT for testing LACP.
- Two advanced servers are in plan to ship
- VPN access request to FD.io Arm servers
- Lijian has got VPN access now
- Juraj singed Jieqiang's key
- mbuf-fast-free dpdk enablement with VPP (DEV_TX_OFFLOAD_MBUF_FAST_FREE)
- VPP Path
-
VPP
- VPP SVE implementation - Lijian
- Vector length specific patch is ready
- SVE patch ready and upstreamed, under review - Lijian
- SVE patch sent to Nitin, Nitin will review the patch when back to work.
- Rebased SVE patch per Nitin's request, waiting for Nitin's feedback on running these patches
- Juan met and fixing some issue running SVE in qemu VM
- SVE validation on FPGA platform - Conflunce page ready
- Run unit tests from DPDK and VPP bihash on FPGA
- Try Lijian's SVE patch to see any cycle count improvement
- SVE patch sent to Nitin, Nitin will review the patch when back to work.
- VPP IPv6 Benchmarking and Profiling
- VPP CLI 'ip route add ipv6_addr/mask' outputs wrong IPv6 routes with mask 123-128, CLI issue only, CSIT's python API works fine.
- Internal patch to resolve this issue under review - upstreamed
- VPP mbuf-fast-free tx offload
- Vector path shows performance improvement, still need to investigate scalar path
- Performance improvement for IPv4 routing test cases using vector path
- VPP Prefetch
- Refactor prefetch implementation in VPP per CPU's actual cache line size - Tianyu & Jieqiang
- Damjan has merged 2 patches, waiting for the last patch, for generic 128B cacheline size.
- For 64B cacheline size native build on Arm, may need to change code.
- Current VPP does not support 64B cacheline size compilation for Arm images.
- Refactor prefetch implementation in VPP per CPU's actual cache line size - Tianyu & Jieqiang
- VPP Classifier - Lijian
- 3 patches: prefetch, key-value compare simd improvement, cache to look up
- 4 loop unrolling decreasing performance
- VPP memif - Tianyu
- CNF PoC proposal preparation
-
- Add support for VPP aarch64 docker image build
- Calico use cases exploration on VPP
-
- CNF PoC proposal preparation
- VPP IPsec on Arm - Govind
- Work on IPsec input/output nodes - Govind & Zach - waiting for maintainer review
- SPD prototype change on ipsec_output/encryption node - Govind
- https://gerrit.fd.io/r/c/vpp/+/31694
- IPSec unit tests done - 'make test' cases implemented & included in patch - Zach
- Testing of flow cache functionality, including hash collisions and stale entry overwrites
- IPSec input node/decryption flow cache implemented in a separate patch - Zach
- Upstreamed https://gerrit.fd.io/r/c/vpp/+/32903
- Internal review for IPsec input node flow cache implementation - Zach & Govind
- Discovered issue with SPD policy counter/statistics on input side, to be fixed in additional standalone patch - merged
- VPP Perfmon plugin enablement on Arm - Zach
- Review perfmon code - Lijian & Govind
- Implemented statistics from PMUv3 - done
- Patch upstream has dependency on kernel patch, waiting for this before upstream to VPP
- Building Intel QAT driver on arm to test IPsec crypt - Zach
- VPP SVE implementation - Lijian
08/03/2021
- Attendees
- Lijian Zhang
- Govindarajan Mohandoss
- Juraj Linkes
- Tianyu Li
- Jieqiang Wang
- Zachary Leaf
- Tina Tsou
- CSIT
- VPP Performance Test
- https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-3n-tsh/
-
https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-report-iterative-2106-3n-tsh/
- 2106 testing partial finished. 21.01.1 ongoing, should be done sometime next week.
- Leverage current spare TX2 server as 2-node topology performance test-bed.
- Add new IPSec NULL encryption & decryption test cases - Juraj
- New IPSec SPD test cases will not have NULL encrypt/decrypt config.
- CSIT Maintainers approved to add new IPSec Policy mode test cases with multiple SPD rules.
- New IPsec test cases - https://gerrit.fd.io/r/c/csit/+/32308 - patch fully tested, waiting for review - Juraj
- Test cases with 1, 10, 100, 1000 SPD entries - still under review
- Try VPP IPsec test cases with a fix on outbound interface - Govind & Juraj
- Outbound IPsec: 10 entries still slower with latest change, related to traffic pattern
- Waiting for new version of patcheset to verify test cases
- Will try the fix on inbound IPsec tests when the Jenkins is back to normal - Juraj
- Inbound IPsec: reproduced and need to investigate
- Flow cache with 1, 10 SPD entries slower, still investigating. Manual test local vs CSIT have different result on 1-10 SPD policies.
- Release testing ongoing
- Comparison between 21.06 and 21.01.1 is ongoing.
- IPsec SPD input/output case ongoing
- Steps to enable test case in CSIT https://gerrit.fd.io/r/c/csit/+/31863
- Add new IPSec NULL encryption & decryption test cases - Juraj
- VPP Performance Test
`
-
- VPP Path
- Voting and working fine.
- CentOS-8 jobs have been removed.
- VPP Device
- https://jenkins.fd.io/view/vpp/job/vpp-csit-verify-device-master-ubuntu2004-aarch64-1n-tx2/
- VPP device testing issues may be caused by XL710 i40e fw or kernel module.
-
- Try to reproduce with another set of firmware and etc but issues still exist
-
https://doc.dpdk.org/guides/nics/i40e.html
- DPDK uses i40evf PMD and it is old and scheduled to be deprecated.
- This is fixed in DPDK 21.05 version by making iavf PMD as default.
- Use iavf PMD instead of i40evf on all VPP branches, waiting for the test results - not reproduced so far.
-
- New issue: different error related moving VF from host to container(not involving dpdk/VPP) - just started investigating
- /usr/bin/vpp[3789]: pci: 0000:91:02.0: open_vfio_iommu_group: open '/dev/vfio/141': Device or resource busy
- Resulting in the same failure as before, only happen on AArch64 platform
- /sys/module/vfio/parameters/enable_unsafe_noiommu_mode affect the behavior in some way
- config option set to N, /dev/vfio device or resource busy error
- config option set to Y & iommu_passthrough = 1, IP packets Rx timeout
- The longer the server runs, more test cases fails
- Next to do
- Need to figure out what arm-smmu-v3.4.auto: event 0x10 means
- Also seen in Intel QAT card from Zach
- Will try to reproduce this issue on local thx2 with 20.04 distro - Lijian
- Will try Mellanox card to see if same issue happens - Juraj
- Need to figure out what arm-smmu-v3.4.auto: event 0x10 means
- /usr/bin/vpp[3789]: pci: 0000:91:02.0: open_vfio_iommu_group: open '/dev/vfio/141': Device or resource busy
- Shipment of new advanced server to the FD.io lab
- One link between TG and DUT, multiple link between DUT for testing LACP.
- Two advanced servers are in plan to ship
- VPN access request to FD.io Arm servers
- Lijian has got VPN access now
- Juraj singed Jieqiang's key
- mbuf-fast-free dpdk enablement with VPP (DEV_TX_OFFLOAD_MBUF_FAST_FREE)
- VPP Path
-
VPP
- VPP SVE implementation - Lijian
- Vector length specific patch is ready
- SVE patch ready and upstreamed, under review - Lijian
- SVE patch sent to Nitin, Nitin will review the patch when back to work.
- Rebased SVE patch per Nitin's request, waiting for Nitin's feedback on running these patches
- Juan met and fixing some issue running SVE in qemu VM
- SVE validation on FPGA platform - Conflunce page ready
- Run unit tests from DPDK and VPP bihash on FPGA
- Try Lijian's SVE patch to see any cycle count improvement
- SVE patch sent to Nitin, Nitin will review the patch when back to work.
- VPP IPv6 Benchmarking and Profiling
- VPP CLI 'ip route add ipv6_addr/mask' outputs wrong IPv6 routes with mask 123-128
- Internal patch to resolve this issue under review
- VPP mbuf-fast-free tx offload
- Vector path shows performance improvement, still need to investigate scalar path
- Performance improvement for IPv4 routing test cases using vector path
- VPP Prefetch
- Refactor prefetch implementation in VPP per CPU's actual cache line size - Tianyu & Jieqiang
- Damjan has merged 2 patches, waiting for the last patch, for generic 128B cacheline size.
- For 64B cacheline size native build on Arm, may need to change code.
- Current VPP does not support 64B cacheline size compilation for Arm images.
- Refactor prefetch implementation in VPP per CPU's actual cache line size - Tianyu & Jieqiang
- VPP Classifier - Lijian
- 3 patches: prefetch, key-value compare simd improvement, cache to look up
- 4 loop unrolling decreasing performance
- VPP memif - Tianyu
- CNF PoC proposal preparation
-
- Add support for VPP aarch64 docker image build
- Calico use cases exploration on VPP
-
- CNF PoC proposal preparation
- VPP IPsec on Arm - Govind
- Work on IPsec input/output nodes - Govind & Zach - waiting for maintainer review
- SPD prototype change on ipsec_output/encryption node - Govind
- https://gerrit.fd.io/r/c/vpp/+/31694
- IPSec unit tests done - 'make test' cases implemented & included in patch - Zach
- Testing of flow cache functionality, including hash collisions and stale entry overwrites
- IPSec input node/decryption flow cache implemented in a separate patch - Zach
- Upstreamed https://gerrit.fd.io/r/c/vpp/+/32903
- Internal review for IPsec input node flow cache implementation - Zach & Govind
- Discovered issue with SPD policy counter/statistics on input side, to be fixed in additional standalone patch - merged
- VPP Perfmon plugin enablement on Arm - Zach
- Review perfmon code - Lijian & Govind
- Implemented statistics from PMUv3 - done
- Patch upstream has dependency on kernel patch, waiting for this before upstream to VPP
- Building Intel QAT driver on arm to test IPsec crypt - Zach
- VPP SVE implementation - Lijian
07/27/2021
-
Attendees
- Lijian Zhang
- Govindarajan Mohandoss
- Juraj Linkes
- Tianyu Li
- Jieqiang Wang
- Zachary Leaf
- Tina Tsou
-
CSIT
- VPP Performance Test
- https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-3n-tsh/
-
https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-report-iterative-2106-3n-tsh/
- 2106 testing partial finished. 21.01.1 ongoing, should be done sometime next week.
- Leverage current spare TX2 server as 2-node topology performance test-bed.
- Add new IPSec NULL encryption & decryption test cases - Juraj
- New IPSec SPD test cases will not have NULL encrypt/decrypt config.
- CSIT Maintainers approved to add new IPSec Policy mode test cases with multiple SPD rules.
- New IPsec test cases - https://gerrit.fd.io/r/c/csit/+/32308 - patch fully tested, waiting for review - Juraj
- Test cases with 1, 10, 100, 1000 SPD entries - still under review
- Try VPP IPsec test cases with a fix on outbound interface - Govind & Juraj
- Outbound IPsec: 10 entries still slower with latest change, related to traffic pattern
- Will try the fix on inbound IPsec tests when the Jenkins is back to normal - Juraj
- Inbound IPsec: reproduced and need to investigate
- Flow cache with 1, 10 SPD entries slower, still investigating. Manual test local vs CSIT have different result on 1-10 SPD policies.
- Release testing ongoing
- Comparison between 21.06 and 21.01.1 is ongoing.
- IPsec SPD input/output case ongoing
- Steps to enable test case in CSIT https://gerrit.fd.io/r/c/csit/+/31863
- 21.06 vs 21.01 see performance drop on https://docs.fd.io/csit/master/report/_static/vpp/performance-changes-3n-tsh-1t1c-pdr.txt
- May need to check VM and IPsec cases
- Add new IPSec NULL encryption & decryption test cases - Juraj
- VPP Path
- Voting and working fine.
- CentOS-8 jobs have been removed.
- VPP Device
- https://jenkins.fd.io/view/vpp/job/vpp-csit-verify-device-master-ubuntu2004-aarch64-1n-tx2/
- VPP device testing issues may be caused by XL710 i40e fw or kernel module.
-
- Try to reproduce with another set of firmware and etc but issues still exist
-
https://doc.dpdk.org/guides/nics/i40e.html
- DPDK uses i40evf PMD and it is old and scheduled to be deprecated.
- This is fixed in DPDK 21.05 version by making iavf PMD as default.
- Use iavf PMD instead of i40evf on all VPP branches, waiting for the test results - not reproduced so far.
- New issue: different error related moving VF from host to container(not involving dpdk/VPP) - just started investigating
- /usr/bin/vpp[3789]: pci: 0000:91:02.0: open_vfio_iommu_group: open '/dev/vfio/141': Device or resource busy
- Resulting in the same failure as before, only happen on AArch64 platform
- /sys/module/vfio/parameters/enable_unsafe_noiommu_mode affect the behavior in some way
- Random issue, more frequently happening on Arm
- Not see in CI recently or manually.
- scapy unexpected timeout issue: packet drop or slow issue?
-
- vfio-pci driver may be the root cause - bind/unbind
-
-
- Connection issue between Jenkins and the build executor in FD.io lab
- Shipment of new advanced server to the FD.io lab
- One link between TG and DUT, multiple link between DUT for testing LACP.
- Two advanced servers are in plan to ship
- VPN access request to FD.io Arm servers
- Lijian has got VPN access now
- Juraj singed Jieqiang's key
- mbuf-fast-free dpdk enablement with VPP (DEV_TX_OFFLOAD_MBUF_FAST_FREE)
- VPP Performance Test
-
VPP
- VPP SVE implementation - Lijian
- Vector length specific patch is ready
- SVE patch ready and upstreamed, under review - Lijian
- SVE patch sent to Nitin, Nitin will review the patch when back to work.
- Rebased SVE patch per Nitin's request, waiting for Nitin's feedback on running these patches
- Juan met and fixing some issue running SVE in qemu VM
- SVE validation on FPGA platform - Conflunce page ready
- Run unit tests from DPDK and VPP bihash on FPGA
- Try Lijian's SVE patch to see any cycle count improvement
- SVE patch sent to Nitin, Nitin will review the patch when back to work.
- VPP mbuf-fast-free tx offload
- Vector path shows performance improvement, still need to investigate scalar path
- Performance improvement for IPv4 routing test cases using vector path
- VPP Prefetch
- Refactor prefetch implementation in VPP per CPU's actual cache line size - Tianyu & Jieqiang
- Damjan has merged 2 patches, waiting for the last patch, for generic 128B cacheline size.
- For 64B cacheline size native build on Arm, may need to change code.
- Refactor prefetch implementation in VPP per CPU's actual cache line size - Tianyu & Jieqiang
- VPP Classifier - Lijian
- 3 patches: prefetch, key-value compare simd improvement, cache to look up
- 4 loop unrolling descreasing performance
- VPP memif - Tianyu
- CNF PoC proposal preparation
-
- Add support for VPP aarch64 docker image build
-
- CNF PoC proposal preparation
- VPP IPsec on Arm - Govind
- Work on IPsec input/output nodes - Govind & Zach - waiting for maintainer review
- SPD prototype change on ipsec_output/encryption node - Govind
- https://gerrit.fd.io/r/c/vpp/+/31694
- IPSec unit tests done - 'make test' cases implemented & included in patch - Zach
- Testing of flow cache functionality, including hash collisions and stale entry overwrites
- IPSec input node/decryption flow cache implemented in a separate patch - Zach
- Upstreamed https://gerrit.fd.io/r/c/vpp/+/32903
- Internal review for IPsec input node flow cache implementation - Zach & Govind
- Discovered issue with SPD policy counter/statistics on input side, to be fixed in additional standalone patch - merged
- VPP Perfmon plugin enablement on Arm - Zach
- Review perfmon code: having some questions/comments, would like a review meeting - Lijian
- Implemented statistics from PMUv3 - done
- Patch upstream has dependency on kernel patch, waiting for this before upstream to VPP
- Building Intel QAT driver on arm to test IPsec crypt - Zach
- VPP SVE implementation - Lijian
07/20/2021
-
Attendees
- Lijian Zhang
- Govindarajan Mohandoss
- Juraj Linkes
- Tianyu Li
- Jieqiang Wang
- Zachary Leaf
- Tina Tsou
-
CSIT
- VPP Performance Test
- https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-3n-tsh/
- https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-report-iterative-2106-3n-tsh/
- Leverage current spare TX2 server as 2-node topology performance test-bed.
- Add new IPSec NULL encryption & decryption test cases - Juraj
- New IPSec SPD test cases will not have NULL encrypt/decrypt config.
- CSIT Maintainers approved to add new IPSec Policy mode test cases with multiple SPD rules.
- New IPsec test cases - https://gerrit.fd.io/r/c/csit/+/32308 - patch fully tested, waiting for review - Juraj
- Test cases with 1, 10, 100, 1000 SPD entries - still under review
- Try VPP IPsec test cases with a fix on outbound interface - Govind & Juraj
- Will try the fix on inbound IPsec tests when the Jenkins is back to normal - Juraj
- Flow cache with 1, 10 SPD entries slower, still investigating. Manual test local vs CSIT have different result on 1-10 SPD policies.
- Release testing ongoing
- Comparison between 21.06 and 21.01.1 is ongoing.
- IPsec SPD input/output case ongoing
- Steps to enable test case in CSIT https://gerrit.fd.io/r/c/csit/+/31863
- 21.06 vs 21.01 see performance drop on https://docs.fd.io/csit/master/report/_static/vpp/performance-changes-3n-tsh-1t1c-pdr.txt
- May need to check VM and IPsec cases
- Add new IPSec NULL encryption & decryption test cases - Juraj
- VPP Path
- Voting and working fine.
- CentOS-8 jobs have been removed.
- VPP Device
- https://jenkins.fd.io/view/vpp/job/vpp-csit-verify-device-master-ubuntu2004-aarch64-1n-tx2/
- VPP device testing issues may be caused by XL710 i40e fw or kernel module.
-
- Try to reproduce with another set of firmware and etc but issues still exist
-
https://doc.dpdk.org/guides/nics/i40e.html
- DPDK uses i40evf PMD and it is old and scheduled to be deprecated.
- This is fixed in DPDK 21.05 version by making iavf PMD as default.
- Use iavf PMD instead of i40evf on all VPP branches, waiting for the test results - not reproduced so far.
- New issue: different error related moving VF from host to container(not involving dpdk/VPP) - just started investigating
- /usr/bin/vpp[3789]: pci: 0000:91:02.0: open_vfio_iommu_group: open '/dev/vfio/141': Device or resource busy
- Resulting in the same failure as before, only happen on AArch64 platform
- /sys/module/vfio/parameters/enable_unsafe_noiommu_mode affect the behavior in some way
- Random issue, more frequently happening on Arm
- vfio-pci driver may be the root cause - bind/unbind
-
- Connection issue between Jenkins and the build executor in FD.io lab
- Shipment of new advanced server to the FD.io lab
- Two advanced servers are in plan to ship
- VPN access request to FD.io Arm servers
- Lijian has got VPN access now
- Juraj singed Jieqiang's key
- mbuf-fast-free dpdk enablement with VPP (DEV_TX_OFFLOAD_MBUF_FAST_FREE)
- VPP Performance Test
-
VPP
- VPP SVE implementation - Lijian
- Vector length specific patch is ready
- SVE patch ready and upstreamed, under review - Lijian
- SVE patch sent to Nitin, Nitin will review the patch when back to work.
- Juan met and fixing some issue running SVE in qemu VM
- SVE validation on FPGA platform
- Run unit tests from DPDK and VPP bihash on FPGA
- Try Lijian's SVE patch to see any cycle count improvement
- VPP mbuf-fast-free tx offload
- Performance improvement for IPv4 routing test cases using vector path
- VPP Prefetch
- Refactor prefetch implementation in VPP per CPU's actual cache line size - Tianyu & Jieqiang
- VPP Classifier - Lijian
- 3 patches: prefetch, key-value compare simd improvement, cache to look up
- VPP memif - Tianyu
- Investigating VPP memif - Tianyu
-
- Patches have been upstreamed and waiting for review
- Prepare the memif readout - Tianyu
- Add support for VPP aarch64 docker image build
-
- Investigating VPP memif - Tianyu
- VPP IPsec on Arm - Govind
- Work on IPsec input/output nodes - Govind & Zach - waiting for maintainer review
- SPD prototype change on ipsec_output/encryption node - Govind
- https://gerrit.fd.io/r/c/vpp/+/31694
- IPSec unit tests done - 'make test' cases implemented & included in patch - Zach
- Testing of flow cache functionality, including hash collisions and stale entry overwrites
- IPSec input node/decryption flow cache implemented in a separate patch - Zach
- Upstreamed https://gerrit.fd.io/r/c/vpp/+/32903
- Internal review for IPsec input node flow cache implementation - Zach & Govind
- Discovered issue with SPD policy counter/statistics on input side, to be fixed in additional standalone patch
- VPP Perfmon plugin enablement on Arm - Zach
- Implemented statistics from PMUv3 - done
- Patch upstream has dependency on kernel patch, waiting for this before upstream to VPP
- Building Intel QAT driver on arm to test IPsec crypt - Zach
- Implemented statistics from PMUv3 - done
- VPP SVE implementation - Lijian
07/13/2021
- Attendees
- Lijian Zhang
- Juraj Linkes
- Tianyu Li
- Jieqiang Wang
- Zachary Leaf
- Tina Tsou
- CSIT
- VPP Performance Test
- https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-3n-tsh/
- https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-report-iterative-2101-3n-tsh/
- Leverage current spare TX2 server as 2-node topology performance test-bed.
- Add new IPSec NULL encryption & decryption test cases - Juraj
- New IPSec SPD test cases will not have NULL encrypt/decrypt config.
- CSIT Maintainers approved to add new IPSec Policy mode test cases with multiple SPD rules.
- New IPsec test cases - https://gerrit.fd.io/r/c/csit/+/32308 - patch fully tested, waiting for review - Juraj
- Test cases with 1, 10, 100, 1000 SPD entries - still under review
- Expected to be merged soon
- Flow cache with 1, 10 SPD entries slower, still investigating. Mannual test local vs CSIT have different result on 1-10 SPD policies.
- Hugepage size, numa-node, core isolation etc. may need to check.
- Enable flow cache option in startup.conf for VPP CSIT IPsec test cases when the patch is ready - Fixed
- Release testing ongoing
- IPsec SPD input/output case ongoing
- Juraj may share the steps how CSIT handle new configuration changes
- 2n-tx2 & 3n-tsh has been upgraded to ubuntu 20.04, everything is working fine now.
- Juraj is investigating running those test cases with 2N-TX2 topology.
- Some container test cases failed on all platform. - 4 cores container case fail on x86 and arm - Fixed and passing.
- Steps to enable test case in CSIT https://gerrit.fd.io/r/c/csit/+/31863
- 21.06 vs 21.01 see performance drop on https://docs.fd.io/csit/master/report/_static/vpp/performance-changes-3n-tsh-1t1c-pdr.txt
- May need to check VM and IPsec cases
- Add new IPSec NULL encryption & decryption test cases - Juraj
- VPP Path
- Voting and working fine.
- CentOS-8 jobs have been removed.
- VPP Device
- https://jenkins.fd.io/view/vpp/job/vpp-csit-verify-device-master-ubuntu2004-aarch64-1n-tx2/
- VPP device testing issues may be caused by XL710 i40e fw or kernel module.
- VPP community is responding this issue actively. - Juraj
- The issue could be reproduced on Arm servers with the NIC with latest firmware version. (still reproduced, no update yet)
- Moved the NIC from arm server to x86 server can easily update firmware, given script to reproduce, intel busy with other stuff, debugging in progress.
- Intel folks debugged, tried updating firmware/drivers, VF driver updated, old driver: i40evf, new driver: iavf. After modified dpdk code by using iavf, issue can be fixed. Need to find proper solution of dpdk.
- Try to reproduce with another set of firmware and etc but issues still exist
-
https://doc.dpdk.org/guides/nics/i40e.html
- DPDK uses i40evf PMD and it is old and scheduled to be deprecated. After using iavf PMD, the issue is not seen. This is fixed in DPDK 21.05 version by making iavf PMD as default. This fix will be ported to all the older VPP LTS release branches. Currently, the fix is planned only for VPP 20.09 release.
- Use iavf PMD instead of i40evf on all VPP branches, waiting for the test results - not reproduced so far.
- New issue: different error related moving VF from host to containter(not involving dpdk/VPP) - just started investigating
- /usr/bin/vpp[3789]: pci: 0000:91:02.0: open_vfio_iommu_group: open '/dev/vfio/141': Device or resource busy
- Resulting in the same failure as before, only happen on AArch64 platform
- /sys/module/vfio/parameters/enable_unsafe_noiommu_mode affect the behavior in some way
- Random issue, more frequently happening on Arm
- vfio-pci driver may be the root cause - bind/unbind
- Moved the NIC from arm server to x86 server can easily update firmware, given script to reproduce, intel busy with other stuff, debugging in progress.
- "Make Test Testcase Error or Failure" --> There was an intermittent VPP anomaly introduced last week with the change from shmem to socket transport for PAPI which causes the MAP unittests to fail [0]. The root cause is being addressed and should be fixed shortly.
- Shipment of new advanced server to the FD.io lab
- New servers are in shortage.
- VPN access request to FD.io Arm servers
- Will remind Machiek to sign Lijian's GPG public key.
- mbuf-fast-free dpdk enablement with VPP (DEV_TX_OFFLOAD_MBUF_FAST_FREE)
- VPP Performance Test
07/06/2021
-
Attendees
- Lijian Zhang
- Juraj Linkes
- Tianyu Li
- Jieqiang Wang
- Zachary Leaf
- Tina Tsou
-
CSIT
- VPP Performance Test
- https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-3n-tsh/
- https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-report-iterative-2101-3n-tsh/
- Leverage current spare TX2 server as 2-node topology performance test-bed.
- Add new IPSec NULL encryption & decryption test cases - Juraj
- New IPSec SPD test cases will not have NULL encrypt/decrypt config.
- CSIT Maintainers approved to add new IPSec Policy mode test cases with multiple SPD rules.
- New IPsec test cases - https://gerrit.fd.io/r/c/csit/+/32308 - patch fully tested, waiting for review - Juraj
- Test cases with 1, 10, 100, 1000 SPD entries - still under review
- Expected to be merged soon
- Flow cache with 1, 10 SPD entries slower, still investigating. Mannual test local vs CSIT have different result on 1-10 SPD policies.
- Hugepage size, numa-node, core isolation etc. may need to check.
- Enable flow cache option in startup.conf for VPP CSIT IPsec test cases when the patch is ready - Fixed
- Release testing ongoing
- IPsec SPD input/output case ongoing
- Juraj may share the steps how CSIT handle new configuration changes
- 2n-tx2 & 3n-tsh has been upgraded to ubuntu 20.04, everything is working fine now.
- Juraj is investigating running those test cases with 2N-TX2 topology.
- Some container test cases failed on all platform. - 4 cores container case fail on x86 and arm - Fixed and passing.
- Steps to enable test case in CSIT https://gerrit.fd.io/r/c/csit/+/31863
- 21.06 vs 21.01 see performance drop on https://docs.fd.io/csit/master/report/_static/vpp/performance-changes-3n-tsh-1t1c-pdr.txt
- May need to check VM and IPsec cases
- Add new IPSec NULL encryption & decryption test cases - Juraj
- VPP Path
- Voting and working fine.
- CentOS-8 jobs have been removed.
- VPP Device
- https://jenkins.fd.io/view/vpp/job/vpp-csit-verify-device-master-ubuntu2004-aarch64-1n-tx2/
- VPP device testing issues may be caused by XL710 i40e fw or kernel module.
- VPP community is responding this issue actively. - Juraj
- The issue could be reproduced on Arm servers with the NIC with latest firmware version. (still reproduced, no update yet)
- Moved the NIC from arm server to x86 server can easily update firmware, given script to reproduce, intel busy with other stuff, debugging in progress.
- Intel folks debugged, tried updating firmware/drivers, VF driver updated, old driver: i40evf, new driver: iavf. After modified dpdk code by using iavf, issue can be fixed. Need to find proper solution of dpdk.
- Try to reproduce with another set of firmware and etc but issues still exist
-
https://doc.dpdk.org/guides/nics/i40e.html
- DPDK uses i40evf PMD and it is old and scheduled to be deprecated. After using iavf PMD, the issue is not seen. This is fixed in DPDK 21.05 version by making iavf PMD as default. This fix will be ported to all the older VPP LTS release branches. Currently, the fix is planned only for VPP 20.09 release.
- Use iavf PMD instead of i40evf on all VPP branches, waiting for the test results - not reproduced so far.
- New issue: different error related moving VF from host to containter(not involving dpdk/VPP) - just started investigating
- /usr/bin/vpp[3789]: pci: 0000:91:02.0: open_vfio_iommu_group: open '/dev/vfio/141': Device or resource busy
- Resulting in the same failure as before, only happen on AArch64 platform
- /sys/module/vfio/parameters/enable_unsafe_noiommu_mode affect the behavior in some way
- Random issue, more frequently happening on Arm
- vfio-pci driver may be the root cause - bind/unbind
- Moved the NIC from arm server to x86 server can easily update firmware, given script to reproduce, intel busy with other stuff, debugging in progress.
- "Make Test Testcase Error or Failure" --> There was an intermittent VPP anomaly introduced last week with the change from shmem to socket transport for PAPI which causes the MAP unittests to fail [0]. The root cause is being addressed and should be fixed shortly.
- Shipment of new advanced server to the FD.io lab
- New servers are in shortage.
- VPN access request to FD.io Arm servers
- Will remind Machiek to sign Lijian's GPG public key.
- mbuf-fast-free dpdk enablement with VPP (DEV_TX_OFFLOAD_MBUF_FAST_FREE)
- VPP Performance Test
-
VPP
- VPP SVE implementation - Lijian
- Vector length specific patch is ready
- SVE patch ready and upstreamed, under review - Lijian
- SVE patch sent to Nitin, Nitin will review the patch when back to work.
- Juan met and fixing some issue running SVE in qemu VM
- SVE validation on FPGA platform
- VPP Prefetch
- Benchmark VPP using prefetch read always vs prefetch write always on N1SDP - Jieqiang
- Repeat the same test on Ampere server - PMU cache-miss less for write always
- Refactor prefetch implementation in VPP per CPU's actual cache line size - Tianyu & Jieqiang
- Benchmark VPP using prefetch read always vs prefetch write always on N1SDP - Jieqiang
- VPP Classifier - Lijian
- 3 patches: prefetch, key-value compare simd improvement, cache to look up
- Investigating VPP classify function, use case, benchmarking - Lijian
- Start with simple use case
- VPP Classify basic inbound L3 src ip / prot case
- Benchmark VPP classifier on Arm/X86 platform
- Done some NEON changes, see some microbranchmark improvement
- investigate CSIT case
- No classify test case in CSIT. - Jieqiang
- may be there is a CSIT case named iacldstbase
- VPP memif - Tianyu
- Investigating VPP memif - Tianyu
- Benchmarking DPDK memif vs VPP memif
- Race condition occurred hen connecting DPDK memif PMD interface(slave) with VPP memif interface(master)
- Performance improvement using loop unrolling for memif nodes - running CSIT perftest
- Patches have been upstreamed and waiting for review
- Prepare the memif readout - Tianyu
- Add support for VPP aarch64 docker image build
- Benchmarking DPDK memif vs VPP memif
- Try to apply C11 weak memory model on VPP memif - Tianyu
- Use 'show runtime'/perfmon to see cycle improvement
- Memif C11 atomics has been updated by maintainer, not using atomic_relaxed - Tianyu
- Investigating VPP memif - Tianyu
- VPP IPsec on Arm - Govind
- Apply prefetches and loop-unrolling with ipsec-out node
- Didn't observe much performance improvement (2%) so far
- Work on IPsec input/output nodes - Govind & Zach - waiting for maintainer review
- VPP uses linear search on SPD lookups
- SPD prototype change on ipsec_output/encryption node, introducing flow cache with hash, has performance improvements, discussing with community - Govind
- https://gerrit.fd.io/r/c/vpp/+/31694
- IPSec unit tests done - 'make test' cases implemented & included in patch - Zach
- Testing of flow cache functionality, including hash collisions and stale entry overwrites
- IPSec input node/decryption flow cache implemented in a separate patch - Zach
- Upstreamed https://gerrit.fd.io/r/c/vpp/+/32903
- Internal review for IPsec input node flow cache implementation - Zach & Govind
- Discovered issue with SPD policy counter/statistics on input side, to be fixed in additional standalone patch
- Apply prefetches and loop-unrolling with ipsec-out node
- VPP Perfmon plugin enablement on Arm - Zach
- Implemented statistics from PMUv3 - done
- Patch upstream has dependency on kernel patch, waiting for this before upstream to VPP
- Building Intel QAT driver on arm to test IPsec crypt - Zach
- Implemented statistics from PMUv3 - done
- VPP SVE implementation - Lijian
06/29/2021
-
Attendees
- Lijian Zhang
- Juraj Linkes
- Tianyu Li
- Jieqiang Wang
- Zachary Leaf
- Tina Tsou
-
CSIT
- VPP Performance Test
- https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-3n-tsh/
- https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-report-iterative-2101-3n-tsh/
- Leverage current spare TX2 server as 2-node topology performance test-bed.
- Add new IPSec NULL encryption & decryption test cases - Juraj
- New IPSec SPD test cases will not have NULL encrypt/decrypt config.
- CSIT Maintainers approved to add new IPSec Policy mode test cases with multiple SPD rules.
- New IPsec test cases - https://gerrit.fd.io/r/c/csit/+/32308 - patch fully tested, waiting for review - Juraj
- Test cases with 1, 10, 100, 1000 SPD entries - still under review
- Expected to be merged soon
- Flow cache with 1, 10 SPD entries slower, still investigating. Mannual test local vs CSIT have different result on 1-10 SPD policies.
- Hugepage size, numa-node, core isolation etc. may need to check.
- Enable flow cache option in startup.conf for VPP CSIT IPsec test cases when the patch is ready - Fixed
- Release testing ongoing
- IPsec SPD input/output case ongoing
- Juraj may share the steps how CSIT handle new configuration changes
- 2n-tx2 & 3n-tsh has been upgraded to ubuntu 20.04, everything is working fine now.
- Juraj is investigating running those test cases with 2N-TX2 topology.
- Some container test cases failed on all platform. - 4 cores container case fail on x86 and arm - Fixed and passing.
- Steps to enable test case in CSIT https://gerrit.fd.io/r/c/csit/+/31863
- 21.06 vs 21.01 see performance drop on https://docs.fd.io/csit/master/report/_static/vpp/performance-changes-3n-tsh-1t1c-pdr.txt
- May need to check VM and IPsec cases
- Add new IPSec NULL encryption & decryption test cases - Juraj
- VPP Path
- Voting and working fine.
- Community plans to drop the support for CentOS-8.
- VPP Device
- https://jenkins.fd.io/view/vpp/job/vpp-csit-verify-device-master-ubuntu2004-aarch64-1n-tx2/
- VPP device testing issues may be caused by XL710 i40e fw or kernel module.
- VPP community is responding this issue actively. - Juraj
- The issue could be reproduced on Arm servers with the NIC with latest firmware version. (still reproduced, no update yet)
- Moved the NIC from arm server to x86 server can easily update firmware, given script to reproduce, intel busy with other stuff, debugging in progress.
- Intel folks debugged, tried updating firmware/drivers, VF driver updated, old driver: i40evf, new driver: iavf. After modified dpdk code by using iavf, issue can be fixed. Need to find proper solution of dpdk.
- Try to reproduce with another set of firmware and etc but issues still exist
-
https://doc.dpdk.org/guides/nics/i40e.html
- DPDK uses i40evf PMD and it is old and scheduled to be deprecated. After using iavf PMD, the issue is not seen. This is fixed in DPDK 21.05 version by making iavf PMD as default. This fix will be ported to all the older VPP LTS release branches. Currently, the fix is planned only for VPP 20.09 release.
- Use iavf PMD instead of i40evf on all VPP branches, waiting for the test results - not reproduced so far.
- New issue: different error related moving VF from host to containter(not involving dpdk/VPP) - just started investigating
- /usr/bin/vpp[3789]: pci: 0000:91:02.0: open_vfio_iommu_group: open '/dev/vfio/141': Device or resource busy
- Resulting in the same failure as before, only happen on AArch64 platform
- /sys/module/vfio/parameters/enable_unsafe_noiommu_mode affect the behavior in some way
- Debugging
- vfio-pci driver may be the root cause - bind/unbind
- Moved the NIC from arm server to x86 server can easily update firmware, given script to reproduce, intel busy with other stuff, debugging in progress.
- "Make Test Testcase Error or Failure" --> There was an intermittent VPP anomaly introduced last week with the change from shmem to socket transport for PAPI which causes the MAP unittests to fail [0]. The root cause is being addressed and should be fixed shortly.
- Shipment of new advanced server to the FD.io lab
- New servers are in shortage.
- VPN access request to FD.io Arm servers
- mbuf-fast-free dpdk enablement with VPP (DEV_TX_OFFLOAD_MBUF_FAST_FREE)
- VPP Performance Test
-
VPP
- VPP SVE implementation - Lijian
- Vector length specific patch is ready
- SVE patch ready and upstreamed, under review - Lijian
- SVE patch sent to Nitin, Nitin will review the patch when back to work.
- Juan met and fixing some issue running SVE in qemu VM
- SVE validation on FPGA platform
- VPP Prefetch
- Benchmark VPP using prefetch read always vs prefetch write always on N1SDP - Jieqiang
- Repeat the same test on Ampere server - PMU cache-miss less for write always
- Refactor prefetch implementation in VPP per CPU's actual cache line size - Tianyu & Jieqiang
- Benchmark VPP using prefetch read always vs prefetch write always on N1SDP - Jieqiang
- VPP Classifier - Lijian
- 3 patches: prefetch, key-value compare simd improvement, cache to look up
- Investigating VPP classify function, use case, benchmarking - Lijian
- Start with simple use case
- VPP Classify basic inbound L3 src ip / prot case
- Benchmark VPP classifier on Arm/X86 platform
- Done some NEON changes, see some microbranchmark improvement
- investigate CSIT case
- No classify test case in CSIT. - Jieqiang
- may be there is a CSIT case named iacldstbase
- VPP memif - Tianyu
- Investigating VPP memif - Tianyu
- Benchmarking DPDK memif vs VPP memif
- Race condition occurred hen connecting DPDK memif PMD interface(slave) with VPP memif interface(master)
- Performance improvement using loop unrolling for memif nodes - running CSIT perftest
- Patches have been upstreamed and waiting for review
- Prepare the memif readout - Tianyu
- Add support for VPP aarch64 docker image build
- Benchmarking DPDK memif vs VPP memif
- Try to apply C11 weak memory model on VPP memif - Tianyu
- Use 'show runtime'/perfmon to see cycle improvement
- Memif C11 atomics has been updated by maintainer, not using atomic_relaxed - Tianyu
- Investigating VPP memif - Tianyu
- VPP IPsec on Arm - Govind
- Apply prefetches and loop-unrolling with ipsec-out node
- Didn't observe much performance improvement (2%) so far
- Work on IPsec input/output nodes - Govind & Zach - waiting for maintainer review
- VPP uses linear search on SPD lookups
- SPD prototype change on ipsec_output/encryption node, introducing flow cache with hash, has performance improvements, discussing with community - Govind
- https://gerrit.fd.io/r/c/vpp/+/31694
- IPSec unit tests done - 'make test' cases implemented & included in patch - Zach
- Testing of flow cache functionality, including hash collisions and stale entry overwrites
- IPSec input node/decryption flow cache implemented in a separate patch - Zach
- Upstreamed https://gerrit.fd.io/r/c/vpp/+/32903
- Internal review for IPsec input node flow cache implementation - Zach & Govind
- Discovered issue with SPD policy counter/statistics on input side, to be fixed in additional standalone patch
- Apply prefetches and loop-unrolling with ipsec-out node
- VPP Perfmon plugin enablement on Arm - Zach
- Implemented statistics from PMUv3 - done
- Patch upstream has dependency on kernel patch, waiting for this before upstream to VPP
- Building Intel QAT driver on arm to test IPsec crypt - Zach
- Implemented statistics from PMUv3 - done
- VPP SVE implementation - Lijian
06/22/2021
- Attendees
- Govindarajan Mohandoss
- Juraj Linkes
- Tianyu Li
- Jieqiang Wang
- Zachary Leaf
- Tina Tsou
- CSIT
- VPP Performance Test
- https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-3n-tsh/
- https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-report-iterative-2101-3n-tsh/
- Leverage current spare TX2 server as 2-node topology performance test-bed.
- Add new IPSec NULL encryption & decryption test cases - Juraj
- New IPSec SPD test cases will not have NULL encrypt/decrypt config.
- CSIT Maintainers approved to add new IPSec Policy mode test cases with multiple SPD rules.
- New IPsec test cases - https://gerrit.fd.io/r/c/csit/+/32308 - patch fully tested, waiting for review - Juraj
- Test cases with 1, 10, 100, 1000 SPD entries
- Expected to be merged soon
- Enable flow cache option in startup.conf for VPP CSIT IPsec test cases when the patch is ready - Fixed
- Release testing ongoing
- IPsec SPD input/output case ongoing
- Juraj may share the steps how CSIT handle new configuration changes
- 2n-tx2 & 3n-tsh has been upgraded to ubuntu 20.04, everything is working fine now.
- Juraj is investigating running those test cases with 2N-TX2 topology.
- Some container test cases failed on all platform. - 4 cores container case fail on x86 and arm
- Steps to enable test case in CSIT https://gerrit.fd.io/r/c/csit/+/31863
- Add new IPSec NULL encryption & decryption test cases - Juraj
- VPP Path
- Voting and working fine.
- Community plans to drop the support for CentOS-8.
- VPP Device
- https://jenkins.fd.io/view/vpp/job/vpp-csit-verify-device-master-ubuntu2004-aarch64-1n-tx2/
- VPP device testing issues may be caused by XL710 i40e fw or kernel module.
- VPP community is responding this issue actively. - Juraj
- The issue could be reproduced on Arm servers with the NIC with latest firmware version. (still reproduced, no update yet)
- Moved the NIC from arm server to x86 server can easily update firmware, given script to reproduce, intel busy with other stuff, debugging in progress.
- Intel folks debugged, tried updating firmware/drivers, VF driver updated, old driver: i40evf, new driver: iavf. After modified dpdk code by using iavf, issue can be fixed. Need to find proper solution of dpdk.
- Try to reproduce with another set of firmware and etc but issues still exist
-
https://doc.dpdk.org/guides/nics/i40e.html
- DPDK uses i40evf PMD and it is old and scheduled to be deprecated. After using iavf PMD, the issue is not seen. This is fixed in DPDK 21.05 version by making iavf PMD as default. This fix will be ported to all the older VPP LTS release branches. Currently, the fix is planned only for VPP 20.09 release.
- Use iavf PMD instead of i40evf on all VPP branches, waiting for the test results - not reproduced so far.
- New issue: different error related moving VF from host to containter(not involving dpdk/VPP) - just started investigating
- /usr/bin/vpp[3789]: pci: 0000:91:02.0: open_vfio_iommu_group: open '/dev/vfio/141': Device or resource busy
- Resulting in the same failure as before, only happen on AArch64 platform
- vfio-pci driver may be the root cause
- Moved the NIC from arm server to x86 server can easily update firmware, given script to reproduce, intel busy with other stuff, debugging in progress.
- "Make Test Testcase Error or Failure" --> There was an intermittent VPP anomaly introduced last week with the change from shmem to socket transport for PAPI which causes the MAP unittests to fail [0]. The root cause is being addressed and should be fixed shortly.
- Shipment of new advanced server to the FD.io lab
- New servers are in shortage.
- VPN access request to FD.io Arm servers
- VPP Performance Test
- VPP
- VPP SVE implementation - Lijian
- Vector length specific patch is ready
- SVE patch ready and upstreamed, under review - Lijian
- SVE patch sent to Nitin, Nitin will review the patch when back to work.
- Juan met and fixing some issue running SVE in qemu VM
- SVE validation on FPGA platform
- VPP Prefetch
- Benchmark VPP using prefetch read always vs prefetch write always on N1SDP - Jieqiang
- Repeat the same test on Ampere server
- Refactor prefetch implementation in VPP per CPU's actual cache line size - Tianyu & Jieqiang
- Benchmark VPP using prefetch read always vs prefetch write always on N1SDP - Jieqiang
- VPP Classifier - Lijian
- Investigating VPP classify function, use case, benchmarking - Lijian
- Start with simple use case
- VPP Classify basic inbound L3 src ip / prot case
- Benchmark VPP classifier on Arm/X86 platform
- Done some NEON changes, see some microbranchmark improvement
- investigate CSIT case
- No classify test case in CSIT. - Jieqiang
- may be there is a CSIT case named iacldstbase
- Investigating VPP classify function, use case, benchmarking - Lijian
- VPP memif - Tianyu
- Investigating VPP memif - Tianyu
- Benchmarking DPDK memif vs VPP memif
- Race condition occurred hen connecting DPDK memif PMD interface(slave) with VPP memif interface(master)
- Performance improvement using loop unrolling for memif nodes - running CSIT perftest
- Patches have been upstreamed and waiting for review
- Prepare the memif readout - Tianyu
- Add support for VPP aarch64 docker image build
- Benchmarking DPDK memif vs VPP memif
- Try to apply C11 weak memory model on VPP memif - Tianyu
- Use 'show runtime'/perfmon to see cycle improvement
- Memif C11 atomics has been updated by maintainer, not using atomic_relaxed - Tianyu
- Investigating VPP memif - Tianyu
- VPP IPsec on Arm - Govind
- Apply prefetches and loop-unrolling with ipsec-out node
- Didn't observe much performance improvement (2%) so far
- Work on IPsec input/output nodes - Govind & Zach - waiting for maintainer review
- VPP uses linear search on SPD lookups
- SPD prototype change on ipsec_output/encryption node, introducing flow cache with hash, has performance improvements, discussing with community - Govind
- https://gerrit.fd.io/r/c/vpp/+/31694
- IPSec unit tests done - 'make test' cases implemented & included in patch - Zach
- Testing of flow cache functionality, including hash collisions and stale entry overwrites
- IPSec input node/decryption flow cache implemented in a separate patch - Zach
- Internal review for IPsec input node flow cache implementation - Zach & Govind
- Discovered issue with SPD policy counter/statistics on input side, to be fixed in additional standalone patch
- Apply prefetches and loop-unrolling with ipsec-out node
- VPP Perfmon plugin enablement on Arm - Zach
- Implemented statistics from PMUv3 - done
- Patch upstream has dependency on kernel patch, waiting for this before upstream to VPP
- Building Intel QAT driver on arm to test IPsec crypt - Zach
- Implemented statistics from PMUv3 - done
- VPP SVE implementation - Lijian
06/15/2021
-
Attendees
- Lijian Zhang
- Govindarajan Mohandoss
- Juraj Linkes
- Tianyu Li
- Jieqiang Wang
- Zachary Leaf
- Tina Tsou
-
CSIT
- VPP Performance Test
- https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-3n-tsh/
- https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-report-iterative-2101-3n-tsh/
- Leverage current spare TX2 server as 2-node topology performance test-bed.
- Add new IPSec NULL encryption & decryption test cases - Juraj
- New IPSec SPD test cases will not have NULL encrypt/decrypt config.
- CSIT Maintainers approved to add new IPSec Policy mode test cases with multiple SPD rules.
- New IPsec test cases - https://gerrit.fd.io/r/c/csit/+/32308 - patch fully tested, waiting for review - Juraj
- VPP exits with the IPsec startup config, try startup config from Zach's email - Juraj
- Enable flow cache option in startup.conf for VPP CSIT IPsec test cases when the patch is ready - Fixed
- Release testing ongoing
- IPsec SPD input/output case ongoing
- Juraj may share the steps how CSIT handle new configuration changes
- 2n-tx2 & 3n-tsh has been upgraded to ubuntu 20.04, everything is working fine now.
- Juraj is investigating running those test cases with 2N-TX2 topology.
- Some container test cases failed on all platform. - 4 cores container case fail on x86 and arm
- Steps to enable test case in CSIT https://gerrit.fd.io/r/c/csit/+/31863
- Add new IPSec NULL encryption & decryption test cases - Juraj
- VPP Performance Test
-
- VPP Path
- Voting and working fine.
- Community plans to drop the support for CentOS-8.
- VPP Device
- https://jenkins.fd.io/view/vpp/job/vpp-csit-verify-device-master-ubuntu2004-aarch64-1n-tx2/
- VPP device testing issues may be caused by XL710 i40e fw or kernel module.
- VPP community is responding this issue actively. - Juraj
- The issue could be reproduced on Arm servers with the NIC with latest firmware version. (still reproduced, no update yet)
- Moved the NIC from arm server to x86 server can easily update firmware, given script to reproduce, intel busy with other stuff, debugging in progress.
- Intel folks debugged, tried updating firmware/drivers, VF driver updated, old driver: i40evf, new driver: iavf. After modified dpdk code by using iavf, issue can be fixed. Need to find proper solution of dpdk.
- Try to reproduce with another set of firmware and etc but issues still exist
-
https://doc.dpdk.org/guides/nics/i40e.html
- DPDK uses i40evf PMD and it is old and scheduled to be deprecated. After using iavf PMD, the issue is not seen. This is fixed in DPDK 21.05 version by making iavf PMD as default. This fix will be ported to all the older VPP LTS release branches. Currently, the fix is planned only for VPP 20.09 release.
- Use iavf PMD instead of i40evf on all VPP branches, waiting for the test results - not reproduced so far.
- New issue: different error related moving VF from host to containter(not involving dpdk/VPP) - just started investigating
- /usr/bin/vpp[3789]: pci: 0000:91:02.0: open_vfio_iommu_group: open '/dev/vfio/141': Device or resource busy
- Moved the NIC from arm server to x86 server can easily update firmware, given script to reproduce, intel busy with other stuff, debugging in progress.
- Will try to reproduce the issue with x86 servers.
- This issue is common to all platforms(Arm & Intel)
- "Make Test Testcase Error or Failure" --> There was an intermittent VPP anomaly introduced last week with the change from shmem to socket transport for PAPI which causes the MAP unittests to fail [0]. The root cause is being addressed and should be fixed shortly. - DaveW
- Shippment of new adavanced server to the FD.io lab
- New servers are in shortage.
- VPP Path
-
VPP
- VPP SVE implementation - Lijian
- Vector length specific patch is ready
- SVE patch ready and upstreamed, under review - Lijian
- SVE patch sent to Nitin, Nitin will review the patch when back to work.
- Juan met and fixing some issue running SVE in qemu VM
- SVE validation on FPGA platform
- VPP Prefetch
- Benchmark VPP using prefetch read always vs prefetch write always on N1SDP - Jieqiang
- Repeat the same test on Ampere server
- Refactor prefetch implementation in VPP per CPU's actual cache line size - Tianyu & Jieqiang
- Benchmark VPP using prefetch read always vs prefetch write always on N1SDP - Jieqiang
- VPP Classifier - Lijian
- Investigating VPP classify function, use case, benchmarking - Lijian
- Start with simple use case
- VPP Classify basic inbound L3 src ip / prot case
- Benchmark VPP classifier on Arm/X86 platform
- Done some NEON changes, see some microbranchmark improvement
- investigate CSIT case
- No classify test case in CSIT. - Jieqiang - may be there is a CSIT case named iacldstbase
- Investigating VPP classify function, use case, benchmarking - Lijian
- VPP memif - Tianyu
- Investigating VPP memif - Tianyu
- Benchmarking DPDK memif vs VPP memif
- Race condition occurred hen connecting DPDK memif PMD interface(slave) with VPP memif interface(master)
- Performance improvement using loop unrolling for memif nodes - running CSIT perftest
- Patches have been upstreamed and waiting for review
- Prepare the memif readout - Tianyu
- Benchmarking DPDK memif vs VPP memif
- Try to apply C11 weak memory model on VPP memif - Tianyu
- Use 'show runtime'/perfmon to see cycle improvement
- Memif C11 atomics has been updated by maintainer, not using atomic_relaxed - Tianyu
- Investigating VPP memif - Tianyu
- VPP IPsec on Arm - Govind
- Apply prefetches and loop-unrolling with ipsec-out node
- Didn't observe much performance improvement (2%) so far
- Work on IPsec input/output nodes - Govind & Zach - waiting for maintainer review
- VPP uses linear search on SPD lookups
- SPD prototype change on ipsec_output/encryption node, introducing flow cache with hash, has performance improvements, discussing with community - Govind
- https://gerrit.fd.io/r/c/vpp/+/31694
- IPSec unit tests done - 'make test' cases implemented & included in patch - Zach
- Testing of flow cache functionality, including hash collisions and stale entry overwrites
- IPSec input node/decryption flow cache implemented in a separate patch - Zach
- Waiting for review comments on outbound side before upstream to VPP
- Discovered issue with SPD policy counter/statistics on input side, to be fixed in additional standalone patch
- Apply prefetches and loop-unrolling with ipsec-out node
- VPP Perfmon plugin enablement on Arm - Zach
- Implemented statistics from PMUv3 - done
- Patch upstream has dependency on kernel patch, waiting for this before upstream to VPP
- Building Intel QAT driver on arm to test IPsec crypt - Zach
- Implemented statistics from PMUv3 - done
- VPP SVE implementation - Lijian
06/08/2021
- Attendees
- Lijian Zhang
- Govindarajan Mohandoss
- Juraj Linkes
- Tianyu Li
- Jieqiang Wang
- Zachary Leaf
- Tina Tsou
- CSIT
- VPP Performance Test
- https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-3n-tsh/
- https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-report-iterative-2101-3n-tsh/
- Leverage current spare TX2 server as 2-node topology performance test-bed.
- Add new IPSec NULL encryption & decryption test cases - Juraj
- New IPSec SPD test cases will not have NULL encrypt/decrypt config.
- CSIT Maintainers approved to add new IPSec Policy mode test cases with multiple SPD rules.
- New IPsec test cases - https://gerrit.fd.io/r/c/csit/+/32308 - patch fully tested, waiting for review - Juraj
- VPP exits with the IPsec startup config, try startup config from Zach's email - Juraj
- Enable flow cache option in startup.conf for VPP CSIT IPsec test cases when the patch is ready - Work in progress.
- 2n-tx2 & 3n-tsh has been upgraded to ubuntu 20.04, everything is working fine now.
- Juraj is investigating running those test cases with 2N-TX2 topology.
- Some container test cases failed on all platform.
- Add new IPSec NULL encryption & decryption test cases - Juraj
- VPP Path
- Voting and working fine.
- Community plans to drop the support for CentOS-8.
- VPP Device
- https://jenkins.fd.io/view/vpp/job/vpp-csit-verify-device-master-ubuntu2004-aarch64-1n-tx2/
- VPP device testing issues may be caused by XL710 i40e fw or kernel module.
- VPP community is responding this issue actively. - Juraj
- The issue could be reproduced on Arm servers with the NIC with latest firmware version. (still reproduced, no update yet)
- Moved the NIC from arm server to x86 server can easily update firmware, given script to reproduce, intel busy with other stuff, debugging in progress.
- Intel folks debugged, tried updating firmware/drivers, VF driver updated, old driver: i40evf, new driver: iavf. After modified dpdk code by using iavf, issue can be fixed. Need to find proper solution of dpdk.
- Try to reproduce with another set of firmware and etc but issues still exist
-
https://doc.dpdk.org/guides/nics/i40e.html
- DPDK uses i40evf PMD and it is old and scheduled to be deprecated. After using iavf PMD, the issue is not seen. This is fixed in DPDK 21.05 version by making iavf PMD as default. This fix will be ported to all the older VPP LTS release branches. Currently, the fix is planned only for VPP 20.09 release.
- Use iavf PMD instead of i40evf on all VPP branches, waiting for the test results.
- Moved the NIC from arm server to x86 server can easily update firmware, given script to reproduce, intel busy with other stuff, debugging in progress.
- Will try to reproduce the issue with x86 servers.
- This issue is common to all platforms(Arm & Intel)
- "Make Test Testcase Error or Failure" --> There was an intermittent VPP anomaly introduced last week with the change from shmem to socket transport for PAPI which causes the MAP unittests to fail [0]. The root cause is being addressed and should be fixed shortly.
- Shippment of new adavanced server to the FD.io lab
- New servers are in shortage.
- VPP Performance Test
- VPP
- VPP default compiler on Arm platform
- In Ubuntu-20.04, VPP on Arm will reprioritize compiler as gcc-10 > gcc-9.2.0 > clang-10
- Plan to benchmark gcc-10 vs clang-12 on arm L2/L3, 1/10k - Jieqiang
- No obvious performance improvement, keep the original default compiler
- Plan to benchmark gcc-10 vs clang-12 on arm L2/L3, 1/10k - Jieqiang
- In Ubuntu-20.04, VPP on Arm will reprioritize compiler as gcc-10 > gcc-9.2.0 > clang-10
- VPP SVE implementation - Lijian
- Vector length specific patch is ready
- SVE patch ready and upstreamed, under review - Lijian
- SVE patch sent to Nitin, Nitin will review the patch when back to work.
- SVE validation on FPGA platform
- VPP Prefetch
- Benchmark VPP using prefetch read always vs prefetch write always - Jieqiang
- Refactor prefetch implementation in VPP per CPU's actual cache line size - Tianyu & Jieqiang
- VPP Classifier - Lijian
- Investigating VPP classify function, use case, benchmarking - Lijian
- Start with simple use case
- VPP Classify basic inbound L3 src ip / prot case
- Benchmark VPP classifier on Arm/X86 platform
- investigate CSIT case
- No classify test case in CSIT. - Jieqiang
- Investigating VPP classify function, use case, benchmarking - Lijian
- VPP memif - Tianyu
- Investigating VPP memif - Tianyu
- Benchmarking DPDK memif vs VPP memif
- Race condition occurred hen connecting DPDK memif PMD interface(slave) with VPP memif interface(master)
- Performance improvement using loop unrolling for memif nodes - running CSIT perftest
- Patches have been upstreamed and waiting for review
- Prepare the memif readout - Tianyu
- Benchmarking DPDK memif vs VPP memif
- Try to apply C11 weak memory model on VPP memif - Tianyu
- Use 'show runtime'/perfmon to see cycle improvement
- Memif C11 atomics has been updated by maintainer, not using atomic_relaxed - Tianyu
- Investigating VPP memif - Tianyu
- VPP IPsec on Arm - Govind
- Apply prefetches and loop-unrolling with ipsec-out node
- Didn't observe much performance improvement (2%) so far
- Work on IPsec input/output nodes - Govind & Zach
- VPP uses linear search on SPD lookups
- SPD prototype change on ipsec_output/encryption node, introducing flow cache with hash, has performance improvements, discussing with community - Govind
- https://gerrit.fd.io/r/c/vpp/+/31694
- IPSec unit tests done - 'make test' cases implemented & included in patch - Zach
- Testing of flow cache functionality, including hash collisions and stale entry overwrites
- IPSec input node/decryption flow cache implemented in a separate patch - Zach
- Waiting for review comments on outbound side before upstream to VPP
- Discovered issue with SPD policy counter/statistics on input side, to be fixed in additional standalone patch
- Apply prefetches and loop-unrolling with ipsec-out node
- VPP Perfmon plugin enablement on Arm - Zach
- Implemented statistics from PMUv3 - done
- Patch upstream has dependency on kernel patch, waiting for this before upstream to VPP
- Implemented statistics from PMUv3 - done
- VPP default compiler on Arm platform
06/01/2021
- Attendees
- Govindarajan Mohandoss
- Juraj Linkes
- Zachary Leaf
- Tina Tsou
- CSIT
- VPP Performance Test
- https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-3n-tsh/
- https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-report-iterative-2101-3n-tsh/
- Leverage current spare TX2 server as 2-node topology performance test-bed.
-
- Start with basic L2/L3/ACL/classifiers tests and use DPDK PMD on 2n-thx2 firstly(daily testing)
- IPSec policy test cases are not running by default.
- Some of the IPSec test cases(Policy tests) has been added to daily testing.
- Enabled the policy tests in mrr-daily testing and it's now running on both 2n-tx2 and 3n-tsh (and also available for per-patch on-demand testing) - Juraj
- Add new IPSec NULL encryption & decryption test cases - Juraj
- New IPSec SPD test cases will not have NULL encrypt/decrypt config.
- IPSec SPD test cases will be ready next week, how to make SPD policy change - IP address range changes - Juraj
- CSIT Maintainers approved to add new IPSec Policy mode test cases with multiple SPD rules.
- New IPsec test cases - https://gerrit.fd.io/r/c/csit/+/32308 - patch fully tested, waiting for review - Juraj
- Enable flow cache option in startup.conf for VPP CSIT IPsec test cases when the patch is ready - Work in progress.
- Send the correct robot framework tags for IPSec policy test cases to Govind - Juraj
- 2n-tx2 & 3n-tsh has been upgraded to ubuntu 20.04, everything is working fine now.
- Juraj is investigating running those test cases with 2N-TX2 topology.
- Juraj will investigate cabling issue on Taishan performance test-bed - resolved.
- Some container case are seems failure on all platform.
-
- VPP Path
- Voting and working fine.
- VPP Device
- https://jenkins.fd.io/view/vpp/job/vpp-csit-verify-device-master-ubuntu2004-aarch64-1n-tx2/
- VPP device testing issues may be caused by XL710 i40e fw or kernel module.
- VPP community is responding this issue actively. - Juraj
- The issue could be reproduced on Arm servers with the NIC with latest firmware version. (still reproduced, no update yet)
- Moved the NIC from arm server to x86 server can easily update firmware, given script to reproduce, intel busy with other stuff, debugging in progress.
- Intel folks debugged, tried updating firmware/drivers, VF driver updated, old driver: i40evf, new driver: iavf. After modified dpdk code by using iavf, issue can be fixed. Need to find proper solution of dpdk.
- Try to reproduce with another set of firmware and etc but issues still exist
- https://doc.dpdk.org/guides/nics/i40e.html
- Internal ticket has been raised
- DPDK uses i40evf PMD and it is old and scheduled to be deprecated. After using iavf PMD, the issue is not seen. This is fixed in DPDK 21.05 version by making iavf PMD as default. This fix will be ported to all the older VPP LTS release branches. Currently, the fix is planned only for VPP 20.09 release.
- Moved the NIC from arm server to x86 server can easily update firmware, given script to reproduce, intel busy with other stuff, debugging in progress.
- Will try to reproduce the issue with x86 servers.
- This issue is common to all platforms(Arm & Intel)
- "Make Test Testcase Error or Failure" --> There was an intermittent VPP anomaly introduced last week with the change from shmem to socket transport for PAPI which causes the MAP unittests to fail [0]. The root cause is being addressed and should be fixed shortly.
- VPP Performance Test
- VPP
- In Ubuntu-20.04, VPP on Arm will reprioritize compiler as gcc-10 > gcc-9.2.0 > clang-10
- Plan to benchmark gcc-10 vs clang-12 on arm L2/L3, 1/10k - Jieqiang
- Vector length specific patch is ready
- Investigating VPP classify function, use case, benchmarking - Lijian
- Start with simple use case
- 128 and 256 fixed size vector wrappers are ready, needs verification
- Jieqiang helped to verify most fixed size vector wrappers - unit test code
- SVE Remaining works - variable type convention - need some workaround for 256bit convention
- VLA patch, coding and verification done. string memcpy/memset, bihash key compare functions, rdma/bonding node
- SVE patch ready and upstreamed, under review - Lijian
- SVE patch sent to Nitin, Nitin will review the patch when back to work.
- Review memif patch
- Memif C11 atomics has been updated by maintainer, not using atomic_relaxed - Tianyu
- VPP Classify basic inbound L3 src ip / prot case, investigate CSIT case - No classify test case in CSIT. - Jieqiang
- Investigating VPP memif - Tianyu
- Benchmarking DPDK memif vs VPP memif
- Review the confluence page and prepare the memif readout - Lijian & Tianyu
- Race condition occurred hen connecting DPDK memif PMD interface(slave) with VPP memif interface(master)
- Performance improvement using loop unrolling for memif nodes - running CSIT perftest
- Prepare the memif readout - Tianyu
- Try to apply C11 weak memory model on VPP memif - Tianyu
- Use 'show runtime'/perfmon to see cycle improvement
- Run memif unit test
- Benchmarking DPDK memif vs VPP memif
- IPsec on Arm platform - Govind
- Apply prefetches and loop-unrolling with ipsec-out node
- Didn't observe much performance improvement (2%) so far
- Apply prefetches and loop-unrolling with ipsec-out node
- Work on IPsec input/output nodes - VPP uses linear search on SPD lookups - Govind & Zach
- SPD prototype change on ipsec_output/encryption node, introducing flow cache with hash, has performance improvements, discussing with community - Govind
- https://gerrit.fd.io/r/c/vpp/+/31694
- IPSec unit tests done - 'make test' cases implemented & included in patch - Zach
- Testing of flow cache functionality, including hash collisions and stale entry overwrites
- IPSec input node/decryption flow cache implemented in a separate patch - Zach
- Waiting for review comments on outbound side before upstream to VPP
- Discovered issue with SPD policy counter/statistics on input side, to be fixed in additional standalone patch
- SPD prototype change on ipsec_output/encryption node, introducing flow cache with hash, has performance improvements, discussing with community - Govind
- Perfmon plugin enablement on Arm - Zach
- Implemented statistics from PMUv3 - done
- Patch upstream has dependency on kernel patch, waiting for this before upstream to VPP
- Investigated CMN-600 stats in perfmon plugin
- Abandoned, CMN-600 only gives system level view, no useful stats at node level - linux perf tool can give the same result
- Implemented statistics from PMUv3 - done
- In Ubuntu-20.04, VPP on Arm will reprioritize compiler as gcc-10 > gcc-9.2.0 > clang-10
05/25/2021
- Attendees
- Lijian Zhang
- Govindarajan Mohandoss
- Juraj Linkes
- Jieqiang Wang
- Zachary Leaf
- Tianyu Li
- Tina Tsou
- CSIT
- VPP Performance Test
- https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-3n-tsh/
- https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-report-iterative-2101-3n-tsh/
- Leverage current spare TX2 server as 2-node topology performance test-bed.
-
- Start with basic L2/L3/ACL/classifiers tests and use DPDK PMD on 2n-thx2 firstly(daily testing)
- IPSec policy test cases are not running by default.
- Some of the IPSec test cases(Policy tests) has been added to daily testing.
- Enabled the policy tests in mrr-daily testing and it's now running on both 2n-tx2 and 3n-tsh (and also available for per-patch on-demand testing) - Juraj
- Add new IPSec NULL encryption & decryption test cases - Juraj
- New IPSec SPD test cases will not have NULL encrypt/decrypt config.
- IPSec SPD test cases will be ready next week, how to make SPD policy change - IP address range changes - Juraj
- CSIT Maintainers approved to add new IPSec Policy mode test cases with multiple SPD rules.
- New IPsec test cases - https://gerrit.fd.io/r/c/csit/+/32308 - patch fully tested, waiting for review - Juraj
- Enable flow cache option in startup.conf for VPP CSIT IPsec test cases when the patch is ready - will look into it
- Send the correct robot framework tags for IPSec policy test cases to Govind - Juraj
- 2n-tx2 & 3n-tsh has been upgraded to ubuntu 20.04, everything is working fine now.
- Juraj is investigating running those test cases with 2N-TX2 topology.
- Juraj will investigate cabling issue on Taishan performance test-bed - resolved.
- Some container case are seems failure on all platform.
-
- VPP Path
- Voting and working fine.
- VPP Device
- https://jenkins.fd.io/view/vpp/job/vpp-csit-verify-device-master-ubuntu2004-aarch64-1n-tx2/
- VPP device testing issues may be caused by XL710 i40e fw or kernel module.
- VPP community is responding this issue actively. - Juraj
- The issue could be reproduced on Arm servers with the NIC with latest firmware version. (still reproduced, no update yet)
- Moved the NIC from arm server to x86 server can easily update firmware, given script to reproduce, intel busy with other stuff, debugging in progress.
- Intel folks debugged, tried updating firmware/drivers, VF driver updated, old driver: i40evf, new driver: iavf. After modified dpdk code by using iavf, issue can be fixed. Need to find proper solution of dpdk.
- Try to reproduce with another set of firmware and etc but issues still exist
- https://doc.dpdk.org/guides/nics/i40e.html
- Internal ticket has been raised
- Try the new version of DPDK but it does not help
- Contact Intel devs for the possible advice
- Workaround may impact too much to all test cases
- Moved the NIC from arm server to x86 server can easily update firmware, given script to reproduce, intel busy with other stuff, debugging in progress.
- Will try to reproduce the issue with x86 servers.
- This issue is common to all platforms(Arm & Intel)
- "Make Test Testcase Error or Failure" --> There was an intermittent VPP anomaly introduced last week with the change from shmem to socket transport for PAPI which causes the MAP unittests to fail [0]. The root cause is being addressed and should be fixed shortly.
- VPP Performance Test
- VPP
- In Ubuntu-20.04, VPP on Arm will reprioritize compiler as gcc-10 > gcc-9.2.0 > clang-10
- Plan to benchmark gcc-10 vs clang-12 on arm L2/L3, 1/10k - Jieqiang
- Vector length specific patch is ready
- Investigating VPP classify function, use case, benchmarking - Lijian
- Start with simple use case
- 128 and 256 fixed size vector wrappers are ready, needs verification
- Jieqiang helped to verify most fixed size vector wrappers - unit test code
- SVE Remaining works - variable type convention - need some workaround for 256bit convention
- VLA patch, coding and verification done. string memcpy/memset, bihash key compare functions, rdma/bonding node
- SVE patch ready and upstreamed, under review - Lijian
- SVE patch sent to Nitin, Nitin will review the patch when back to work.
- Review memif patch
- Memif C11 atomics has been updated by maintainer, not using atomic_relaxed - Tianyu
- VPP Classify basic inbound L3 src ip / prot case, investigate CSIT case - No classify test case in CSIT. - Jieqiang
- Investigating VPP memif - Tianyu
- Benchmarking DPDK memif vs VPP memif
- Review the confluence page and prepare the memif readout - Lijian & Tianyu
- Race condition occurred hen connecting DPDK memif PMD interface(slave) with VPP memif interface(master)
- Performance improvement using loop unrolling for memif nodes - running CSIT perftest
- Prepare the memif readout - Tianyu
- Try to apply C11 weak memory model on VPP memif - Tianyu
- Use 'show runtime'/perfmon to see cycle improvement
- Run memif unit test
- Benchmarking DPDK memif vs VPP memif
- IPsec on Arm platform. - Govind
- IPsec input node optimization work in progress - Zach & Govind
- Apply prefetches and loop-unrolling with ipsec-out node
- Didn't observe much performance improvement (2%) so far
- Work on IPsec input node and VPP uses linear search on SPD lookup.
- SPD prototype change, introducing flow cache with hash, have performance improvement, discussing with community.
- https://gerrit.fd.io/r/c/vpp/+/31694
- IPSec unit test - make test new cases implementation
- Make test cases for IPSec policy mode - Done, included in Govind's patch, waiting for maintainer review - Zach
- Add/Remove/Add+Remove+Readd/Hash collisions/Multiple interfaces & rules
- Review the patch and grasp the basics about IPSec - Lijian
- Perfmon plugin enablement on Arm - Zach
- patch upstream has dependency on kernel patch
- Will try loop unrolling on the SPD lookup.
- Will investigate the prefetches and loop unrolling on IPsec input node. - Govind
- SPD prototype change, introducing flow cache with hash, have performance improvement, discussing with community.
- Working on IPsec flow cache, discussed with Neal, maintainer agree with the change
- perfmon CMN-600 investigating - Zach
- VPP perfmon CMN-600 patch abandon, system level, not vpp node level, linux perf can give the same result - Zach
- Plan to upstream perfmon plugin - resolving review comments - Zach
- IPSec flow cache outbound done, working on inbound side in seperate patch - Zach
- IPSec decryption / input node - Zach
- In Ubuntu-20.04, VPP on Arm will reprioritize compiler as gcc-10 > gcc-9.2.0 > clang-10
05/18/2021
- Attendees
- Lijian Zhang
- Govindarajan Mohandoss
- Juraj Linkes
- Zachary Leaf
- Tianyu Li
- Tina Tsou
- CSIT
- VPP Performance Test
- https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-3n-tsh/
- https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-report-iterative-2101-3n-tsh/
- Leverage current spare TX2 server as 2-node topology performance test-bed.
-
- Start with basic L2/L3/ACL/classifiers tests and use DPDK PMD on 2n-thx2 firstly(daily testing)
- IPSec policy test cases are not running by default.
- Some of the IPSec test cases(Policy tests) has been added to daily testing.
- Enabled the policy tests in mrr-daily testing and it's now running on both 2n-tx2 and 3n-tsh (and also available for per-patch on-demand testing) - Juraj
- Add new IPSec NULL encryption & decryption test cases - Juraj
- New IPSec SPD test cases will not have NULL encrypt/decrypt config.
- IPSec SPD test cases will be ready next week, how to make SPD policy change - IP address range changes - Juraj
- CSIT Maintainers approved to add new IPSec Policy mode test cases with multiple SPD rules.
- New IPsec test cases - https://gerrit.fd.io/r/c/csit/+/32308
- Enable flow cache option in startup.conf for VPP CSIT IPsec test cases when the patch is ready
- Send the correct robot framework tags for IPSec policy test cases to Govind - Juraj
- 2n-tx2 & 3n-tsh has been upgraded to ubuntu 20.04, everything is working fine now.
- Juraj is investigating running those test cases with 2N-TX2 topology.
- Juraj will investigate adding IPSec test cases on Taishan performance test-bed.
-
- VPP Path
- Voting and working fine.
- VPP Device
- https://jenkins.fd.io/view/vpp/job/vpp-csit-verify-device-master-ubuntu2004-aarch64-1n-tx2/
- VPP device testing issues may be caused by XL710 i40e fw or kernel module.
- VPP community is responding this issue actively. - Juraj
- The issue could be reproduced on Arm servers with the NIC with latest firmware version. (still reproduced, no update yet)
- Moved the NIC from arm server to x86 server can easily update firmware, given script to reproduce, intel busy with other stuff, debugging in progress.
- Try to reproduce with another set of firmware and etc but issues still exist
- https://doc.dpdk.org/guides/nics/i40e.html
- Internal ticket has been raised
- Try the new version of DPDK but it does not help
- Contact Intel devs for the possible advice
- Workaround may impact too much to all test cases
- Will try to reproduce the issue with x86 servers.
- This issue is common to all platforms(Arm & Intel)
- "Make Test Testcase Error or Failure" --> There was an intermittent VPP anomaly introduced last week with the change from shmem to socket transport for PAPI which causes the MAP unittests to fail [0]. The root cause is being addressed and should be fixed shortly.
- Lab move for the fd.io lab
- Lab moving started stage 2, moved part of the servers to make sure ci service not down.
- Lab move is done, some issues with taishan testbed
- Arm servers documented in https://gerrit.fd.io/r/c/csit/+/30662
- VPP Performance Test
- VPP
- In Ubuntu-20.04, VPP on Arm will reprioritize compiler as gcc-10 > gcc-9.2.0 > clang-10
- Plan to benchmark gcc-10 vs clang-12
- Vector length specific patch is ready
- 128 and 256 fixed size vector wrappers are ready, needs verification
- Jieqiang helped to verify most fixed size vector wrappers - unit test code
- SVE Remaining works - variable type convention - need some workaround for 256bit convention
- VLA patch, coding and verification done. string memcpy/memset, bihash key compare functions, rdma/bonding node
- SVE patch ready and upstreamed, under review - Lijian
- SVE patch sent to Nitin, Nitin will review the patch when back to work.
- Review memif patch
- Functional bug related to C11 atomics has been resolved by VPP maintainer.
- VPP Classify basic inbound L3 src ip / prot case, investigate CSIT case. - Jieqiang
- Make test cases for IPSec policy mode - Zach
- Add/Remove/Add+Remove+Readd/Hash collisions/Multiple interfaces & rules - Add more test cases
- Investigating VPP memif - Tianyu
- Benchmarking DPDK memif vs VPP memif
- Review the confluence page and prepare the memif readout - Lijian & Tianyu
- Race condition occurred hen connecting DPDK memif PMD interface(slave) with VPP memif interface(master)
- Performance improvement using loop unrolling for memif nodes
- Prepare the memif readout - Tianyu
- Try to apply C11 weak memory model on VPP memif - Tianyu
- Use 'show runtime'/perfmon to see cycle improvement
- Run memif unit test
- Benchmarking DPDK memif vs VPP memif
- IPsec on Arm platform. - Govind
- IPsec input node optimization work in progress - Zach & Govind
- Apply prefetches and loop-unrolling with ipsec-out node
- Didn't observe much performance improvement (2%) so far
- Work on IPsec input node and VPP uses linear search on SPD lookup.
- SPD prototype change, introducing flow cache with hash, have performance improvement, discussing with community.
- https://gerrit.fd.io/r/c/vpp/+/31694
- IPSec unit test - make test new cases implementation
- Review the patch and grasp the basics about IPSec - Lijian
- Perfmon plugin enablement on Arm - Zach
- patch upstream has dependency on kernel patch
- Will try loop unrolling on the SPD lookup.
- Will investigate the prefetches and loop unrolling on IPsec input node. - Govind
- SPD prototype change, introducing flow cache with hash, have performance improvement, discussing with community.
- Working on IPsec flow cache, discussed with Neal, maintainer agree with the change
- perfmon CMN-600 investigating - Zach
- VPP perfmon CMN-600 patch abandon, system level, not vpp node level, linux perf can give the same result - Zach
- Plan to upstream perfmon plugin - resolving review comments - Zach
- IPSec flow cache outbound done, working on inbound side in seperate patch - Zach
- IPSec decryption / input node - Zach
- In Ubuntu-20.04, VPP on Arm will reprioritize compiler as gcc-10 > gcc-9.2.0 > clang-10
05/11/2021
- Attendees
- Lijian Zhang
- Govindarajan Mohandoss
- Juraj Linkes
- Zachary Leaf
- Tianyu Li
- Tina Tsou
- CSIT
- VPP Performance Test
- https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-3n-tsh/
- https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-report-iterative-2101-3n-tsh/
- Leverage current spare TX2 server as 2-node topology performance test-bed.
-
- Start with basic L2/L3/ACL/classifiers tests and use DPDK PMD on 2n-thx2 firstly(daily testing)
- IPSec policy test cases are not running by default.
- Some of the IPSec test cases(Policy tests) has been added to daily testing.
- Enabled the policy tests in mrr-daily testing and it's now running on both 2n-tx2 and 3n-tsh (and also available for per-patch on-demand testing) - Juraj
- Add new IPSec NULL encryption & decryption test cases - Juraj
- New IPSec SPD test cases will not have NULL encrypt/decrypt config.
- IPSec SPD test cases will be ready next week, how to make SPD policy change - IP address range changes - Juraj
- CSIT Maintainers approved to add new IPSec Policy mode test cases with multiple SPD rules.
- Send the correct robot framework tags for IPSec policy test cases to Govind - Juraj
- 2n-tx2 & 3n-tsh has been upgraded to ubuntu 20.04, everything is working fine now.
- Juraj is investigating running those test cases with 2N-TX2 topology.
- Juraj will investigate adding IPSec test cases on Taishan performance test-bed.
-
- VPP Path
- Voting and working fine.
- VPP Device
- https://jenkins.fd.io/view/vpp/job/vpp-csit-verify-device-master-ubuntu2004-aarch64-1n-tx2/
- VPP device testing issues may be caused by XL710 i40e fw or kernel module.
- VPP community is responding this issue actively. - Juraj
- The issue could be reproduced on Arm servers with the NIC with latest firmware version. (still reproduced, no update yet)
- Moved the NIC from arm server to x86 server can easily update firmware, given script to reproduce, intel busy with other stuff, debugging in progress.
- https://doc.dpdk.org/guides/nics/i40e.html
- Internal ticket has been raised
- Try the new version of DPDK but it does not help
- Contact Intel devs for the possible advice
- Workaround may impact too much to all test cases
- Will try to reproduce the issue with x86 servers.
- This issue is common to all platforms(Arm & Intel)
- "Make Test Testcase Error or Failure" --> There was an intermittent VPP anomaly introduced last week with the change from shmem to socket transport for PAPI which causes the MAP unittests to fail [0]. The root cause is being addressed and should be fixed shortly.
- Lab move for the fd.io lab
- Lab moving started stage 2, moved part of the servers to make sure ci service not down.
- Arm servers documented in https://gerrit.fd.io/r/c/csit/+/30662
- Almost all except performance testbed, which will be moved this week, everything is smooth so far.
- ubuntu 1804 -> 2004
- VPP Performance Test
- VPP
- In Ubuntu-20.04, VPP on Arm will reprioritize compiler as gcc-10 > gcc-9.2.0 > clang-10
- Multi-arch support - Lijian
- SOC ID will be available on /proc entry starting from kernel version 5.9 to differentiate vendor CPUs.
- Vector length specific patch is ready
- 128 and 256 fixed size vector wrappers are ready, needs verification
- Jieqiang helped to verify most fixed size vector wrappers - unit test code
- SVE Remaining works - variable type convention - need some workaround for 256bit convention
- VLA patch, coding and verification done. string memcpy/memset, bihash key compare functions, rdma/bonding node
- SVE patch ready and upstreamed, under review - Lijian
- SVE patch sent to Nitin, Nitin will review the patch when back to work.
- Review memif patch
- VPP Classify basic inbound L3 src ip / prot case, investigate CSIT case.
- Make test cases for IPSec policy mode - Jieqiang
- Review memif test cases/memif cases
- Finished coding of SVE string library, bihash key compare functions
- Verify SVE vector length specific wrappers - Jieqiang
-
https://gerrit.fd.io/r/c/vpp/+/30326 - second proposal - not preferred
- Extend vector length agnostic opportunities
- SVE unit testing based on test_vec, fix test_vec issues
- Investigate VPP test cases in container
- Focus more on data-plane performance benchmarking and optimization - Tianyu
- Record the benchmarking results of VPP CNF 3 test cases in excel template
- Will do readout presentation with extended people - Tianyu
- Record the benchmarking results of VPP CNF 3 test cases in excel template
- Focus more on data-plane performance benchmarking and optimization - Tianyu
- Investigating VPP memif - Tianyu
- Benchmarking DPDK memif vs VPP memif
- Review the confluence page and prepare the memif readout - Lijian & Tianyu
- Race condition occurred hen connecting DPDK memif PMD interface(slave) with VPP memif interface(master)
- Performance improvement using loop unrolling for memif nodes
- Prepare the memif readout - Tianyu
- Try to apply C11 weak memory model on VPP memif - Tianyu
- Use 'show runtime'/perfmon to see cycle improvement
- Run memif unit test
- Benchmarking DPDK memif vs VPP memif
- IPsec on Arm platform. - Govind
- IPsec input node optimization work in progress - Zach & Govind
- Apply prefetches and loop-unrolling with ipsec-out node
- Didn't observe much performance improvement (2%) so far
- Work on IPsec input node and VPP uses linear search on SPD lookup.
- SPD prototype change, introducing flow cache with hash, have performance improvement, discussing with community.
- https://gerrit.fd.io/r/c/vpp/+/31694
- IPSec unit test - make test new cases implementation
- Review the patch and grasp the basics about IPSec - Lijian
- Perfmon plugin enablement on Arm - Zach
- patch upstream has dependency on kernel patch
- Will try loop unrolling on the SPD lookup.
- Will investigate the prefetches and loop unrolling on IPsec input node. - Govind
- SPD prototype change, introducing flow cache with hash, have performance improvement, discussing with community.
- Working on IPsec flow cache, discussed with Neal, maintainer agree with the change
- Using startup parameter to enable the IPsec flow cache feature
- Adding Python test case to test IPSec node behavior - Jieqiang
- perfmon CMN-600 investigating - Zach
- VPP perfmon CMN-600 patch abandon, system level, not vpp node level, linux perf can give the same result - Zach
- Plan to upstream perfmon plugin - resolving review comments - Zach
- IPSec flow cache outbound done, working on inbound side in seperate patch - Zach
- IPSec decryption / input node - Zach
04/27/2021
- Attendees
- Govindarajan Mohandoss
- Lijian Zhang
- Juraj Linkes
- Tianyu Li
- Jieqiang Wang
- Tina Tsou
- CSIT
- VPP Performance Test
- https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-3n-tsh/
- https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-report-iterative-2101-3n-tsh/
- Leverage current spare TX2 server as 2-node topology performance test-bed.
-
- Start with basic L2/L3/ACL/classifiers tests and use DPDK PMD on 2n-thx2 firstly(daily testing)
- IPSec policy test cases are not running by default.
- Some of the IPSec test cases(Policy tests) has been added to daily testing.
- Enabled the policy tests in mrr-daily testing and it's now running on both 2n-tx2 and 3n-tsh (and also available for per-patch on-demand testing) - Juraj
- Add new IPSec NULL encryption & decryption test cases - Juraj
- Send the correct robot framework tags for IPSec policy test cases to Govind - Juraj
- 2n-tx2 & 3n-tsh has been upgraded to ubuntu 20.04, everything is working fine now.
- Juraj is investigating running those test cases with 2N-TX2 topology.
- Juraj will investigate adding IPSec test cases on Taishan performance test-bed.
-
- VPP Path
- CentOS-8 on Arm Jenkins is created and could be triggered manually with 'beta-verify' and 'beta-merge'
-
-
https://jenkins.fd.io/job/vpp-verify-master-centos8-aarch64/
- Its voting right is enabled on Arm.
-
https://jenkins.fd.io/job/vpp-verify-master-centos8-aarch64/
-
- CentOS-8 on Arm Jenkins is created and could be triggered manually with 'beta-verify' and 'beta-merge'
- VPP Device
- https://jenkins.fd.io/view/vpp/job/vpp-csit-verify-device-master-ubuntu2004-aarch64-1n-tx2/
- VPP device testing issues may be caused by XL710 i40e fw or kernel module.
- VPP community is responding this issue actively. - Juraj
- The issue could be reproduced on Arm servers with the NIC with latest firmware version. (still reproduced, no update yet)
- https://doc.dpdk.org/guides/nics/i40e.html
- Internal ticket has been raised
- Try the new version of DPDK but it does not help
- Contact Intel devs for the possible advice
- Workaround may impact too much to all test cases
- Will try to reproduce the issue with x86 servers.
- This issue is common to all platforms(Arm & Intel)
- "Make Test Testcase Error or Failure" --> There was an intermittent VPP anomaly introduced last week with the change from shmem to socket transport for PAPI which causes the MAP unittests to fail [0]. The root cause is being addressed and should be fixed shortly.
- Lab move for the fd.io lab
- Lab moving started stage 2, moved part of the servers to make sure ci service not down.
- Arm servers documented in https://gerrit.fd.io/r/c/csit/+/30662
- VPP Performance Test
- VPP
- In Ubuntu-20.04, VPP on Arm will reprioritize compiler as gcc-10 > gcc-9.2.0 > clang-10
- Multi-arch support - Lijian
- SOC ID will be available on /proc entry starting from kernel version 5.9 to differentiate vendor CPUs.
- Vector length specific patch is ready
- 128 and 256 fixed size vector wrappers are ready, needs verification
- Jieqiang helped to verify most fixed size vector wrappers - unit test code
- SVE Remaining works - variable type convention - need some workaround for 256bit convention
- VLA patch, coding and verification done. string memcpy/memset, bihash key compare functions, rdma/bonding node
- SVE patch ready and upstreamed, under review - Lijian
- Make test cases for IPSec policy mode - Jieqiang
- Review memif test cases/memif cases
- Finished coding of SVE string library, bihash key compare functions
- Verify SVE vector length specific wrappers - Jieqiang
-
https://gerrit.fd.io/r/c/vpp/+/30326 - second proposal - not preferred
- Extend vector length agnostic opportunities
- SVE unit testing based on test_vec, fix test_vec issues
- Investigate VPP test cases in container
- Focus more on data-plane performance benchmarking and optimization - Tianyu
- Record the benchmarking results of VPP CNF 3 test cases in excel template
- Will do readout presentation with extended people - Tianyu
- Record the benchmarking results of VPP CNF 3 test cases in excel template
- Focus more on data-plane performance benchmarking and optimization - Tianyu
- Investigating VPP memif - Tianyu
- Benchmarking DPDK memif vs VPP memif
- Review the confluence page and prepare the memif readout - Lijian & Tianyu
- Race condition occurred hen connecting DPDK memif PMD interface(slave) with VPP memif interface(master)
- Performance improvement using loop unrolling for memif nodes
- Prepare the memif readout - Tianyu
- Try to apply C11 weak memory model on VPP memif - Tianyu
- Use 'show runtime'/perfmon to see cycle improvement
- Run memif unit test
- Benchmarking DPDK memif vs VPP memif
- IPsec on Arm platform. - Govind
- IPsec input node optimization work in progress - Zach & Govind
- Apply prefetches and loop-unrolling with ipsec-out node
- Didn't observe much performance improvement (2%) so far
- Work on IPsec input node and VPP uses linear search on SPD lookup.
- SPD prototype change, introducing flow cache with hash, have performance improvement, discussing with community.
- https://gerrit.fd.io/r/c/vpp/+/31694
- IPSec unit test - make test new cases implementation
- Review the patch and grasp the basics about IPSec - Lijian
- Perfmon plugin enablement on Arm - Zach
- patch upstream has dependency on kernel patch
- Will try loop unrolling on the SPD lookup.
- Will investigate the prefetches and loop unrolling on IPsec input node. - Govind
- SPD prototype change, introducing flow cache with hash, have performance improvement, discussing with community.
- Working on IPsec flow cache, discussed with Neal, maintainer agree with the change
- Using startup parameter to enable the IPsec flow cache feature
- Adding Python test case to test IPSec node behavior - Jieqiang
- perfmon CMN-600 investigating - Zach
- Plan to upstream perfmon plugin - resolving review comments - Zach
- IPSec decryption / input node - Zach
04/13/2021
- Attendees
- Govindarajan Mohandoss
- Lijian Zhang
- Juraj Linkes
- Tianyu Li
- Jieqiang Wang
- Tina Tsou
- CSIT
- VPP Performance Test
- https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-3n-tsh/
- https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-report-iterative-2101-3n-tsh/
- Leverage current spare TX2 server as 2-node topology performance test-bed.
-
- Start with basic L2/L3/ACL/classifiers tests and use DPDK PMD on 2n-thx2 firstly(daily testing)
- IPSec policy test cases are not running by default.
- Some of the IPSec test cases(Policy tests) has been added to daily testing.
- Enabled the policy tests in mrr-daily testing and it's now running on both 2n-tx2 and 3n-tsh (and also available for per-patch on-demand testing) - Juraj
- Add new IPSec NULL encryption & decryption test cases - Juraj
- Send the correct robot framework tags for IPSec policy test cases to Govind - Juraj
- 2n-tx2 & 3n-tsh has been upgraded to ubuntu 20.04, everything is working fine now.
- Some issues occurred during the upgrade.
- Patch to resolve the building error of DPDK on 3n-tsh testbed.
- Root cause is the change of build system of DPDK on 3n-tsh related to SOC id detection.
- Juraj is investigating running those test cases with 2N-TX2 topology.
- Juraj will investigate adding IPSec test cases on Taishan performance test-bed.
-
- VPP Path
- CentOS-8 on Arm Jenkins is created and could be triggered manually with 'beta-verify' and 'beta-merge'
-
-
https://jenkins.fd.io/job/vpp-verify-master-centos8-aarch64/
- Its voting right is enabled on Arm.
-
https://jenkins.fd.io/job/vpp-verify-master-centos8-aarch64/
-
- CentOS-8 on Arm Jenkins is created and could be triggered manually with 'beta-verify' and 'beta-merge'
- VPP Device
- https://jenkins.fd.io/view/vpp/job/vpp-csit-verify-device-master-ubuntu2004-aarch64-1n-tx2/
- VPP device testing issues may be caused by XL710 i40e fw or kernel module.
- VPP community is responding this issue actively. - Juraj
- The issue could be reproduced on Arm servers with the NIC with latest firmware version. (still reproduced, no update yet)
- https://doc.dpdk.org/guides/nics/i40e.html
- Internal ticket has been raised
- Try the new version of DPDK but it does not help
- Contact Intel devs for the possible advice
- Will try to reproduce the issue with x86 servers.
- This issue is common to all platforms(Arm & Intel)
- "Make Test Testcase Error or Failure" --> There was an intermittent VPP anomaly introduced last week with the change from shmem to socket transport for PAPI which causes the MAP unittests to fail [0]. The root cause is being addressed and should be fixed shortly.
- Lab move for the fd.io lab
- Arm servers documented in https://gerrit.fd.io/r/c/csit/+/30662
- VPP Performance Test
- VPP
- In Ubuntu-20.04, VPP on Arm will reprioritize compiler as gcc-10 > gcc-9.2.0 > clang-10
- Multi-arch support - Lijian
- SOC ID will be available on /proc entry starting from kernel version 5.9 to differentiate vendor CPUs.
- Vector length specific patch is ready
- 128 and 256 fixed size vector wrappers are ready, needs verification
- Jieqiang helped to verify most fixed size vector wrappers - unit test code
- SVE Remaining works - variable type convention - need some workaround for 256bit convention
- VLA patch, coding and verification done. string memcpy/memset, bihash key compare functions, rdma/bonding node
- Make test cases for IPSec policy mode - Jieqiang
- Review memif test cases/memif cases
- Finished coding of SVE string library, bihash key compare functions
- Verify SVE vector length specific wrappers - Jieqiang
-
https://gerrit.fd.io/r/c/vpp/+/30326 - second proposal - not preferred
- Extend vector length agnostic opportunities
- SVE unit testing based on test_vec, fix test_vec issues
- Test template update - Jieqiang
- Investigate VPP test cases in container
- Focus more on data-plane performance benchmarking and optimization - Tianyu
- Record the benchmarking results of VPP CNF 3 test cases in excel template
- Will do readout presentation with extended people - Tianyu
- Record the benchmarking results of VPP CNF 3 test cases in excel template
- Focus more on data-plane performance benchmarking and optimization - Tianyu
- Investigating VPP memif - Tianyu
- Benchmarking DPDK memif vs VPP memif
- Review the confluence page and prepare the memif readout - Lijian & Tianyu
- Race condition occurred hen connecting DPDK memif PMD interface(slave) with VPP memif interface(master)
- Prepare the memif readout - Tianyu
- Try to apply C11 weak memory model on VPP memif - Tianyu
- Use 'show runtime'/perfmon to see cycle improvement
- Run memif unit test
- Benchmarking DPDK memif vs VPP memif
- IPsec on Arm platform. - Govind
- Apply prefetches and loop-unrolling with ipsec-out node
- Didn't observe much performance improvement (2%) so far
- Work on IPsec input node and VPP uses linear search on SPD lookup.
- SPD prototype change, introducing flow cache with hash, have performance improvement, discussing with community.
- https://gerrit.fd.io/r/c/vpp/+/31694
- Review the patch and grasp the basics about IPSec - Lijian
- Perfmon plugin enablement on Arm - Zach
- patch upstream has dependency on kernel patch
- Will try loop unrolling on the SPD lookup.
- Will investigate the prefetches and loop unrolling on IPsec input node. - Govind
- SPD prototype change, introducing flow cache with hash, have performance improvement, discussing with community.
- Apply prefetches and loop-unrolling with ipsec-out node
- Working on IPsec flow cache, discussed with Neal, maintainer agree with the change
- Using startup parameter to enable the IPsec flow cache feature
- Adding Python test case to test IPSec node behavior - Jieqiang
- perfmon CMN-600 investigating - Zach
- Plan to upstream perfmon plugin - resolving review comments - Zach
- IPSec decryption - Zach
03/30/2021
- Attendees
- Govindarajan Mohandoss
- Lijian Zhang
- Juraj Linkes
- Tianyu Li
- Jieqiang Wang
- Tina Tsou
- CSIT
- VPP Performance Test
- https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-3n-tsh/
- https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-report-iterative-2101-3n-tsh/
- Leverage current spare TX2 server as 2-node topology performance test-bed.
-
- Start with basic L2/L3/ACL/classifiers tests and use DPDK PMD on 2n-thx2 firstly(daily testing)
- IPSec policy test cases are not running by default.
- 2 node IPsec SPD policy test case patch is ready, starting with 1 and 1k tunnels. (40, 400 tunnels in seperate patch)
- https://gerrit.fd.io/r/c/csit/+/31605
- Fix the wrong CLI commands but configuration still has problems.
- Send the correct robot framework tags for IPSec policy test cases to Govind - Juraj
- 2n-tx2 & 3n-tsh has been upgraded to ubuntu 20.04, everything is working fine now.
- Some issues occurred during the upgrade.
- Patch to resolve the building error of DPDK on arm testbed.(taishan dpdk cases still have issues, investigating)
- Juraj is investigating running those test cases with 2N-TX2 topology.
- Juraj will investigate adding IPSec test cases on Taishan performance test-bed.
-
- VPP Path
- CentOS-8 on Arm Jenkins is created and could be triggered manually with 'beta-verify' and 'beta-merge'
-
-
https://jenkins.fd.io/job/vpp-verify-master-centos8-aarch64/
- Its voting right is enabled on Arm.
-
https://jenkins.fd.io/job/vpp-verify-master-centos8-aarch64/
-
- CentOS-8 on Arm Jenkins is created and could be triggered manually with 'beta-verify' and 'beta-merge'
- VPP Device
- VPP device testing issues may be caused by XL710 i40e fw or kernel module.
- VPP community is responding this issue actively. - Juraj
- The issue could be reproduced on Arm servers with the NIC with latest firmware version. (still reproduced, no update yet)
- https://doc.dpdk.org/guides/nics/i40e.html
- Internal ticket has been raised
- Try the new version of DPDK but it does not help
- Contact Intel devs for the possible advice
- Will try to reproduce the issue with x86 servers.
- "Make Test Testcase Error or Failure" --> There was an intermittent VPP anomaly introduced last week with the change from shmem to socket transport for PAPI which causes the MAP unittests to fail [0]. The root cause is being addressed and should be fixed shortly.
- Lab move for the fd.io lab
- Arm servers documented in https://gerrit.fd.io/r/c/csit/+/30662
- VPP device testing issues may be caused by XL710 i40e fw or kernel module.
- VPP Performance Test
- VPP
- In Ubuntu-20.04, VPP on Arm will reprioritize compiler as gcc-10 > gcc-9.2.0 > clang-10
- Multi-arch support - Lijian
- SOC ID will be available on /proc entry starting from kernel version 5.9 to differentiate vendor CPUs.
- Vector length specific patch is ready
- 128 and 256 fixed size vector wrappers are ready, needs verification
- Review memif test cases/memif cases
- Finished coding of SVE string library, bihash key compare functions
- Verify SVE vector length specific wrappers - Jieqiang
-
https://gerrit.fd.io/r/c/vpp/+/30326 - second proposal - not preferred
- Extend vector length agnostic opportunities
- SVE unit testing based on test_vec, fix test_vec issues
- Test template update
- SVE unit test in qemu-vm, met compiling issue, investigating
- Investigate VPP test cases in container
- Focus more on data-plane performance benchmarking and optimization - Tianyu
- Record the benchmarking results of VPP CNF 3 test cases in excel template
- Will do readout presentation with extended people - Tianyu
- Record the benchmarking results of VPP CNF 3 test cases in excel template
- Focus more on data-plane performance benchmarking and optimization - Tianyu
- Investigating VPP memif - Tianyu
- Benchmarking DPDK memif vs VPP memif
- Review the confluence page and prepare the memif readout - Lijian & Tianyu
- Race condition occurred hen connecting DPDK memif PMD interface(slave) with VPP memif interface(master)
- Prepare the memif readout - Tianyu
- Benchmarking DPDK memif vs VPP memif
- IPsec on Arm platform. - Govind
- Apply prefetches and loop-unrolling with ipsec-out node
- Didn't observe much performance improvement (2%) so far
- Work on IPsec input node and VPP uses linear search on SPD lookup.
- SPD prototype change, introducing flow cache with hash, have performance improvement, discussing with community.
- https://gerrit.fd.io/r/c/vpp/+/31694
- Review the patch and grasp the basics about IPSec - Lijian
- Perfmon plugin enablement on Arm - Zach
- patch upstream has dependency on kernel patch
- Will try loop unrolling on the SPD lookup.
- Will investigate the prefetches and loop unrolling on IPsec input node. - Govind
- SPD prototype change, introducing flow cache with hash, have performance improvement, discussing with community.
- Apply prefetches and loop-unrolling with ipsec-out node
- Working on IPsec flow cache, discussed with Neal, maintainer agree with the change
- Using startup parameter to enable the IPsec flow cache feature
- Discuss with jieqiang adding python test case to test ipsec node behavior
- perfmon CMN-600 investigating - Zach
03/16/2021
- Attendees
- Govindarajan Mohandoss
- Lijian Zhang
- Juraj Linkes
- Tianyu Li
- Jieqiang Wang
- Tina Tsou
- CSIT
- VPP Performance Test
- https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-3n-tsh/
- https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-report-iterative-2101-3n-tsh/
- Leverage current spare TX2 server as 2-node topology performance test-bed.
-
- Start with basic L2/L3/ACL/classifiers tests and use DPDK PMD on 2n-thx2 firstly(daily testing)
- IPSec policy test cases are not running by default.
- Juraj is investigating running those test cases with 2N-TX2 topology.
-
- Juraj will investigate adding IPSec test cases on Taishan performance test-bed.
- VPP Path
- CentOS-8 on Arm Jenkins is created and could be triggered manually with 'beta-verify' and 'beta-merge'
- CentOS-8 docker image on Arm is working fine, but not triggered per patch yet.
-
https://jenkins.fd.io/job/vpp-verify-master-centos8-aarch64/
- Its voting right is enabled on Arm.
-
https://jenkins.fd.io/job/vpp-verify-master-centos8-aarch64/
- CentOS-8 docker image on Arm is working fine, but not triggered per patch yet.
- CentOS-8 on Arm Jenkins is created and could be triggered manually with 'beta-verify' and 'beta-merge'
- VPP Device
- VPP device testing issues may be caused by XL710 i40e fw or kernel module.
- VPP community is responding this issue actively. - Juraj
- The issue could be reproduced on Arm servers with the NIC with latest firmware version.
- Will try to reproduce the issue with x86 servers.
- Lab move for the fd.io lab
- Arm servers documented in https://gerrit.fd.io/r/c/csit/+/30662
- VPP device testing issues may be caused by XL710 i40e fw or kernel module.
- VPP Performance Test
- VPP
- In Ubuntu-20.04, VPP on Arm will reprioritize compiler as gcc-10 > gcc-9.2.0 > clang-10
- Multi-arch support - Lijian
- SOC ID will be available on /proc entry starting from kernel version 5.9 to differentiate vendor CPUs.
- Vector length specific patch is ready
- 128 and 256 fixed size vector wrappers are ready, needs verification
- Verify SVE vector length specific wrappers - Jieqiang
-
https://gerrit.fd.io/r/c/vpp/+/30326 - second proposal - not preferred
- Extend vector length agnostic opportunities
- Investigate VPP test cases in container
- Focus more on data-plane performance benchmarking and optimization - Tianyu
- Record the benchmarking results of VPP CNF 3 test cases in excel template
- Will do readout presentation with extented people - Tianyu
- Record the benchmarking results of VPP CNF 3 test cases in excel template
- Focus more on data-plane performance benchmarking and optimization - Tianyu
- Investigating VPP memif - Tianyu
- Benchmarking DPDK memif vs VPP memif
- IPsec on Arm platform. - Govind
- Apply prefetches and loop-unrolling with ipsec-out node
- Didn't observe much performance improvement (2%) so far
- Work on IPsec input node and VPP uses linear search on SPD lookup.
- SPD prototype change, introducing flow cache with hash, have performance improvement, discussing with community.
- Perfmon plugin enablement on Arm - Zach
- patch upstream has dependency on kernel patch
- Will try loop unrolling on the SPD lookup.
- Will investigate the prefetches and loop unrolling on IPsec input node. - Govind
- Apply prefetches and loop-unrolling with ipsec-out node
03/09/2021
-
Attendees
- Govindarajan Mohandoss
- Lijian Zhang
- Juraj Linkes
- Tianyu Li
- Jieqiang Wang
- Tina Tsou
-
CSIT
- VPP Performance Test
- https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-3n-tsh/
-
https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-report-iterative-2101-3n-tsh/
- CSIT official release 21.01 is available
-
https://docs.fd.io/csit/rls2101/report/
- Will investigate test case eth-l2xcbase-eth-2vhostvr1024-1vm - Jieqiang
- 20.09 vs 21.01 show run vector per call drop from 256 to 200 - need to check dpdk version changes
- Perf drop only observed for VM cases
- DPDK testpmd running inside VM, l2 cross connect running inside VPP.
- Check the number for CSIT 2101 release
- Will investigate test case eth-l2xcbase-eth-2vhostvr1024-1vm - Jieqiang
-
https://docs.fd.io/csit/rls2101/report/
- CSIT official release 21.01 is available
- Leverage current spare TX2 server as 2-node topology performance test-bed.
-
- Start with basic L2/L3/ACL/classifiers tests and use DPDK PMD on 2n-thx2 firstly(daily testing)
- Tests are running fine
- L2/L3 tests are running fine, IPSec tests are not supported on 2-node topo, ACL/Classifiers needs investigation.
- Suitable time to run release testing on 2n-tx2 testbed.
- Will investigate IPSec test cases on 2n-tx2 - Juraj
- Maintainer confirm that it is feasible
- Patch merged, https://gerrit.fd.io/r/c/csit/+/31309 p
- Patch created for daily running https://gerrit.fd.io/r/c/csit/+/31478
- crypto tests will be enabled on daily and report Jenkins job
- IPv6 / policy mode crypto test cases to be investigated and added
- Add memif test case to 2n-tx2 once the release testing is done.
- Take ~ 1 or 1.5 hour for one round of memif testing.
- Maintainer confirm that it is feasible
- release testing for 2n-tx2
- Performance data added to daily trending page
- https://docs.fd.io/csit/master/trending/introduction/dashboard.html#n-tx2
- Test cases include L2/IPv4/IPv6/Classifier/ACL
- Performance data added to daily trending page
-
- VPP Path
- CentOS-7 will be enabled with master branch for support lts release
- CentOS-7 Jenkins on Arm will not be supported.
- CentOS-8 will be supported by the end of this year by Redhat.
- CentOS-8 on Arm Jenkins is created and could be triggered manually with 'beta-verify' and 'beta-merge'
- CentOS-8 docker image on Arm is working fine, but not triggered per patch yet.
- Job is enabled https://jenkins.fd.io/job/vpp-verify-master-centos8-aarch64/
- Running per patch and voting right is enabled
- Maintainer ask for more servers for sake of redundancy
- Sync with Dave for ARM server requirement
- Job is enabled https://jenkins.fd.io/job/vpp-verify-master-centos8-aarch64/
- CentOS-8 docker image on Arm is working fine, but not triggered per patch yet.
- CentOS-7 will be enabled with master branch for support lts release
- VPP Device
- VPP device testing issues may be caused by XL710 i40e fw or kernel module.
- VPP community is responding this issue actively. - Juraj
- Working on a workaround to make sure not starting multiple VPP instances at the same time - Juraj
- Dependency on maintainers to fix this issue
- Intel will ship a new NIC with latest firmware
- Shipment takes a long time empirically
- NIC has been shipped to vexxhost, wait for NIC arrival.
- Try to reproduce the issue on this NIC on Arm platform
- Updating firmware on the current NIC is risky
- Shipment takes a long time empirically
- Voting rights will be enabled once this issue is fixed
-
- Maintainer raised the ticket to get intel people involved
- Will not update the firmware because the release testing is ongoing
-
- Current VPP device testing on TX2 is around 40 mins - 45 mins
- Which is acceptable by CSIT maintainers
- Arm is required to present Arm achievement and plan to TSC.
- Govind will prepare the slides
- Will show Arm roadmap in the next TSC meeting
- Proposals have been sent to VPP maintainer on verifying SVE/SVE2
- Lab move for the fd.io lab
- Arm servers documented in https://gerrit.fd.io/r/c/csit/+/30662
- VPP device testing issues may be caused by XL710 i40e fw or kernel module.
- VPP Performance Test
-
VPP
- In Ubuntu-20.04, VPP on Arm will reprioritize compiler as gcc-10 > gcc-9.2.0 > clang-10
- Multi-arch support - Lijian
- SOC ID will be available on /proc entry starting from kernel version 5.9 to differentiate vendor CPUs.
- Investigate VPP Intel AVF PMD driver - Lijian
- Initial benchmarking and analysis is done, and profiling result is recorded.
- Optimize ethernet-input and avf-input node with NEON intrinsics
- Benchmarking result shows some improvement from vectorization with ethernet-input and avf-input node
- avf-input node with neon optimization is merged.
- ethernet-input patch needs to split into two parts required by VPP maintainer
- Try cross-connect with AVF PMD driver to check avf-input node only - Lijian
- SVE/SVE2 two proposals are upstreamed, will discuss the proposals with maintainers
- Try to capture some software benchmarking results
-
https://gerrit.fd.io/r/c/vpp/+/29942 - first proposal - preferred
- 128 and 256 fixed size vector wrappers are ready, needs verification
- Verify SVE vector length specific wrappers - Jieqiang
-
https://gerrit.fd.io/r/c/vpp/+/30326 - second proposal - not preferred
- Extend vector length agnostic opportunities
- Investigate the scalable SIMD instructions on RISC-V - Lijian
- Investigate how to run traffic tests for VPP in docker - Lijian
- Plan to talk with VPP maintainers on this topic
- Benchmark the performance using AVF PMD driver on Ampere server. - Jieqiang
- Performance on Altra is about 30%-40% lower than 8268.
- Performance on Altra is slightly better than N1SDP.
- Analysis of benchmarking results for Ampere Altra
- A lot of context switches occur on Ampere Altra compared to N1SDP
- perf tools used to capture the perf events
- Talk with Ampere or N1 team on how to enable CMN-600 counters for ampere altra
- Remove interrupts on altra but no performance improvement seen
- instruction cache misses are higher on altra than N1
- Analysis of benchmarking results for Ampere Altra
- IO testing is doable with specific PCIe slots, with which PCIe bandwidth is not the bottle-neck.
- Investigate VPP test cases in container
- Focus more on data-plane performance benchmarking and optimization - Tianyu
- Record the benchmarking results of VPP CNF 3 test cases in excel template
- Focus more on data-plane performance benchmarking and optimization - Tianyu
- VPP compiling error on CentOS 7 - Jieqiang
-
https://gerrit.fd.io/r/c/vpp/+/31421
- CentOS 7 build issue has been fixed
- Developing NEON wrapper to SVE 128/256bit on qemu
-
https://gerrit.fd.io/r/c/vpp/+/31421
-
- IPsec on Arm platform. - Govind
- Apply prefetches and loop-unrolling with ipsec-out node
- Didn't observe much performance improvement (2%) so far
- Work on IPsec input node and VPP uses linear search on SPD lookup.
- SPD prototype change, introducing flow cache with hash, have performance improvement, discussing with community.
- perfmon plugin enablement on Arm - Zach
- patch upstream has dependency on kernel patch
- Will try loop unrolling on the SPD lookup.
- Will investigate the prefetches and loop unrolling on IPsec input node. - Govind
- Apply prefetches and loop-unrolling with ipsec-out node
- IPsec on Arm platform. - Govind
02/23/2021
-
Attendees
- Govindarajan Mohandoss
- Lijian Zhang
- Juraj Linkes
- Tianyu Li
- Jieqiang Wang
- Tina Tsou
-
CSIT
- VPP Performance Test
- https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-3n-tsh/
-
https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-report-iterative-2101-3n-tsh/
- CSIT official release 21.01 is available
-
https://docs.fd.io/csit/rls2101/report/
- Will investigate test case eth-l2xcbase-eth-2vhostvr1024-1vm - Jieqiang
- DPDK testpmd running inside VM, l2 cross connect running inside VPP.
- Check the number for CSIT 2101 release
-
https://docs.fd.io/csit/rls2101/report/
- CSIT official release 21.01 is available
- Leverage current spare TX2 server as 2-node topology performance test-bed.
-
- Start with basic L2/L3/ACL/classifiers tests and use DPDK PMD on 2n-thx2 firstly(daily testing)
- Tests are running fine
- L2/L3 tests are running fine, IPSec tests are not supported on 2-node topo, ACL/Classifiers needs investigation.
- Suitable time to run release testing on 2n-tx2 testbed.
- Will investigate IPSec test cases on 2n-tx2 - Juraj
- Maintainer confirm that it is feasible
- Patch created, https://gerrit.fd.io/r/c/csit/+/31309
- crypto tests will be enabled on daily and report Jenkins job
- Add memif test case to 2n-tx2 once the release testing is done.
- Take ~ 1 or 1.5 hour for one round of memif testing.
- release testing for 2n-tx2
- Performance data added to daily trending page
- https://docs.fd.io/csit/master/trending/introduction/dashboard.html#n-tx2
- Test cases include L2/IPv4/IPv6/Classifier/ACL
- Release testing done for 2n-tx2, ongoing for 3n-tsh(due to next week)
- Release report plan to be published on 10th Feb
- Performance data added to daily trending page
-
- VPP Path
- CentOS-7 will be enabled with master branch for support lts release
- CentOS-7 Jenkins on Arm will be supported.
- CentOS-8 on Arm Jenkins is created and could be triggered manually with 'beta-verify' and 'beta-merge'
- CentOS-8 docker image on Arm is working fine, but not triggered per patch yet.
- Job is enabled https://jenkins.fd.io/job/vpp-verify-master-centos8-aarch64/
- Running per patch and voting right is enabled
- Maintainer ask for more servers for sake of redundancy
- Job is enabled https://jenkins.fd.io/job/vpp-verify-master-centos8-aarch64/
- CentOS-8 docker image on Arm is working fine, but not triggered per patch yet.
- CentOS-7 will be enabled with master branch for support lts release
- VPP Device
- VPP device testing issues may be caused by XL710 i40e fw or kernel module.
- VPP community is responding this issue actively. - Juraj
- Working on a workaround to make sure not starting multiple VPP instances at the same time - Juraj
- Dependency on maintainers to fix this issue
- Intel will ship a new NIC with latest firmware
- Shipment takes a long time empirically
- Try to reproduce the issue on this NIC on Arm platform
- Updating firmware on the current NIC is risky
- Voting rights will be enabled once this issue is fixed
-
- Maintainer raised the ticket to get intel people involved
- Will not update the firmware because the release testing is ongoing
-
- Current VPP device testing on TX2 is around 40 mins - 45 mins
- Which is acceptable by CSIT maintainers
- Arm is required to present Arm achievement and plan to TSC.
- Govind will prepare the slides
- Verify SVE/SVE2 features inside QEMU docker - Lijian
- Latest VPP binary crash on the QEMU docker
- System call fails inside QEMU docker when running VPP
- Verify SVE/SVE2 features inside ARM QEMU VM
- VPP maintainers want real hardware to verify SVE code
- This solution will be abandoned.
- 'make test' execution is slow
- Sync with DPDK team/VPP community to decide the solution
- Proposals have been sent to VPP maintainer on verifying SVE/SVE2
- Latest VPP binary crash on the QEMU docker
- Lab move for the fd.io lab
- Arm servers documented in https://gerrit.fd.io/r/c/csit/+/30662
- VPP device testing issues may be caused by XL710 i40e fw or kernel module.
- VPP Performance Test
-
VPP
- In Ubuntu-20.04, VPP on Arm will reprioritize compiler as gcc-10 > gcc-9.2.0 > clang-10
- Multi-arch support - Lijian
- SOC ID will be available on /proc entry starting from kernel version 5.9 to differentiate vendor CPUs.
- Investigate VPP Intel AVF PMD driver - Lijian
- Initial benchmarking and analysis is done, and profiling result is recorded.
- Optimize ethernet-input and avf-input node with NEON intrinsics
- Benchmarking result shows some improvement from vectorization with ethernet-input and avf-input node
- avf-input node with neon optimization is merged.
- ethernet-input patch needs to split into two parts required by VPP maintainer
- Try cross-connect with AVF PMD driver to check avf-input node only - Lijian
- SVE/SVE2 two proposals are upstreamed, will discuss the proposals with maintainers
- Try to capture some software benchmarking results
-
https://gerrit.fd.io/r/c/vpp/+/29942 - first proposal - preferred
- 128 and 256 fixed size vector wrappers are ready, needs verification
- Verify SVE vector length specific wrappers - Jieqiang
-
https://gerrit.fd.io/r/c/vpp/+/30326 - second proposal - not preferred
- Extend vector length agnostic opportunities
- Investigate the scalable SIMD instructions on RISC-V - Lijian
- Investigate how to run traffic tests for VPP in docker - Lijian
- Plan to talk with VPP maintainers on this topic
- Repeat the 4x and 2x loop unrolling tests on Ampere server for L3 forwarding with Lijian's patch.
- No positive for 4x loop unrolling on Ampere, so will keep 2x unrolling for Neoverse N1
- Benchmark the performance using AVF PMD driver on Ampere server. - Jieqiang
- Performance on Altra is about 30%-40% lower than 8268.
- Performance on Altra is slightly better than N1SDP.
- Analysis of benchmarking results for Ampere Altra
- A lot of context switches occur on Ampere Altra compared to N1SDP
- perf tools used to capture the perf events
- Talk with Ampere or N1 team on how to enable CMN-600 counters for ampere altra
- Remove interrupts on altra but no performance improvement seen
- instruction cache misses are higher on altra than N1
- Analysis of benchmarking results for Ampere Altra
- IO testing is doable with specific PCIe slots, with which PCIe bandwidth is not the bottle-neck.
- Investigate VPP test cases in container
- Investigate VPP memif test cases in VPP CSIT - Jieqiang
- Investigate VPP use cases proposals in containers - Tianyu
- 3 use cases are investigated.
- Will explore the memif logic and share the progress.
- Will share the link on details about how to run VPP in container.
- Investigate VPP agent usage - Tianyu
- Focus more on data-plane performance benchmarking and optimization - Tianyu
- VPP compiling error on CentOS 7 - Jieqiang
- IPsec on Arm platform. - Govind
- Apply prefetches and loop-unrolling with ipsec-out node
- Didn't observe much performance improvement (2%) so far
- Work on IPsec input node and VPP uses linear search on SPD lookup.
- perfmon plugin enablement on Arm - Zach
- patch upstream has dependency on kernel patch
- Will try loop unrolling on the SPD lookup.
- Will investigate the prefetches and loop unrolling on IPsec input node. - Govind
- perfmon plugin enablement on Arm - Zach
- Apply prefetches and loop-unrolling with ipsec-out node
02/09/2021
- Attendees
- Govindarajan Mohandoss
- Lijian Zhang
- Juraj Linkes
- Tianyu Li
- Jieqiang Wang
- Tina Tsou
- CSIT
- VPP Performance Test
- https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-3n-tsh/
-
https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-report-iterative-2101-3n-tsh/
- CSIT official release 20.09 is available
- CSIT official release 21.01 is ongoing
-
- Will investigate test case eth-l2xcbase-eth-2vhostvr1024-1vm - Jieqiang
- DPDK testpmd running inside VM, l2 cross connect running inside VPP.
- Check the number for CSIT 2101 release
-
- Leverage current spare TX2 server as 2-node topology performance test-bed.
-
- Start with basic L2/L3/ACL/classifiers tests and use DPDK PMD on 2n-thx2 firstly(daily testing)
- Tests are running fine
- L2/L3 tests are running fine, IPSec tests are not supported on 2-node topo, ACL/Classifiers needs investigation.
- Suitable time to run release testing on 2n-tx2 testbed.
- Will investigate IPSec test cases on 2n-tx2 - Juraj
- Maintainer confirm that it is feasible
- Add memif test case to 2n-tx2 once the release testing is done.
- release testing for 2n-tx2
- Performance data added to daily trending page
- https://docs.fd.io/csit/master/trending/introduction/dashboard.html#n-tx2
- Test cases include L2/IPv4/IPv6/Classifier/ACL
- Release testing done for 2n-tx2, ongoing for 3n-tsh(due to next week)
- Release report plan to be published on 10th Feb
- Performance data added to daily trending page
-
- VPP Path
- CentOS-7 will be enabled with master branch for support lts release
- CentOS-7 Jenkins on Arm will be supported.
- CentOS-8 on Arm Jenkins is created and could be triggered manually with 'beta-verify' and 'beta-merge'
- CentOS-8 docker image on Arm is working fine, but not triggered per patch yet.
- Job is enabled https://jenkins.fd.io/job/vpp-verify-master-centos8-aarch64/
- Running per patch and voting right is enabled
- Will ask Dave if he needs help with testing CentOS-8 on Arm - Juraj.
- Will verify the image uploaded by Dave if it is ready.
- Job is enabled https://jenkins.fd.io/job/vpp-verify-master-centos8-aarch64/
- https://gerrit.fd.io/r/c/ci-management/+/28960
- https://gerrit.fd.io/r/c/ci-management/+/28022
- Jenkins job to verify runs fine but slow
- https://gerrit.fd.io/r/c/ci-management/+/31083
- Maintainer ask for more servers for sake of redundancy
- 'make test' failure on ubuntu 20.04 AARCH64
- Dave has sent email for the details
- CentOS-8 docker image on Arm is working fine, but not triggered per patch yet.
- CentOS-7 will be enabled with master branch for support lts release
- VPP Device
- VPP device testing issues may be caused by XL710 i40e fw or kernel module.
- VPP community is responding this issue actively. - Juraj
- Working on a workaround to make sure not starting multiple VPP instances at the same time - Juraj
- Dependency on maintainers to fix this issue
- Intel will ship a new NIC with latest firmware
- Shippment takes a long time empirically
- Try to reproduce the issue on this NIC on Arm platform
- Updating firmware on the current NIC is risky
- Voting rights will be enabled once this issue is fixed
-
- Maintainer raised the ticket to get intel people involved
- Will not update the firmware because the release testing is ongoing
-
- Current VPP device testing on TX2 is around 40 mins - 45 mins
- Which is acceptable by CSIT maintainers
- Arm is required to present Arm achievement and plan to TSC.
- Govind will prepare the slides
- Verify SVE/SVE2 features inside QEMU docker - Lijian
- Latest VPP binary crash on the QEMU docker
- System call fails inside QEMU docker when running VPP
- Verify SVE/SVE2 features inside ARM QEMU VM
- 'make test' execution is slow
- Sync with DPDK team/VPP community to decide the solution
- Proposals have been sent to VPP maintainer on verifying SVE/SVE2
- Latest VPP binary crash on the QEMU docker
- Lab move for the fd.io lab
- Arm servers documented in https://gerrit.fd.io/r/c/csit/+/30662
- VPP device testing issues may be caused by XL710 i40e fw or kernel module.
- VPP Performance Test
- VPP
- In Ubuntu-20.04, VPP on Arm will reprioritize compiler as gcc-10 > gcc-9.2.0 > clang-10
- Multi-arch support - Lijian
- SOC ID will be available on /proc entry starting from kernel version 5.9 to differentiate vendor CPUs.
- Investigate VPP Intel AVF PMD driver - Lijian
- Initial benchmarking and analysis is done, and profiling result is recorded.
- Optimize ethernet-input and avf-input node with NEON intrinsics
- Benchmarking result shows some improvement from vectorization with ethernet-input and avf-input node
- avf-input node with neon optimization is merged.
- ethernet-input patch needs to split into two parts required by VPP maintainer
- Try cross-connect with AVF PMD driver to check avf-input node only - Lijian
- SVE/SVE2 two proposals are upstreamed, will discuss the proposals with maintainers
- Try to capture some software benchmarking results
- https://gerrit.fd.io/r/c/vpp/+/29942 - first proposal - preferred
- https://gerrit.fd.io/r/c/vpp/+/30326 - second proposal - not preferred
- Investigate the scalable SIMD instructions on RISC-V - Lijian
- Investigate how to run traffic tests for VPP in docker - Lijian
- Plan to talk with VPP maintainers on this topic
- Repeat the 4x and 2x loop unrolling tests on Ampere server for L3 forwarding with Lijian's patch.
- No positive for 4x loop unrolling on Ampere, so will keep 2x unrolling for Neoverse N1
- Benchmark the performance using AVF PMD driver on Ampere server. - Jieqiang
- Performance on Altra is about 30%-40% lower than 8268.
- Performance on Altra is slightly better than N1SDP.
- Analysis of benchmarking results for Ampere Altra
- A lot of context switches occur on Ampere Altra compared to N1SDP
- perf tools used to capture the perf events
- Talk with Ampere or N1 team on how to enable CMN-600 counters for ampere altra
- Remove interrupts on altra but no performance improvement seen
- instruction cache misses are higher on altra than N1
- Analysis of benchmarking results for Ampere Altra
- IO testing is doable with specific PCIe slots, with which PCIe bandwidth is not the bottle-neck.
- Investigate VPP test cases in container
- Investigate VPP memif test cases in VPP CSIT - Jieqiang
- Investigate VPP use cases proposals in containers - Tianyu
- 3 use cases are investigated.
- Will explore the memif logic and share the progress.
- Will share the link on details about how to run VPP in container.
- Investigate VPP agent usage - Tianyu
- Benchmark Altra vs Cascade 8268
- IPsec on Arm platform. - Govind
- Apply prefetches and loop-unrolling with ipsec-out node
- Didn't observe much performance improvement (2%) so far
- Work on IPsec input node and VPP uses linear search on SPD lookup.
- Will try loop unrolling on the SPD lookup.
- Will investigate the prefetches and loop unrolling on IPsec input node. - Govind
- Apply prefetches and loop-unrolling with ipsec-out node
02/02/2021
- Attendees
- Govindarajan Mohandoss
- Lijian Zhang
- Juraj Linkes
- Tianyu Li
- Jieqiang Wang
- Tina Tsou
- CSIT
- VPP Performance Test
- https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-3n-tsh/
-
https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-report-iterative-2101-3n-tsh/
- CSIT official release 20.09 is available
- CSIT official release 21.01 is ongoing
-
- Will investigate test case eth-l2xcbase-eth-2vhostvr1024-1vm - Jieqiang
- DPDK testpmd running inside VM, l2 cross connect running inside VPP.
- Check the number for CSIT 2101 release
-
- Leverage current spare TX2 server as 2-node topology performance test-bed.
-
- Start with basic L2/L3/ACL/classifiers tests and use DPDK PMD on 2n-thx2 firstly(daily testing)
- Tests are running fine
- L2/L3 tests are running fine, IPSec tests are not supported on 2-node topo, ACL/Classifiers needs investigation.
- Suitable time to run release testing on 2n-tx2 testbed.
- Will investigate IPSec test cases on 2n-tx2 - Juraj
- Maintainer confirm that it is feasible
- Add memif test case to 2n-tx2 once the release testing is done.
- release testing for 2n-tx2
- Performance data added to daily trending page
- https://docs.fd.io/csit/master/trending/introduction/dashboard.html#n-tx2
- Test cases include L2/IPv4/IPv6/Classifier/ACL
- Release report plan to be published on 10th Feb
- Performance data added to daily trending page
-
- VPP Path
- CentOS-7 will be enabled with master branch for support lts release
- CentOS-7 Jenkins on Arm will be supported.
- CentOS-8 on Arm Jenkins is created and could be triggered manually with 'beta-verify' and 'beta-merge'
- CentOS-8 docker image on Arm is working fine, but not triggered per patch yet.
- Will ask Dave if he needs help with testing CentOS-8 on Arm - Juraj.
- Will verify the image uploaded by Dave if it is ready.
- Will ask Dave if he needs help with testing CentOS-8 on Arm - Juraj.
- https://gerrit.fd.io/r/c/ci-management/+/28960
- https://gerrit.fd.io/r/c/ci-management/+/28022
- Jenkins job to verify runs fine but slow
- https://gerrit.fd.io/r/c/ci-management/+/31083
- Maintainer ask for more servers for sake of redundancy
- CentOS-8 docker image on Arm is working fine, but not triggered per patch yet.
- CentOS-7 will be enabled with master branch for support lts release
- VPP Device
- VPP device testing issues may be caused by XL710 i40e fw or kernel module.
- VPP community is responding this issue actively. - Juraj
- Working on a workaround to make sure not starting multiple VPP instances at the same time - Juraj
- Dependency on maintainers to fix this issue
- Voting rights will be enabled once this issue is fixed
- Implementation is ready, and will do test it with actual patches.
- Apply file locking mechanism to allow that only one VPP instances are running.
- https://gerrit.fd.io/r/c/csit/+/30425
- Patches are under review
- Maintainer raised the ticket to get intel people involved
- Will not update the firmware because the release testing is ongoing
- Current VPP device testing on TX2 is around 40 mins - 45 mins
- Which is acceptable by CSIT maintainers
- Arm is required to present Arm achievement and plan to TSC.
- Govind will prepare the slides
- Verify SVE/SVE2 features inside QEMU docker - Lijian
- Latest VPP binary crash on the QEMU docker
- System call fails inside QEMU docker when running VPP
- Verify SVE/SVE2 features inside ARM QEMU VM
- 'make test' execution is slow
- Sync with DPDK team/VPP community to decide the solution
- Proposals have been sent to VPP maintainer on verifying SVE/SVE2
- Latest VPP binary crash on the QEMU docker
- Lab move for the fd.io lab
- Arm servers documented in https://gerrit.fd.io/r/c/csit/+/30662
- VPP device testing issues may be caused by XL710 i40e fw or kernel module.
- VPP Performance Test
- VPP
- In Ubuntu-20.04, VPP on Arm will reprioritize compiler as gcc-10 > gcc-9.2.0 > clang-10
- Multi-arch support - Lijian
- SOC ID will be available on /proc entry starting from kernel version 5.9 to differentiate vendor CPUs.
- Investigate VPP Intel AVF PMD driver - Lijian
- Initial benchmarking and analysis is done, and profiling result is recorded.
- Optimize ethernet-input and avf-input node with NEON intrinsics
- Benchmarking result shows some improvement from vectorization with ethernet-input and avf-input node
- avf-input node with neon optimization is merged.
- ethernet-input patch needs to split into two parts required by VPP maintainer
- Try cross-connect with AVF PMD driver to check avf-input node only - Lijian
- SVE/SVE2 two proposals are upstreamed, will discuss the proposals with maintainers
- Try to capture some software benchmarking results
- https://gerrit.fd.io/r/c/vpp/+/29942 - first proposal - preferred
- https://gerrit.fd.io/r/c/vpp/+/30326 - second proposal - not preferred
- Investigate the scalable SIMD instructions on RISC-V - Lijian
- Investigate how to run traffic tests for VPP in docker - Lijian
- Plan to talk with VPP maintainers on this topic
- Repeat the 4x and 2x loop unrolling tests on Ampere server for L3 forwarding with Lijian's patch.
- No positive for 4x loop unrolling on Ampere, so will keep 2x unrolling for Neoverse N1
- Benchmark the performance using AVF PMD driver on Ampere server. - Jieqiang
- Performance on Altra is about 30%-40% lower than 8268.
- Performance on Altra is slightly better than N1SDP.
- Analysis of benchmarking results for Ampere Altra
- A lot of context switches occur on Ampere Altra compared to N1SDP
- perf tools used to capture the perf events
- Talk with Ampere or N1 team on how to enable CMN-600 counters for ampere altra
- Remove interrupts on altra but no performance improvement seen
- instruction cache misses are higher on altra than N1
- Analysis of benchmarking results for Ampere Altra
- IO testing is doable with specific PCIe slots, with which PCIe bandwidth is not the bottle-neck.
- Investigate VPP test cases in container
- Investigate VPP memif test cases in VPP CSIT - Jieqiang
- Investigate VPP use cases proposals in containers - Tianyu
- 3 use cases are investigated.
- Will explore the memif logic and share the progress.
- Will share the link on details about how to run VPP in container.
- Investigate VPP agent usage - Tianyu
- Benchmark Altra vs Cascade 8268
- IPsec on Arm platform. - Govind
- Apply prefetches and loop-unrolling with ipsec-out node
- Didn't observe much performance improvement (2%) so far
- Work on IPsec input node and VPP uses linear search on SPD lookup.
- Will try loop unrolling on the SPD lookup.
- Will investigate the prefetches and loop unrolling on IPsec input node. - Govind
- Apply prefetches and loop-unrolling with ipsec-out node
01/19/2021
- Attendees
- Govindarajan Mohandoss
- Lijian Zhang
- Juraj Linkes
- Tianyu Li
- Jieqiang Wang
- Tina Tsou
- CSIT
- VPP Performance Test
- https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-3n-tsh/
-
https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-report-iterative-2101-3n-tsh/
- CSIT official release 20.09 is available
- https://docs.fd.io/csit/rls2009/report/
- Jieqiang will compare the performance data with release 20.09
- Will investigate test case eth-l2xcbase-eth-2vhostvr1024-1vm - Jieqiang
- DPDK testpmd running inside VM, l2 cross connect running inside VPP.
- Check the number for CSIT 2101 release
- CSIT official release 20.09 is available
- Leverage current spare TX2 server as 2-node topology performance test-bed.
- Hardware configurations/wiring are done; Physical connection to the TG is done.
- almost done, two steps need to be done
- start with basic L2/L3/ACL/classifiers tests and use DPDK PMD on 2n-thx2 firstly(daily testing)
- Take the execution time into consideration if we want run release testing on 2n-thx2.
- It takes 9 hours to finish the one round testing.
- Tests are running fine
- L2/L3 tests are running fine, IPSec tests are not supported on 2-node topo, ACL/Classifiers needs investigation.
- Suitable time to run release testing on 2n-tx2 testbed.
- Will investigate IPSec test cases on 2n-tx2 - Juraj
- Add memif test case to 2n-tx2 once the release testing is done.
- VPP Path
- CentOS-7 will be enabled with master branch for support lts release
- CentOS-7 Jenkins on Arm will be supported.
- CentOS-8 on Arm Jenkins is created and could be triggered manually with 'beta-verify' and 'beta-merge'
- CentOS-8 docker image on Arm is working fine, but not triggered per patch yet.
- Will ask Dave if he needs help with testing CentOS-8 on Arm - Juraj.
- Will verify the image uploaded by Dave if it is ready.
- Will ask Dave if he needs help with testing CentOS-8 on Arm - Juraj.
- https://gerrit.fd.io/r/c/ci-management/+/28960
- https://gerrit.fd.io/r/c/ci-management/+/28022
- CentOS-8 docker image on Arm is working fine, but not triggered per patch yet.
- CentOS-7 will be enabled with master branch for support lts release
- VPP Device
- VPP device testing issues may be caused by XL710 i40e fw or kernel module.
- VPP community is responding this issue actively. - Juraj
- Working on a workaround to make sure not starting multiple VPP instances at the same time - Juraj
- Implementation is ready, and will do test it with actual patches.
- Apply file locking mechanism to allow that only one VPP instances are running.
- https://gerrit.fd.io/r/c/csit/+/30425
- Patches are under review
- Machiek raised the ticket to get intel people involved
- Will not update the firmaware because the release testing is ongoing
- Current VPP device testing on TX2 is around 40 mins - 45 mins
- Which is acceptable by CSIT maintainers
- Arm is required to present Arm achievement and plan to TSC.
- Govind will prepare the slides
- Verify SVE/SVE2 features inside QEMU docker
- Latest VPP binary crash on the QEMU docker - Lijian
- Lab move for the fd.io lab
- Arm servers documented in https://gerrit.fd.io/r/c/csit/+/30662
- VPP device testing issues may be caused by XL710 i40e fw or kernel module.
- VPP Performance Test
- VPP
- In Ubuntu-20.04, VPP on Arm will reprioritize compiler as gcc-10 > gcc-9.2.0 > clang-10
- Multi-arch support - Lijian
- SOC ID will be available on /proc entry starting from kernel version 5.9 to differentiate vendor CPUs.
- Investigate VPP Intel AVF PMD driver - Lijian
- Initial benchmarking and analysis is done, and profiling result is recorded.
- Optimize ethernet-input and avf-input node with NEON intrinsics
- Benchmarking result shows some improvement from vectorization with ethernet-input and avf-input node
- avf-input node with neon optimization is merged.
- ethernet-input patch needs to split into two parts required by VPP maintainer
- Try cross-connect with AVF PMD driver to check avf-input node only - Lijian
- SVE/SVE2 two proposals are upstreamed, will discuss the proposals with maintainers
- Try to capture some software benchmarking results
- https://gerrit.fd.io/r/c/vpp/+/29942 - first proposal - preferred
- https://gerrit.fd.io/r/c/vpp/+/30326 - second proposal - not preferred
- Investigate the scalable SIMD instructions on RISC-V - Lijian
- Investigate how to run traffic tests for VPP in docker - Lijian
- Plan to talk with VPP maintainers on this topic
- Repeat the 4x and 2x loop unrolling tests on Ampere server for L3 forwarding with Lijian's patch.
- No positive for 4x loop unrolling on Ampere, so will keep 2x unrolling for Neoverse N1
- Benchmark the performance using AVF PMD driver on Ampere server. - Jieqiang
- Performance on Altra is about 30%-40% lower than 8268.
- Performance on Altra is slightly better than N1SDP.
- Analysis of benchmarking results for Ampere Altra
- A lot of context switches occur on Ampere Altra compared to N1SDP
- perf tools used to capture the perf events
- Talk with Ampere or N1 team on how to enable CMN-600 counters for ampere altra
- Analysis of benchmarking results for Ampere Altra
- IO testing is doable with specific PCIe slots, with which PCIe bandwidth is not the bottle-neck.
- Investigate VPP test cases in container
- Investigate VPP memif test cases in VPP CSIT - Jieqiang
- Investigate VPP use cases proposals in containers - Tianyu
- 3 use cases are investigated.
- Will explore the memif logic and share the progress.
- Will share the link on details about how to run VPP in container.
- Benchmark Altra vs Cascade 8268
- IPsec on Arm platform. - Govind
- Apply prefetches and loop-unrolling with ipsec-out node
- Didn't observe much performance improvement (2%) so far
- Work on IPsec input node and VPP uses linear search on SPD lookup.
- Will try loop unrolling on the SPD lookup.
- Will investigate the prefetches and loop unrolling on IPsec input node. - Govind
- Apply prefetches and loop-unrolling with ipsec-out node
01/05/2021
- Attendees
- Govindarajan Mohandoss
- Lijian Zhang
- Juraj Linkes
- Jieqiang Wang
- Tianyu Li
- Tina Tsou
- CSIT
- VPP Performance Test
- https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-3n-tsh/
-
https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-report-iterative-2009-3n-tsh/
- CSIT official release 20.09 is available
- https://docs.fd.io/csit/rls2009/report/
- Jieqiang will compare the performance data with release 20.09
- Will investigate test case eth-l2xcbase-eth-2vhostvr1024-1vm - Jieqiang
- DPDK testpmd running inside VM, l2 cross connect running inside VPP.
- CSIT official release 20.09 is available
- Leverage current spare TX2 server as 2-node topology performance test-bed.
- Hardware configurations/wiring are done; Physical connection to the TG is done.
- almost done, two steps need to be done
- start with basic L2/L3/IPSec/ACL/classifiers tests and use DPDK PMD on 2n-thx2 firstly(daily testing)
- Take the execution time into consideration if we want run release testing on 2n-thx2.
- Tests are running fine
- L2/L3 tests are running fine, IPSec tests are not supported on 2-node topo, ACL/Classifiers needs investigation.
- Suitable time to run release testing on 2n-tx2 testbed.
- VPP Path
- CentOS-7 will be enabled with master branch for support lts release
- CentOS-7 Jenkins on Arm will be supported.
- CentOS-8 on Arm Jenkins is created and could be triggered manually with 'beta-verify' and 'beta-merge'
- CentOS-8 docker image on Arm is working fine, but not triggered per patch yet.
- Will ask Dave if he needs help with testing CentOS-8 on Arm - Juraj.
- https://gerrit.fd.io/r/c/ci-management/+/28960
- https://gerrit.fd.io/r/c/ci-management/+/28022
- CentOS-8 docker image on Arm is working fine, but not triggered per patch yet.
- CentOS-7 will be enabled with master branch for support lts release
- VPP Device
- VPP device testing issues may be caused by XL710 i40e fw or kernel module.
- VPP community is responding this issue actively. - Juraj
- Working on a workaround to make sure not starting multiple VPP instances at the same time - Juraj
- Implementation is ready, and will do test it with actual patches.
- Apply file locking mechanism to allow that only one VPP instances are running.
- https://gerrit.fd.io/r/c/csit/+/30425
- Patches are under review
- Current VPP device testing on TX2 is around 40 mins - 45 mins
- Which is acceptable by CSIT maintainers
- Arm is required to present Arm achievement and plan to TSC.
- Govind will prepare the slides
- Verify SVE/SVE2 features inside QEMU docker
- Latest VPP binary crash on the QEMU docker - Lijian
- VPP device testing issues may be caused by XL710 i40e fw or kernel module.
- VPP Performance Test
- VPP
- In Ubuntu-20.04, VPP on Arm will reprioritize compiler as gcc-10 > gcc-9.2.0 > clang-10
- Multi-arch support - Lijian
- SOC ID will be available on /proc entry starting from kernel version 5.9 to differentiate vendor CPUs.
- Investigate VPP Intel AVF PMD driver - Lijian
- Initial benchmarking and analysis is done, and profiling result is recorded.
- Optimize ethernet-input and avf-input node with NEON intrinsics
- Benchmarking result shows some improvement from vectorization with ethernet-input and avf-input node
- avf-input node with neon optimization is merged.
- ethernet-input patch needs to split into two parts required by VPP maintainer
- Try cross-connect with AVF PMD driver to check avf-input node only - Lijian
- SVE/SVE2 two proposals are upstreamed, will discuss the proposals with maintainers
- Try to capture some software benchmarking results
- https://gerrit.fd.io/r/c/vpp/+/29942 - first proposal - preferred
- https://gerrit.fd.io/r/c/vpp/+/30326 - second proposal - not preferred
- Investigate the scalable SIMD instructions on RISC-V - Lijian
- Investigate how to run traffic tests for VPP in docker - Lijian
- Plan to talk with VPP maintainers on this topic
- Repeat the 4x and 2x loop unrolling tests on Ampere server for L3 forwarding with Lijian's patch.
- No positive for 4x loop unrolling on Ampere, so will keep 2x unrolling for Neoverse N1
- Benchmark the performance using AVF PMD driver on Ampere server. - Jieqiang
- Performance on Altra is about 30%-40% lower than 8268.
- Performance on Altra is slightly better than N1SDP.
- IO testing is doable with specific PCIe slots, with which PCIe bandwidth is not the bottle-neck.
- Investigate VPP test cases in container
- Investigate VPP test cases in VPP CSIT - Jieqiang
- Investigate VPP use cases proposals in containers - Tianyu
- Benchmark Altra vs Cascade 8268
- IPsec on Arm platform. - Govind
- Apply prefetches and loop-unrolling with ipsec-out node
- Didn't observe much performance improvement (2%) so far
- Work on IPsec input node and VPP uses linear search on SPD lookup.
- Will try loop unrolling on the SPD lookup.
- Will investigate the prefetches and loop unrolling on IPsec input node. - Govind
- Apply prefetches and loop-unrolling with ipsec-out node
12/22/2020
- Attendees
- Govindarajan Mohandoss
- Lijian Zhang
- Juraj Linkes
- Jieqiang Wang
- Tina Tsou
- General
- Will cancel the meeting on Dec 29th;
- CSIT
- VPP Performance Test
- https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-3n-tsh/
-
https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-report-iterative-2009-3n-tsh/
- CSIT official release 20.09 is available
- https://docs.fd.io/csit/rls2009/report/
- Jieqiang will compare the performance data with release 20.05
- CSIT official release 20.09 is available
- Leverage current spare TX2 server as 2-node topology performance test-bed.
- Hardware configurations/wiring are done; Physical connection to the TG is done.
- almost done, two steps need to be done
- codes to update Jenkins job needs to be merged
- start with basic L2/L3/IPSec/ACL/classifiers tests and use DPDK PMD on 2n-thx2 firstly(daily testing)
- Take the execution time into consideration if we want run release testing on 2n-thx2.
- VPP Path
- CentOS-7 will be enabled with master branch for support lts release
- CentOS-7 Jenkins on Arm will be supported.
- CentOS-8 on Arm Jenkins is created and could be triggered manually with 'beta-verify' and 'beta-merge'
- CentOS-8 docker image on Arm is working fine, but not triggered per patch yet.
- Will ask Dave if he needs help with testing CentOS-8 on Arm - Juraj.
- https://gerrit.fd.io/r/c/ci-management/+/28960
- https://gerrit.fd.io/r/c/ci-management/+/28022
- CentOS-8 docker image on Arm is working fine, but not triggered per patch yet.
- CentOS-7 will be enabled with master branch for support lts release
- VPP Device
- VPP device testing issues may be caused by XL710 i40e fw or kernel module.
- VPP community is responding this issue actively. - Juraj
- Working on a workaround to make sure not starting multiple VPP instances at the same time - Juraj
- Implementation is ready, and will do test it with actual patches.
- Apply file locking mechanism to allow that only one VPP instances are running.
- Current VPP device testing on TX2 is around 40 mins - 45 mins
- Which is acceptable by CSIT maintainers
- LF will provide QSFP+ fiber switch for FD.io lab.
- Basically done. LF just procured the existing fiber switch currently rent by Arm in FD.io lab.
- Send the progress to relavent people in Arm - Lijian
- Confirm with Tina to ensure Arm is not charged - Lijian
- Arm is required to present Arm achievement and plan to TSC.
- Govind will prepare the slides
- Verify SVE/SVE2 features on VPP CSIT
- VPP device testing issues may be caused by XL710 i40e fw or kernel module.
- VPP Performance Test
- VPP
- In Ubuntu-20.04, VPP on Arm will reprioritize compiler as gcc-10 > gcc-9.2.0 > clang-10
- Multi-arch support - Lijian
- SOC ID will be available on /proc entry starting from kernel version 5.9 to differentiate vendor CPUs.
- Investigate VPP Intel AVF PMD driver - Lijian
- Initial benchmarking and analysis is done, and profiling result is recorded.
- Optimize ethernet-input and avf-input node with NEON intrinsics
- Benchmarking result shows some improvement from vectorization with ethernet-input and avf-input node
- avf-input node with neon optimization is merged.
- ethernet-input patch needs to split into two parts required by VPP maintainer
- Try cross-connect with AVF PMD driver to check avf-input node only - Lijian
- SVE/SVE2 two proposals are upstreamed, will discuss the proposals with maintainers
- Try to capture some software benchmarking results
- https://gerrit.fd.io/r/c/vpp/+/29942 - first proposal - preferred
- https://gerrit.fd.io/r/c/vpp/+/30326 - second proposal - not preferred
- Investigate the scalable SIMD instructions on RISC-V - Lijian
- Investigate how to run traffic tests for VPP in docker - Lijian
- Repeat the 4x and 2x loop unrolling tests on Ampere server for L3 forwarding with Lijian's patch.
- No positive for 4x loop unrolling on Ampere, so will keep 2x unrolling for Neoverse N1
- Benchmark the performance using AVF PMD driver on Ampere server. - Jieqiang
- Performance on Altra is about 30%-40% lower than 8268.
- Performance on Altra is slightly better than N1SDP.
- IO testing is doable with specific PCIe slots, with which PCIe bandwidth is not the bottle-neck.
- Try the VFs with DPDK plugin. - Jieqiang
- Repeat scalability testing on N1SDP, e.g., 1T1C, 2T2C, 3T3C
- Benchmark Altra vs Cascade 8268
- IPsec on Arm platform. - Govind
- Apply prefetches and loop-unrolling with ipsec-out node
- Didn't observe much performance improvement (2%) so far
- Work on IPsec input node and VPP uses linear search on SPD lookup.
- Will try loop unrolling on the SPD lookup.
- Tune PMD config to achieve zero cache eviction for System Level Cache on N1SDP - kamalakshitha
- Tuned version has higher SLC eviction than the default version, talk with CPU team to figure out the reason.
- Review akshitha's PPT on SLC eviction and share it with the team. - Govind
- Prefetches on IPsec and learn about the cache behavior. - kamalakshitha
- Will investigate the prefetches and loop unrolling on IPsec input node. - Govind
- Apply prefetches and loop-unrolling with ipsec-out node
12/15/2020
- Attendees
- Govindarajan Mohandoss
- Lijian Zhang
- Juraj Linkes
- Jieqiang Wang
- Tina Tsou
- General
- Will cancel the meeting on Dec 29th;
- CSIT
- VPP Performance Test
- https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-3n-tsh/
-
https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-report-iterative-2009-3n-tsh/
- CSIT official release 20.09 is available
- https://docs.fd.io/csit/rls2009/report/
- Jieqiang will compare the performance data with release 20.05
- CSIT official release 20.09 is available
- Leverage current spare TX2 server as 2-node topology performance test-bed.
- Hardware configurations/wiring are done; Physical connection to the TG is done.
- VPP Path
- CentOS-7 will be enabled with master branch for support lts release
- CentOS-7 Jenkins on Arm will be supported.
- CentOS-8 on Arm Jenkins is created and could be triggered manually with 'beta-verify' and 'beta-merge'
- CentOS-8 docker image on Arm is working fine, but not triggered per patch yet.
- https://gerrit.fd.io/r/c/ci-management/+/28960
- https://gerrit.fd.io/r/c/ci-management/+/28022
- CentOS-7 will be enabled with master branch for support lts release
- VPP Device
- VPP device testing issues may be caused by XL710 i40e fw or kernel module.
- VPP community is responding this issue actively. - Juraj
- Working on a workaround to make sure not starting multiple VPP instances at the same time - Juraj
- Implementation is ready, and will do test it with actual patches.
- Current VPP device testing on TX2 is around 40 mins - 45 mins
- Which is acceptable by CSIT maitainers
- LF will provide QSFP+ fiber switch for FD.io lab.
- Basically done. LF just procured the existing fiber switch currently rent by Arm in FD.io lab.
- Send the progress to relavent people in Arm - Lijian
- Arm is required to present Arm achievement and plan to TSC.
- Govind will prepare the slides
- VPP device testing issues may be caused by XL710 i40e fw or kernel module.
- VPP Performance Test
- VPP
- In Ubuntu-20.04, VPP on Arm will reprioritize compiler as gcc-10 > gcc-9.2.0 > clang-10
- Multi-arch support - Lijian
- SOC ID will be available on /proc entry starting from kernel version 5.9 to differentiate vendor CPUs.
- Investigate VPP Intel AVF PMD driver - Lijian
- Initial benchmarking and analysis is done, and profiling result is recorded.
- Optimize ethernet-input and avf-input node with NEON intrinsics
- Benchmarking result shows some improvement from vectorization with ethernet-input and avf-input node
- Try cross-connect with AVF PMD driver to check avf-input node only - Lijian
- SVE/SVE2 two proposals are upstreamed, will discuss the proposals with maintainers
- Try to capture some software benchmarking results
- https://gerrit.fd.io/r/c/vpp/+/29942 - first proposal - preferred
- https://gerrit.fd.io/r/c/vpp/+/30326 - second proposal - not preferred
- Repeat the 4x and 2x loop unrolling tests on Ampere server for L3 forwarding with Lijian's patch.
- No positive for 4x loop unrolling on Ampere, so will keep 2x unrolling for Neoverse N1
- Benchmark the performance using AVF PMD driver on Ampere server. - Jieqiang
- Performance on Altra is about 30%-40% lower than 8268.
- Performance on Altra is slightly better than N1SDP.
- IO testing is doable with specific PCIe slots, with which PCIe bandwidth is not the bottle-neck.
- Try the VFs with DPDK plugin. - Jieqiang
- Repeat scalability testing on N1SDP, e.g., 1T1C, 2T2C, 3T3C
- Benchmark Altra vs Cascade 8268
- IPsec on Arm platform. - Govind
- Apply prefetches and loop-unrolling with ipsec-out node
- Didn't observe much performance improvement (2%) so far
- Work on IPsec input node and VPP uses linear search on SPD lookup.
- Will try loop unrolling on the SPD lookup.
- Tune PMD config to achieve zero cache eviction for System Level Cache on N1SDP - kamalakshitha
- Tuned version has higher SLC eviction than the default version, talk with CPU team to figure out the reason.
- Review akshitha's PPT on SLC eviction and share it with the team. - Govind
- Prefetches on IPsec and learn about the cache behavior. - kamalakshitha
- Will investigate the prefetches and loop unrolling on IPsec input node. - Govind
- Apply prefetches and loop-unrolling with ipsec-out node
12/08/2020
- Attendees
- Govindarajan Mohandoss
- Lijian Zhang
- Juraj Linkes
- Tina Tsou
- General
- CSIT
- VPP Performance Test
- https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-3n-tsh/
- https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-report-iterative-2009-3n-tsh/
- Use the spare TX2 server as 2-node topology performance test-bed.
- Hardware configurations/wiring are done.
- Physical connection to the TG is done.
- Software installation for the perf tests is pending.
- Execution time is much slower due to thunderx
- Code changes related to SSH calls speed up 4x.
- VPP Path
- Dave will add CentOS-8 Jenkins on Arm job
- Dave is preparing scripts to generate docker images automatically on both x86 and Arm - First step;
- CentOS-8 on Arm Jenkins is created and could be triggered manually with 'beta-verify' and 'beta-merge'
- CentOS-8 docker image on Arm is working fine, but not triggered per patch yet.
- https://gerrit.fd.io/r/c/ci-management/+/28960
- https://gerrit.fd.io/r/c/ci-management/+/28022
- VPP Device
- VPP device testing issues may be caused by XL710 i40e fw or kernel module.
- Working with VPP/DPDK/Intel to root cause this issue. - Juraj
- Current VPP device testing on TX2 is around 40 mins - 45 mins
- Which is acceptable by CSIT maitainers
- LF will provide QSFP+ fiber switch for FD.io lab.
- Vexxhost just has a spare one, and LF will buy it for FD.io lab, which will probably happen this month.
- N1SDP shipment to FD.io
- Govind will track the status
- CSIT budget plan for 10G switch purchase in FD.io lab. - Juraj, Tina
- Trishan de Lanerolle from LF is working with Machiek to provide 10G switch.
- Arm is required to present Arm achievement and plan to TSC.
- Govind will prepare the slides
- VPP device testing issues may be caused by XL710 i40e fw or kernel module.
- VPP Performance Test
- VPP
- In Ubuntu-20.04, VPP on Arm will reprioritize compiler as gcc-10 > gcc-9.2.0 > clang-10
- Multi-arch support - Lijian
- Key point is how to differentiate vendor CPUs with other Perseus CPUs
- SOC id will be available on /proc entry starting from kernel version 5.9
- Will investigate the details - Lijian
- Investigate VPP Intel AVF PMD driver - Lijian
- Benchmarked cross-connect and TX queue is dropping packets
- Initial benchmarking and analysis is done, and profiling result is recorded.
- To optimize ethernet-input and avf-input node with NEON intrinsics
- Try cross-connect with AVF PMD driver to check avf-input node only - Lijian
- SVE/SVE2 two proposals upstreamed
- Repeat the 4x and 2x loop unrolling tests on Ampere server for L3 forwarding with Lijian's patch.
- Have to repeat the testing in the future.
- Benchmark the performance using AVF PMD driver on Ampere server. - Jieqiang
- Ampere altra server has some PCIe bugs.
- Try the VFs with DPDK plugin. - Jieqiang
- Repeat scalability testing on N1SDP, e.g., 1T1C, 2T2C, 3T3C
- IPsec on Arm platform. - Govind
- Apply prefetches and loop-unrolling with ipsec-out node
- Didn't observe much performance improvement (2%) so far
- Work on IPsec input node and VPP uses linear search on SPD lookup.
- Will try loop unrolling on the SPD lookup.
- Tune PMD config to achieve zero cache eviction for System Level Cache on N1SDP - kamalakshitha
- Tuned version has higher SLC eviction than the default version, talk with CPU team to figure out the reason.
- Review akshitha's PPT on SLC eviction and share it with the team. - Govind
- Prefetches on IPsec and learn about the cache behavior. - kamalakshitha
- Will investigate the prefetches and loop unrolling on IPsec input node. - Govind
- Apply prefetches and loop-unrolling with ipsec-out node
12/1/2020
- Attendees
- Govindarajan Mohandoss
- Jieqiang Wang
- Juraj Linkes
- Tina Tsou
- General
- CSIT
- VPP Performance Test
- https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-3n-tsh/ - Done
- https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-report-iterative-2009-3n-tsh/ - Done
- 20% perf-drop with L2 learning 1Mx flows, 4T4C, in release-2005
- Issue caused by - https://gerrit.fd.io/r/c/vpp/+/26549 - Sync up with Lijian
- Perf data capture for CSIT official release is done, so MRR testing with Taishan server is resolved.
- Huge-pages are not configured on Taishan, or previous 4K huge-pages are not enough.
- The issues are gone with 32k huge pages configured on the Taishan servers.
- Some random failed test cases due to SSH connection failures.
- Huge-pages are not configured on Taishan, or previous 4K huge-pages are not enough.
- Use the spare TX2 server as 2-node topology performance test-bed.
- Hardware configurations/wiring are done.
- Physical connection to the TG is done.
- Software installation for the perf tests is pending.
- Execution time is much slower due to thunderx
- Code changes related to SSH calls speed up 4x.
- VPP Path
- Dave is preparing scripts to generate docker images automatically on both x86 and Arm - First step;
- CentOS-8 on Arm Jenkins is created and could be triggered manually with 'beta-verify' and 'beta-merge'
- CentOS-8 docker image on Arm is working fine, but not triggered per patch yet.
- https://gerrit.fd.io/r/c/ci-management/+/28960
- https://gerrit.fd.io/r/c/ci-management/+/28022 - auto-generate docker image
- Will keep the CentOS 7 with master branch.
- VPP Device
- Current VPP device testing on TX2 is around 40 mins - 45 mins
- LF will provide QSFP+ fiber switch for FD.io lab.
- CSIT will install normally used os distro and kernel.
- 2x ThunderX2 servers are setup in FD.io lab. SSH and IPMI connections are working.
- To enable voting right for the VPP device jobs. - Juraj
- Failed tests due to sw_interface_dump api issue. - Juraj
- VPP device job is unstable
- Race condition occurs when multiple VPP instances are starting.
- Will try to update the i40e driver & firmware.
- To enable voting right for the VPP device jobs. - Juraj
- N1SDP shipment to FD.io
- Govind will update the shippment status to Juraj and Machiek.
- Will still have to ship N1SDP to FD.io lab, and Machiek confirmed that there will be enough rack space.
- CSIT budget plan for 10G switch purchase in FD.io lab. - Juraj, Tina
- Trishan de Lanerolle from LF is working with Machiek to provide 10G switch.
- Arm is required to present Arm achievement and plan to TSC.
- VPP Performance Test
- VPP
- In Ubuntu-20.04, VPP on Arm will reprioritize compiler as gcc-10 > gcc-9.2.0 > clang-10
- Multi-arch support - Lijian
- Key point is how to differentiate vendor CPUs with other Perseus CPUs
- SOC id will be available on /proc entry starting from kernel version 5.9
- Will investigate the details - Lijian
- Investigate VPP Intel AVF PMD driver - Lijian
- Initial benchmarking and analysis is done, and profiling result is recorded.
- To optimize ethernet-input and avf-input node with NEON intrinsics
- Try cross-connect with AVF PMD driver to check avf-input node only - Lijian
- SVE/SVE2 proposal
- Refactored ethernet-input node with SVE/SVE2 intrinsics per Damjan's suggestion
- Patches are upstreamed for comments
- Repeat the 4x and 2x loop unrolling tests on Ampere server for L3 forwarding with Lijian's patch. - Jieqiang
- Benchmark the performance using AVF PMD driver on Ampere server. - Jieqiang
- Ampere altra server has some PCIe bugs.
- Try the VFs with DPDK plugin. - Jieqiang
- Repeat scalability testing on N1SDP, e.g., 1T1C, 2T2C, 3T3C
- IPsec on Arm platform. - Govind
- Apply prefetches with ipsec-out node
- Work on IPsec input node and VPP uses linear search on SPD lookup.
- Will try loop unrolling on the SPD lookup.
- Tune PMD config to achieve zero cache eviction for System Level Cache on N1SDP - kamalakshitha
- Tuned version has higher SLC eviction than the default version, talk with CPU team to figure out the reason.
- Review akshitha's PPT on SLC eviction and share it with the team. - Govind
- Prefetches on IPsec and learn about the cache behavior. - kamalakshitha
- Will investigate the prefetches and loop unrolling on IPsec input node. - Govind
11/24/2020
- Attendees
- Govindarajan Mohandoss
- Lijian Zhang
- Tina Tsou
- General
- CSIT
- VPP Performance Test
- https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-3n-tsh/
- https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-report-iterative-2009-3n-tsh/
- 20% perf-drop with L2 learning 1Mx flows, 4T4C, in release-2005
- Issue caused by - https://gerrit.fd.io/r/c/vpp/+/26549
- Perf data capture for CSIT official release is done, so MRR testing with Taishan server is resolved.
- Huge-pages are not configured on Taishan, or previous 4K huge-pages are not enough.
- Use the spare TX2 server as 2-node topology performance test-bed.
- Hardware configurations/wiring are done.
- VPP Path
- Dave is preparing scripts to generate docker images automatically on both x86 and Arm - First step;
- CentOS-8 on Arm Jenkins is created and could be triggered manually with 'beta-verify' and 'beta-merge'
- CentOS-8 docker image on Arm is working fine, but not triggered per patch yet.
- https://gerrit.fd.io/r/c/ci-management/+/28960
- https://gerrit.fd.io/r/c/ci-management/+/28022 - auto-generate docker image
- VPP Device
- Current VPP device testing on TX2 is around 40 mins - 45 mins
- LF will provide QSFP+ fiber switch for FD.io lab.
- CSIT will install normally used os distro and kernel.
- 2x ThunderX2 servers are setup in FD.io lab. SSH and IPMI connections are working.
- To enable voting right for the VPP device jobs. - Juraj
- Failed tests due to sw_interface_dump api issue. - Juraj
- To enable voting right for the VPP device jobs. - Juraj
- N1SDP shipment to FD.io
- Govind will update the shippment status to Juraj and Machiek.
- Will still have to ship N1SDP to FD.io lab, and Machiek confirmed that there will be enough rack space.
- CSIT budget plan for 10G switch purchase in FD.io lab. - Juraj, Tina
- Trishan de Lanerolle from LF is working with Machiek to provide 10G switch.
- Arm is required to present Arm achievement and plan to TSC.
- VPP Performance Test
- VPP
- In Ubuntu-20.04, VPP on Arm will reprioritize compiler as gcc-10 > gcc-9.2.0 > clang-10
- Multi-arch support - Lijian
- Key point is how to differentiate vendor CPUs with other Perseus CPUs
- SOC id will be available on /proc entry starting from kernel version 5.9
- Will investigate the details - Lijian
- Investigate VPP Intel AVF PMD driver - Lijian
- Initial benchmarking and analysis is done, and profiling result is recorded.
- To optimize ethernet-input and avf-input node with NEON intrinsics
- Try cross-connect with AVF PMD driver to check avf-input node only - Lijian
- SVE/SVE2 proposal
- Refactored ethernet-input node with SVE/SVE2 intrinsics per Damjan's suggestion
- Patches are upstreamed for comments
- Repeat the 4x and 2x loop unrolling tests on Ampere server for L3 forwarding with Lijian's patch. - Jieqiang
- Benchmark the performance using AVF PMD driver on Ampere server. - Jieqiang
- Ampere altra server has some PCIe bugs.
- Try the VFs with DPDK plugin. - Jieqiang
- Repeat scalability testing on N1SDP, e.g., 1T1C, 2T2C, 3T3C
- IPsec on Arm platform. - Govind
- Apply prefetches with ipsec-out node
- Work on IPsec input node and VPP uses linear search on SPD lookup.
- Will try loop unrolling on the SPD lookup.
- Tune PMD config to achieve zero cache eviction for System Level Cache on N1SDP - kamalakshitha
- Tuned version has higher SLC eviction than the default version, talk with CPU team to figure out the reason.
- Review akshitha's PPT on SLC eviction and share it with the team. - Govind
- Prefetches on IPsec and learn about the cache behavior. - kamalakshitha
- Will investigate the prefetches and loop unrolling on IPsec input node. - Govind
11/17/2020
- Attendees
- Govindarajan Mohandoss
- Lijian Zhang
- Tina Tsou
- General
- CSIT
- VPP Performance Test
- https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-3n-tsh/
- https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-report-iterative-2009-3n-tsh/
- 20% perf-drop with L2 learning 1Mx flows, 4T4C, in release-2005
- Issue caused by - https://gerrit.fd.io/r/c/vpp/+/26549
- Use the spare TX2 server as 2-node topology performance test-bed.
- Juraj to check with Peter about the feasibility.
- VPP Path
- Dave is preparing scripts to generate docker images automatically on both x86 and Arm - First step;
- CentOS-8 on Arm Jenkins is created and could be triggered manually with 'beta-verify' and 'beta-merge'
- https://gerrit.fd.io/r/c/ci-management/+/28960
- https://gerrit.fd.io/r/c/ci-management/+/28022 - auto-generate docker image
- VPP Device
- CSIT will install normally used os distro and kernel.
- 2x ThunderX2 servers are setup in FD.io lab. SSH and IPMI connections are working.
- To enable voting right for the VPP device jobs. - Juraj
- Failed tests due to sw_interface_dump api issue. - Juraj
- To enable voting right for the VPP device jobs. - Juraj
- N1SDP shipment to FD.io
- Will still have to ship N1SDP to FD.io lab, and Machiek confirmed that there will be enough rack space.
- CSIT budget plan for 10G switch purchase in FD.io lab. - Juraj, Tina
- Trishan de Lanerolle from LF is working with Machiek to provide 10G switch.
- Arm is required to present Arm achievement and plan to TSC.
- VPP Performance Test
- VPP
- In Ubuntu-20.04, VPP on Arm will reprioritize compiler as gcc-10 > gcc-9.2.0 > clang-10
- Multi-arch support - Lijian
- Key point is how to differentiate vendor CPUs with other Perseus CPUs
- SOC id will be available on /proc entry starting from kernel version 5.9
- Will investigate the details - Lijian
- Investigate VPP Intel AVF PMD driver - Lijian
- Initial benchmarking and analysis is done, and profiling result is recorded.
- To optimize ethernet-input and avf-input node with NEON intrinsics
- Try cross-connect with AVF PMD driver to check avf-input node only - Lijian
- SVE/SVE2 proposal
- Refactored ethernet-input node with SVE/SVE2 intrinsics per Damjan's suggestion
- Patches are upstreamed for comments
- Repeat the 4x and 2x loop unrolling tests on Ampere server for L3 forwarding with Lijian's patch. - Jieqiang
- Benchmark the performance using AVF PMD driver on Ampere server. - Jieqiang
- Ampere altra server has some PCIe bugs.
- Try the VFs with DPDK plugin. - Jieqiang
- Repeat scalability testing on N1SDP, e.g., 1T1C, 2T2C, 3T3C
- IPsec on Arm platform. - Govind
- Apply prefetches with ipsec-out node
- Work on IPsec input node and VPP uses linear search on SPD lookup.
- Will try loop unrolling on the SPD lookup.
- Tune PMD config to achieve zero cache eviction for System Level Cache on N1SDP - kamalakshitha
- Tuned version has higher SLC eviction than the default version, talk with CPU team to figure out the reason.
- Review akshitha's PPT on SLC eviction and share it with the team. - Govind
- Prefetches on IPsec and learn about the cache behavior. - kamalakshitha
- Will investigate the prefetches and loop unrolling on IPsec input node. - Govind
11/10/2020
- Attendees
- Govindarajan Mohandoss
- Lijian Zhang
- Jieqiang Wang
- Juraj Linkes
- Tina Tsou
- General
- CSIT
- VPP Performance Test
- CSIT DPDK test cases will be enabled on Arm servers and data will be included next CSIT release report.
- https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-mrr-weekly-master-3n-tsh/
-
https://jenkins.fd.io/view/csit/job/csit-dpdk-perf-report-iterative-2009-3n-tsh/
- Already done by juraj, the data is published on CSIT 2009 report.
- L2 learning 1Mx flows, 4T4C, with release-2005, about 20% performance drop.
- The patch caused this issue has been identified - https://gerrit.fd.io/r/c/vpp/+/26549
- Repeat tests on local N1SDP and cascade server. - Jieqiang
- Repeat the test case with latest master branch. - Jieqiang
- The patch introduced this perf drop need to be analyzed. - Jieqiang, Lijian
- This patch needs to be analysed on VPP 2005 and 2001 releases. - Jieqiang, Lijian
- The perf drop rate is ~5-8% on latest VPP code compared to the original data.
-
https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-3n-tsh/
- The iterative jobs for VPP 2009 are still running.
- https://jenkins.fd.io/view/csit/job/csit-vpp-perf-report-iterative-2009-3n-tsh/
- Still running for one more weeks.
- Still running for more time due to Jenkins issues like Jenkins restart.
- Daily performance jobs only run MRR tests, while iterative jobs run MRR tests and NDR/PDR tests, etc, which takes longer time.
- 1 Spare TX2 server can be used to create 2 node topology to run performance tests.
- Juraj to check with Peter about the feasibility.
- Move the thx2 to the same rack for tg and install the same nic on tg.
- 1g NIC for management installed on thx2, but cannot be net-booted.
- Able to net-boot from the built-in 10G NIC.
- The tx2 has been moved to the same rack where the tg is located.
- Plan to set up the weekly perf tests on the new topo.
- Port the robotframe configuration steps for tsh testbeds from thx1 to thx2 to speed up perf tests. - Juraj
- CSIT DPDK test cases will be enabled on Arm servers and data will be included next CSIT release report.
- VPP Path
- Totally 6x ThunderX1 servers in Nomad cluster
- Investigate adding CentOS-7 on Arm Jenkins jobs - Juraj & Jieqiang
- Check with Dave about what we should do with CentOS-7 on Arm Jenkins if CentOS-8 is the main distro for VPP verification. - Juraj
- Dave is preparing scripts to generate docker images automatically on both x86 and Arm - First step;
- CentOS-8 on Arm Jenkins is created and could be triggered manually with 'beta-verify' and 'beta-merge' (Ubuntu-20.04 job will also be triggered)
- https://gerrit.fd.io/r/c/ci-management/+/28960
- https://gerrit.fd.io/r/c/ci-management/+/28022 automate the generation of docker builder images.
- Plan to drop the support for CentOS 7 from Dave.
- Tried Dave's patch to generate docker image on Arm and saw some errors. - Juraj
- Test arm centos7 jenkins builder image. - Juraj.
- Check with CSIT maintainers about the concrete plan for the enablement of Centos8 gerrit Jenkins job on aarch64. - Juraj
- VPP Device
- CSIT will install normally used os distro and kernel.
- 2x ThunderX2 servers are setup in FD.io lab. SSH and IPMI connections are working.
- The servers are physically installed. Packages are installed. CSIT tests are run on these servers outside of jenkins. pxeboot (N/W boot) works fine with 10G NIC (Inband) and not with 1G NIC. One of the server works after reboot and the other server loses N/W connectivity.
- Figure out which host of two hosts to run the Jenkins job.
- Revert to old kernel version 4.15.0-55 to avoid AVF issue.
- AVF issue is common across the platform.
- Differences between avf driver versions may be the root cause of behavior changes.
- New VPP device job takes about 55 minutes to finish, which needs to be reduced to 45 minutes around.
- Python runs slower on new thx2 servers than 1-node skylake.
- Try new version of Python(such as 3.8) or split the device tests into two parts.
- Check how many CPUs get utilized for robot framework execution on thx2 server.
- Two thunderx2 are running fine right now and the VPP device jobs are almost done.
- Disabling hyperthreading on new thx2 will speed up the VPP device tests.
- Enable the voting right for the VPP device jobs. - Juraj
- Failed tests due to sw_interface_dump api issue. - Juraj
- N1SDP shippment to FD.io
- Get response from Maciek about the rack space and traffic generator availability.
- CSIT budget plan for 10g switch purchase in FD.io lab. - Juraj, Tina
- VPP Performance Test
- VPP
- In Ubuntu-20.04, VPP on Arm will reprioritize compiler as gcc-10 > gcc-9.2.0 > clang-10
- Multi-arch support - Lijian
- Key point is how to differentiate vendor CPUs with other Perseus CPUs
- Nitin requires VPP generic image supporting 64B and 128B cache line CPU optimally at the same time - cannot be satisfied so far.
- Detect the CPU type from firmware for Perseus-CPU servers, need to confirm with customers. - Lijian
- Summarize the meeting minutes and action items. - Lijian
- SOC id will be available on /proc entry from kernel version 5.9 - Ljian, Honnappa
- Investigate VPP Intel AVF PMD driver - Lijian
- Start investigating AVF code in VPP.
- Start benchmarking AVF PMD driver in VPP on N1SDP.
- Investigate the performance gap using AVF PMD driver between N1SDP and Cascade Lake. - Lijian
- avf_input and avf_output nodes don't comsume lots of CPU cycles than dpdk-related nodes do.
- SVE/SVE2 proposal
- Will send email to Damjan asking him to review
- SVE proposal patch is upstreamed, call for comments - https://gerrit.fd.io/r/c/vpp/+/28986
- No further comments from VPP community.
- Apply the SVE/SVE2 intrinsics on refactoring ethernet-input node. - Lijian
- SVE/SVE2 functionality to be tested on the new development platform.
- Verify SVE/SVE2 code changes on simulator.
- Try to run standalone SVE codes on the new FPGA platform.
- Repeat the 4x and 2x loop unrolling tests on Ampere server for L3 forwarding with Lijian's patch. - Jieqiang
- Benchmark the performance of L2/L3/ACL tests using AVF PMD driver on Ampere server. - Jieqiang
- Ampere altra server has some PCIe bugs.
- Try the VFs with DPDK plugin. - Jieqiang
- Repeat scalability testing on N1SDP, e.g., 1T1C, 2T2C, 3T3C
- Find out the tuned configuration for cross connect test cases using AVF PMD driver.
- Figure out corresponding configurations in CSIT scripts.
- Repeat the ACL ingress SL test cases locally for N1SDP.
- Transport use cases on VPP. - Govind
- Discussed the node graph and topology.
- IPsec on Arm platform. - Govind
- Work on IPsec input node and VPP uses linear search on SPD lookup.
- Will try loop unrolling on the SPD lookup.
- Tune PMD config to achieve zero cache eviction for System Level Cache on N1SDP - kamalakshitha
- Tuned version has higher SLC eviction than the default version, talk with CPU team to figure out the reason.
- Review akshitha's PPT on SLC eviction and share it with the team. - Govind
- Prefetches on IPsec and learn about the cache behavior. - kamalakshitha
- Will investigate the prefetches and loop unrolling on IPsec input node. - Govind
- Work on IPsec input node and VPP uses linear search on SPD lookup.
- Plans
11/03/2020
- Attendees
- Govindarajan Mohandoss
- Lijian Zhang
- Jieqiang Wang
- Juraj Linkes
- Tina Tsou
- Honnappa Nagarahalli
- General
- CSIT
- VPP Performance Test
- CSIT DPDK test cases will be enabled on Arm servers and data will be included next CSIT release report.
- L2 learning 1Mx flows, 4T4C, with release-2005, about 20% performance drop.
- The patch caused this issue has been identified - https://gerrit.fd.io/r/c/vpp/+/26549
- Repeat tests on local N1SDP and cascade server. - Jieqiang
- Repeat the test case with latest master branch. - Jieqiang
- The patch introduced this perf drop need to be analyzed. - Jieqiang, Lijian
- Look into the patch to get some ideas about the code changes. - Jieqiang, Lijian
-
https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-3n-tsh/
- The iterative jobs for VPP 2009 are still running.
- Still running for one more weeks.
- https://jenkins.fd.io/view/csit/job/csit-vpp-perf-report-iterative-2009-3n-tsh/
- Daily performance jobs only run MRR tests, while iterative jobs run MRR tests and NDR/PDR tests, etc, which takes longer time.
- 1 Spare TX2 server can be used to create 2 node topology to run performance tests.
- Juraj to check with Peter about the feasibility.
- Move the thx2 to the same rack for tg and install the same nic on tg.
- 1g NIC for management installed on thx2, but cannnot be net-booted.
- VPP Path
- Totally 6x ThunderX1 servers in Nomad cluster
- Investigate adding CentOS-7 on Arm Jenkins jobs - Juraj & Jieqiang
- Check with Dave about what we should do with CentOS-7 on Arm Jenkins if CentOS-8 is the main distro for VPP verification. - Juraj
- Dave is preparing scripts to generate docker images automatically on both x86 and Arm - First step;
- CentOS-8 on Arm Jenkins is created and could be triggered manually with 'beta-verify' and 'beta-merge' (Ubuntu-20.04 job will also be triggered)
- https://gerrit.fd.io/r/c/ci-management/+/28960
- https://gerrit.fd.io/r/c/ci-management/+/28022 automate the generation of docker builder images.
- Test arm centos7 jenkins builder image. - Juraj.
- Check with CSIT maintainers about the concrete plan for the enablement of Centos8 gerrit Jenkins job on aarch64. - Juraj
- VPP Device
- CSIT will install normally used os distro and kernel.
- 2x ThunderX2 servers are setup in FD.io lab. SSH and IPMI connections are working.
- The servers are physically installed. Packages are installed. CSIT tests are run on these servers outside of jenkins. pxeboot (N/W boot) works fine with 10G NIC (Inband) and not with 1G NIC. One of the server works after reboot and the other server loses N/W connectivity.
- Figure out which host of two hosts to run the Jenkins job.
- Revert to old kernel version 4.15.0-55 to avoid AVF issue.
- AVF issue is common across the platform.
- Differences between avf driver versions may be the root cause of behavior changes.
- New VPP device job takes about 55 minutes to finish, which needs to be reduced to 45 minutes around.
- Python runs slower on new thx2 servers than 1-node skylake.
- Try new version of Python(such as 3.8) or split the device tests into two parts.
- Check how many CPUs get utilized for robot framework execution on thx2 server.
- Two thunderx2 are running fine right now and the VPP device jobs are almost done.
- N1SDP shippment to FD.io
- Get response from Machiek about the rack space and traffic generator avalability.
- CSIT budget plan for 10g switch purchase in FD.io lab. - Juraj, Tina
- VPP Performance Test
- VPP
- In Ubuntu-20.04, VPP on Arm will reprioritize compiler as gcc-10 > gcc-9.2.0 > clang-10
- Multi-arch support - Lijian
- Key point is how to differentiate vendor CPUs with other Perseus CPUs
- Nitin requires VPP generic image supporting 64B and 128B cache line CPU optimally at the same time - cannot be satisfied so far.
- Detect the CPU type from firmware for Perseus-CPU servers, need to confirm with customers. - Lijian
- Summarize the meeting minutes and action items. - Lijian
- SOC id will be available on /proc entry from kernel version 5.9 - Ljian, Honnappa
- Investigate VPP Intel AVF PMD driver - Lijian
- Start investigating AVF code in VPP.
- Start benchmarking AVF PMD driver in VPP on N1SDP.
- Investigate the performance gap using AVF PMD driver between N1SDP and Cascade Lake. - Lijian
- avf_input and avf_output nodes don't comsume lots of CPU cycles than dpdk-related nodes do.
- SVE/SVE2 proposal
- Will send email to Damjan asking him to review
- SVE proposal patch is upstreamed, call for comments - https://gerrit.fd.io/r/c/vpp/+/28986
- No further comments from VPP community.
- Apply the SVE/SVE2 intrinsics on refractoring ethernet-input node. - Lijian
- SVE/SVE2 functionality to be tested on the new development platform.
- Repeat the 4x and 2x loop unrolling tests on Ampere server for L3 forwarding with Lijian's patch. - Jieqiang
- Benchmark the performance of L2/L3/ACL tests using AVF PMD driver on Ampere server. - Jieqiang
- Repeat scalability testing on N1SDP, e.g., 1T1C, 2T2C, 3T3C
- Find out the tuned configuration for cross connect test cases using AVF PMD driver.
- Figure out corresponding configurations in CSIT scripts
- Repeat the ACL ingress SL test cases locally for N1SDP.
- Transport use cases on VPP. - Govind
- Discussed the node graph and topology.
- IPsec on Arm platform. - Govind
- Work on IPsec input node and VPP uses linear search on SPD lookup.
- Will try loop unrolling on the SPD lookup.
- Tune PMD config to achieve zero cache eviction for System Level Cache on N1SDP - kamalakshitha
- Tuned version has higher SLC eviction than the default version, talk with CPU team to figure out the reason.
- Review akshitha's PPT on SLC eviction and share it with the team. - Govind.
- Prefetches on IPsec and learn about the cache behavior. - kamalakshitha
- Work on IPsec input node and VPP uses linear search on SPD lookup.
- Plans
10/27/2020
-
Attendees
- Govindarajan Mohandoss
- Lijian Zhang
- Jieqiang Wang
- Juraj Linkes
- Tina Tsou
- Honnappa Nagarahalli
-
General
-
CSIT
- VPP Performance Test
- CSIT DPDK test cases will be enabled on Arm servers and data will be included next CSIT release report.
- L2 learning 1Mx flows, 4T4C, with release-2005, about 20% performance drop.
- The patch caused this issue has been identified - https://gerrit.fd.io/r/c/vpp/+/26549
- Repeat tests on local N1SDP and cascade server. - Jieqiang
- Look into the patch to get some ideas about the code changes. - Jieqiang, Lijian
-
https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-3n-tsh/
- The iterative jobs for VPP 2009 are still running.
- Still running for one or two weeks.
- https://jenkins.fd.io/view/csit/job/csit-vpp-perf-report-iterative-2009-3n-tsh/
- Daily performance jobs only run MRR tests, while iterative jobs run MRR tests and NDR/PDR tests, etc, which takes longer time.
- 1 Spare TX2 server can be used to create 2 node topology to run performance tests.
- Juraj to check with Peter about the feasibility.
- Move the thx2 to the same rack for tg and install the same nic on tg.
- VPP Path
- Totally 6x ThunderX1 servers in Nomad cluster
- Investigate adding CentOS-7 on Arm Jenkins jobs - Juraj & Jieqiang
- Check with Dave about what we should do with CentOS-7 on Arm Jenkins if CentOS-8 is the main distro for VPP verification. - Juraj
- Dave is preparing scripts to generate docker images automatically on both x86 and Arm - First step;
- CentOS-8 on Arm Jenkins is created and could be triggered manually with 'beta-verify' and 'beta-merge' (Ubuntu-20.04 job will also be triggered)
- https://gerrit.fd.io/r/c/ci-management/+/28960
- Check with CSIT maintainers about the concrete plan for the enablement of Centos8 gerrit Jenkins job on aarch64. - Juraj
- VPP Device
- CSIT will install normally used os distro and kernel.
- 2x ThunderX2 servers are setup in FD.io lab. SSH and IPMI connections are working.
- The servers are physically installed. Packages are installed. CSIT tests are run on these servers outside of jenkins. pxeboot (N/W boot) works fine with 10G NIC (Inband) and not with 1G NIC. One of the server works after reboot and the other server loses N/W connectivity.
- Figure out which host of two hosts to run the Jenkins job.
- Revert to old kernel version 4.15.0-55 to aviod AVF issue.
- Differences between avf driver versions may be the root cause of behavior changes.
- New VPP device job takes about 55 minutes to finish, which needs to be reduced to 40 minutes around.
- Python runs slower on new thx2 servers than 1-node skylake.
- Try new version of Python(such as 3.8) or split the device tests into two parts.
- Check how many CPUs get utilized for robot framework execution on thx2 server.
- VPP Performance Test
-
VPP
- In Ubuntu-20.04, VPP on Arm will reprioritize compiler as gcc-10 > gcc-9.2.0 > clang-10
- Multi-arch support - Lijian
- Key point is how to differentiate vendor CPUs with other Perseus CPUs
- Nitin requires VPP generic image supporting 64B and 128B cache line CPU optimally at the same time - cannot be satisfied so far.
- Detect the CPU type from firmware for Perseus-CPU servers, need to confirm with customers. - Lijian
- Summarize the meeting minutes and action items. - Lijian
- Investigate VPP Intel AVF PMD driver - Lijian
- Start investigating AVF code in VPP.
- Start benchmarking AVF PMD driver in VPP on N1SDP.
- Investigate the performance gap using AVF PMD driver between N1SDP and Cascade Lake. - Lijian
- avf_input and avf_output nodes don't comsume lots of CPU cycles than dpdk-related nodes do.
- SVE/SVE2 proposal
- Will send email to Damjan asking him to review
- SVE proposal patch is upstreamed, call for comments - https://gerrit.fd.io/r/c/vpp/+/28986
- No further comments from VPP community.
- Apply the SVE/SVE2 on ethernet-input node. - Lijian
- Repeat the 4x and 2x loop unrolling tests on Ampere server. - Jieqiang
- Benchmark the performance of L2/L3/ACL tests using AVF PMD driver on Ampere server. - Jieqiang
- Repeat scalability testing on N1SDP, e.g., 1T1C, 2T2C, 3T3C
- Figure out corresponding configurations in CSIT scripts
- Repeat the ACL ingress SL test cases locally for N1SDP.
- Repeat scalability testing on N1SDP, e.g., 1T1C, 2T2C, 3T3C
- Transport use cases on VPP. - Govind
- Discussed the node graph and topology.
- IPsec on Arm platform. - Govind
- Work on IPsec input node and VPP uses linear search on SPD lookup.
- Will try loop unrolling on the SPD lookup.
- Tune PMD config to achieve zero cache eviction for System Level Cache on N1SDP - kamalakshitha
- Tuned version has higher SLC eviction than the default version, talk with CPU team to figure out the reason.
- Prefetches on IPsec and learn about the cache behavior. - kamalakshitha
- Work on IPsec input node and VPP uses linear search on SPD lookup.
- Plans
10/20/2020
- Attendees
- Govindarajan Mohandoss
- Lijian Zhang
- Jieqiang Wang
- Juraj Linkes
- Tina Tsou
- Honnappa Nagarahalli
- General
- CSIT
- VPP Performance Test
- CSIT DPDK test cases will be enabled on Arm servers and data will be included next CSIT release report.
- L2 learning 1Mx flows, 4T4C, with release-2005, about 20% performance drop.
- The patch caused this issue has been identified - https://gerrit.fd.io/r/c/vpp/+/26549
-
https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-3n-tsh/
- The iterative jobs for VPP 2009 are still running.
- https://jenkins.fd.io/view/csit/job/csit-vpp-perf-report-iterative-2009-3n-tsh/
- Daily performance jobs only run MRR tests, while iterative jobs run MRR tests and NDR/PDR tests and etc, which takes longer time.
- The iterative jobs for VPP 2009 are still running.
- 1 Spare TX2 server can be used to create 2 node topology to run performance tests.
- Juraj to check with Peter about the feasibility.
- VPP Path
- Totally 6x ThunderX1 servers in Nomad cluster
- Investigate adding CentOS-7 on Arm Jenkins jobs - Juraj & Jieqiang
- Check with Dave about what we should do with CentOS-7 on Arm Jenkins if CentOS-8 is the main distro for VPP verification. - Juraj
- Dave is preparing scripts to generate docker images automatically on both x86 and Arm - First step;
- CentOS-8 on Arm Jenkins is created and could be triggered manually with 'beta-verify' and 'beta-merge' (Ubuntu-20.04 job will also be triggered)
- https://gerrit.fd.io/r/c/ci-management/+/28960
- Check with CSIT maintainers about the concrete plan for the enablement of Centos8 gerrit Jenkins job on aarch64. - Juraj
- Errors happen when running latest VPP debug image, which was introduced by https://gerrit.fd.io/r/c/vpp/+/29490 - Lijian
- VPP Device
- CSIT will install normally used os distro and kernel.
- 2x ThunderX2 servers are setup in FD.io lab. SSH and IPMI connections are working.
- The servers are physically installed. Packages are installed. CSIT tests are run on these servers outside of jenkins. pxeboot (N/W boot) works fine with 10G NIC (Inband) and not with 1G NIC. One of the server works after reboot and the other server loses N/W connectivity.
- Figure out which host of two hosts to run the Jenkins job.
- Two failed test cases related to AVF plugin.
- The root cause is the newer kernel version - 4.15.0-118-generic fails, 4.15.0-72-generic works.
- Downgrade the kernel version to 4.15.0-72-generic and continue the VPP device testing.
- Try the same experiment on X86 to see if this issue is arm-specific or not. - Juraj
- VPP Performance Test
- VPP
- In Ubuntu-20.04, VPP on Arm will reprioritize compiler as gcc-10 > gcc-9.2.0 > clang-10
- Multi-arch support - Lijian
- Key point is how to differentiate vendor CPUs with other Perseus CPUs
- Nitin requires VPP generic image supporting 64B and 128B cache line CPU optimally at the same time - cannot be satisfied so far.
- Detect the CPU type from firmware for Perseus-CPU servers, need to confirm with customers. - Lijian
- Investigate VPP Intel AVF PMD driver - Lijian
- Start investigating AVF code in VPP.
- Start benchmarking AVF PMD driver in VPP on N1SDP.
- Investigate the performance gap using AVF PMD driver between N1SDP and Cascade Lake. - Lijian
- SVE/SVE2 proposal
- Will send email to Damjan asking him to review
- SVE proposal patch is upstreamed, call for comments - https://gerrit.fd.io/r/c/vpp/+/28986
- No further comments from VPP community.
- Benchmark VPP scalability on N1SDP vs CascadeLake, with 3x CPUs.
- Finished the benchmarking and shared the data to team. - Jieqiang
- Repeat scalability testing on N1SDP, e.g., 1T1C, 2T2C, 3T3C
- Figure out corresponding configurations in CSIT scripts
- Repeat the ACL ingress SL test cases locally for N1SDP.
- Transport use cases on VPP. - Govind
- Discussed the node graph and topology.
- IPsec on Arm platform. - Govind
- Tune PMD config to achieve zero cache eviction for System Level Cache on N1SDP - kamalakshitha
- Prefetches on IPsec and learn about the cache behavior. - kamalakshitha
- Plans
10/13/2020
- Attendees
- Govindarajan Mohandoss
- Lijian Zhang
- Jieqiang Wang
- Juraj Linkes
- Tina Tsou
- Honnappa Nagarahalli
- General
- CSIT
- VPP Performance Test
- CSIT DPDK test cases will be enabled on Arm servers and data will be included next CSIT release report.
- L2 learning 1Mx flows, 4T4C, with release-2005, about 20% performance drop.
- The patch caused this issue has been identified - https://gerrit.fd.io/r/c/vpp/+/26549
- https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-3n-tsh/
- 1 Spare TX2 server can be used to create 2 node topology to run performance tests.
- Juraj to check with Peter about the feasibility.
- VPP Path
- Totally 6x ThunderX1 servers in Nomad cluster
- Investigate adding CentOS-7 on Arm Jenkins jobs - Juraj & Jieqiang
- Dave is preparing scripts to generate docker images automatically on both x86 and Arm - First step;
- CentOS-8 on Arm Jenkins is created and could be triggered manually with 'beta-verify' and 'beta-merge' (Ubuntu-20.04 job will also be triggered)
- https://gerrit.fd.io/r/c/ci-management/+/28960
- Check with CSIT maintainers about the concrete plan for the enablement of Centos8 gerrit Jenkins job on aarch64. - Juraj
- VPP Device
- CSIT will install normally used os distro and kernel.
- 2x ThunderX2 servers are setup in FD.io lab. SSH and IPMI connections are working.
- The servers are physically installed. Packages are installed. CSIT tests are run on these servers outside of jenkins. pxeboot (N/W boot) works fine with 10G NIC (Inband) and not with 1G NIC. One of the server works after reboot and the other server loses N/W connectivity.
- Figure out which host of two hosts to run the Jenkins job.
- Two failed test cases related to AVF plugin.
- VPP Performance Test
- VPP
- In Ubuntu-20.04, VPP on Arm will reprioritize compiler as gcc-10 > gcc-9.2.0 > clang-10
- Multi-arch support - Lijian
- Key point is how to differentiate vendor CPUs with other Perseus CPUs
- Nitin requires VPP generic image supporting 64B and 128B cache line CPU optimally at the same time - cannot be satisfied so far.
- Detect the CPU type from firmware for Perseus-CPU servers, need to confirm with customers. - Lijian
- Investigate VPP Intel AVF PMD driver - Lijian
- Start investigating AVF code in VPP.
- SVE/SVE2 proposal
- Will send email Damjan asking him to review
- SVE proposal patch is upstreamed, call for comments - https://gerrit.fd.io/r/c/vpp/+/28986
- No further comments from VPP community.
- Benchmark VPP scalability on N1SDP vs CascadeLake, with 3x CPUs.
- Finished the benchmarking and shared the data to team.
- Repeat scalability testing on N1SDP, e.g., 1T1C, 2T2C, 3T3C
- Figure out corresponding configurations in CSIT scripts
- Repeat the ACL ingress SL test cases locally for N1SDP.
- Transport use cases on VPP. - Govind
- Discussed the node graph and topology.
- IPsec on Arm platform. - Govind
- Plans
10/06/2020
- Attendees
- Govindarajan Mohandoss
- Juraj Linkes
- Tina Tsou
- Honnappa Nagarahalli
- General
- CSIT
- VPP Performance Test
- CSIT DPDK test cases will be enabled on Arm servers and data will be included next CSIT release report.
- L2 learning 1Mx flows, 4T4C, with release-2005, about 20% performance drop.
- The patch caused this issue has been identified - https://gerrit.fd.io/r/c/vpp/+/26549
- https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-3n-tsh/
- 1 Spare TX2 server can be used to create 2 node topology to run performance tests.
- Juraj to check with Peter about the feasibility.
- VPP Path
- Totally 6x ThunderX1 servers in Nomad cluster
- Investigate adding CentOS-7 on Arm Jenkins jobs - Juraj & Jieqiang
- Dave is preparing scripts to generate docker images automatically on both x86 and Arm - First step;
- CentOS-8 on Arm Jenkins is created and could be triggered manually with 'beta-verify' and 'beta-merge' (Ubuntu-20.04 job will also be triggered)
- https://gerrit.fd.io/r/c/ci-management/+/28960
- VPP Device
- CSIT will install normally used os distro and kernel.
- 2x ThunderX2 servers are setup in FD.io lab. SSH and IPMI connections are working.
- The servers are physically installed. Packages are installed. CSIT tests are run on these servers outside of jenkins. pxeboot (N/W boot) works fine with 10G NIC (Inband) and not with 1G NIC. One of the server works after reboot and the other server loses N/W connectivity.
- VPP Performance Test
- VPP
- In Ubuntu-20.04, VPP on Arm will reprioritize compiler as gcc-10 > gcc-9.2.0 > clang-10
- Multi-arch support - Lijian
- Key point is how to differentiate vendor CPUs and other Perseus CPUs
- Nitin requires VPP generic image supporting 64B and 128B cache line CPU optimally at the same time - cannot be satisfied so far.
- Investigate VPP Intel AVF PMD driver - Lijian
- Start investigating AVF code in VPP.
- SVE/SVE2 proposal
- Will send email Damjan asking him to review
- SVE proposal patch is upstreamed, call for comments - https://gerrit.fd.io/r/c/vpp/+/28986
- Benchmark VPP scalability on N1SDP vs CascadeLake, with 3x CPUs.
- Finished the benchmarking and shared the data to team.
- Repeat scalability testing on N1SDP, e.g., 1T1C, 2T2C, 3T3C
- Figure out corresponding configurations in CSIT scripts
- Transport use cases on VPP. - Govind
- Discussed the node graph and topology.
- IPsec on Arm platform. - Govind
- Plans
09/29/2020
- Attendees
- Govindarajan Mohandoss
- Juraj Linkes
- Jieqiang Wang
- Tina Tsou
- Lijian Zhang
- General
- CSIT
- VPP Performance Test
- CSIT DPDK test cases will be enabled on Arm servers and data will be included next CSIT release report.
- L2 learning 1Mx flows, 4T4C, with release-2005, about 20% performance drop.
- The patch caused this issue has been identified - https://gerrit.fd.io/r/c/vpp/+/26549
- https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-3n-tsh/
- VPP Path
- Totally 6x ThunderX1 servers in Nomad cluster
- Investigate adding CentOS-7 on Arm Jenkins jobs - Juraj & Jieqiang
- Dave is preparing scripts to generate docker images automatically on both x86 and Arm - First step;
- CentOS-8 on Arm Jenkins is created and could be triggered manually with 'beta-verify' and 'beta-merge' (Ubuntu-20.04 job will also be triggered)
- https://gerrit.fd.io/r/c/ci-management/+/28960
- VPP Device
- CSIT will install normally used os distro and kernel.
- 2x ThunderX2 servers are setup in FD.io lab. SSH and IPMI connections are working.
- VPP Performance Test
- VPP
- In Ubuntu-20.04, VPP on Arm will reprioritize compiler as gcc-10 > gcc-9.2.0 > clang-10
- Multi-arch support - Lijian
- Key point is how to differentiate Vendor CPUs with other Perseus CPUs
- Nitin requires VPP generic image supporting 64B and 128B cache line CPU optimally at the same time - cannot be satisfied so far.
- Investigate VPP Intel AVF PMD driver - Lijian
- Start investigating AVF code in VPP.
- SVE/SVE2 proposal
- Will send email Damjan asking him to review
- SVE proposal patch is upstreamed, call for comments - https://gerrit.fd.io/r/c/vpp/+/28986
- Benchmark VPP scalability on N1SDP vs CascadeLake, with 3x CPUs.
- Finished the benchmarking and shared the data to team.
- Repeat scalability testing on N1SDP, e.g., 1T1C, 2T2C, 3T3C
- Figure out corresponding configurations in CSIT scripts
- Transport use cases on VPP. - Govind
- Discussed the node graph and topology.
- IPsec on Arm platform. - Govind
- Plans
09/22/2020
- Attendees
- Govindarajan Mohandoss
- Juraj Linkes
- Jieqiang Wang
- Tina Tsou
- Lijian Zhang
- General
- CSIT
- VPP Performance Test
- CSIT DPDK test cases will be enabled on Arm servers and data will be included next CSIT release report.
- L2 learning 1Mx flows, 4T4C, with release-2005, about 20% performance drop.
- The patch caused this issue has been identified - https://gerrit.fd.io/r/c/vpp/+/26549
- https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-3n-tsh/
- VPP Path
- VexxHost will replace the faulty RAM with a new one, and get the expense reimbursed by LF.
- Issue is resolved by replugining back the previous RAM, and server is alive now.
- Investigate adding CentOS-7 on Arm Jenkins jobs - Juraj & Jieqiang
- Dave is preparing scripts to generate docker images automatically on both x86 and Arm - First step;
- Add CentOS-7 on Arm - Second step;
- https://gerrit.fd.io/r/c/ci-management/+/28960
- VexxHost will replace the faulty RAM with a new one, and get the expense reimbursed by LF.
- VPP Device
- 3x SoftIron servers will be decommissioned directly to free rack space for 2x ThunderX2 servers.
- Two ThunderX2 servers are received by Vexx host and currently in the storage warehouse.
- VexxHost people will setup the servers and provide IP connectivity.
- VPP Performance Test
- VPP
- In Ubuntu-20.04, VPP on Arm will reprioritize compiler as gcc-10 > gcc-9.2.0 > clang-10
- Multi-arch support - Lijian
- Key point is how to differentiate Vendor CPUs with other Perseus CPUs
- Investigate VPP Intel AVF PMD driver - Lijian
- Start investigating AVF code in VPP.
- SVE/SVE2 proposal
- SVE proposal patch is upstreamed, call for comments - https://gerrit.fd.io/r/c/vpp/+/28986
- Benchmark VPP scalability on N1SDP vs CascadeLake, with 3x CPUs.
- Repeat scalability testing on N1SDP, e.g., 1T1C, 2T2C, 3T3C
- Figure out corresponding configurations in CSIT scripts
- Transport use cases on VPP. - Govind
- Discussed the node graph and topology.
- IPsec on Arm platform. - Govind
- Plans
09/15/2020
- Attendees
- Govindarajan Mohandoss
- Juraj Linkes
- Jieqiang Wang
- Tina Tsou
- General
- CSIT
- VPP Performance Test
- CSIT DPDK test cases will be enabled on Arm servers and data will be included next CSIT release report.
- On L2 learning 1Mx flows, 4T4C, with release-2005, there is about 20% performance drop.
- The patch caused this issue has been identified.
- https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-3n-tsh/
- They have finished collecting data with performance testing setup, and the mrr daily is resumed
- FD.io CSIT-2005 Release Report was released, https://docs.fd.io/csit/rls2005/report/
- VPP Path
- LF will pay for the expense, and Vexhost has or will make the order for new RAM module.
- Will confirm with Dean if Arm can pay for the expense. If yes, will send the proposal to vexhost.
- One ThunderX1 has faulty RAM. Will try to replace all the RAMs.
- Juraj will follow or create new vexxhost ticket to replace faulty RAM.
- Check with Juraj with the latest news about the faulty RAMs.
- Investigate adding CentOS-7 on Arm Jenkins jobs - Juraj & Jieqiang
- Dave is preparing scripts to generate docker images automatically on both x86 and Arm - first step;
- Add CentOS-7 on Arm will be second step.
- Will confirm with Dave W. if he will add this Jenkins job and if he requires any help - Jieqiang
- Questions on the docker file upload.
- The Dockerfile has been upstreamed for review and merge.
- Juraj/Jieqiang to help Dave Wallace to fix the script issues. Currently, the build process is done manually and will be automated.
- VPP Device
- We can decommission 3x SoftIron servers directly, but for the existing ThunderX2 servers, the decommission with it could be temporarily. We probably will reinstall it in the near future.
- Mention the rack space request for the two ThunderX2 servers in CSIT meeting. - Juraj
- Two ThunderX2 servers are received by Vexx host and currently in the storage warehouse.
- Vexx host people will setup the servers and provide IP connectivity. Juraj will install the necessary software after that.
- Budget plan for CSIT FD.io lab.
- We have enough servers for VPP path & device tests.
- We can ask the CSIT FD.io lab folks for saving rack space for arm servers.
- We may plan to send new advanced servers for perf tests in future but we won't mention the specific server type.
- VPP Performance Test
- VPP
- In Ubuntu-20.04, VPP on Arm will reprioritize compiler as gcc-10 > gcc-9.2.0 > clang-10
- Benchmarking btw gcc-10.1.0, clang-10/clang-9 and gcc-9.2.0 on Arm machines - Jieqiang
- N1SDP enablement. - Lijian
- Preparing patches to enable creating big tables on huge-pages
- IP4-rewrite refactor patch brings performance improvement especially with 10K flows
- Upstreamed and are using csit testing to verify the patch.
- Vendor CPU server enablement in VPP - Lijian
- Ready for internal review
- Will discuss with VPP maintainer
- Investigate VPP Intel AVF driver - Lijian
- SVE
- SVE intrinsics wrapper is done. Proposal patch is ready for review.
- ACLE, architecture, sve-sve2-programming-example
- SVE intrinsics is preferred.
- Share dpdk team with SVE knowledge.
- Benchmarked VPP on n1sdp on scalability, on 3x CPUs.
- Will repeat scalability testing on N1SDP.
- Benchmark AVF driver btw Cascade Lake and N1SDP - Jieqiang
- Will investigate AVF drivers on Arm. - Lijian
- Jieqiang will figure out performance data for 1x, 10Kx flows on Cascade Lake in CSIT.
- Conform if the system is same for the local dell server and cascade server in CSIT. - Jieqiang
- Check if there are any test cases with 1t1c/2t2c/4t4c configured for 2n-clx testbed in CSIT - Jieqiang
- Performance data; Configurations;
- VM2VM
- Transport use cases on VPP. - Govind
- Discussed the node graph and topology.
- Trying to make IPsec enabled with Arm platform. - Govind
- Started system tuning on PMD TX direction.
- Investigate mempool configuration.
- Change the descriptor size by modifying the DPDK source code.
- Currently working on non-encryption optimization with PMD driver.
- Focus both non-encryption and encryption cases.
- Basic IPsec functions are working. Will do benchmarking per CPU core.
- Trying to reproduce the CSIT VPP device testing on local servers - Jieqiang
- Will try to run VPP device testing on local ThunderX2 servers with XL710 25G NICs.
- After VPP device scripts can run well on local servers, Jieqiang can investigate more features, IPv4, IPv6, Tunnel, IPSec.
- Plans
09/08/2020
- Attendees
- Govindarajan Mohandoss
- Juraj Linkes
- Jieqiang Wang
- Tina Tsou
- General
- CSIT
- VPP Performance Test
- CSIT DPDK test cases will be enabled on Arm servers and data will be included next CSIT release report.
- On L2 learning 1Mx flows, 4T4C, with release-2005, there is about 20% performance drop.
- The patch caused this issue has been identified.
- https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-3n-tsh/
- They have finished collecting data with performance testing setup, and the mrr daily is resumed
- FD.io CSIT-2005 Release Report was released, https://docs.fd.io/csit/rls2005/report/
- VPP Path
- LF will pay for the expense, and Vexhost has or will make the order for new RAM module.
- Will confirm with Dean if Arm can pay for the expense. If yes, will send the proposal to vexhost.
- One ThunderX1 has faulty RAM. Will try to replace all the RAMs.
- Juraj will follow or create new vexxhost ticket to replace faulty RAM.
- Investigate adding CentOS-7 on Arm Jenkins jobs - Juraj & Jieqiang
- Dave is preparing scripts to generate docker images automatically on both x86 and Arm - first step;
- Add CentOS-7 on Arm will be second step.
- Will confirm with Dave W. if he will add this Jenkins job and if he requires any help - Jieqiang
- Questions on the docker file upload.
- The Dockerfile has been upstreamed for review and merge.
- Juraj/Jieqiang to help Dave Wallace to fix the script issues. Currently, the build process is done manually and will be automated.
- VPP Device
- We can decommission 3x SoftIron servers directly, but for the existing ThunderX2 servers, the decommission with it could be temporarily. We probably will reinstall it in the near future.
- Mention the rack space request for the two ThunderX2 servers in CSIT meeting. - Juraj
- Two ThunderX2 servers are received by Vexx host and currently in the storage warehouse.
- Vexx host people will setup the servers and provide IP connectivity. Juraj will install the necessary software after that.
- VPP Performance Test
- VPP
- In Ubuntu-20.04, VPP on Arm will reprioritize compiler as gcc-10 > gcc-9.2.0 > clang-10
- Benchmarking btw gcc-10.1.0, clang-10/clang-9 and gcc-9.2.0 on Arm machines - Jieqiang
- N1SDP enablement. - Lijian
- Preparing patches to enable creating big tables on huge-pages
- IP4-rewrite refactor patch brings performance improvement especially with 10K flows
- Upstreamed and are using csit testing to verify the patch.
- SVE
- SVE intrinsics wrapper is done. Proposal patch is ready for review.
- ACLE, architecture, sve-sve2-programming-example
- SVE intrinsics is preferred.
- Benchmarked VPP on n1sdp on scalability, on 3x CPUs.
- Will repeat scalability testing on N1SDP.
- Benchmark AVF driver btw Cascade Lake and N1SDP - Jieqiang
- Will investigate AVF drivers on Arm. - Lijian
- Jieqiang will figure out performance data for 1x, 10Kx flows on Cascade Lake in CSIT.
- Performance data; Configurations;
- VM2VM
- Transport use cases on VPP. - Govind
- Discussed the node graph and topology.
- Trying to make IPsec enabled with Arm platform. - Govind
- Started system tuning on PMD TX direction.
- Currently working on non-encryption optimization with PMD driver.
- Focus both non-encryption and encryption cases.
- Basic IPsec functions are working. Will do benchmarking per CPU core.
- Trying to reproduce the CSIT VPP device testing on local servers - Jieqiang
- Will try to run VPP device testing on local ThunderX2 servers with XL710 25G NICs.
- After VPP device scripts can run well on local servers, Jieqiang can investigate more features, IPv4, IPv6, Tunnel, IPSec.
- Plans
09/01/2020
- Attendees
- Govindarajan Mohandoss
- Juraj Linkes
- Jieqiang Wang
- Tina Tsou
- General
- CSIT
- VPP Performance Test
- On L2 learning 1Mx flows, 4T4C, with release-2005, there is about 20% performance drop.
- The patch caused this issue has been identified.
- https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-3n-tsh/
- They have finished collecting data with performance testing setup, and the mrr daily is resumed
- FD.io CSIT-2005 Release Report was released, https://docs.fd.io/csit/rls2005/report/
- On L2 learning 1Mx flows, 4T4C, with release-2005, there is about 20% performance drop.
- VPP Path
- Will confirm with Dean if Arm can pay for the expense. If yes, will send the proposal to vexhost.
- One ThunderX1 has faulty RAM. Will try to replace all the RAMs.
- Seems plugin working RAMs into empty slots will resolve the problem.
- Juraj will send email to Machiek about the ownership of any FD.io lab servers, and who should pay for the charge.
- The second ThunderX1 has IPMI problem, but SSH is working fine.
- IPMI IP is configured via SSH Linux prompt. It's working fine now.
- Juraj will follow or create new vexxhost ticket to replace faulty RAM.
- 3x spare ThunderX servers are used for CI and included in Nomad cluster.
- Faulty RAM on TX server is not fixed and yet to be debugged.
- Pending with Vexx host to proceed further.
- Investigate adding CentOS-7 on Arm Jenkins jobs - Juraj & Jieqiang
- Will confirm with Dave W. if he will add this Jenkins job and if he requires any help - Jieqiang
- Questions on the docker file upload.
- The Dockerfile has been upstreamed for review and merge.
- Juraj/Jieqiang to help Dave Wallace to fix the script issues. Currently, the build process is done manually and will be automated.
- VPP Device
- Mention the rack space request for the two ThunderX2 servers in CSIT meeting. - Juraj
- Two ThunderX2 servers are received by Vexx host and currently in the storage warehouse.
- Vexx host people will setup the servers and provide IP connectivity. Juraj will install the necessary software after that.
- VPP Performance Test
- VPP
- Benchmarking btw gcc-10.1.0, clang-10/clang-9 and gcc-9.2.0 on Arm machines - Jieqiang
- gcc-10.1.0 has compiling errors with latest VPP source code.
- This issue is fixed by Jieqiang and available for internal review.
- gcc-10.1.0 has compiling errors with latest VPP source code.
- N1SDP enablement. - Lijian
- Preparing patches to enable creating big tables on huge-pages
- IP4-rewrite refactor patch brings performance improvement especially with 10K flows
- Upstreamed and are using csit testing to verify the patch.
- gcc-10 compiling issue is resolved and merged.
- SVE
- ACLE, architecture, sve-sve2-programming-example
- SVE intrinsics is preferred.
- Benchmarked VPP on n1sdp on scalability, on 3x CPUs.
- VM2VM
- Transport use cases on VPP. - Govind
- Discussed the node graph and topology.
- Trying to make IPsec enabled with Arm platform. - Govind
- Started system tuning on PMD TX direction.
- Currently working on non-encryption optimization with PMD driver.
- Focus both non-encryption and encryption cases.
- Basic IPsec functions are working. Will do benchmarking per CPU core.
- Trying to reproduce the CSIT VPP device testing on local servers - Jieqiang
- Will try to run VPP device testing on local ThunderX2 servers with XL710 25G NICs.
- After VPP device scripts can run well on local servers, Jieqiang can investigate more features, IPv4, IPv6, Tunnel, IPSec.
- Plans
- Benchmarking btw gcc-10.1.0, clang-10/clang-9 and gcc-9.2.0 on Arm machines - Jieqiang
08/25/2020
- Attendees
- Govindarajan Mohandoss
- Juraj Linkes
- Jieqiang Wang
- Tina Tsou
- General
- CSIT
- VPP Performance Test
- On L2 learning 1Mx flows, 4T4C, with release-2005, there is about 20% performance drop.
- Jieqiang is trying to narrow down the patch that causes the issue.
- https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-3n-tsh/
- They have finished collecting data with performance testing setup, and the mrr daily is resumed
- FD.io CSIT-2005 Release Report was released, https://docs.fd.io/csit/rls2005/report/
- On L2 learning 1Mx flows, 4T4C, with release-2005, there is about 20% performance drop.
- VPP Path
- One ThunderX1 has faulty RAM. Will try to replace all the RAMs.
- Seems plugin working RAMs into empty slots will resolve the problem.
- Juraj will send email to Machiek about the ownership of any FD.io lab servers, and who should pay for the charge.
- The second ThunderX1 has IPMI problem, but SSH is working fine.
- IPMI IP is configured via SSH Linux prompt. It's working fine now.
- Juraj will follow or create new vexxhost ticket to replace faulty RAM.
- 3x spare ThunderX servers are used for CI and included in Nomad cluster.
- Faulty RAM on TX server is not fixed and yet to be debugged.
- Pending with Vexx host to proceed further.
- Investigate adding CentOS-7 on Arm Jenkins jobs - Juraj & Jieqiang
- Will confirm with Dave W. if he will add this Jenkins job and if he requires any help - Jieqiang
- Questions on the docker file upload.
- The Dockerfile has been upstreamed for review and merge.
- Juraj/Jieqiang to help Dave Wallace to fix the script issues. Currently, the build process is done manually and will be automated.
- One ThunderX1 has faulty RAM. Will try to replace all the RAMs.
- VPP Device
- Mention the rack space request for the two ThunderX2 servers in CSIT meeting. - Juraj
- Two ThunderX2 servers are received by Vexx host and currently in the storage warehouse.
- Vexx host people will setup the servers and provide IP connectivity. Juraj will install the necessary software after that.
- VPP Performance Test
- VPP
- Benchmarking btw gcc-10.1.0, clang-10/clang-9 and gcc-9.2.0 on Arm machines - Jieqiang
- gcc-10.1.0 has compiling errors with latest VPP source code.
- This issue is fixed by Jieqiang and available for internal review.
- gcc-10.1.0 has compiling errors with latest VPP source code.
- N1SDP enablement. - Lijian
- Preparing patches to enable creating big tables on huge-pages
- IP4-rewrite refactor patch brings performance improvement especially with 10K flows
- Upstreamed and are using csit testing to verify the patch.
- SVE
- ACLE, architecture, sve-sve2-programming-example
- SVE intrinsics is preferred.
- Benchmarked VPP on n1sdp on scalability, on 3x CPUs.
- VM2VM
- Transport use cases on VPP. - Govind
- Discussed the node graph and topology.
- Trying to make IPsec enabled with Arm platform. - Govind
- Currently working on non-encryption optimization with PMD driver.
- Focus both non-encryption and encryption cases.
- Basic IPsec functions are working. Will do benchmarking per CPU core.
- Trying to reproduce the CSIT VPP device testing on local servers - Jieqiang
- Will try to run VPP device testing on local ThunderX2 servers with XL710 25G NICs.
- After VPP device scripts can run well on local servers, Jieqiang can investigate more features, IPv4, IPv6, Tunnel, IPSec.
- Plans
- Benchmarking btw gcc-10.1.0, clang-10/clang-9 and gcc-9.2.0 on Arm machines - Jieqiang
08/18/2020
- Attendees
- Govindarajan Mohandoss
- Juraj Linkes
- Jieqiang Wang
- Tina Tsou
- General
- CSIT
- VPP Performance Test
- Jieqiang is investigating some performance drop (between 2005 and 2008 releases) cases on Taishan servers.
- https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-3n-tsh/
- They have finished collecting data with performance testing setup, and the mrr daily is resumed
- FD.io CSIT-2005 Release Report was released, https://docs.fd.io/csit/rls2005/report/
- Jieqiang will share investigation report, but so far there is no apparent performance differences.
- Some 4T4C test cases on Taishan have obvious performance drop. Will compare the trending with x86 machines.
- VPP Path
- One ThunderX1 has faulty RAM. Will try to replace all the RAMs.
- The second ThunderX1 has IPMI problem, but SSH is working fine.
- Juraj will follow or create new vexxhost ticket to replace faulty RAM.
- 3x spare ThunderX servers are used for CI and included in Nomad cluster.
- Faulty RAM on TX server is not fixed and yet to be debugged.
- Pending with Vexx host to proceed further.
- Investigate adding CentOS-7 on Arm Jenkins jobs - Juraj & Jieqiang
- Will confirm with Dave W. if he will add this Jenkins job and if he requires any help - Jieqiang
- Questions on the docker file upload.
- The Dockerfile has been upstreamed for review and merge.
- Juraj/Jieqiang to help Dave Wallace to fix the script issues. Currently, the build process is done manually and will be automated.
- VPP Device
- Two ThunderX2 servers are received by Vexx host and currently in the storage warehouse.
- Vexx host people will setup the servers and provide IP connectivity. Juraj will install the necessary software after that.
- VPP Performance Test
- VPP
- VPP hoststack TCP/CPS(Connnection per Second) investigation;
- Initial benchmarking with VPP hostack on N1SDP was done. 29Gb/s on N1SDP and 26Gb/s on Haswell.
- Investigating vlib_timer and timer wheel in VPP.
- Benchmarking btw gcc-10.1.0, clang-10/clang-9 and gcc-9.2.0 on Arm machines - Jieqiang
- gcc-10.1.0 has compiling errors with latest VPP source code.
- This issue is fixed by Jieqiang and available for internal review.
- gcc-10.1.0 has compiling errors with latest VPP source code.
- N1SDP enablement. - Lijian
- Preparing patches to enable creating big tables on huge-pages
- IP4-rewrite refactor patch brings performance improvement especially with 10K flows
- Upstreamed and are using csit testing to verify the patch.
- ACL optimization investigation on n1sdp - Govind
- Patch to remove redundancy prefetches are committed - Govind
- Trying to make IPsec enabled with Arm platform. - Govind
- Currently working on non-encryption optimization with PMD driver.
- Focus both non-encryption and encryption cases.
- Basic IPsec functions are working. Will do benchmarking per CPU core.
- Trying to reproduce the CSIT VPP device testing on local servers - Jieqiang
- Will try to run VPP device testing on local ThunderX2 servers with XL710 25G NICs.
- After VPP device scripts can run well on local servers, Jieqiang can investigate more features, IPv4, IPv6, Tunnel, IPSec.
- Plans
- VPP hoststack TCP/CPS(Connnection per Second) investigation;
08/11/2020
- Attendees
- Honnappa Nagarahalli
- Govindarajan Mohandoss
- Juraj Linkes
- Jieqiang Wang
- Tina Tsou
- Lijian Zhang
- Filip Varga
- General
- CSIT
- VPP Performance Test
- Jieqiang is investigating some performance drop cases on Taishan servers.
- https://jenkins.fd.io/view/csit/job/csit-vpp-perf-mrr-daily-master-3n-tsh/
- They have finished collecting data with performance testing setup, and the mrr daily is resumed
- FD.io CSIT-2005 Release Report was released, https://docs.fd.io/csit/rls2005/report/
- Jieqiang will share investigation report, but so far there is no apparent performance differences.
- Some 4T4C test cases on Taishan have obvious performance drop. Will compare the trending with x86 machines.
- VPP Path
- One ThunderX1 has faulty RAM. Will try to replace all the RAMs.
- The second ThunderX1 has IPMI problem, but SSH is working fine.
- Juraj will follow or create new vexxhost ticket to replace faulty RAM.
- 3x spare ThunderX servers are used for CI and included in Nomad cluster.
- Faulty RAM on TX server is not fixed and yet to be debugged.
- Investigate adding CentOS-7 on Arm Jenkins jobs - Juraj & Jieqiang
- Will confirm with Dave W. if he will add this Jenkins job and if he requires any help - Jieqiang
- Questions on the docker file upload.
- The Dockerfile has been upstreamed for review and merge.
- VPP Device
- Two ThunderX2 information will be confirmed with FD.io CSIT lab admin. - Jieqiang
- Commit internal patch to support ThunderX2 in VPP device testing. - Jieqiang
- VPP device job is running now and will be triggered per VPP patch and CSIT patch
- VPP Performance Test
- FD.io lab
- Two ThunderX2 servers have been collected and shipped, and will target to arrive on July 30th.
- VPP
- VPP hoststack TCP/CPS(Connnection per Second) investigation;
- Initial benchmarking with VPP hostack on N1SDP was done. 29Gb/s on N1SDP and 26Gb/s on Haswell.
- Investigating vlib_timer and timer wheel in VPP.
- Benchmarking btw gcc-10.1.0, clang-10/clang-9 and gcc-9.2.0 on Arm machines - Jieqiang
- gcc-10.1.0 has compiling errors with latest VPP source code.
- N1SDP enablement. - Lijian
- Preparing patches to enable creating big tables on huge-pages
- IP4-rewrite refactor patch brings performance improvement especially with 10K flows
- Upstreamed and are using csit testing to verify the patch.
- ACL optimization investigation on n1sdp - Govind
- Patch to remove redundancy prefetches are committed - Govind
- Trying to make IPsec enabled with Arm platform. - Govind
- Currently working on non-encryption optimization with PMD driver.
- Focus both non-encryption and encryption cases.
- Basic IPsec functions are working. Will do benchmarking per CPU core.
- Trying to reproduce the CSIT VPP device testing on local servers - Jieqiang
- Will try to run VPP device testing on local ThunderX2 servers with XL710 25G NICs.
- After VPP device scripts can run well on local servers, Jieqiang can investigate more features, IPv4, IPv6, Tunnel, IPSec.
- Plans
- VPP hoststack TCP/CPS(Connnection per Second) investigation;
08/04/2020
- Attendees
- Honnappa Nagarahalli
- Govindarajan Mohandoss
- Juraj Linkes
- Jieqiang Wang
- Tina Tsou
- Lijian Zhang
- Filip Varga
- General
- CSIT
- VPP Performance Test
- They have finished collecting data with performance testing setup, and the mrr daily is resumed
- FD.io CSIT-2005 Release Report was released, https://docs.fd.io/csit/rls2005/report/
- Jieqiang will share investigation report, but so far there is no apparent performance differences.
- Some 4T4C test cases on Taishan have obvious performance drop. Will compare the trending with x86 machines.
- VPP Path
- One ThunderX1 has faulty RAM. Will try to replace all the RAMs.
- The second ThunderX1 has IPMI problem, but SSH is working fine.
- Juraj will follow or create new vexxhost ticket to replace faulty RAM.
- 3x spare ThunderX servers are used for CI and included in Nomad cluster.
- Faulty RAM on TX server is not fixed and yet to be debugged.
- Investigate adding CentOS-7 on Arm Jenkins jobs - Juraj & Jieqiang
- Will confirm with Dave W. if he will add this Jenkins job and if he requires any help - Jieqiang
- Questions on the docker file upload.
- The Dockerfile has been upstreamed for review and merge.
- VPP Device
- Two ThunderX2 information will be confirmed with FD.io CSIT lab admin. - Jieqiang
- Commit internal patch to support ThunderX2 in VPP device testing. - Jieqiang
- VPP device job is running now and will be triggered per VPP patch and CSIT patch
- VPP Performance Test
- FD.io lab
- Two ThunderX2 servers have been collected and shipped, and will target to arrive on July 30th.
- VPP
- VPP hoststack TCP/CPS(Connnection per Second) investigation;
- Initial benchmarking with VPP hostack on N1SDP was done. 29Gb/s on N1SDP and 26Gb/s on Haswell.
- Investigating vlib_timer and timer wheel in VPP.
- Benchmarking btw gcc-10.1.0, clang-10/clang-9 and gcc-9.2.0 on Arm machines - Jieqiang
- gcc-10.1.0 has compiling errors with latest VPP source code.
- N1SDP enablement. - Lijian
- Preparing patches to enable creating big tables on huge-pages
- IP4-rewrite refactor patch brings performance improvement especially with 10K flows
- Upstreamed and are using csit testing to verify the patch.
- ACL optimization investigation on n1sdp - Govind
- Patch to remove redundancy prefetches are committed - Govind
- Trying to make IPsec enabled with Arm platform. - Govind
- Currently working on non-encryption optimization with PMD driver.
- Focus both non-encryption and encryption cases.
- Basic IPsec functions are working. Will do benchmarking per CPU core.
- Trying to reproduce the CSIT VPP device testing on local servers - Jieqiang
- Will try to run VPP device testing on local ThunderX2 servers with XL710 25G NICs.
- After VPP device scripts can run well on local servers, Jieqiang can investigate more features, IPv4, IPv6, Tunnel, IPSec.
- Plans
- VPP hoststack TCP/CPS(Connnection per Second) investigation;
07/28/2020
- Attendees
- Honnappa Nagarahalli
- Govindarajan Mohandoss
- Juraj Linkes
- Jieqiang Wang
- Tina Tsou
- Lijian Zhang
- General
- CSIT
- VPP Performance Test
- They have finished collecting data with performance testing setup, and the mrr daily is resumed
- FD.io CSIT-2005 Release Report was released, https://docs.fd.io/csit/rls2005/report/
- Jieqiang will share investigation report, but so far there is no apparent performance differences.
- VPP performance testing is running once a week.
- VPP Path
- One ThunderX1 has faulty RAM. Will try to replace all the RAMs.
- The second ThunderX1 has IPMI problem, but SSH is working fine.
- Juraj will follow or create new vexxhost ticket to replace faulty RAM.
- 3x spare ThunderX servers are used for CI and included in Nomad cluster.
- Faulty RAM on TX server is not fixed and yet to be debugged.
- Investigate adding CentOS-7 on Arm Jenkins jobs - Juraj & Jieqiang
- Will confirm with Dave W. if he will add this Jenkins job and if he requires any help - Jieqiang
- Questions on the docker file upload.
- The Dockerfile has been upstreamed for review and merge.
- VPP Device
- Two ThunderX2 information will be confirmed with FD.io CSIT lab admin. - Jieqiang
- Commit internal patch to support ThunderX2 in VPP device testing. - Jieqiang
- VPP device job is running now and will be triggered per VPP patch and CSIT patch
- VPP Performance Test
- FD.io lab
- Two ThunderX2 servers have been collected and shipped, and will target to arrive on July 30th.
- VPP
- VPP hoststack TCP/CPS(Connnection per Second) investigation;
- Initial benchmarking with VPP hostack on N1SDP was done. 29Gb/s on N1SDP and 26Gb/s on Haswell.
- Investigating vlib_timer and timer wheel in VPP.
- Benchmarking btw gcc-10.1.0, clang-10/clang-9 and gcc-9.2.0 on Arm machines - Jieqiang
- N1SDP enablement. - Lijian
- Preparing patches to enable creating big tables on huge-pages
- IP4-rewrite refactor patch brings performance improvement especially with 10K flows
- Upstreamed and are using csit testing to verify the patch.
- ACL optimization investigation on n1sdp - Govind
- Patch to remove redundancy prefetches are committed - Govind
- Trying to make IPsec enabled with Arm platform. - Govind
- Focus both non-encryption and encryption cases.
- Basic IPsec functions are working. Will do benchmarking per CPU core.
- Trying to reproduce the CSIT VPP device testing on local servers - Jieqiang
- Will try to run VPP device testing on local ThunderX2 servers with XL710 25G NICs.
- After VPP device scripts can run well on local servers, Jieqiang can investigate more features, IPv4, IPv6, Tunnel, IPSec.
- Plans
- VPP hoststack TCP/CPS(Connnection per Second) investigation;
07/21/2020
- Attendees
- Honnappa Nagarahalli
- Govindarajan Mohandoss
- Juraj Linkes
- Jieqiang Wang
- Tina Tsou
- Lijian Zhang
- General
- CSIT
- VPP Performance Test
- FD.io CSIT-2005 Release Report was released, https://docs.fd.io/csit/rls2005/report/
- VPP performance testing is running once a week.
- VPP Path
- Juraj will follow or create new vexxhost ticket to replace faulty RAM.
- 3x spa
- Juraj will follow or create new vexxhost ticket to replace faulty RAM.
- VPP Performance Test
- VPP 2022 Make Test Use Case Poll
- VPP-AArch64
- VPP-ABF
- VPP Alternative Builds
- VPP API Concepts
- VPP API Versioning
- VPP-ApiChangeProcess
- VPP-ArtifactVersioning
- VPP-BIER
- VPP-Bihash
- VPP-BugReports
- VPP Build System Deep Dive
- VPP Build, Install, And Test Images
- VPP-BuildArtifactRetentionPolicy
- VPP-c2cpel
- VPP Code Walkthrough VoD
- VPP Code Walkthrough VoD Topic Index
- VPP Code Walkthrough VoDs
- VPP-CodeStyleConventions
- VPP-CodingTips
- VPP Command Line Arguments
- VPP Command Line Interface CLI Guide
- VPP-CommitMessages
- VPP-Committers-SMEs
- VPP-CommitterTasks-ApiFreeze
- VPP CommitterTasks Compare API Changes
- VPP-CommitterTasks-CutPointRelease
- VPP-CommitterTasks-CutRelease
- VPP-CommitterTasks-FinalReleaseCandidate
- VPP-CommitterTasks-PullThrottleBranch
- VPP-CommitterTasks-ReleasePlan
- VPP Configuration Tool
- VPP Configure An LW46 MAP E Terminator
- VPP Configure VPP As A Router Between Namespaces
- VPP Configure VPP TAP Interfaces For Container Routing
- VPP-CoreFileMismatch
- VPP-cpel
- VPP-cpeldump
- VPP-CurrentData
- VPP-DHCPKit
- VPP-DHCPv6
- VPP-DistributedOwnership
- VPP-Documentation
- VPP DPOs And Feature Arcs
- VPP EC2 Instance With SRIOV
- VPP-elog
- VPP-FAQ
- VPP Feature Arcs
- VPP-Features
- VPP-Features-IPv6
- VPP-FIB
- VPP-g2
- VPP Getting VPP 16.06
- VPP Getting VPP Release Binaries
- VPP-HA
- VPP-HostStack
- VPP-HostStack-BuiltinEchoClientServer
- VPP-HostStack-EchoClientServer
- VPP-HostStack-ExternalEchoClientServer
- VPP HostStack Hs Test
- VPP-HostStack-LDP-iperf
- VPP-HostStack-LDP-nginx
- VPP-HostStack-LDP-sshd
- VPP-HostStack-nginx
- VPP-HostStack-SessionLayerArchitecture
- VPP-HostStack-TestHttpServer
- VPP-HostStack-TestProxy
- VPP-HostStack-TLS
- VPP-HostStack-VCL
- VPP-HostStack-VclEchoClientServer
- VPP-Hotplug
- VPP How To Add A Tunnel Encapsulation
- VPP How To Build The Sample Plugin
- VPP How To Connect A PCI Interface To VPP
- VPP How To Create A VPP Binary Control Plane API
- VPP How To Deploy VPP In EC2 Instance And Use It To Connect Two Different VPCs
- VPP How To Optimize Performance %28System Tuning%29
- VPP How To Use The API Trace Tools
- VPP How To Use The C API
- VPP How To Use The Packet Generator And Packet Tracer
- VPP-Howtos
- VPP-index
- VPP Installing VPP Binaries From Packages
- VPP Interconnecting vRouters With VPP
- VPP Introduction To IP Adjacency
- VPP Introduction To N Tuple Classifiers
- VPP IP Adjacency Introduction
- VPP-IPFIX
- VPP-IPSec
- VPP IPSec And IKEv2
- VPP IPv6 SR VIRL Topology File
- VPP Java API
- VPP Java API Plugin Support
- VPP Jira Workflow
- VPP-Macswapplugin
- VPP-MakeTestFramework
- VPP-Meeting
- VPP-MFIB
- VPP Missing Prefetches
- VPP Modifying The Packet Processing Directed Graph
- VPP MPLS FIB
- VPP-NAT
- VPP Nataas Test
- VPP-OVN
- VPP Per Feature Notes
- VPP Performance Analysis Tools
- VPP-perftop
- VPP Progressive VPP Tutorial
- VPP Project Meeting Minutes
- VPP Pulling, Building, Running, Hacking And Pushing VPP Code
- VPP Pure L3 Between Namespaces With 32s
- VPP Pure L3 Container Networking
- VPP Pushing And Testing A Tag
- VPP Python API
- VPP-PythonVersionPolicy
- VPP-QuickTrexSetup
- VPP Random Hints And Kinks For KVM Usage
- VPP Release Plans Release Plan 16.09
- VPP Release Plans Release Plan 17.01
- VPP Release Plans Release Plan 17.04
- VPP Release Plans Release Plan 17.07
- VPP Release Plans Release Plan 17.10
- VPP Release Plans Release Plan 18.01
- VPP Release Plans Release Plan 18.04
- VPP Release Plans Release Plan 18.07
- VPP Release Plans Release Plan 18.10
- VPP Release Plans Release Plan 19.01
- VPP Release Plans Release Plan 19.04
- VPP Release Plans Release Plan 19.08
- VPP Release Plans Release Plan 20.01
- VPP Release Plans Release Plan 20.05
- VPP Release Plans Release Plan 20.09
- VPP Release Plans Release Plan 21.01
- VPP Release Plans Release Plan 21.06
- VPP Release Plans Release Plan 21.10
- VPP Release Plans Release Plan 22.02
- VPP Release Plans Release Plan 22.06
- VPP Release Plans Release Plan 22.10
- VPP Release Plans Release Plan 23.02
- VPP Release Plans Release Plan 23.06
- VPP Release Plans Release Plan 23.10
- VPP Release Plans Release Plan 24.02
- VPP Release Plans Release Plan 24.06
- VPP Release Plans Release Plan 24.10
- VPP Release Plans Release Plan 25.02
- VPP Release Plans Release Plan 25.06
- VPP Release Plans Release Plan 25.10
- VPP Release Plans Release Plan 26.02
- VPP Release Plans Release Plan 26.06
- VPP-RM
- VPP-SecurityGroups
- VPP Segment Routing For IPv6
- VPP Segment Routing For MPLS
- VPP Setting Up Your Dev Environment
- VPP-SNAT
- VPP Software Architecture
- VPP STN Testing
- VPP The VPP API
- VPP Training Events
- VPP-Troubleshooting
- VPP-Troubleshooting-BuildIssues
- VPP-Troubleshooting-Vagrant
- VPP Tutorial DPDK And MacSwap
- VPP Tutorial Routing And Switching
- VPP-Tutorials
- VPP Use VPP To Chain VMs Using Vhost User Interface
- VPP Use VPP To Connect VMs Using Vhost User Interface
- VPP Using mTCP User Mode TCP Stack With VPP
- VPP Using VPP As A VXLAN Tunnel Terminator
- VPP Using VPP In A Multi Thread Model
- VPP-VOM
- VPP VPP BFD Nexus
- VPP VPP Home Gateway
- VPP VPP WIKI DEPRECATED CONTENT
- VPP-VPPCommunicationsLibrary
- VPP-VPPConfig
- VPP What Is ODP4VPP
- VPP What Is VPP
- VPP Working Environments
- VPP Working With The 16.06 Throttle Branch