Performance Analyses for DPDK Applications Using the Ethdev Library #83

adel-belkhiri · 2024-09-26T17:15:09Z

This PR introduces 5 performance analyses for DPDK applications implemented using the ethdev library. These analyses provide insights into packet distribution, rate, throughput, and PMD thread busyness, thus helping monitor and optimize application performance.

New Features:

Packet Rate Analysis:
Computes the packet rate (in pps) for transmitted and received Ethernet traffic on a per-queue basis. It helps monitor the efficiency of traffic processing by calculating the number of packets received or transmitted in a given period.
Packet Throughput Analysis:
Analyzes packet throughput (in bps) for transmitted and received Ethernet traffic. This is done on a per-queue basis to evaluate bandwidth usage.
Busyness Analysis for PMD Threads:
Estimates how busy PMD threads are by comparing the times they successfully retrieve packets from the NIC queue versus when they are merely spinning without processing any packets. This can be used to evaluate the efficiency of polling operations.
Packet Distribution Analysis:
Analyzes the distribution of packets retrieved in a single rte_eth_rx_burst() call, on a per-thread and per-queue basis. This analysis provides insights into packet retrieval patterns, including how evenly or unevenly the workload is distributed across queues.
Packet Distribution Statistics:
Computes various statistics related to packet retrieval distribution, including the minimum, maximum, average, and standard deviation for packets retrieved in a single rte_eth_rx_burst() call, which offers more granular visibility into traffic distribution patterns.

MatthewKhouzam

minor changes.

...c/org/eclipse/tracecompass/incubator/internal/dpdk/core/ethdev/rate/analysis/Attributes.java

MatthewKhouzam · 2024-09-26T19:14:36Z

...c/org/eclipse/tracecompass/incubator/internal/dpdk/core/ethdev/rate/analysis/Attributes.java

+ *
+ * @author Adel Belkhiri
+ */
+@SuppressWarnings({ "nls" })


@NonNullByDefault

MatthewKhouzam · 2024-09-26T19:17:23Z

...e/tracecompass/incubator/internal/dpdk/core/ethdev/rate/analysis/DpdkEthdevEventHandler.java

+        try {
+            StateSystemBuilderUtils.incrementAttributeLong(ssb, ts, queueQuark, nbPkts);
+        } catch (StateValueTypeException e) {
+            Activator.getInstance().logWarning(getClass().getName() + ": problem accessing the state of a NIC queue (Quark =" + String.valueOf(queueQuark) + ")"); //$NON-NLS-1$ //$NON-NLS-2$


I really like this... can you put it in the activator to have it everywhere?

...e/tracecompass/incubator/internal/dpdk/core/ethdev/rate/analysis/DpdkEthdevEventHandler.java

...se/tracecompass/incubator/internal/dpdk/core/ethdev/rate/analysis/DpdkEthdevEventLayout.java

...racecompass/incubator/internal/dpdk/core/ethdev/rate/analysis/DpdkEtherRateDataProvider.java

...rg/eclipse/tracecompass/incubator/internal/dpdk/core/ethdev/rate/analysis/IODataPalette.java

arfio

I think all the analyses look very interesting but there is a lot of duplicated that can easily be reused throughout. In particular the throughput and rate analysis that are very similar, it could be merged into one analysis with multiple views.

...ss/incubator/internal/dpdk/core/ethdev/poll/distribution/analysis/DpdkEthdevEventLayout.java

...mpass/incubator/internal/dpdk/core/ethdev/poll/distribution/analysis/TmfEtherPollAspect.java

...cecompass/incubator/internal/dpdk/core/ethdev/poll/stats/analysis/DpdkEthdevEventLayout.java

...e/tracecompass/incubator/internal/dpdk/core/ethdev/rate/analysis/DpdkEthdevEventHandler.java

...ecompass/incubator/internal/dpdk/core/ethdev/spin/analysis/DpdkEthdevSpinAnalysisModule.java

...racecompass/incubator/internal/dpdk/core/ethdev/spin/analysis/DpdkEtherSpinDataProvider.java

MatthewKhouzam

Small changes needed

.../internal/dpdk/core/ethdev/throughput/analysis/AbstractDpdkEthdevThroughputDataProvider.java

...bator/internal/dpdk/core/ethdev/throughput/analysis/DpdkEthdevThroughputBpsDataProvider.java

...bator/internal/dpdk/core/ethdev/throughput/analysis/DpdkEthdevThroughputPpsDataProvider.java

...g/eclipse/tracecompass/incubator/internal/dpdk/core/ethdev/throughput/analysis/Messages.java

.../src/org/eclipse/tracecompass/incubator/internal/dpdk/ui/ethdev/throughput/bps/Messages.java

...tracecompass/incubator/internal/dpdk/ui/ethdev/throughput/bps/NicQueueThroughputBpsView.java

.../src/org/eclipse/tracecompass/incubator/internal/dpdk/ui/ethdev/throughput/pps/Messages.java

arfio

A lot of small comments, once they are fixed, I will be happy to approve !

analyses/org.eclipse.tracecompass.incubator.dpdk.core/META-INF/MANIFEST.MF

...bator/internal/dpdk/core/ethdev/poll/distribution/analysis/DpdkPollDistributionAnalysis.java

tracetypes/org.eclipse.tracecompass.incubator.dpdk.core/META-INF/MANIFEST.MF

...rc/org/eclipse/tracecompass/incubator/internal/dpdk/core/analysis/DpdkEthdevEventLayout.java

...bator/internal/dpdk/core/ethdev/throughput/analysis/DpdkEthdevThroughputPpsDataProvider.java

...rg/eclipse/tracecompass/incubator/internal/dpdk/ui/ethdev/spin/ThreadSpinStatisticsView.java

...tracecompass/incubator/internal/dpdk/ui/ethdev/throughput/bps/NicQueueThroughputBpsView.java

...tracecompass/incubator/internal/dpdk/ui/ethdev/throughput/pps/NicQueueThroughputPpsView.java

arfio · 2025-01-07T16:41:25Z

I just need a small trace to test the view and the remaining comments to be fixed.

Introduces packet throughput analysis for DPDK applications using the ethdev library. The analysis calculates packet throughput in both bits per second (bps) and packets per second (pps) for transmitted and received Ethernet traffic on a per-queue basis. Note that the calculation of throughput in bps requires the existence of custom profiling events in the trace Signed-off-by: Adel Belkhiri <[email protected]>

PMD threads continuously poll NIC queues, leading to constant 100% CPU usage. This analysis provides a rough estimation of how busy PMD threads are with actual work, by comparing the times they successfully retrieve packets from the NIC queue versus the times they are merely spinning without processing any packets. Signed-off-by: Adel Belkhiri <[email protected]>

Introduce packet distribution analysis for PMD threads based on ethdev library. This analysis computes the distribution of packets retrieved in a single rte_eth_rx_burst() call, on a per-queue basis. Signed-off-by: Adel Belkhiri <[email protected]>

Introduce packet distribution statistics analysis for PMD threads in the ethdev library. This analysis calculates various statistics related to the distribution of packets retrieved in a single rte_eth_rx_burst() call, on a per-thread and per-queue basis. The computed statistics include the minimum, maximum, average number of packets retrieved, as well as the standard deviation. Signed-off-by: Adel Belkhiri <[email protected]>

adel-belkhiri · 2025-01-29T17:03:45Z

I just need a small trace to test the view and the remaining comments to be fixed.

For anyone who wants to try the analyses, here is a link to download a test trace: https://drive.google.com/file/d/1w9hZUSRihlgyPEBdYDWMZZ10wlYpByUH/view?usp=sharing

arfio

Perfect for me! Thank you!
Matthew may still have some comments.

MatthewKhouzam

My issues were addressed. Thank you for the great contribution!

MatthewKhouzam requested changes Sep 26, 2024

View reviewed changes

arfio self-requested a review September 27, 2024 19:04

arfio requested changes Sep 30, 2024

View reviewed changes

adel-belkhiri force-pushed the dpdk-ethdev branch 2 times, most recently from 0b9fd6f to d2f9125 Compare October 16, 2024 15:54

MatthewKhouzam requested changes Nov 4, 2024

View reviewed changes

adel-belkhiri force-pushed the dpdk-ethdev branch from d2f9125 to 12cfb9a Compare November 12, 2024 17:25

adel-belkhiri mentioned this pull request Nov 12, 2024

Move the initial dpdk analysis to the 'tracetypes' directory #117

Merged

adel-belkhiri force-pushed the dpdk-ethdev branch from 12cfb9a to 7a8e39d Compare November 22, 2024 20:03

arfio requested changes Nov 28, 2024

View reviewed changes

adel-belkhiri force-pushed the dpdk-ethdev branch 2 times, most recently from 451df36 to d21e82b Compare December 16, 2024 20:08

adel-belkhiri force-pushed the dpdk-ethdev branch 2 times, most recently from a36ba23 to 0b0f9b6 Compare January 23, 2025 16:54

adel-belkhiri added 4 commits January 29, 2025 11:46

adel-belkhiri force-pushed the dpdk-ethdev branch from 0b0f9b6 to 422de92 Compare January 29, 2025 16:51

arfio self-requested a review February 3, 2025 16:47

arfio approved these changes Feb 3, 2025

View reviewed changes

MatthewKhouzam approved these changes Feb 3, 2025

View reviewed changes

MatthewKhouzam merged commit 8917c4c into eclipse-tracecompass-incubator:master Feb 3, 2025
2 checks passed

Performance Analyses for DPDK Applications Using the Ethdev Library #83

Performance Analyses for DPDK Applications Using the Ethdev Library #83

Uh oh!

Conversation

adel-belkhiri commented Sep 26, 2024

Uh oh!

MatthewKhouzam left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

MatthewKhouzam Sep 26, 2024

Choose a reason for hiding this comment

Uh oh!

adel-belkhiri Oct 15, 2024

Choose a reason for hiding this comment

Uh oh!

MatthewKhouzam Sep 26, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

arfio left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

MatthewKhouzam left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

arfio left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

arfio commented Jan 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

adel-belkhiri commented Jan 29, 2025

Uh oh!

arfio left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MatthewKhouzam left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

arfio commented Jan 7, 2025 •

edited

Loading

arfio left a comment •

edited

Loading