performance testing: number of subscriptions vs. function latency by diana-qing · Pull Request #111 · stanford-esrg/retina

diana-qing · 2025-06-12T03:28:05Z

This PR adds scripts to measure how the latency of a function when running the ip_subs application changes as the number of subscriptions changes.

…already profiled the given sub count

thearossman · 2025-06-12T14:11:16Z

Cargo.toml

    "examples/log_ssh",
-    "examples/streaming",
+    "examples/streaming", 
+    "examples/ip_subs",


Can you move this into the tests/perf folder?

thearossman · 2025-06-12T14:11:41Z

examples/ip_subs/src/main.rs

@@ -0,0 +1,49 @@
+use retina_core::{Runtime, config::load_config};


Add a README for this example.

thearossman · 2025-06-12T14:12:05Z

tests/perf/README.md

@@ -0,0 +1,31 @@
+# Performance Testing


Add an intro with the high-level motivation for this and what it does! What you shared at the EOQ lab meeting was great.

Also mention the initial testing you did to ensure that this approach is accurate!

Here's my best understanding of what you found:

You compared results with Retina's current timing infrastructure, which inlines cycle counts. You found that the uprobes add a constant overhead. That is, this will accurately surface patterns for the use-case of comparing function latency across different implementations or applications.

You can't run this at super high throughputs. IIRC, we were able to handle ~5Gbps of live traffic (unless you got more on the passive box). This gives plenty of data points for saying something about function latency.

You confirmed this separates entry/exit points by thread, so it'll be accurate even if there are multiple cores. (IMO this was a bit unclear in the documentation.)

thearossman · 2025-06-12T14:13:43Z

tests/perf/README.md

+```
+
+## Number of Subscriptions vs. Function Latency
+`generate_ip_subs.py` shards the IPv4 address space into `n` subnets to generate `n` Retina subscriptions, where `n` is passed in by the user. The subscriptions are written to `spec.toml`.


Maybe clarify that this is a sample / basic application and more can easily be added. The main goal of your project was to set up the infrastructure.

thearossman · 2025-06-12T14:18:18Z

tests/perf/README.md

+
+`run_app.py` runs the `ip_subs` application and measures how the latency of a function changes as the number of subscriptions changes. It generates subscriptions using `generate_ip_subs.py`, then runs `ip_subs` with these subscriptions and measures latency using `func_latency.py`. The latencies are written to `stats/ip_subs_latency_stats.csv` and plots on the number of subscriptions vs. latency for different stats (e.g. average, 99th percentile) can be found in the `figs` directory. The `stats` and `figs` directory get created by the script if they don't already exist.
+
+When running `run_app.py`, you can specify which function to profile, the number of subscriptions, and the config file path. For example, to measure the latency of the `process_packet` function in online mode when the number of subscriptions is 64 and 256, you can run:


Probably mention that you can profile multiple functions, but because it just records entry/exit timestamps, keep in mind that profiling functions that overlap will cause interference. (You observed this!)

thearossman · 2025-06-12T14:19:14Z

tests/perf/README.md

+
+`func_latency.py` uses bcc to profile function latency when running an application by attaching eBPF programs to uprobes at the entry and exit point of functions. Latency is measured in nanoseconds by default. The code for profiling function latency was based on the [example provided by bcc](https://github.com/iovisor/bcc/blob/master/tools/funclatency.py).
+
+`run_app.py` runs the `ip_subs` application and measures how the latency of a function changes as the number of subscriptions changes. It generates subscriptions using `generate_ip_subs.py`, then runs `ip_subs` with these subscriptions and measures latency using `func_latency.py`. The latencies are written to `stats/ip_subs_latency_stats.csv` and plots on the number of subscriptions vs. latency for different stats (e.g. average, 99th percentile) can be found in the `figs` directory. The `stats` and `figs` directory get created by the script if they don't already exist.


Should this be run from a specific directory within the Retina repo?

thearossman · 2025-06-12T14:22:56Z

tests/perf/func_latency.py

@@ -0,0 +1,206 @@
+# code for profiling function latency with bcc based on https://github.com/iovisor/bcc/blob/master/tools/funclatency.py


We had talked a bit about managing output a couple of weeks ago in online mode by consuming the subprocess output and filtering it before printing:

Making it so that the "samples lost" alert isn't printed

Consuming the output and printing the updates on Gbps processed, packets lost, etc.

Did you try this and run into challenges? (I think this is not critical for accuracy, but it is extremely helpful for usability if it's reasonably easy to do.)

thearossman · 2025-06-12T14:23:13Z

tests/perf/generate_ip_subs.py

@@ -0,0 +1,49 @@
+import argparse


File header comment

thearossman · 2025-06-12T14:23:27Z

tests/perf/run_app.py

@@ -0,0 +1,128 @@
+import argparse


File header comment

diana-qing added 30 commits May 16, 2025 21:39

set up scripts for generating num subs vs. runtime plots

64a9bf3

use Python's ipaddress library to shard IPv4 addr space

bbe39d5

add process_packet run results

22867a9

update scripts

43a394a

clean and rebuild before each run

35be0b3

use ld lib path env var. force rebuild binary on each run in run_app

1f9540e

add signal to profile func latency and terminate after 10 sec

8d1224b

code cleanup

e63d328

edit path

650ce41

log conn records to file

49cd708

remove signaling code

de07253

rename app to ip_subs, script generate_subs.py to generate_ip_subs.py

d9d7eca

update num subs

60af5f7

remove spec.toml

ca7c739

remove benchmark_app

ff3b447

code cleanup

2dba884

delete figs

da68a4b

write stats to csv

eee480e

code cleanup

86e6da6

more code cleanup

3988168

code cleanup x3

c139245

use abs path to python interpreter

b422b44

code cleanup

82352bd

get latency per pid

e86c03c

add force-execute arg. if force-execute is false, profile if haven't …

12df91a

…already profiled the given sub count

code cleanup

22cba67

code cleanup x2

e06ffa7

code cleanup x3

512151d

delete comment

5e29652

add documentation

33ab2b3

diana-qing added 4 commits June 12, 2025 03:16

update docs

ad709a1

tests/perf/README.md

0ef64a0

fix wording

3f573fb

fmt and clippy

b3a7a02

thearossman reviewed Jun 12, 2025

View reviewed changes

tests/perf/generate_ip_subs.py

@@ -0,0 +1,49 @@

import argparse

Copy link

Collaborator

thearossman Jun 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

File header comment

thearossman reviewed Jun 12, 2025

View reviewed changes

tests/perf/run_app.py

@@ -0,0 +1,128 @@

import argparse

Copy link

Collaborator

thearossman Jun 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

File header comment

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

performance testing: number of subscriptions vs. function latency#111

performance testing: number of subscriptions vs. function latency#111
diana-qing wants to merge 34 commits intostanford-esrg:mainfrom
diana-qing:num-sub

diana-qing commented Jun 12, 2025

Uh oh!

thearossman Jun 12, 2025

Uh oh!

thearossman Jun 12, 2025

Uh oh!

thearossman Jun 12, 2025

Uh oh!

thearossman Jun 12, 2025 •

edited

Loading

Uh oh!

thearossman Jun 12, 2025

Uh oh!

thearossman Jun 12, 2025

Uh oh!

thearossman Jun 12, 2025

Uh oh!

thearossman Jun 12, 2025

Uh oh!

thearossman Jun 12, 2025

Uh oh!

thearossman Jun 12, 2025

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

2 participants

		@@ -0,0 +1,49 @@
		use retina_core::{Runtime, config::load_config};


		`run_app.py` runs the `ip_subs` application and measures how the latency of a function changes as the number of subscriptions changes. It generates subscriptions using `generate_ip_subs.py`, then runs `ip_subs` with these subscriptions and measures latency using `func_latency.py`. The latencies are written to `stats/ip_subs_latency_stats.csv` and plots on the number of subscriptions vs. latency for different stats (e.g. average, 99th percentile) can be found in the `figs` directory. The `stats` and `figs` directory get created by the script if they don't already exist.

		When running `run_app.py`, you can specify which function to profile, the number of subscriptions, and the config file path. For example, to measure the latency of the `process_packet` function in online mode when the number of subscriptions is 64 and 256, you can run:


		`func_latency.py` uses bcc to profile function latency when running an application by attaching eBPF programs to uprobes at the entry and exit point of functions. Latency is measured in nanoseconds by default. The code for profiling function latency was based on the [example provided by bcc](https://github.com/iovisor/bcc/blob/master/tools/funclatency.py).

		`run_app.py` runs the `ip_subs` application and measures how the latency of a function changes as the number of subscriptions changes. It generates subscriptions using `generate_ip_subs.py`, then runs `ip_subs` with these subscriptions and measures latency using `func_latency.py`. The latencies are written to `stats/ip_subs_latency_stats.csv` and plots on the number of subscriptions vs. latency for different stats (e.g. average, 99th percentile) can be found in the `figs` directory. The `stats` and `figs` directory get created by the script if they don't already exist.

		@@ -0,0 +1,206 @@
		# code for profiling function latency with bcc based on https://github.com/iovisor/bcc/blob/master/tools/funclatency.py

Conversation

diana-qing commented Jun 12, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

thearossman Jun 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

2 participants

thearossman Jun 12, 2025 •

edited

Loading