Skip to content

Commit 1c752cc

Browse files
authored
Rename PetClinic and Agent (#81)
In this commit, we are simply renaming PetClinic and Agent to VehicleInventoryService and DistroConfig. We are doing this change so we can have multiple people working on this in parallel without stepping on toes. Note that there are a few occurences of PetClinic/Agent that were not changed - in these cases, we know that we will be deleting this logic, so we did not bother updating now (e.g. AgentResolver). Testing: ``` dev-dsk-thp-2a-bd3a87ed % ./gradlew test > Task :compileTestJava Note: Some input files use or override a deprecated API. Note: Recompile with -Xlint:deprecation for details. Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0. You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins. For more on this, please refer to https://docs.gradle.org/8.6/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation. BUILD SUCCESSFUL in 5m 45s 3 actionable tasks: 3 executed ``` By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.
1 parent 4a38939 commit 1c752cc

14 files changed

+150
-134
lines changed

performance-tests/README.md

Lines changed: 16 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -3,14 +3,14 @@
33
- [Process](#process)
44
- [What do we measure?](#what-do-we-measure)
55
- [Config](#config)
6-
- [Agents](#agents)
6+
- [DistroConfigs](#distroConfigs)
77
- [Automation](#automation)
88
- [Setup and Usage](#setup-and-usage)
99
- [Visualization](#visualization)
1010

1111
This directory will contain tools and utilities
1212
that help us to measure the performance overhead introduced by
13-
the agent and to measure how this overhead changes over time.
13+
the distro and to measure how this overhead changes over time.
1414

1515
The overhead tests here should be considered a "macro" benchmark. They serve to measure high-level
1616
overhead as perceived by the operator of a "typical" application. Tests are performed on a Java 11
@@ -20,25 +20,25 @@ distribution from [Eclipse Temurin](https://projects.eclipse.org/projects/adopti
2020

2121
There is one dynamic test here called [OverheadTests](https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/benchmark-overhead/src/test/java/io/opentelemetry/OverheadTests.java).
2222
The `@TestFactory` method creates a test pass for each of the [defined configurations](https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/benchmark-overhead/src/test/java/io/opentelemetry/config/Configs.java).
23-
Before the tests run, a single collector instance is started. Each test pass has one or more agents configured and those are tested in series.
24-
For each agent defined in a configuration, the test runner (using [testcontainers](https://www.testcontainers.org/)) will:
23+
Before the tests run, a single collector instance is started. Each test pass has one or more distroConfigs and those are tested in series.
24+
For each distro defined in a configuration, the test runner (using [testcontainers](https://www.testcontainers.org/)) will:
2525

2626
1. create a fresh postgres instance and populate it with initial data.
27-
2. create a fresh instance of [spring-petclinic-rest](https://github.com/spring-petclinic/spring-petclinic-rest) instrumented with the specified agent
27+
2. create a fresh instance of [spring-petclinic-rest](https://github.com/spring-petclinic/spring-petclinic-rest) instrumented with the specified distroConfig
2828
3. measure the time until the petclinic app is marked "healthy" and then write it to a file.
2929
4. if configured, perform a warmup phase. During the warmup phase, a bit of traffic is generated in order to get the application into a steady state (primarily helping facilitate jit compilations). Currently, we use a 30 second warmup time.
3030
5. start a JFR recording by running `jcmd` inside the petclinic container
3131
6. run the [k6 test script](https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/benchmark-overhead/k6/basic.js) with the configured number of iterations through the file and the configured number of concurrent virtual users (VUs).
3232
7. after k6 completes, petclinic is shut down
3333
8. after petclinic is shut down, postgres is shut down
3434

35-
And this repeats for every agent configured in each test configuration.
35+
And this repeats for every distro configured in each test configuration.
3636

3737
After all the tests are complete, the results are collected and committed back to the `/results` subdirectory as csv and summary text files.
3838

3939
## What do we measure?
4040

41-
For each test pass, we record the following metrics in order to compare agents and determine
41+
For each test pass, we record the following metrics in order to compare distroConfigs and determine
4242
relative overhead.
4343

4444
| metric name | units | description |
@@ -53,7 +53,7 @@ relative overhead.
5353
| Request p95 | ms | 95th percentile time to handle a single web requ4st (measured at the caller) |
5454
| Iteration mean | ms | average time to do a single pass through the k6 test script |
5555
| Iteration p95 | ms | 95th percentile time to do a single pass through the k6 test script |
56-
| Peak threads | # | Highest number of running threads in the VM, including agent threads |
56+
| Peak threads | # | Highest number of running threads in the VM, including distroConfig threads |
5757
| Network read mean | bits/s | Average network read rate |
5858
| Network write mean | bits/s | Average network write rate |
5959
| Average JVM user CPU | % | Average observed user CPU (range 0.0-1.0) |
@@ -68,26 +68,26 @@ Each config contains the following:
6868

6969
- name
7070
- description
71-
- list of agents (see below)
71+
- list of distroConfigs (see below)
7272
- maxRequestRate (optional, used to throttle traffic)
7373
- concurrentConnections (number of concurrent virtual users [VUs])
7474
- totalIterations - the number of passes to make through the k6 test script
7575
- warmupSeconds - how long to wait before starting conducting measurements
7676

7777
Currently, we test:
7878

79-
- no agent versus latest released agent
80-
- no agent versus latest snapshot
79+
- no distro versus latest released distro
80+
- no distro versus latest snapshot
8181
- latest release vs. latest snapshot
8282

8383
Additional configurations can be created by submitting a PR against the `Configs` class.
8484

85-
### Agents
85+
### DistroConfigs
8686

87-
An agent is defined in code as a name, description, optional URL, and optional additional
88-
arguments to be passed to the JVM (not including `-javaagent:`). New agents may be defined
89-
by creating new instances of the `Agent` class. The `AgentResolver` is used to download
90-
the relevant agent jar for an `Agent` definition.
87+
An distroConfig is defined in code as a name, description, optional URL, and optional additional
88+
arguments to be passed to the JVM (not including `-javaagent:`). New distroConfigs may be defined
89+
by creating new instances of the `Distro` class. The `AgentResolver` is used to download
90+
the relevant distroConfig jar for an `Distro` definition.
9191

9292
## Automation
9393

performance-tests/src/test/java/io/opentelemetry/OverheadTests.java

Lines changed: 27 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -8,13 +8,13 @@
88
import static org.junit.jupiter.api.Assertions.fail;
99
import static org.junit.jupiter.api.DynamicTest.dynamicTest;
1010

11-
import io.opentelemetry.agents.Agent;
1211
import io.opentelemetry.config.Configs;
1312
import io.opentelemetry.config.TestConfig;
1413
import io.opentelemetry.containers.CollectorContainer;
1514
import io.opentelemetry.containers.K6Container;
16-
import io.opentelemetry.containers.PetClinicRestContainer;
1715
import io.opentelemetry.containers.PostgresContainer;
16+
import io.opentelemetry.containers.VehicleInventoryServiceContainer;
17+
import io.opentelemetry.distros.DistroConfig;
1818
import io.opentelemetry.results.AppPerfResults;
1919
import io.opentelemetry.results.MainResultsPersister;
2020
import io.opentelemetry.results.ResultsCollector;
@@ -63,11 +63,11 @@ Stream<DynamicTest> runAllTestConfigurations() {
6363
void runTestConfig(TestConfig config) {
6464
runDurations.clear();
6565
config
66-
.getAgents()
66+
.getDistroConfigs()
6767
.forEach(
68-
agent -> {
68+
distroConfig -> {
6969
try {
70-
runAppOnce(config, agent);
70+
runAppOnce(config, distroConfig);
7171
} catch (Exception e) {
7272
fail("Unhandled exception in " + config.getName(), e);
7373
}
@@ -77,40 +77,43 @@ void runTestConfig(TestConfig config) {
7777
new MainResultsPersister(config).write(results);
7878
}
7979

80-
void runAppOnce(TestConfig config, Agent agent) throws Exception {
80+
void runAppOnce(TestConfig config, DistroConfig distroConfig) throws Exception {
8181
GenericContainer<?> postgres = new PostgresContainer(NETWORK).build();
8282
postgres.start();
8383

84-
GenericContainer<?> petclinic =
85-
new PetClinicRestContainer(NETWORK, collector, agent, namingConventions).build();
84+
GenericContainer<?> vehicleInventoryService =
85+
new VehicleInventoryServiceContainer(NETWORK, collector, distroConfig, namingConventions)
86+
.build();
8687
long start = System.currentTimeMillis();
87-
petclinic.start();
88-
writeStartupTimeFile(agent, start);
88+
vehicleInventoryService.start();
89+
writeStartupTimeFile(distroConfig, start);
8990

9091
if (config.getWarmupSeconds() > 0) {
91-
doWarmupPhase(config, petclinic);
92+
doWarmupPhase(config, vehicleInventoryService);
9293
}
9394

9495
long testStart = System.currentTimeMillis();
95-
startRecording(agent, petclinic);
96+
startRecording(distroConfig, vehicleInventoryService);
9697

97-
GenericContainer<?> k6 = new K6Container(NETWORK, agent, config, namingConventions).build();
98+
GenericContainer<?> k6 =
99+
new K6Container(NETWORK, distroConfig, config, namingConventions).build();
98100
k6.start();
99101

100102
long runDuration = System.currentTimeMillis() - testStart;
101-
runDurations.put(agent.getName(), runDuration);
103+
runDurations.put(distroConfig.getName(), runDuration);
102104

103105
// This is required to get a graceful exit of the VM before testcontainers kills it forcibly.
104106
// Without it, our jfr file will be empty.
105-
petclinic.execInContainer("kill", "1");
106-
while (petclinic.isRunning()) {
107+
vehicleInventoryService.execInContainer("kill", "1");
108+
while (vehicleInventoryService.isRunning()) {
107109
TimeUnit.MILLISECONDS.sleep(500);
108110
}
109111
postgres.stop();
110112
}
111113

112-
private void startRecording(Agent agent, GenericContainer<?> petclinic) throws Exception {
113-
Path outFile = namingConventions.container.jfrFile(agent);
114+
private void startRecording(
115+
DistroConfig distroConfig, GenericContainer<?> vehicleInventoryService) throws Exception {
116+
Path outFile = namingConventions.container.jfrFile(distroConfig);
114117
String[] command = {
115118
"jcmd",
116119
"1",
@@ -120,10 +123,10 @@ private void startRecording(Agent agent, GenericContainer<?> petclinic) throws E
120123
"name=petclinic",
121124
"filename=" + outFile
122125
};
123-
petclinic.execInContainer(command);
126+
vehicleInventoryService.execInContainer(command);
124127
}
125128

126-
private void doWarmupPhase(TestConfig testConfig, GenericContainer<?> petclinic)
129+
private void doWarmupPhase(TestConfig testConfig, GenericContainer<?> vehicleInventoryService)
127130
throws IOException, InterruptedException {
128131
System.out.println(
129132
"Performing startup warming phase for " + testConfig.getWarmupSeconds() + " seconds...");
@@ -139,7 +142,7 @@ private void doWarmupPhase(TestConfig testConfig, GenericContainer<?> petclinic)
139142
"name=warmup",
140143
"filename=warmup.jfr"
141144
};
142-
petclinic.execInContainer(startCommand);
145+
vehicleInventoryService.execInContainer(startCommand);
143146

144147
long deadline =
145148
System.currentTimeMillis() + TimeUnit.SECONDS.toMillis(testConfig.getWarmupSeconds());
@@ -155,14 +158,14 @@ private void doWarmupPhase(TestConfig testConfig, GenericContainer<?> petclinic)
155158

156159
System.out.println("Stopping disposable JFR warmup recording...");
157160
String[] stopCommand = {"jcmd", "1", "JFR.stop", "name=warmup"};
158-
petclinic.execInContainer(stopCommand);
161+
vehicleInventoryService.execInContainer(stopCommand);
159162

160163
System.out.println("Warmup complete.");
161164
}
162165

163-
private void writeStartupTimeFile(Agent agent, long start) throws IOException {
166+
private void writeStartupTimeFile(DistroConfig distroConfig, long start) throws IOException {
164167
long delta = System.currentTimeMillis() - start;
165-
Path startupPath = namingConventions.local.startupDurationFile(agent);
168+
Path startupPath = namingConventions.local.startupDurationFile(distroConfig);
166169
Files.writeString(startupPath, String.valueOf(delta));
167170
}
168171
}

performance-tests/src/test/java/io/opentelemetry/config/Configs.java

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@
55

66
package io.opentelemetry.config;
77

8-
import io.opentelemetry.agents.Agent;
8+
import io.opentelemetry.distros.DistroConfig;
99
import java.util.Arrays;
1010
import java.util.stream.Stream;
1111

@@ -14,8 +14,9 @@ public enum Configs {
1414
RELEASE(
1515
TestConfig.builder()
1616
.name("release")
17-
.description("compares no agent, latest stable, and latest snapshot agents")
18-
.withAgents(Agent.NONE, Agent.LATEST_RELEASE, Agent.LATEST_SNAPSHOT)
17+
.description("compares no distro, latest stable, and latest snapshot distro")
18+
.withDistroConfigs(
19+
DistroConfig.NONE, DistroConfig.LATEST_RELEASE, DistroConfig.LATEST_SNAPSHOT)
1920
.warmupSeconds(60)
2021
.build());
2122

performance-tests/src/test/java/io/opentelemetry/config/TestConfig.java

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@
55

66
package io.opentelemetry.config;
77

8-
import io.opentelemetry.agents.Agent;
8+
import io.opentelemetry.distros.DistroConfig;
99
import java.util.ArrayList;
1010
import java.util.Arrays;
1111
import java.util.Collections;
@@ -20,7 +20,7 @@ public class TestConfig {
2020

2121
private final String name;
2222
private final String description;
23-
private final List<Agent> agents;
23+
private final List<DistroConfig> distroConfigs;
2424
private final int maxRequestRate;
2525
private final int concurrentConnections;
2626
private final int totalIterations;
@@ -29,7 +29,7 @@ public class TestConfig {
2929
public TestConfig(Builder builder) {
3030
this.name = builder.name;
3131
this.description = builder.description;
32-
this.agents = Collections.unmodifiableList(builder.agents);
32+
this.distroConfigs = Collections.unmodifiableList(builder.distroConfigs);
3333
this.maxRequestRate = builder.maxRequestRate;
3434
this.concurrentConnections = builder.concurrentConnections;
3535
this.totalIterations = builder.totalIterations;
@@ -44,8 +44,8 @@ public String getDescription() {
4444
return description;
4545
}
4646

47-
public List<Agent> getAgents() {
48-
return Collections.unmodifiableList(agents);
47+
public List<DistroConfig> getDistroConfigs() {
48+
return Collections.unmodifiableList(distroConfigs);
4949
}
5050

5151
public int getMaxRequestRate() {
@@ -71,7 +71,7 @@ public static Builder builder() {
7171
static class Builder {
7272
private String name;
7373
private String description;
74-
private List<Agent> agents = new ArrayList<>();
74+
private List<DistroConfig> distroConfigs = new ArrayList<>();
7575
private int maxRequestRate = DEFAULT_MAX_REQUEST_RATE;
7676
private int concurrentConnections = DEFAULT_CONCURRENT_CONNECTIONS;
7777
private int totalIterations = DEFAULT_TOTAL_ITERATIONS;
@@ -87,8 +87,8 @@ Builder description(String description) {
8787
return this;
8888
}
8989

90-
Builder withAgents(Agent... agents) {
91-
this.agents.addAll(Arrays.asList(agents));
90+
Builder withDistroConfigs(DistroConfig... distroConfigs) {
91+
this.distroConfigs.addAll(Arrays.asList(distroConfigs));
9292
return this;
9393
}
9494

performance-tests/src/test/java/io/opentelemetry/containers/K6Container.java

Lines changed: 8 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -5,8 +5,8 @@
55

66
package io.opentelemetry.containers;
77

8-
import io.opentelemetry.agents.Agent;
98
import io.opentelemetry.config.TestConfig;
9+
import io.opentelemetry.distros.DistroConfig;
1010
import io.opentelemetry.util.NamingConventions;
1111
import java.nio.file.Path;
1212
import java.time.Duration;
@@ -22,20 +22,23 @@
2222
public class K6Container {
2323
private static final Logger logger = LoggerFactory.getLogger(K6Container.class);
2424
private final Network network;
25-
private final Agent agent;
25+
private final DistroConfig distroConfig;
2626
private final TestConfig config;
2727
private final NamingConventions namingConventions;
2828

2929
public K6Container(
30-
Network network, Agent agent, TestConfig config, NamingConventions namingConvention) {
30+
Network network,
31+
DistroConfig distroConfig,
32+
TestConfig config,
33+
NamingConventions namingConvention) {
3134
this.network = network;
32-
this.agent = agent;
35+
this.distroConfig = distroConfig;
3336
this.config = config;
3437
this.namingConventions = namingConvention;
3538
}
3639

3740
public GenericContainer<?> build() {
38-
Path k6OutputFile = namingConventions.container.k6Results(agent);
41+
Path k6OutputFile = namingConventions.container.k6Results(distroConfig);
3942
return new GenericContainer<>(DockerImageName.parse("loadimpact/k6"))
4043
.withNetwork(network)
4144
.withNetworkAliases("k6")

performance-tests/src/test/java/io/opentelemetry/containers/PetClinicRestContainer.java renamed to performance-tests/src/test/java/io/opentelemetry/containers/VehicleInventoryServiceContainer.java

Lines changed: 14 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -5,8 +5,8 @@
55

66
package io.opentelemetry.containers;
77

8-
import io.opentelemetry.agents.Agent;
9-
import io.opentelemetry.agents.AgentResolver;
8+
import io.opentelemetry.distros.AgentResolver;
9+
import io.opentelemetry.distros.DistroConfig;
1010
import io.opentelemetry.util.NamingConventions;
1111
import java.nio.file.Path;
1212
import java.util.ArrayList;
@@ -24,28 +24,32 @@
2424
import org.testcontainers.utility.DockerImageName;
2525
import org.testcontainers.utility.MountableFile;
2626

27-
public class PetClinicRestContainer {
27+
public class VehicleInventoryServiceContainer {
2828

29-
private static final Logger logger = LoggerFactory.getLogger(PetClinicRestContainer.class);
29+
private static final Logger logger =
30+
LoggerFactory.getLogger(VehicleInventoryServiceContainer.class);
3031
private static final int PETCLINIC_PORT = 9966;
3132
private final AgentResolver agentResolver = new AgentResolver();
3233

3334
private final Network network;
3435
private final Startable collector;
35-
private final Agent agent;
36+
private final DistroConfig distroConfig;
3637
private final NamingConventions namingConventions;
3738

38-
public PetClinicRestContainer(
39-
Network network, Startable collector, Agent agent, NamingConventions namingConventions) {
39+
public VehicleInventoryServiceContainer(
40+
Network network,
41+
Startable collector,
42+
DistroConfig distroConfig,
43+
NamingConventions namingConventions) {
4044
this.network = network;
4145
this.collector = collector;
42-
this.agent = agent;
46+
this.distroConfig = distroConfig;
4347
this.namingConventions = namingConventions;
4448
}
4549

4650
public GenericContainer<?> build() throws Exception {
4751

48-
Optional<Path> agentJar = agentResolver.resolve(this.agent);
52+
Optional<Path> agentJar = agentResolver.resolve(this.distroConfig);
4953

5054
GenericContainer<?> container =
5155
new GenericContainer<>(
@@ -89,7 +93,7 @@ private String[] buildCommandline(Optional<Path> agentJar) {
8993
"-Dotel.exporter.otlp.insecure=true",
9094
"-Dotel.exporter.otlp.endpoint=http://collector:4317",
9195
"-Dotel.resource.attributes=service.name=petclinic-otel-overhead"));
92-
result.addAll(this.agent.getAdditionalJvmArgs());
96+
result.addAll(this.distroConfig.getAdditionalJvmArgs());
9397
agentJar.ifPresent(path -> result.add("-javaagent:/app/" + path.getFileName()));
9498

9599
result.add("-jar");

0 commit comments

Comments
 (0)