Skip to content

Commit 4ceeceb

Browse files
giarc3giarc3
authored
Expand benchmarks (#136)
1 parent fe39ff6 commit 4ceeceb

File tree

5 files changed

+263
-204
lines changed

5 files changed

+263
-204
lines changed

‎benchmarks/README.md‎

Lines changed: 20 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -4,45 +4,53 @@ This directory contains a benchmark suite for the Java version of the Tenant Sec
44
To build and run the benchmark, just execute the following commands in this directory:
55

66
To show single threaded latency of a roundtrip (encrypt/decrypt):
7+
78
```
89
mvn clean install
9-
java -Xms1024m -Xmx1024m -jar target/benchmarks.jar -f0 -wi 1
10+
java -Xms1024m -Xmx1024m -jar target/benchmarks.jar -f 0 -wi 1
1011
```
1112

12-
A variation that will show throughput using key leasing:
13+
A variation that will show throughput instead:
14+
1315
```
1416
mvn clean install
15-
TENANT_ID=tenant-gcp-l java -Xms1024m -Xmx1024m -jar target/benchmarks.jar -f 0 -wi 1 -bm thrpt -tu s
17+
java -Xms1024m -Xmx1024m -jar target/benchmarks.jar -f 0 -wi 1 -bm thrpt -tu s
1618
```
1719

1820
You have to benchmark an actual version of the TSC, though this can be a `SNAPSHOT` version published locally.
1921
Update the `pom.xml` to whatever version you'd like to test.
2022

21-
2223
## Tenant Security Proxy
2324

2425
In order to run the benchmarks, the TSC needs to connect to a _Tenant Security Proxy (TSP)_.
2526
This service is provided as a Docker container, so it is easy to run the proxy on any computer that has Docker
2627
installed. IronCore Labs hosts the Docker container on a publicly accessible container registry, so you can pull
2728
the image from there and run it locally.
2829

29-
The TSP has a companion Docker container, the _Tenant Security Logdriver_ (LD) that runs alongside it in your environment. It is also hosted on the same publicly accessible container registry.
30-
31-
3230
In addition to the Docker containers, you need a configuration file that specifies how the TSP should communicate
3331
with the IronCore Labs Configuration Broker and Data Control Platform, which work together to enable the end-to-end
3432
encryption that keeps all of the tenant KMS configuration information secure. To simplify the process of running
3533
these examples, we have created a demo vendor and tenants that you can use for the examples; all the necessary
3634
configuration information is included in the [demo-tsp.conf](demo-tsp.conf) file in this directory.
35+
3736
**NOTE:** Normally, the file containing the configuration would be generated by the vendor and loaded into a
3837
Kubernetes secret or similar mechanism for securely loading the configuration into the docker container. We
3938
have included this configuration in the repository as a convenience. Also note that these accounts are all
4039
created in IronCore's staging infrastructure.
4140

42-
The following `docker-compose` command will get a TSP+LD running on your computer with the provided configuration:
41+
Production TSPs will often be accompanied by one or more
42+
[Tenant Security Logdriver](https://ironcorelabs.com/docs/saas-shield/tenant-security-logdriver/overview/) instances.
43+
Because the purpose of this benchmark is to demonstrate the capabilities of the Tenant Security _Client_, we have chosen to not include
44+
Logdriver in it. If you wish to modify the Docker Compose file to include Logdriver, be sure to consult its
45+
[Deployment](https://ironcorelabs.com/docs/saas-shield/tenant-security-logdriver/deployment/) page to learn how to properly configure it
46+
based on the resources you have available.
47+
48+
The following `docker compose` command will get a TSP running on your computer with the provided configuration:
49+
4350
```
44-
docker-compose -f docker-compose.yml up
51+
docker compose up
4552
```
53+
4654
This starts the TSP locally listening on port 32804. The benchmark expects to connect to the TSP at that address.
4755

4856
To connect with and use the TSP, you need to supply a couple more configuration values:
@@ -66,7 +74,7 @@ There are six tenants defined; their IDs are the following:
6674

6775
The last three are similar to the first three, but they have _key leasing_ enabled.
6876

69-
By default, the benchmark will use the `tenant-gcp` tenant. If you would like to experiment with a different tenant, just do:
77+
By default, the benchmark will use the `tenant-gcp-l` tenant. If you would like to experiment with a different tenant, just do:
7078

7179
```bash
7280
export TENANT_ID=<select tenant ID>
@@ -78,6 +86,6 @@ before running the benchmark.
7886

7987
Since TSC-java is a library that interacts with a back-end service (TSP), the benchmark results are not always straightforward to interpret. Most API calls in the TSC make a round-trip to the TSP, and the TSP also does some computation. If testing on a single machine, it is good to monitor the CPU/RAM usage of the TSP processes in addition to the Java benchmark process to make sure you aren't resource constrained.
8088

81-
In general, operation latency is a function of latency to the TSP + latency to the tenant's KMS (if key-leasing is disabled).
89+
In general, operation latency is a function of latency to the TSP + latency to the tenant's KMS (if key-leasing is disabled).
8290

83-
The TSP's tenant logging mechanism has some tunable limits. By default, a TSP should be able to sustain 500 ops/sec/tenant, with the ability to burst higher for a limited time. The benchmark is using a single tenant, and (depending on your machine and benchmark config) can easily be executing a few thousand ops/sec. If you run a benchmark long enough you will overwhelm the TSP. In a real application, you would scale-out the TSP at this point. See [the TSP documentation](https://ironcorelabs.com/docs/saas-shield/tenant-security-proxy/overview/) for more details.
91+
The TSP's tenant logging mechanism has some tunable limits. By default, a TSP should be able to sustain 500 ops/sec/tenant, with the ability to burst higher for a limited time. The benchmark is using a single tenant, and (depending on your machine and benchmark config) can easily be executing a few thousand ops/sec. If you run a benchmark long enough you will overwhelm the TSP. In a real application, you would scale-out the TSP at this point. See [the TSP documentation](https://ironcorelabs.com/docs/saas-shield/tenant-security-proxy/deployment/) for more details.

‎benchmarks/demo-tsp.conf‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,4 +5,5 @@ SERVICE_SIGNING_PRIVATE_KEY=9lAJ3a1Cb4xTL9jD/qdt53pBSG6m9ZNWWFs2WsMN2Mpp+huxRRRM
55
SERVICE_SEGMENT_ID=698
66
API_KEY=0WUaXesNgbTAuLwn
77
IRONCORE_ENV=stage
8-
RUST_LOG=info
8+
RUST_LOG=warn
9+
TSP_ENABLE_LOGDRIVER_INTEGRATION=false

‎benchmarks/docker-compose.yml‎

Lines changed: 0 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,3 @@
1-
version: "3.3"
21
services:
32
tenant-security-proxy:
43
env_file:
@@ -7,17 +6,3 @@ services:
76
- 32804:7777
87
- 32805:9000
98
image: gcr.io/ironcore-images/tenant-security-proxy:4
10-
links:
11-
- tenant-security-logdriver
12-
tenant-security-logdriver:
13-
environment:
14-
- LOGDRIVER_EVENT_PRODUCER_URL=tcp://tenant-security-proxy:5555
15-
env_file:
16-
- ./demo-tsp.conf
17-
ports:
18-
- 32806:9001
19-
image: gcr.io/ironcore-images/tenant-security-logdriver:4
20-
volumes:
21-
- type: bind
22-
source: /tmp
23-
target: /logdriver

‎benchmarks/pom.xml‎

Lines changed: 81 additions & 117 deletions
Original file line numberDiff line numberDiff line change
@@ -1,124 +1,88 @@
1-
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
2-
<modelVersion>4.0.0</modelVersion>
1+
<project xmlns="http://maven.apache.org/POM/4.0.0"
2+
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
3+
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
4+
<modelVersion>4.0.0</modelVersion>
35

4-
<groupId>com.ironcorelabs</groupId>
5-
<artifactId>benchmarks</artifactId>
6-
<version>1.0</version>
7-
<packaging>jar</packaging>
6+
<groupId>com.ironcorelabs</groupId>
7+
<artifactId>benchmarks</artifactId>
8+
<version>1.0</version>
9+
<packaging>jar</packaging>
810

9-
<name>Tenant Security Proxy Benchmarks</name>
11+
<name>Tenant Security Proxy Benchmarks</name>
1012

11-
<dependencies>
12-
<dependency>
13-
<groupId>org.openjdk.jmh</groupId>
14-
<artifactId>jmh-core</artifactId>
15-
<version>${jmh.version}</version>
16-
</dependency>
17-
<dependency>
18-
<groupId>org.openjdk.jmh</groupId>
19-
<artifactId>jmh-generator-annprocess</artifactId>
20-
<version>${jmh.version}</version>
21-
<scope>provided</scope>
22-
</dependency>
23-
<dependency>
24-
<groupId>com.ironcorelabs</groupId>
25-
<artifactId>tenant-security-java</artifactId>
26-
<version>4.0.0</version>
27-
</dependency>
28-
</dependencies>
13+
<dependencies>
14+
<dependency>
15+
<groupId>org.openjdk.jmh</groupId>
16+
<artifactId>jmh-core</artifactId>
17+
<version>${jmh.version}</version>
18+
</dependency>
19+
<dependency>
20+
<groupId>org.openjdk.jmh</groupId>
21+
<artifactId>jmh-generator-annprocess</artifactId>
22+
<version>${jmh.version}</version>
23+
<scope>provided</scope>
24+
</dependency>
25+
<dependency>
26+
<groupId>com.ironcorelabs</groupId>
27+
<artifactId>tenant-security-java</artifactId>
28+
<version>7.2.0</version>
29+
</dependency>
30+
</dependencies>
2931

30-
<properties>
31-
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
32-
<jmh.version>1.23</jmh.version>
33-
<javac.target>1.8</javac.target>
34-
<uberjar.name>benchmarks</uberjar.name>
35-
</properties>
32+
<properties>
33+
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
34+
<jmh.version>1.37</jmh.version>
35+
<javac.target>17</javac.target>
36+
<uberjar.name>benchmarks</uberjar.name>
37+
</properties>
3638

37-
<build>
38-
<plugins>
39-
<plugin>
40-
<groupId>org.apache.maven.plugins</groupId>
41-
<artifactId>maven-compiler-plugin</artifactId>
42-
<version>3.8.0</version>
43-
<configuration>
44-
<compilerVersion>${javac.target}</compilerVersion>
45-
<source>${javac.target}</source>
46-
<target>${javac.target}</target>
47-
</configuration>
48-
</plugin>
49-
<plugin>
50-
<groupId>org.apache.maven.plugins</groupId>
51-
<artifactId>maven-shade-plugin</artifactId>
52-
<version>3.2.1</version>
53-
<executions>
54-
<execution>
55-
<phase>package</phase>
56-
<goals>
57-
<goal>shade</goal>
58-
</goals>
59-
<configuration>
60-
<finalName>${uberjar.name}</finalName>
61-
<transformers>
62-
<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
63-
<mainClass>org.openjdk.jmh.Main</mainClass>
64-
</transformer>
65-
<transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer" />
66-
</transformers>
67-
<filters>
68-
<filter>
69-
<artifact>*:*</artifact>
70-
<excludes>
71-
<exclude>META-INF/*.SF</exclude>
72-
<exclude>META-INF/*.DSA</exclude>
73-
<exclude>META-INF/*.RSA</exclude>
74-
</excludes>
75-
</filter>
76-
</filters>
77-
</configuration>
78-
</execution>
79-
</executions>
80-
</plugin>
81-
</plugins>
82-
<pluginManagement>
83-
<plugins>
84-
<plugin>
85-
<artifactId>maven-clean-plugin</artifactId>
86-
<version>2.5</version>
87-
</plugin>
88-
<plugin>
89-
<artifactId>maven-deploy-plugin</artifactId>
90-
<version>2.8.1</version>
91-
</plugin>
92-
<plugin>
93-
<artifactId>maven-install-plugin</artifactId>
94-
<version>2.5.1</version>
95-
</plugin>
96-
<plugin>
97-
<artifactId>maven-jar-plugin</artifactId>
98-
<version>2.4</version>
99-
</plugin>
100-
<plugin>
101-
<artifactId>maven-javadoc-plugin</artifactId>
102-
<version>2.9.1</version>
103-
</plugin>
104-
<plugin>
105-
<artifactId>maven-resources-plugin</artifactId>
106-
<version>2.6</version>
107-
</plugin>
108-
<plugin>
109-
<artifactId>maven-site-plugin</artifactId>
110-
<version>3.3</version>
111-
</plugin>
112-
<plugin>
113-
<artifactId>maven-source-plugin</artifactId>
114-
<version>2.2.1</version>
115-
</plugin>
116-
<plugin>
117-
<artifactId>maven-surefire-plugin</artifactId>
118-
<version>2.17</version>
119-
</plugin>
120-
</plugins>
121-
</pluginManagement>
122-
</build>
39+
<build>
40+
<plugins>
41+
<plugin>
42+
<groupId>org.apache.maven.plugins</groupId>
43+
<artifactId>maven-compiler-plugin</artifactId>
44+
<version>3.13.0</version>
45+
<configuration>
46+
<compilerVersion>${javac.target}</compilerVersion>
47+
<source>${javac.target}</source>
48+
<target>${javac.target}</target>
49+
</configuration>
50+
</plugin>
51+
<plugin>
52+
<groupId>org.apache.maven.plugins</groupId>
53+
<artifactId>maven-shade-plugin</artifactId>
54+
<version>3.5.2</version>
55+
<executions>
56+
<execution>
57+
<phase>package</phase>
58+
<goals>
59+
<goal>shade</goal>
60+
</goals>
61+
<configuration>
62+
<finalName>${uberjar.name}</finalName>
63+
<transformers>
64+
<transformer
65+
implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
66+
<mainClass>org.openjdk.jmh.Main</mainClass>
67+
</transformer>
68+
<transformer
69+
implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer" />
70+
</transformers>
71+
<filters>
72+
<filter>
73+
<artifact>*:*</artifact>
74+
<excludes>
75+
<exclude>META-INF/*.SF</exclude>
76+
<exclude>META-INF/*.DSA</exclude>
77+
<exclude>META-INF/*.RSA</exclude>
78+
</excludes>
79+
</filter>
80+
</filters>
81+
</configuration>
82+
</execution>
83+
</executions>
84+
</plugin>
85+
</plugins>
86+
</build>
12387

12488
</project>

0 commit comments

Comments
 (0)