Skip to content

Commit 45fbfaf

Browse files
giarc3giarc3
authored
Update benchmarks for all languages (#209)
1 parent c03f705 commit 45fbfaf

File tree

7 files changed

+57
-67
lines changed

7 files changed

+57
-67
lines changed

‎flake.lock‎

Lines changed: 6 additions & 6 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎java/src/jmh/java/com/ironcorelabs/ironcore_alloy_java/README.md‎

Lines changed: 13 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -83,20 +83,20 @@ The benchmark is using a single tenant, and (depending on your machine and bench
8383

8484
There are also benchmarks available in [Rust](https://github.com/IronCoreLabs/ironcore-alloy/tree/main/benches), [Kotlin](https://github.com/IronCoreLabs/ironcore-alloy/tree/main/kotlin/benchmarks) and [Python](https://github.com/IronCoreLabs/ironcore-alloy/blob/main/python/ironcore-alloy/bench.py).
8585

86-
The following were benchmarks run on 8/15/2024 on a Macbook M1 Max.
86+
The following were benchmarks run on March 24th, 2025 on a Macbook M2 Max.
8787

8888
```text
8989
Benchmark Mode Cnt Score Error Units
90-
SaasShieldBenchmark.batchEncrypt10DocsOf100B avgt 5 3880.803 ± 373.629 us/op
91-
SaasShieldBenchmark.tspDecrypt100B avgt 5 899.365 ± 151.952 us/op
92-
SaasShieldBenchmark.tspDecrypt10KB avgt 5 6204.417 ± 804.504 us/op
93-
SaasShieldBenchmark.tspDecrypt1B avgt 5 753.186 ± 53.981 us/op
94-
SaasShieldBenchmark.tspDecrypt1MB avgt 5 344116.886 ± 11127.442 us/op
95-
SaasShieldBenchmark.tspEncrypt100B avgt 5 919.766 ± 249.134 us/op
96-
SaasShieldBenchmark.tspEncrypt10KB avgt 5 6705.966 ± 646.834 us/op
97-
SaasShieldBenchmark.tspEncrypt1B avgt 5 1211.361 ± 900.329 us/op
98-
SaasShieldBenchmark.tspEncrypt1MB avgt 5 340480.214 ± 6824.536 us/op
99-
StandaloneBenchmark.standaloneRoundtripStandard100Kb avgt 5 525.981 ± 169.361 us/op
100-
StandaloneBenchmark.standaloneRoundtripStandard10B avgt 5 427.260 ± 164.509 us/op
101-
StandaloneBenchmark.standaloneRoundtripStandard10Kb avgt 5 445.747 ± 161.576 us/op
90+
SaasShieldBenchmark.batchEncrypt10DocsOf100B avgt 5 1004.374 ± 176.667 us/op
91+
SaasShieldBenchmark.tspDecrypt100B avgt 5 309.908 ± 58.527 us/op
92+
SaasShieldBenchmark.tspDecrypt10KB avgt 5 492.581 ± 183.462 us/op
93+
SaasShieldBenchmark.tspDecrypt1B avgt 5 299.011 ± 15.632 us/op
94+
SaasShieldBenchmark.tspDecrypt1MB avgt 5 11287.560 ± 235.532 us/op
95+
SaasShieldBenchmark.tspEncrypt100B avgt 5 303.053 ± 37.513 us/op
96+
SaasShieldBenchmark.tspEncrypt10KB avgt 5 458.719 ± 85.330 us/op
97+
SaasShieldBenchmark.tspEncrypt1B avgt 5 299.640 ± 39.078 us/op
98+
SaasShieldBenchmark.tspEncrypt1MB avgt 5 11580.426 ± 575.643 us/op
99+
StandaloneBenchmark.standaloneRoundtripStandard100Kb avgt 5 2177.326 ± 83.221 us/op
100+
StandaloneBenchmark.standaloneRoundtripStandard10B avgt 5 133.866 ± 50.676 us/op
101+
StandaloneBenchmark.standaloneRoundtripStandard10Kb avgt 5 337.769 ± 49.548 us/op
102102
```

‎java/src/jmh/java/com/ironcorelabs/ironcore_alloy_java/StandaloneBenchmark.java‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -54,9 +54,9 @@ public void setUp() throws Exception {
5454

5555
standaloneConfig = new StandaloneConfiguration(standardSecrets, deterministicSecrets, vectorSecrets);
5656
standaloneSdk = new Standalone(standaloneConfig);
57-
smallWord = new PlaintextBytes(randomWord(1).getBytes());
58-
mediumWord = new PlaintextBytes(randomWord(10).getBytes());
59-
largeWord = new PlaintextBytes(randomWord(100).getBytes());
57+
smallWord = new PlaintextBytes(randomWord(10).getBytes());
58+
mediumWord = new PlaintextBytes(randomWord(10 * 1000).getBytes());
59+
largeWord = new PlaintextBytes(randomWord(100 * 1000).getBytes());
6060
}
6161

6262
private String randomWord(int length) {

‎kotlin/benchmarks/src/README.md‎

Lines changed: 15 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -84,20 +84,21 @@ There are also benchmarks available in [Rust](https://github.com/IronCoreLabs/ir
8484

8585
## Results
8686

87-
The following benchmarking run was done on May 7, 2024 on a MacBook Pro (2023) with an Apple M2 Max chip. It uses a locally-built TSP running with the configuration from `demo-tsp.conf`.
87+
The following benchmarking run was done on March 24th, 2025 on a Macbook M2 Max.
8888

89-
```
89+
```text
9090
Benchmark Mode Cnt Score Error Units
91-
SaasShieldBenchmark.batchEncrypt10DocsOf100B avgt 5 545.626 ± 19.272 us/op
92-
SaasShieldBenchmark.tspDecrypt100B avgt 5 148.846 ± 11.811 us/op
93-
SaasShieldBenchmark.tspDecrypt10KB avgt 5 242.892 ± 2.902 us/op
94-
SaasShieldBenchmark.tspDecrypt1B avgt 5 141.517 ± 7.603 us/op
95-
SaasShieldBenchmark.tspDecrypt1MB avgt 5 9104.683 ± 74.820 us/op
96-
SaasShieldBenchmark.tspEncrypt100B avgt 5 142.432 ± 5.696 us/op
97-
SaasShieldBenchmark.tspEncrypt10KB avgt 5 246.627 ± 5.616 us/op
98-
SaasShieldBenchmark.tspEncrypt1B avgt 5 139.817 ± 5.732 us/op
99-
SaasShieldBenchmark.tspEncrypt1MB avgt 5 9322.918 ± 136.650 us/op
100-
StandaloneBenchmark.standaloneRoundtripStandard100Kb avgt 5 117.444 ± 56.447 us/op
101-
StandaloneBenchmark.standaloneRoundtripStandard10B avgt 5 112.489 ± 56.958 us/op
102-
StandaloneBenchmark.standaloneRoundtripStandard10Kb avgt 5 113.051 ± 51.750 us/op
91+
SaasShieldBenchmark.batchEncrypt10DocsOf100B avgt 5 972.373 ± 81.365 us/op
92+
SaasShieldBenchmark.tspDecrypt100B avgt 5 300.325 ± 18.856 us/op
93+
SaasShieldBenchmark.tspDecrypt10KB avgt 5 422.159 ± 55.192 us/op
94+
SaasShieldBenchmark.tspDecrypt1B avgt 5 290.425 ± 20.535 us/op
95+
SaasShieldBenchmark.tspDecrypt1MB avgt 5 11429.166 ± 206.175 us/op
96+
SaasShieldBenchmark.tspEncrypt100B avgt 5 292.691 ± 19.519 us/op
97+
SaasShieldBenchmark.tspEncrypt10KB avgt 5 414.124 ± 45.703 us/op
98+
SaasShieldBenchmark.tspEncrypt1B avgt 5 292.419 ± 19.750 us/op
99+
SaasShieldBenchmark.tspEncrypt1MB avgt 5 11721.616 ± 1013.550 us/op
100+
StandaloneBenchmark.standaloneRoundtripStandard100Kb avgt 5 2177.714 ± 85.201 us/op
101+
StandaloneBenchmark.standaloneRoundtripStandard10B avgt 5 127.618 ± 70.516 us/op
102+
StandaloneBenchmark.standaloneRoundtripStandard10Kb avgt 5 328.130 ± 98.322 us/op
103+
StandaloneBenchmark.standaloneVectorEncrypt384d avgt 5 80.240 ± 38.963 us/op
103104
```

‎kotlin/benchmarks/src/StandaloneBenchmark.kt‎

Lines changed: 5 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -2,8 +2,8 @@ package test
22

33
import com.ironcorelabs.ironcore_alloy.*
44
import java.util.Base64
5-
import java.util.concurrent.*
65
import java.util.Random
6+
import java.util.concurrent.*
77
import kotlin.ByteArray
88
import kotlin.system.*
99
import kotlinx.coroutines.*
@@ -58,9 +58,9 @@ class StandaloneBenchmark {
5858

5959
@Setup
6060
fun setUp() {
61-
smallWord = randomWord(1).toByteArray()
62-
mediumWord = randomWord(10).toByteArray()
63-
largeWord = randomWord(100).toByteArray()
61+
smallWord = randomWord(10).toByteArray()
62+
mediumWord = randomWord(10 * 1000).toByteArray()
63+
largeWord = randomWord(100 * 1000).toByteArray()
6464
}
6565

6666
@State(Scope.Thread)
@@ -70,9 +70,7 @@ class StandaloneBenchmark {
7070

7171
@Benchmark
7272
fun standaloneVectorEncrypt384d(s: Vector384State) {
73-
runBlocking {
74-
standaloneSdk.vector().encrypt(PlaintextVector(s.vector, "", ""), metadata)
75-
}
73+
runBlocking { standaloneSdk.vector().encrypt(PlaintextVector(s.vector, "", ""), metadata) }
7674
}
7775

7876
@Benchmark

‎python/ironcore-alloy/BENCHMARKS.md‎

Lines changed: 11 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -9,25 +9,16 @@ See `project_root/benches/README.md` for more general information about benchmar
99

1010
## Results
1111

12-
The following benchmarking run was done on August 30th, 2024 on a Lenovo Thinkpad X1 Extreme 2nd Gen with an i9-9880H CPU.
12+
The following benchmarking run was done on March 24th, 2025 on a Macbook M2 Max.
1313

14-
```
15-
.....................
16-
vector encrypt d=384: Mean +- std dev: 2.66 ms +- 0.14 ms
17-
.....................
18-
vector encrypt d=768: Mean +- std dev: 5.28 ms +- 0.25 ms
19-
.....................
20-
vector encrypt d=1536: Mean +- std dev: 10.3 ms +- 0.6 ms
21-
.....................
22-
vector encrypt d=2048: Mean +- std dev: 13.6 ms +- 0.7 ms
23-
.....................
24-
vector batch (100) encrypt d=768: Mean +- std dev: 416 ms +- 25 ms
25-
.....................
26-
vector batch (1000) encrypt d=768: Mean +- std dev: 4.14 sec +- 0.19 sec
27-
.....................
28-
standard_roundtrip_small: Mean +- std dev: 616 us +- 23 us
29-
.....................
30-
standard_roundtrip_medium: Mean +- std dev: 13.1 ms +- 0.6 ms
31-
.....................
32-
standard_roundtrip_large: Mean +- std dev: 124 ms +- 6 ms
14+
```text
15+
vector encrypt d=384: Mean +- std dev: 1.07 ms +- 0.02 ms
16+
vector encrypt d=768: Mean +- std dev: 2.07 ms +- 0.05 ms
17+
vector encrypt d=1536: Mean +- std dev: 4.00 ms +- 0.05 ms
18+
vector encrypt d=2048: Mean +- std dev: 5.26 ms +- 0.06 ms
19+
vector batch (100) encrypt d=768: Mean +- std dev: 192 ms +- 2 ms
20+
vector batch (1000) encrypt d=768: Mean +- std dev: 1.91 sec +- 0.01 sec
21+
standard_roundtrip_10b: Mean +- std dev: 178 us +- 12 us
22+
standard_roundtrip_10kb: Mean +- std dev: 3.75 ms +- 0.05 ms
23+
standard_roundtrip_100kb: Mean +- std dev: 35.2 ms +- 0.4 ms
3324
```

‎python/ironcore-alloy/bench.py‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -144,17 +144,17 @@ def random_floats(dimensions: int):
144144
### standalone standard
145145
random_small_word = random_word(10).encode("utf-8")
146146
runner.bench_async_func(
147-
"standard_roundtrip_small",
147+
"standard_roundtrip_10b",
148148
standard_roundtrip,
149149
sdk,
150150
random_small_word,
151151
metadata,
152152
)
153153
random_medium_word = random_word(10 * 1000).encode("utf-8")
154154
runner.bench_async_func(
155-
"standard_roundtrip_medium", standard_roundtrip, sdk, random_medium_word, metadata
155+
"standard_roundtrip_10kb", standard_roundtrip, sdk, random_medium_word, metadata
156156
)
157-
random_large_word = random_word(10 * 10000).encode("utf-8")
157+
random_large_word = random_word(100 * 1000).encode("utf-8")
158158
runner.bench_async_func(
159-
"standard_roundtrip_large", standard_roundtrip, sdk, random_large_word, metadata
159+
"standard_roundtrip_100kb", standard_roundtrip, sdk, random_large_word, metadata
160160
)

0 commit comments

Comments
 (0)