Skip to content

Commit dc4f85b

Browse files
stevehustevehu
authored
Merge pull request #12 from justin-tay/benchmark
Add benchmark action and implementations
2 parents d6762e1 + c768a62 commit dc4f85b

39 files changed

+29013
-292
lines changed

‎.github/workflows/benchmark.yml‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
name: Benchmarks
2+
on:
3+
push:
4+
branches:
5+
- master
6+
7+
permissions:
8+
contents: write
9+
deployments: write
10+
11+
jobs:
12+
benchmark:
13+
name: Run benchmark
14+
runs-on: ubuntu-latest
15+
steps:
16+
- name: Checkout
17+
uses: actions/checkout@v5
18+
- name: Setup JDK
19+
uses: actions/setup-java@v5
20+
with:
21+
distribution: 'temurin'
22+
java-version: '21'
23+
cache: 'maven'
24+
- name: Build with Maven
25+
run: |
26+
mvn --batch-mode --update-snapshots verify -Dstyle.color=always -Dmaven.javadoc.skip=true
27+
- name: Run benchmark
28+
run: |
29+
java -jar target/benchmarks.jar Comparison -wi 3 -i 3 -f 1 -rf json
30+
- name: Store raw benchmark result as artifact
31+
uses: actions/upload-artifact@v4
32+
with:
33+
name: benchmark-result
34+
path: jmh-result.json
35+
- name: Store benchmark result
36+
uses: benchmark-action/github-action-benchmark@v1
37+
with:
38+
name: JSON Schema Validator Benchmark
39+
tool: 'jmh'
40+
output-file-path: jmh-result.json
41+
# Use personal access token instead of GITHUB_TOKEN due to https://github.community/t/github-action-not-triggering-gh-pages-upon-push/16096
42+
github-token: ${{ secrets.GITHUB_TOKEN }}
43+
auto-push: true
44+
comment-on-alert: false
45+
max-items-in-chart: 50
46+
summary-always: true

‎.gitignore‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,12 +7,14 @@ dist/
77
.project
88
.classpath
99
.settings
10+
.factorypath
1011
.metadata/
1112
*.iml
1213
*.log
1314
*.tmp
1415
*.zip
1516
*.bak
17+
bin/
1618

1719
# virtual machine crash logs, see http://www.java.com/en/download/help/error_hotspot.xml
1820
hs_err_pid*

‎README.md‎

Lines changed: 37 additions & 56 deletions
Original file line numberDiff line numberDiff line change
@@ -1,77 +1,58 @@
11
# json-schema-validator-perftest
2-
A performance test project that compares networknt, fge and everit json-schema-validator
32

4-
This library provides a skeletal performance test suite for this JSON Schema Validator library:
3+
This project is the performance test suite for the [NetworkNT JSON Schema Validator](https://github.com/networknt/json-schema-validator).
54

6-
https://github.com/networknt/json-schema-validator
5+
## Benchmark results
76

8-
It contains the same test suite for the other two Java schema validators:
7+
The benchmark results can be found [here](https://networknt.github.io/json-schema-validator-perftest/dev/bench/).
98

10-
https://github.com/fge/json-schema-validator/wiki/Performance
9+
These benchmarks are run on [Standard GitHub-hosted runners for public repositories](https://docs.github.com/en/actions/reference/runners/github-hosted-runners#standard-github-hosted-runners-for-public-repositories) which are not dedicated machines. Some variance between runs are to be expected.
1110

12-
https://github.com/everit-org/json-schema
11+
## Data set
1312

13+
| Name | Description |
14+
| ------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
15+
| `draft4_basic` | This is the same test used by the [fge json-schema validator](https://github.com/fge/json-schema-validator/wiki/Performance). |
16+
| `draft7_krakend` | From [krakend-schema](https://github.com/krakend/krakend-schema). |
17+
| `draft2020_12_cql2` | From [cql2](https://schemas.opengis.net/cql2/1.0/cql2.json). Note that this uses `oneOf` which [requires verification of every sub-schema](https://json-schema.org/understanding-json-schema/reference/combining#oneOf). |
18+
| `draft2020_12_cql2_fix` | Modified from [cql2](https://schemas.opengis.net/cql2/1.0/cql2.json) to convert `oneOf` to `allOf` as [recommended](https://github.com/json-schema-org/json-schema-spec/issues/1082) to compare performance. |
19+
| `draft2020_12_openapi` | Validation of OpenAPI document. |
20+
| `draft2020_12_unevaluatedProperties` | For the `unevaluatedProperties` keyword. |
1421

15-
# Test results comparison
22+
## Implementations
1623

17-
For testing the performance of the `fge/json-schema-validator` library I've used [this main class](https://github.com/fge/json-schema-validator/blob/master/src/test/java/com/github/fge/jsonschema/NewAPIPerfTest.java) .
24+
| Name | Repository | Dialects | Remarks |
25+
| --------------------------------- | ---------------------------------------------------- | ---------------------------------------------------------------- | ----------------------- |
26+
| NetworkNT JSON Schema Validator | https://github.com/networknt/json-schema-validator | Draft 4, Draft 6, Draft 7, Draft 2019-09, Draft 2020-12 |
27+
| dev.harrel:json-schema | https://github.com/harrel56/json-schema | Draft 7, Draft 2019-09, Draft 2020-12 |
28+
| jsonschemafriend | https://github.com/jimblackler/jsonschemafriend | Draft 3, Draft 4, Draft 6, Draft 7, Draft 2019-09, Draft 2020-12 |
29+
| OptimumCode json-schema-validator | https://github.com/OptimumCode/json-schema-validator | Draft 4, Draft 6, Draft 7, Draft 2019-09, Draft 2020-12 |
30+
| json-sKema | https://github.com/erosb/json-sKema | Draft 2020-12 |
31+
| Everit JSON Schema Validator | https://github.com/everit-org/json-schema | Draft 4, Draft 6, Draft 7 | Replaced by json-sKema. |
1832

19-
## results 2017-07-09
20-
fge: 7130 ms
21-
everit-org: 1168 ms
22-
networknt: 223 ms
33+
## Running the benchmarks
2334

24-
I ran the testsuite of both libraries 10 times (each execution performing 500 validations).
25-
Tests results on my local computer:
26-
Running the tests on other workstations gave a bit different numbers but similar ratios.
35+
### Clone this repository
2736

28-
## results 2021-02-18
29-
fge: 4741 ms
30-
everit-org: 564 ms
31-
networknt: 543 ms
32-
33-
34-
## results 2022-05-02
35-
The fge validator is removed and the runner is removed as well. Both libraries are upgraded to the latest version.
36-
37-
To run it from the IDE, the result:
38-
```
39-
everit-org(1.14.1): 260 ms
40-
networknt(1.0.69): 196 ms
37+
```shell
38+
git clone https://github.com/networknt/json-schema-validator-perftest.git
4139
```
4240

43-
To run it from the jar file.
41+
### Build the project with Maven
4442

45-
```
46-
java -cp target/perftest.jar com.networknt.schema.perftest.EveritPerf
47-
java -cp target/perftest.jar com.networknt.schema.perftest.NetworkntPerf
48-
```
49-
The result:
50-
51-
```
52-
everit-org(1.14.1): 281 ms
53-
networknt(1.0.69): 448 ms
43+
```shell
44+
cd json-schema-validator-perftest/
45+
mvn clean package
5446
```
5547

56-
Single run
57-
58-
# Running the tests
59-
60-
Clone this repository:
61-
62-
`git clone https://github.com/networknt/json-schema-validator-perftest.git`
63-
64-
Build the project with Maven:
48+
### Run all the benchmarks
6549

50+
```shell
51+
java -jar target/benchmarks.jar
6652
```
67-
cd json-schema-perftest/
68-
mvn clean install -Pdist
69-
```
70-
71-
Run the test:
7253

73-
`java -jar target/perftest.jar`
54+
### Run a specific benchmark
7455

75-
# Test results
76-
77-
Check [test results](test_results)
56+
```shell
57+
java -jar target/benchmarks.jar Networknt
58+
```

‎pom.xml‎

Lines changed: 48 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -15,11 +15,19 @@
1515
<java.version>21</java.version>
1616
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
1717
<argLine>-Xmx512m -XX:MaxPermSize=256m</argLine>
18-
<version.slf4j>2.0.17</version.slf4j>
19-
<version.logback>1.5.18</version.logback>
18+
<slf4j.version>2.0.17</slf4j.version>
19+
<logback.version>1.5.19</logback.version>
20+
<everit-json-schema.version>1.14.6</everit-json-schema.version>
21+
<json-schema-validator.version>1.5.9</json-schema-validator.version>
22+
<json-schema.version>1.8.2</json-schema.version>
23+
<jsonschemafriend.version>0.12.5</jsonschemafriend.version>
24+
<json-schema-validator-jvm.version>0.5.1</json-schema-validator-jvm.version>
25+
<json-sKema.version>0.25.0</json-sKema.version>
2026
<jmh.version>1.37</jmh.version>
27+
<nexus-staging-maven.version>1.7.0</nexus-staging-maven.version>
28+
<maven-compiler-plugin.version>3.14.0</maven-compiler-plugin.version>
29+
<maven-shade-plugin.version>3.6.1</maven-shade-plugin.version>
2130
<uberjar.name>benchmarks</uberjar.name>
22-
<version.nexus-staging-maven>1.7.0</version.nexus-staging-maven>
2331
</properties>
2432

2533
<repositories>
@@ -52,22 +60,42 @@
5260
<dependency>
5361
<groupId>com.github.erosb</groupId>
5462
<artifactId>everit-json-schema</artifactId>
55-
<version>1.14.4</version>
63+
<version>${everit-json-schema.version}</version>
5664
</dependency>
5765
<dependency>
5866
<groupId>com.networknt</groupId>
5967
<artifactId>json-schema-validator</artifactId>
60-
<version>1.4.2</version>
68+
<version>${json-schema-validator.version}</version>
6169
</dependency>
70+
<dependency>
71+
<groupId>dev.harrel</groupId>
72+
<artifactId>json-schema</artifactId>
73+
<version>${json-schema.version}</version>
74+
</dependency>
75+
<dependency>
76+
<groupId>net.jimblackler.jsonschemafriend</groupId>
77+
<artifactId>core</artifactId>
78+
<version>${jsonschemafriend.version}</version>
79+
</dependency>
80+
<dependency>
81+
<groupId>io.github.optimumcode</groupId>
82+
<artifactId>json-schema-validator-jvm</artifactId>
83+
<version>${json-schema-validator-jvm.version}</version>
84+
</dependency>
85+
<dependency>
86+
<groupId>com.github.erosb</groupId>
87+
<artifactId>json-sKema</artifactId>
88+
<version>${json-sKema.version}</version>
89+
</dependency>
6290
<dependency>
6391
<groupId>org.slf4j</groupId>
6492
<artifactId>slf4j-api</artifactId>
65-
<version>${version.slf4j}</version>
93+
<version>${slf4j.version}</version>
6694
</dependency>
6795
<dependency>
6896
<groupId>ch.qos.logback</groupId>
6997
<artifactId>logback-classic</artifactId>
70-
<version>${version.logback}</version>
98+
<version>${logback.version}</version>
7199
</dependency>
72100
</dependencies>
73101
<build>
@@ -94,6 +122,7 @@
94122
<plugin>
95123
<groupId>org.apache.maven.plugins</groupId>
96124
<artifactId>maven-shade-plugin</artifactId>
125+
<version>${maven-shade-plugin.version}</version>
97126
<executions>
98127
<execution>
99128
<phase>package</phase>
@@ -133,7 +162,7 @@
133162
<plugin>
134163
<groupId>org.sonatype.plugins</groupId>
135164
<artifactId>nexus-staging-maven-plugin</artifactId>
136-
<version>${version.nexus-staging-maven}</version>
165+
<version>${nexus-staging-maven.version}</version>
137166
<extensions>true</extensions>
138167
<configuration>
139168
<serverId>ossrh</serverId>
@@ -154,41 +183,32 @@
154183
</execution>
155184
</executions>
156185
</plugin>
157-
<plugin>
158-
<groupId>org.apache.maven.plugins</groupId>
159-
<artifactId>maven-javadoc-plugin</artifactId>
160-
<version>3.2.0</version>
161-
<executions>
162-
<execution>
163-
<id>attach-javadocs</id>
164-
<goals>
165-
<goal>jar</goal>
166-
</goals>
167-
<configuration>
168-
<source>8</source>
169-
</configuration>
170-
</execution>
171-
</executions>
172-
</plugin>
173186
<plugin>
174187
<groupId>org.apache.maven.plugins</groupId>
175188
<artifactId>maven-compiler-plugin</artifactId>
176-
<version>3.6.1</version>
189+
<version>${maven-compiler-plugin.version}</version>
177190
<configuration>
178191
<source>${java.version}</source>
179192
<target>${java.version}</target>
193+
<annotationProcessorPaths>
194+
<path>
195+
<groupId>org.openjdk.jmh</groupId>
196+
<artifactId>jmh-generator-annprocess</artifactId>
197+
<version>${jmh.version}</version>
198+
</path>
199+
</annotationProcessorPaths>
180200
</configuration>
181201
</plugin>
182202
<plugin>
183203
<groupId>org.apache.maven.plugins</groupId>
184204
<artifactId>maven-surefire-plugin</artifactId>
185-
<version>2.22.2</version>
205+
<version>3.5.4</version>
186206
</plugin>
187207
<!-- JACOCO added for code coverage -->
188208
<plugin>
189209
<groupId>org.jacoco</groupId>
190210
<artifactId>jacoco-maven-plugin</artifactId>
191-
<version>0.8.6</version>
211+
<version>0.8.13</version>
192212
<executions>
193213
<execution>
194214
<id>pre-unit-test</id>
@@ -251,7 +271,7 @@
251271
<plugin>
252272
<groupId>org.apache.maven.plugins</groupId>
253273
<artifactId>maven-compiler-plugin</artifactId>
254-
<version>3.8.0</version>
274+
<version>${maven-compiler-plugin.version}</version>
255275
<configuration>
256276
<release>11</release>
257277
</configuration>

0 commit comments

Comments
 (0)