Skip to content

Commit 619dde9

Browse files
Stranger6667Stranger6667
committed
docs: Update benchmarks
Signed-off-by: Dmitry Dygalo <[email protected]>
1 parent c6c6550 commit 619dde9

File tree

3 files changed

+30
-37
lines changed

3 files changed

+30
-37
lines changed

‎crates/benchmark-suite/README.md‎

Lines changed: 10 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -26,45 +26,37 @@ $ cargo bench
2626
| GeoJSON | Canadian border in GeoJSON format | 4.8 KB | 2.1 MB |
2727
| CITM | Concert data catalog with inferred schema | 2.3 KB | 501 KB |
2828
| Fast | From fastjsonschema benchmarks (valid/invalid) | 595 B | 55 B / 60 B |
29+
| FHIR | Patient example validated against FHIR schema | 3.3 MB | 2.1 KB |
2930

3031
Sources:
3132
- OpenAPI: [Zuora](https://github.com/APIs-guru/openapi-directory/blob/1afd351ddf50e050acdb52937a819ef1927f417a/APIs/zuora.com/2021-04-23/openapi.yaml), [Schema](https://spec.openapis.org/oas/3.0/schema/2021-09-28)
3233
- Swagger: [Kubernetes](https://raw.githubusercontent.com/APIs-guru/openapi-directory/master/APIs/kubernetes.io/v1.10.0/swagger.yaml), [Schema](https://github.com/OAI/OpenAPI-Specification/blob/main/_archive_/schemas/v2.0/schema.json)
3334
- GeoJSON: [Schema](https://geojson.org/schema/FeatureCollection.json)
3435
- CITM: Schema inferred via [infers-jsonschema](https://github.com/Stranger6667/infers-jsonschema)
3536
- Fast: [fastjsonschema benchmarks](https://github.com/horejsek/python-fastjsonschema/blob/master/performance.py#L15)
37+
- FHIR: [Schema](http://hl7.org/fhir/R4/fhir.schema.json.zip) (R4 v4.0.1), [Example](http://hl7.org/fhir/R4/patient-example-d.json.html)
3638

3739
## Results
3840

3941
### Comparison with Other Libraries
4042

4143
| Benchmark | jsonschema_valid | valico | boon | jsonschema (validate) |
4244
|---------------|------------------|---------------|---------------|------------------------|
43-
| OpenAPI | - | - | 6.60 ms (**x3.17**) | 2.0813 ms |
44-
| Swagger | - | 114.26 ms (**x51.90**) | 10.06 ms (**x4.57**) | 2.2021 ms |
45-
| GeoJSON | 19.56 ms (**x23.37**) | 299.53 ms (**x358.00**) | 16.59 ms (**x19.82**) | 836.93 µs |
46-
| CITM Catalog | 2.84 ms (**x7.40**) | 28.30 ms (**x73.74**) | 1.11 ms (**x2.89**) | 383.98 µs |
47-
| Fast (Valid) | 1.11 µs (**x14.67**) | 3.78 µs (**x49.94**) | 332.39 ns (**x4.39**) | 75.748 ns |
48-
| Fast (Invalid)| 247.88 ns (**x4.67**) | 3.82 µs (**x71.93**) | 383.79 ns (**x7.22**) | 53.176 ns |
49-
50-
### jsonschema Performance: `validate` vs `is_valid`
51-
52-
| Benchmark | validate | is_valid | Speedup |
53-
|---------------|------------|------------|---------|
54-
| OpenAPI | 2.0813 ms | 2.0612 ms | **1.01x** |
55-
| Swagger | 2.2021 ms | 2.0729 ms | **1.06x** |
56-
| GeoJSON | 836.93 µs | 796.28 µs | **1.05x** |
57-
| CITM Catalog | 383.98 µs | 313.13 µs | **1.23x** |
58-
| Fast (Valid) | 75.748 ns | 53.642 ns | **1.41x** |
59-
| Fast (Invalid)| 53.176 ns | 3.4919 ns | **15.23x** |
45+
| OpenAPI | - | - | 6.7333 ms (**x3.58**) | 1.8807 ms |
46+
| Swagger | - | 105.07 ms (**x49.51**) | 9.6542 ms (**x4.55**) | 2.1220 ms |
47+
| GeoJSON | 15.718 ms (**x20.29**) | 273.29 ms (**x352.93**) | 16.785 ms (**x21.67**) | 774.53 µs |
48+
| CITM Catalog | 2.4505 ms (**x2.24**) | 25.139 ms (**x23.00**) | 981.26 µs (**x0.90**) | 1.0928 ms |
49+
| Fast (Valid) | 990.10 ns (**x5.16**) | 3.1914 µs (**x16.64**) | 298.32 ns (**x1.56**) | 191.79 ns |
50+
| Fast (Invalid)| 233.43 ns (**x0.85**) | 3.2628 µs (**x11.87**) | 422.37 ns (**x1.54**) | 274.98 ns |
51+
| FHIR | 578.97 ms (**x152930.85**) | 1.5964 ms (**x421.62**) | 173.39 µs (**x45.80**) | 3.7863 µs |
6052

6153
Notes:
6254

6355
1. `jsonschema_valid` and `valico` do not handle valid path instances matching the `^\\/` regex.
6456

6557
2. `jsonschema_valid` fails to resolve local references (e.g. `#/definitions/definitions`).
6658

67-
You can find benchmark code in [benches/](benches/), Rust version is `1.82`.
59+
You can find benchmark code in [benches/](benches/), Rust version is `1.91.1`.
6860

6961
## Contributing
7062

‎crates/jsonschema-py/BENCHMARKS.md‎

Lines changed: 11 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -31,43 +31,35 @@ $ pytest benches/bench.py
3131
| GeoJSON | Canadian border in GeoJSON format | 4.8 KB | 2.1 MB |
3232
| CITM | Concert data catalog with inferred schema | 2.3 KB | 501 KB |
3333
| Fast | From fastjsonschema benchmarks (valid/invalid) | 595 B | 55 B / 60 B |
34+
| FHIR | Patient example validated against FHIR schema | 3.3 MB | 2.1 KB |
3435

3536
Sources:
3637
- OpenAPI: [Zuora](https://github.com/APIs-guru/openapi-directory/blob/1afd351ddf50e050acdb52937a819ef1927f417a/APIs/zuora.com/2021-04-23/openapi.yaml), [Schema](https://spec.openapis.org/oas/3.0/schema/2021-09-28)
3738
- Swagger: [Kubernetes](https://raw.githubusercontent.com/APIs-guru/openapi-directory/master/APIs/kubernetes.io/v1.10.0/swagger.yaml), [Schema](https://github.com/OAI/OpenAPI-Specification/blob/main/_archive_/schemas/v2.0/schema.json)
3839
- GeoJSON: [Schema](https://geojson.org/schema/FeatureCollection.json)
3940
- CITM: Schema inferred via [infers-jsonschema](https://github.com/Stranger6667/infers-jsonschema)
4041
- Fast: [fastjsonschema benchmarks](https://github.com/horejsek/python-fastjsonschema/blob/master/performance.py#L15)
42+
- FHIR: [Schema](http://hl7.org/fhir/R4/fhir.schema.json.zip) (R4 v4.0.1), [Example](http://hl7.org/fhir/R4/patient-example-d.json.html)
4143

4244
## Results
4345

4446
### Comparison with Other Libraries
4547

46-
| Benchmark | fastjsonschema | jsonschema | jsonschema-rs |
48+
| Benchmark | fastjsonschema | jsonschema | jsonschema-rs (validate) |
4749
|---------------|----------------|---------------|----------------|
48-
| OpenAPI | - (1) | 640.34 ms (**x94.35**) | 6.79 ms |
49-
| Swagger | - (1) | 1134.76 ms (**x232.81**)| 4.88 ms |
50-
| Canada (GeoJSON) | 10.43 ms (**x4.33**) | 785.21 ms (**x325.59**) | 2.41 ms |
51-
| CITM Catalog | 4.97 ms (**x3.66**) | 82.42 ms (**x60.67**) | 1.36 ms |
52-
| Fast (Valid) | 1.95 µs (**x6.49**) | 35.81 µs (**x119.15**) | 300.55 ns |
53-
| Fast (Invalid)| 2.17 µs (**x4.14**) | 35.83 µs (**x68.31**) | 524.50 ns |
54-
55-
### jsonschema-rs Performance: `validate` vs `is_valid`
56-
57-
| Benchmark | validate | is_valid | Speedup |
58-
|---------------|------------|------------|---------|
59-
| OpenAPI | 6.79 ms | 6.84 ms | **0.99x** |
60-
| Swagger | 4.88 ms | 4.73 ms | **1.03x** |
61-
| Canada (GeoJSON) | 2.41 ms | 2.34 ms | **1.03x** |
62-
| CITM Catalog | 1.36 ms | 1.28 ms | **1.06x** |
63-
| Fast (Valid) | 300.55 ns | 249.95 ns | **1.20x** |
64-
| Fast (Invalid)| 524.50 ns | 561.00 ns | **0.93x** |
50+
| OpenAPI | - (1) | 533.80 ms (**x72.88**) | 7.3264 ms |
51+
| Swagger | - (1) | 970.32 ms (**x193.34**)| 5.0193 ms |
52+
| Canada (GeoJSON) | 9.521 ms (**x3.88**) | 738.28 ms (**x300.61**) | 2.4559 ms |
53+
| CITM Catalog | 4.0752 ms (**x2.85**) | 76.05 ms (**x53.26**) | 1.4281 ms |
54+
| Fast (Valid) | 1.933 µs (**x6.90**) | 32.124 µs (**x114.75**) | 279.95 ns |
55+
| Fast (Invalid)| 2.073 µs (**x4.24**) | 32.245 µs (**x65.89**) | 489.35 ns |
56+
| FHIR | 1.9352 ms (**x452.40**) | 11.463 ms (**x2679.55**) | 4.2780 µs |
6557

6658
Notes:
6759

6860
1. `fastjsonschema` fails to compile the Open API spec due to the presence of the `uri-reference` format (that is not defined in Draft 4). However, unknown formats are explicitly supported by the spec.
6961

70-
You can find benchmark code in [benches/](benches/), Python version `3.13.0`, Rust version `1.82`.
62+
You can find benchmark code in [benches/](benches/), Python version `3.14.0`, Rust version `1.91.1`.
7163

7264
## Contributing
7365

‎crates/jsonschema-py/benches/bench.py‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,8 @@ def load_from_benches(filename, loader=load_json):
3939
CITM_CATALOG_SCHEMA = load_from_benches("citm_catalog_schema.json")
4040
CITM_CATALOG = load_from_benches("citm_catalog.json")
4141
FAST_SCHEMA = load_from_benches("fast_schema.json")
42+
FHIR_SCHEMA = load_from_benches("fhir.schema.json")
43+
FHIR_INSTANCE = load_from_benches("patient-example-d.json")
4244
FAST_INSTANCE_VALID = [
4345
9,
4446
"hello",
@@ -100,6 +102,7 @@ def args(request, variant):
100102
"geojson.json",
101103
"citm_catalog_schema.json",
102104
"fast_schema.json",
105+
"fhir.schema.json",
103106
),
104107
)
105108
@pytest.mark.parametrize(
@@ -173,3 +176,9 @@ def test_canada(benchmark, args):
173176
@pytest.mark.benchmark(group="citm_catalog")
174177
def test_citm_catalog(benchmark, args):
175178
benchmark(*args)
179+
180+
181+
@pytest.mark.data(FHIR_SCHEMA, FHIR_INSTANCE)
182+
@pytest.mark.benchmark(group="fhir")
183+
def test_fhir(benchmark, args):
184+
benchmark(*args)

0 commit comments

Comments
 (0)