Skip to content

Commit a89eed3

Browse files
authored
added jsl-vision benchmarks (#2105)
1 parent a265f64 commit a89eed3

File tree

1 file changed

+46
-12
lines changed

1 file changed

+46
-12
lines changed

docs/en/ocr_benchmark.md

Lines changed: 46 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -8,25 +8,59 @@ key: docs-benchmark
88
modify_date: "2024-06-24"
99
show_nav: true
1010
sidebar:
11-
nav: sparknlp-healthcare
11+
nav: sparknlp-healthcare
1212
---
1313

1414
<div class="h3-box" markdown="1">
1515

16-
## Speed Benchmarks
16+
## Benchmarks
17+
18+
### JSL-Vision Benchmarks
19+
Speed benchmarks ran on a H100 machine with 112 Cores and 2TB Ram
20+
21+
{:.table-model-big}
22+
| model | Avg Speed | |
23+
| ----------------------------------- | --------- | --- |
24+
| jsl_vision_ocr_1.0_light | 1.11s | |
25+
| jsl_vision_ocr_1.0 | 2.53s | |
26+
| jsl_vision_ocr_1.0_structured_light | 13.5s | |
27+
| jsl_vision_ocr_structured_1.0 | 58.64 | |
28+
29+
30+
31+
[FUNSD](https://guillaumejaume.github.io/FUNSD/) simple OCR metrics.
32+
For comparison against [open source see here](https://medium.com/john-snow-labs/jsl-vision-state-of-the-art-document-understanding-on-your-hardware-f4862f15d9f9) and against [closed source see here](https://medium.com/john-snow-labs/jsl-vision-vs-closed-source-models-document-intelligence-without-compromise-62728afe0c5b)
33+
34+
{:.table-model-big}
35+
| model | funsd_OCR_cer (Lower is better) |
36+
| ------------------------ | ------------------------------- |
37+
| jsl_vision_ocr_1.0 | 0.147 |
38+
| jsl_vision_ocr_1.0_light | 0.203 |
39+
40+
Metrics on [OmniOCR Benchmark](https://github.com/getomni-ai/benchmark) using Img2Markdown2Json flow.
41+
For comparison against [open source see here](https://medium.com/john-snow-labs/jsl-vision-state-of-the-art-document-understanding-on-your-hardware-f4862f15d9f9) and against [closed source see here](https://medium.com/john-snow-labs/jsl-vision-vs-closed-source-models-document-intelligence-without-compromise-62728afe0c5b)
42+
43+
{:.table-model-big}
44+
| model | omni_json_accuracy (Higher is better) |
45+
| ----------------------------------- | ------------------------------------- |
46+
| jsl_vision_ocr_structured_1.0 | 0.805 |
47+
| jsl_vision_ocr_structured_1.0_light | 0.759 |
48+
49+
50+
<div class="h3-box" markdown="1">
1751

1852
### PDF De-identification Benchmark
1953

2054
- **Dataset:** 1000 scanned PDF pages.
21-
- **Instance :**
22-
- m5n.4xlarge (16 vCPUs, 64 GiB memory)
23-
- m5n.8xlarge (32 vCPUs, 128 GiB memory)
55+
- **Instance :**
56+
- m5n.4xlarge (16 vCPUs, 64 GiB memory)
57+
- m5n.8xlarge (32 vCPUs, 128 GiB memory)
2458
- **AMI:** ubuntu/images/hvm-ssd/ubuntu-jammy-22.04-amd64-server-20240411
2559
- **Versions:**
26-
- **spark-nlp Version:** v5.4.0
27-
- **visual-nlp Version:** v5.3.2
28-
- **spark-nlp-jsl Version :** v5.3.2
29-
- **Spark Version :** v3.4.1
60+
- **spark-nlp Version:** v5.4.0
61+
- **visual-nlp Version:** v5.3.2
62+
- **spark-nlp-jsl Version :** v5.3.2
63+
- **Spark Version :** v3.4.1
3064
- **Visual NLP Pipeline:** 'pdf_deid_subentity_context_augmented_pipeline'
3165

3266
</div><div class="h3-box" markdown="1">
@@ -43,7 +77,7 @@ sidebar:
4377
</div><div class="h3-box" markdown="1">
4478

4579
### Dicom De-identification Benchmark
46-
This section contains benchmarks for de-ideintification of dicom files, both for GPU and CPU.
80+
This section contains benchmarks for de-ideintification of dicom files, both for GPU and CPU.
4781
Note: file sizes are included as reference, but *they are not* the best proxy for estimating running time, as the final figures will depend on image size than in turn depends on the actual compression that is being used in the file.
4882
The numbers reported are average *time per file*.
4983

@@ -105,7 +139,7 @@ Note: don't forget to count multi-frame images.
105139
</div><div class="h3-box" markdown="1">
106140

107141
#### I don't know the dimensions of my images
108-
If you don't know the dimensions, but you know that your images are also uncompressed, you can use file size as proxy.
142+
If you don't know the dimensions, but you know that your images are also uncompressed, you can use file size as proxy.
109143
_Keep in mind that if you actually have compressed images you will be underestimating the processing time_.
110144

111145

@@ -119,7 +153,7 @@ But not all hope is lost, you can work with frames!.
119153
</div><div class="h3-box" markdown="1">
120154

121155
#### Using total frame count as a proxy for estimation
122-
If you don't know the size of the images, or the compression, you can estimate the number of frames and use the time it takes for a single frame(like the numbers we've shared in this document) as a proxy.
156+
If you don't know the size of the images, or the compression, you can estimate the number of frames and use the time it takes for a single frame(like the numbers we've shared in this document) as a proxy.
123157
</br>
124158
Reasons for doing this:
125159
* You can compute frame count and sizes very efficiently using Visual NLP.

0 commit comments

Comments
 (0)