Skip to content

Commit 0ee6b07

Browse files
msluszniakmsluszniak
committed
Remove benchmarks from hook API
1 parent eece898 commit 0ee6b07

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

50 files changed

+4
-1555
lines changed

‎docs/docs/02-hooks/01-natural-language-processing/useLLM.md‎

Lines changed: 0 additions & 37 deletions
Original file line numberDiff line numberDiff line change
@@ -498,40 +498,3 @@ Depending on selected model and the user's device generation speed can be above
498498
| [Phi 4 Mini](https://huggingface.co/software-mansion/react-native-executorch-phi-4-mini) | 4B | ✅ |
499499
| [SmolLM 2](https://huggingface.co/software-mansion/react-native-executorch-smolLm-2) | 135M, 360M, 1.7B | ✅ |
500500
| [LLaMA 3.2](https://huggingface.co/software-mansion/react-native-executorch-llama-3.2) | 1B, 3B | ✅ |
501-
502-
## Benchmarks
503-
504-
### Model size
505-
506-
| Model | XNNPACK [GB] |
507-
| --------------------- | :----------: |
508-
| LLAMA3_2_1B | 2.47 |
509-
| LLAMA3_2_1B_SPINQUANT | 1.14 |
510-
| LLAMA3_2_1B_QLORA | 1.18 |
511-
| LLAMA3_2_3B | 6.43 |
512-
| LLAMA3_2_3B_SPINQUANT | 2.55 |
513-
| LLAMA3_2_3B_QLORA | 2.65 |
514-
515-
### Memory usage
516-
517-
| Model | Android (XNNPACK) [GB] | iOS (XNNPACK) [GB] |
518-
| --------------------- | :--------------------: | :----------------: |
519-
| LLAMA3_2_1B | 3.2 | 3.1 |
520-
| LLAMA3_2_1B_SPINQUANT | 1.9 | 2 |
521-
| LLAMA3_2_1B_QLORA | 2.2 | 2.5 |
522-
| LLAMA3_2_3B | 7.1 | 7.3 |
523-
| LLAMA3_2_3B_SPINQUANT | 3.7 | 3.8 |
524-
| LLAMA3_2_3B_QLORA | 4 | 4.1 |
525-
526-
### Inference time
527-
528-
| Model | iPhone 16 Pro (XNNPACK) [tokens/s] | iPhone 13 Pro (XNNPACK) [tokens/s] | iPhone SE 3 (XNNPACK) [tokens/s] | Samsung Galaxy S24 (XNNPACK) [tokens/s] | OnePlus 12 (XNNPACK) [tokens/s] |
529-
| --------------------- | :--------------------------------: | :--------------------------------: | :------------------------------: | :-------------------------------------: | :-----------------------------: |
530-
| LLAMA3_2_1B | 16.1 | 11.4 | ❌ | 15.6 | 19.3 |
531-
| LLAMA3_2_1B_SPINQUANT | 40.6 | 16.7 | 16.5 | 40.3 | 48.2 |
532-
| LLAMA3_2_1B_QLORA | 31.8 | 11.4 | 11.2 | 37.3 | 44.4 |
533-
| LLAMA3_2_3B | ❌ | ❌ | ❌ | ❌ | 7.1 |
534-
| LLAMA3_2_3B_SPINQUANT | 17.2 | 8.2 | ❌ | 16.2 | 19.4 |
535-
| LLAMA3_2_3B_QLORA | 14.5 | ❌ | ❌ | 14.8 | 18.1 |
536-
537-
❌ - Insufficient RAM.

‎docs/docs/02-hooks/01-natural-language-processing/useSpeechToText.md‎

Lines changed: 0 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -322,22 +322,3 @@ function App() {
322322
| [whisper-base](https://huggingface.co/openai/whisper-base) | Multilingual |
323323
| [whisper-small.en](https://huggingface.co/openai/whisper-small.en) | English |
324324
| [whisper-small](https://huggingface.co/openai/whisper-small) | Multilingual |
325-
326-
## Benchmarks
327-
328-
### Model size
329-
330-
| Model | XNNPACK [MB] |
331-
| ---------------- | :----------: |
332-
| WHISPER_TINY_EN | 151 |
333-
| WHISPER_TINY | 151 |
334-
| WHISPER_BASE_EN | 290.6 |
335-
| WHISPER_BASE | 290.6 |
336-
| WHISPER_SMALL_EN | 968 |
337-
| WHISPER_SMALL | 968 |
338-
339-
### Memory usage
340-
341-
| Model | Android (XNNPACK) [MB] | iOS (XNNPACK) [MB] |
342-
| ------------ | :--------------------: | :----------------: |
343-
| WHISPER_TINY | 410 | 375 |

‎docs/docs/02-hooks/01-natural-language-processing/useTextEmbeddings.md‎

Lines changed: 0 additions & 40 deletions
Original file line numberDiff line numberDiff line change
@@ -116,43 +116,3 @@ function App() {
116116
:::info
117117
For the supported models, the returned embedding vector is normalized, meaning that its length is equal to 1. This allows for easier comparison of vectors using cosine similarity, just calculate the dot product of two vectors to get the cosine similarity score.
118118
:::
119-
120-
## Benchmarks
121-
122-
### Model size
123-
124-
| Model | XNNPACK [MB] |
125-
| -------------------------- | :----------: |
126-
| ALL_MINILM_L6_V2 | 91 |
127-
| ALL_MPNET_BASE_V2 | 438 |
128-
| MULTI_QA_MINILM_L6_COS_V1 | 91 |
129-
| MULTI_QA_MPNET_BASE_DOT_V1 | 438 |
130-
| CLIP_VIT_BASE_PATCH32_TEXT | 254 |
131-
132-
### Memory usage
133-
134-
| Model | Android (XNNPACK) [MB] | iOS (XNNPACK) [MB] |
135-
| -------------------------- | :--------------------: | :----------------: |
136-
| ALL_MINILM_L6_V2 | 95 | 110 |
137-
| ALL_MPNET_BASE_V2 | 405 | 455 |
138-
| MULTI_QA_MINILM_L6_COS_V1 | 120 | 140 |
139-
| MULTI_QA_MPNET_BASE_DOT_V1 | 435 | 455 |
140-
| CLIP_VIT_BASE_PATCH32_TEXT | 200 | 280 |
141-
142-
### Inference time
143-
144-
:::warning
145-
Times presented in the tables are measured as consecutive runs of the model. Initial run times may be up to 2x longer due to model loading and initialization.
146-
:::
147-
148-
| Model | iPhone 17 Pro (XNNPACK) [ms] | OnePlus 12 (XNNPACK) [ms] |
149-
| -------------------------- | :--------------------------: | :-----------------------: |
150-
| ALL_MINILM_L6_V2 | 7 | 21 |
151-
| ALL_MPNET_BASE_V2 | 24 | 90 |
152-
| MULTI_QA_MINILM_L6_COS_V1 | 7 | 19 |
153-
| MULTI_QA_MPNET_BASE_DOT_V1 | 24 | 88 |
154-
| CLIP_VIT_BASE_PATCH32_TEXT | 14 | 39 |
155-
156-
:::info
157-
Benchmark times for text embeddings are highly dependent on the sentence length. The numbers above are based on a sentence of around 80 tokens. For shorter or longer sentences, inference time may vary accordingly.
158-
:::

‎docs/docs/02-hooks/01-natural-language-processing/useVAD.md‎

Lines changed: 0 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -164,31 +164,3 @@ export default function App() {
164164
## Supported models
165165

166166
- [fsmn-vad](https://huggingface.co/funasr/fsmn-vad)
167-
168-
## Benchmarks
169-
170-
### Model size
171-
172-
| Model | XNNPACK [MB] |
173-
| -------- | :----------: |
174-
| FSMN_VAD | 1.83 |
175-
176-
### Memory usage
177-
178-
| Model | Android (XNNPACK) [MB] | iOS (XNNPACK) [MB] |
179-
| -------- | :--------------------: | :----------------: |
180-
| FSMN_VAD | 97 | 45,9 |
181-
182-
### Inference time
183-
184-
<!-- TODO: MEASURE INFERENCE TIME FOR SAMSUNG GALAXY S24 WHEN POSSIBLE -->
185-
186-
:::warning warning
187-
Times presented in the tables are measured as consecutive runs of the model. Initial run times may be up to 2x longer due to model loading and initialization.
188-
:::
189-
190-
Inference time were measured on a 60s audio, that can be found [here](https://models.silero.ai/vad_models/en.wav).
191-
192-
| Model | iPhone 16 Pro (XNNPACK) [ms] | iPhone 14 Pro Max (XNNPACK) [ms] | iPhone SE 3 (XNNPACK) [ms] | OnePlus 12 (XNNPACK) [ms] |
193-
| -------- | :--------------------------: | :------------------------------: | :------------------------: | :-----------------------: |
194-
| FSMN_VAD | 151 | 171 | 180 | 109 |

‎docs/docs/02-hooks/02-computer-vision/useClassification.md‎

Lines changed: 0 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -87,27 +87,3 @@ function App() {
8787
| Model | Number of classes | Class list |
8888
| ----------------------------------------------------------------------------------------------------------------- | ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
8989
| [efficientnet_v2_s](https://pytorch.org/vision/stable/models/generated/torchvision.models.efficientnet_v2_s.html) | 1000 | [ImageNet1k_v1](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/common/rnexecutorch/models/classification/Constants.h) |
90-
91-
## Benchmarks
92-
93-
### Model size
94-
95-
| Model | XNNPACK [MB] | Core ML [MB] |
96-
| ----------------- | :----------: | :----------: |
97-
| EFFICIENTNET_V2_S | 85.6 | 43.9 |
98-
99-
### Memory usage
100-
101-
| Model | Android (XNNPACK) [MB] | iOS (Core ML) [MB] |
102-
| ----------------- | :--------------------: | :----------------: |
103-
| EFFICIENTNET_V2_S | 230 | 87 |
104-
105-
### Inference time
106-
107-
:::warning
108-
Times presented in the tables are measured as consecutive runs of the model. Initial run times may be up to 2x longer due to model loading and initialization.
109-
:::
110-
111-
| Model | iPhone 17 Pro (Core ML) [ms] | iPhone 16 Pro (Core ML) [ms] | iPhone SE 3 (Core ML) [ms] | Samsung Galaxy S24 (XNNPACK) [ms] | OnePlus 12 (XNNPACK) [ms] |
112-
| ----------------- | :--------------------------: | :--------------------------: | :------------------------: | :-------------------------------: | :-----------------------: |
113-
| EFFICIENTNET_V2_S | 64 | 68 | 217 | 205 | 198 |

‎docs/docs/02-hooks/02-computer-vision/useImageEmbeddings.md‎

Lines changed: 0 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -102,31 +102,3 @@ try {
102102
:::info
103103
For the supported models, the returned embedding vector is normalized, meaning that its length is equal to 1. This allows for easier comparison of vectors using cosine similarity, just calculate the dot product of two vectors to get the cosine similarity score.
104104
:::
105-
106-
## Benchmarks
107-
108-
### Model size
109-
110-
| Model | XNNPACK [MB] |
111-
| --------------------------- | :----------: |
112-
| CLIP_VIT_BASE_PATCH32_IMAGE | 352 |
113-
114-
### Memory usage
115-
116-
| Model | Android (XNNPACK) [MB] | iOS (XNNPACK) [MB] |
117-
| --------------------------- | :--------------------: | :----------------: |
118-
| CLIP_VIT_BASE_PATCH32_IMAGE | 350 | 340 |
119-
120-
### Inference time
121-
122-
:::warning
123-
Times presented in the tables are measured as consecutive runs of the model. Initial run times may be up to 2x longer due to model loading and initialization. Performance also heavily depends on image size, because resize is expansive operation, especially on low-end devices.
124-
:::
125-
126-
| Model | iPhone 17 Pro (XNNPACK) [ms] | OnePlus 12 (XNNPACK) [ms] |
127-
| --------------------------- | :--------------------------: | :-----------------------: |
128-
| CLIP_VIT_BASE_PATCH32_IMAGE | 18 | 55 |
129-
130-
:::info
131-
Image embedding benchmark times are measured using 224×224 pixel images, as required by the model. All input images, whether larger or smaller, are resized to 224×224 before processing. Resizing is typically fast for small images but may be noticeably slower for very large images, which can increase total inference time.
132-
:::

‎docs/docs/02-hooks/02-computer-vision/useImageSegmentation.md‎

Lines changed: 0 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -87,31 +87,3 @@ function App() {
8787
| Model | Number of classes | Class list |
8888
| -------------------------------------------------------------------------------------------------------------------------------- | ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- |
8989
| [deeplabv3_resnet50](https://pytorch.org/vision/stable/models/generated/torchvision.models.segmentation.deeplabv3_resnet50.html) | 21 | [DeeplabLabel](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/src/types/imageSegmentation.ts) |
90-
91-
## Benchmarks
92-
93-
### Model size
94-
95-
| Model | XNNPACK [MB] |
96-
| ----------------- | ------------ |
97-
| DEELABV3_RESNET50 | 168 |
98-
99-
### Memory usage
100-
101-
:::warning
102-
Data presented in the following sections is based on inference with non-resized output. When resize is enabled, expect higher memory usage and inference time with higher resolutions.
103-
:::
104-
105-
| Model | Android (XNNPACK) [MB] | iOS (XNNPACK) [MB] |
106-
| ----------------- | ---------------------- | ------------------ |
107-
| DEELABV3_RESNET50 | 930 | 660 |
108-
109-
### Inference time
110-
111-
:::warning
112-
Times presented in the tables are measured as consecutive runs of the model. Initial run times may be up to 2x longer due to model loading and initialization.
113-
:::
114-
115-
| Model | iPhone 16 Pro (Core ML) [ms] | iPhone 14 Pro Max (Core ML) [ms] | Samsung Galaxy S24 (XNNPACK) [ms] |
116-
| ----------------- | ---------------------------- | -------------------------------- | --------------------------------- |
117-
| DEELABV3_RESNET50 | 1000 | 670 | 700 |

‎docs/docs/02-hooks/02-computer-vision/useOCR.md‎

Lines changed: 0 additions & 42 deletions
Original file line numberDiff line numberDiff line change
@@ -266,45 +266,3 @@ You need to make sure the recognizer model you pass in `recognizerSource` matche
266266
| ------------------------------------------------- | :--------: |
267267
| [CRAFT](https://github.com/clovaai/CRAFT-pytorch) | Detector |
268268
| [CRNN](https://www.jaided.ai/easyocr/modelhub/) | Recognizer |
269-
270-
## Benchmarks
271-
272-
### Model size
273-
274-
| Model | XNNPACK [MB] |
275-
| -------------------------- | :-----------: |
276-
| Detector (CRAFT_QUANTIZED) | 20.9 |
277-
| Recognizer (CRNN) | 18.5 - 25.2\* |
278-
279-
\* - The model weights vary depending on the language.
280-
281-
### Memory usage
282-
283-
| Model | Android (XNNPACK) [MB] | iOS (XNNPACK) [MB] |
284-
| ------------------------------------ | :--------------------: | :----------------: |
285-
| Detector (CRAFT) + Recognizer (CRNN) | 1400 | 1320 |
286-
287-
### Inference time
288-
289-
**Image Used for Benchmarking:**
290-
291-
| ![Alt text](../../../static/img/harvard.png) | ![Alt text](../../../static/img/harvard-boxes.png) |
292-
| -------------------------------------------- | -------------------------------------------------- |
293-
| Original Image | Image with detected Text Boxes |
294-
295-
:::warning
296-
Times presented in the tables are measured as consecutive runs of the model. Initial run times may be up to 2x longer due to model loading and initialization.
297-
:::
298-
299-
**Time measurements:**
300-
301-
Notice that the recognizer models were executed between 3 and 7 times during a single recognition.
302-
The values below represent the averages across all runs for the benchmark image.
303-
304-
| Model | iPhone 17 Pro [ms] | iPhone 16 Pro [ms] | iPhone SE 3 | Samsung Galaxy S24 [ms] | OnePlus 12 [ms] |
305-
| ------------------------------- | ------------------ | ------------------ | ----------- | ----------------------- | --------------- |
306-
| **Total Inference Time** | 652 | 600 | 2855 | 1092 | 1034 |
307-
| Detector (CRAFT) `forward_800` | 220 | 221 | 1740 | 521 | 492 |
308-
| Recognizer (CRNN) `forward_512` | 45 | 38 | 110 | 40 | 38 |
309-
| Recognizer (CRNN) `forward_256` | 21 | 18 | 54 | 20 | 19 |
310-
| Recognizer (CRNN) `forward_128` | 11 | 9 | 27 | 10 | 10 |

‎docs/docs/02-hooks/02-computer-vision/useObjectDetection.md‎

Lines changed: 0 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -126,27 +126,3 @@ function App() {
126126
| Model | Number of classes | Class list |
127127
| --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
128128
| [SSDLite320 MobileNetV3 Large](https://pytorch.org/vision/stable/models/generated/torchvision.models.detection.ssdlite320_mobilenet_v3_large.html#torchvision.models.detection.SSDLite320_MobileNet_V3_Large_Weights) | 91 | [COCO](https://github.com/software-mansion/react-native-executorch/blob/main/packages/react-native-executorch/common/rnexecutorch/models/object_detection/Constants.h) |
129-
130-
## Benchmarks
131-
132-
### Model size
133-
134-
| Model | XNNPACK [MB] |
135-
| ------------------------------ | :----------: |
136-
| SSDLITE_320_MOBILENET_V3_LARGE | 13.9 |
137-
138-
### Memory usage
139-
140-
| Model | Android (XNNPACK) [MB] | iOS (XNNPACK) [MB] |
141-
| ------------------------------ | :--------------------: | :----------------: |
142-
| SSDLITE_320_MOBILENET_V3_LARGE | 164 | 132 |
143-
144-
### Inference time
145-
146-
:::warning
147-
Times presented in the tables are measured as consecutive runs of the model. Initial run times may be up to 2x longer due to model loading and initialization.
148-
:::
149-
150-
| Model | iPhone 17 Pro (XNNPACK) [ms] | iPhone 16 Pro (XNNPACK) [ms] | iPhone SE 3 (XNNPACK) [ms] | Samsung Galaxy S24 (XNNPACK) [ms] | OnePlus 12 (XNNPACK) [ms] |
151-
| ------------------------------ | :--------------------------: | :--------------------------: | :------------------------: | :-------------------------------: | :-----------------------: |
152-
| SSDLITE_320_MOBILENET_V3_LARGE | 71 | 74 | 257 | 115 | 109 |

0 commit comments

Comments
 (0)