Skip to content

Commit b2b21e5

Browse files
committed
add benchmark comparison
1 parent 68f1ca1 commit b2b21e5

File tree

1 file changed

+36
-0
lines changed

1 file changed

+36
-0
lines changed

examples/readme.md

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
# Benchmark comparison for the models
2+
- Hardware: Macbook pro 14 inches with m1 pro and 16 GB of ram
3+
4+
### Example 1: personal portfolio
5+
**URL**: https://perinim.github.io/projects
6+
**Task**: List me all the projects with their description.
7+
8+
| Name | Execution time (seconds) | total_tokens | prompt_tokens | completion_tokens | successful_requests | total_cost_USD |
9+
| ----------------------------------- | ------------------------ | ------------ | ------------- | ----------------- | ------------------- | -------------- |
10+
| gpt-3.5-turbo | 35.98 | 858 | 512 | 346 | 2 | 0.00146 |
11+
| gpt-4-turbo-preview | 13.907 | 866 | 512 | 354 | 2 | 0.01574 |
12+
| Ollama with Mistral and embeddings | 26.10 | 0 | 0 | 0 | 0 | 0 |
13+
| Docker with Mistral and embeddings | 240.22 | 0 | 0 | 0 | 0 | 0 |
14+
### Example 2: Wired
15+
**URL**: https://www.wired.com
16+
**Task**: List me all the articles with their description.
17+
18+
| Name | Execution time (seconds) | total_tokens | prompt_tokens | completion_tokens | successful_requests | total_cost_USD |
19+
| ----------------------------------- | ------------------------ | ------------ | ------------- | ----------------- | ------------------- | -------------- |
20+
| gpt-3.5-turbo | 87.03 | 3780 | 3760 | 3000 | 2 | 0.01319 |
21+
| gpt-4-turbo-preview | 74.90 | 5306 | 3060 | 2246 | 2 | 0.09798 |
22+
| Ollama with Mistral and embeddings | 60.915 | 0 | 0 | 0 | 0 | 0 |
23+
| Docker with Mistral and embeddings | 612.48 | 0<br> | 0<br> | 0<br> | 0<br> | 0<br> |
24+
25+
### Example 3: Amazon product page
26+
**URL**: https://www.amazon.it/s?k=alexa&__mk_it_IT=ÅMÅŽÕÑ&crid=1WWVF1RGDBBSB&sprefix=alex%2Caps%2C114&ref=nb_sb_noss_2
27+
**Task**: List me all the articles with their the costs and image url.
28+
29+
| Name | Execution time (seconds) | total_tokens | prompt_tokens | completion_tokens | successful_requests | total_cost_USD |
30+
| ----------------------------------- | ------------------------ | ------------ | ------------- | ----------------- | ------------------- | -------------- |
31+
| gpt-3.5-turbo | 145.55 | 26038 | 18091 | 7947 | 5 | 0.04303 |
32+
| gpt-4-turbo-preview | 82.38 | 15640 | 13698 | 1942 | 2 | 0.19524 |
33+
| Ollama with Llama2 and embeddings | 200.77 | 0<br> | 0<br> | 0<br> | 0<br> | 0<br> |
34+
| Docker with Mistral and embeddings | 2008.32 | 0<br> | 0<br> | 0<br> | 0<br> | 0<br> |
35+
## Hosting services
36+
[[💻 Provider costs informations]]

0 commit comments

Comments
 (0)