Merge pull request #139 from cohere-ai/add_public_embed_evals

added public benchmarks folder

Author: mahjongmen

public_embedding_benchmarks/beir.md

@@ -0,0 +1,7 @@
+| Model | Dimensions| msmarco | arguana | climate-fever | cqadupstack | dbpedia-entity | fever | fiqa | hotpotqa | nfcorpus | nq | quora | scidocs | scifact | trec-covid | webis-touche20 | bioasq | signal1m | trec-news | robust04 | avg BEIR (18 datasets) | avg Open BEIR (14 datasets)
+| -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- |
+| embed-english-v3.0 | 1024 | 42.9 | 61.5 | 38.4 | 41.5 | 43.4 | 89.0 | 42.2 | 70.7 | 38.6 | 61.6 | 88.7 | 20.3 | 71.8 | 81.9 | 32.4 | 45.6 | 26.3 | 50.5 | 54.0 | 53.2 | 55.9 |
+| embed-multilingual-v3.0 | 1024 | 43.4 | 55.1 | 30.0 | 40.6 | 41.0 | 88.5 | 44.1 | 70.6 | 36.4 | 63.4 | 88.9 | 19.3 | 70.0 | 83.4 | 32.7 | 42.8 | 27.2 | 47.6 | 53.4 | 51.9 | 54.6 |
+| OpenAI ada-002 | 1536 | 40.9 | 57.4 | 21.6 | 41.7 | 39.4 | 75.0 | 44.4 | 60.9 | 37.0 | 51.6 | 87.6 | 18.4 | 72.7 | 68.5 | 21.6 | N/A | N/A | N/A | N/A | N/A | 49.8 |
+
+Note: OpenAI did not report BEIR benchmarks for text-embedding-3-large

public_embedding_benchmarks/miracl.md

@@ -0,0 +1,7 @@
+| Model | Dimensions| Arabic (ar) | Bengali (bn) | English (en) | Spanish (es) | Persian (fa) | Finnish (fi) | French (fr) | Hindi (hi) | Indonesian (id) | Japanese (ja) | Korean (ko) | Russian (ru) | Swahili (sw) | Telugu (te) | Thai (th) | Chinese (zh) | Germany (de) | Yoruba (yo) | Avg (18 datasets) | Avg (excl. de and yo) |
+| -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- |
+| embed-multilingual-v3.0 | 1024 | 76.5 | 75.8 | 57.0 | 55.1 | 57.5 | 77.1 | 57.4 | 61.7 | 52.5 | 69.6 | 66.0 | 68.8 | 75.7 | 83.3 | 79.5 | 58.9 | 58.7 | 61.8 | 66.3 | 67.0 |
+| BM25 | N/A | 48.1 | 50.8 | 35.1 | 31.9 | 33.3 | 55.1 | 18.3 | 45.8 | 44.9 | 36.9 | 41.9 | 33.4 | 38.3 | 49.4 | 48.8 | 18.0 | N/A | N/A | N/A | 39.4 |
+| OpenAI text-embedding-large  | 3072 | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | 54.9 | N/A |
+
+Note: OpenAI has only reported on the average of their Miracl's Scores; Cohere's embed-multilingual-v3.0 model is SOTA in Multilingual Retrieval

public_embedding_benchmarks/mteb.md

@@ -0,0 +1,8 @@
+| Model |Dimensions| Classification (12 datasets) | Clustering (11 datasets) | Pair Classification (3 datasets) | Reranking (4 datasets) | STS (10 datasets) | Summariziation (1 dataset) | Retrieval (15 datasets)| Avg (56 datasets) |
+| -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- |
+| embed-english-v3.0  | 1024   | 76.5   | 47.4  | 85.8  | 58.0   | 82.6   | 30.2 | 55.0   | 64.5   |
+| embed-multilingual-v3.0   | 1024   | 76.0   | 46.6   | 86.1   | 57.9   | 83.2   | 31.0   | 53.8   | 64.0   |
+| OpenAI ada-002   | 1536   | 70.9   | 45.9   | 84.9   | 56.3   | 81.0   | 30.8   | 49.3   | 61.0   |
+|OpenAI text-embedding-large   | 3072   | 75.5   | 49.0   | 85.7   | 59.2   | 81.7   | 29.9   | 59.2   | 64.6   |
+
+Note: Cohere's Embedding models achieves comparable performance to text-embedding-large at 1/3 of the dimension size