Merge pull request #275664 from s-polly/stp-ai-studio-follow

Update to model benchmarks for Build

Author: Stacyrch140

articles/ai-studio/how-to/model-benchmarks.md

@@ -20,22 +20,49 @@ In Azure AI Studio, you can compare benchmarks across models and datasets availa
 
 :::image type="content" source="../media/explore/model-benchmarks-dashboard-view.png" alt-text="Screenshot of dashboard view graph of model accuracy." lightbox="../media/explore/model-benchmarks-dashboard-view.png":::
 
-Model benchmarks help you make informed decisions about the sustainability of models and datasets prior to initiating any job. The benchmarks are a curated list of the best performing models for a given task, based on a comprehensive comparison of benchmarking metrics. Currently, Azure AI Studio provides benchmarks based on quality, via the metrics listed below.
+Model benchmarks help you make informed decisions about the sustainability of models and datasets before initiating any job. The benchmarks are a curated list of the best performing models for a given task, based on a comprehensive comparison of benchmarking metrics. Currently, Azure AI Studio provides benchmarking across the following types of models based on our model catalog collections: 
+
+- Benchmarks across LLMs and SLMs  
+- Benchmarks across embeddings models 
+
+You can switch between the **Quality benchmarks** and **Embeddings benchmarks** views by clicking on the corresponding tabs within the model benchmarks experience in AI Studio.
+
+
+### Benchmarking of LLMs and SLMs 
+
+Model benchmarks assess the quality of LLMs and SLMs across various metrics listed below:  
 
 | Metric       | Description |
 |--------------|-------|
 | Accuracy     |Accuracy scores are available at the dataset and the model levels. At the dataset level, the score is the average value of an accuracy metric computed over all examples in the dataset. The accuracy metric used is exact-match in all cases except for the *HumanEval* dataset that uses a `pass@1` metric. Exact match simply compares model generated text with the correct answer according to the dataset, reporting one if the generated text matches the answer exactly and zero otherwise. `Pass@1` measures the proportion of model solutions that pass a set of unit tests in a code generation task. At the model level, the accuracy score is the average of the dataset-level accuracies for each model.|
 | Coherence    |Coherence evaluates how well the language model can produce output that flows smoothly, reads naturally, and resembles human-like language.|
 | Fluency      |Fluency evaluates the language proficiency of a generative AI's predicted answer. It assesses how well the generated text adheres to grammatical rules, syntactic structures, and appropriate usage of vocabulary, resulting in linguistically correct and natural-sounding responses.|
-| GPTSimilarity|GPTSimilarity is a measure that quantifies the similarity between a ground truth sentence (or document) and the prediction sentence generated by an AI model. It is calculated by first computing sentence-level embeddings using the embeddings API for both the ground truth and the model's prediction. These embeddings represent high-dimensional vector representations of the sentences, capturing their semantic meaning and context.|
+| GPTSimilarity|GPTSimilarity is a measure that quantifies the similarity between a ground truth sentence (or document) and the prediction sentence generated by an AI model. It's calculated by first computing sentence-level embeddings using the embeddings API for both the ground truth and the model's prediction. These embeddings represent high-dimensional vector representations of the sentences, capturing their semantic meaning and context.|
+|Groundedness| Groundedness measures how well the language model's generated answers align with information from the input source. |
+|Relevance| Relevance measures the extent to which the language model's generated responses are pertinent and directly related to the given questions. |
 
 The benchmarks are updated regularly as new metrics and datasets are added to existing models, and as new models are added to the model catalog.
 
+### Benchmarking of embedding models
+
+Model benchmarks assess embeddings models across various metrics listed in the table:  
+
+
+|Metric  |Description  |
+|---------|---------|
+|Accuracy      | Accuracy is the proportion of correct predictions among the total number of predictions processed.         |
+|F1 Score      | F1 Score is the weighted mean of the precision and recall, where the best value is 1 (perfect precision and recall), and worst is 0.         |
+|Mean Average Precision (MAP)      |Mean Average Precision (MAP) evaluates the quality of ranking and recommender systems. It measures both the relevance of suggested items and how good the system is at placing more relevant items at the top. Values can range from 0 to 1. The higher the MAP, the better the system can place relevant items high in the list.          |
+|Normalized Discounted Cumulative Gain (NDCG)      |Normalized Discounted Cumulative Gain evaluates a machine learning algorithm's ability to sort items based on relevance. It compares rankings to an ideal order where all relevant items are at the top of the list, where k is the list length while evaluating ranking quality. In our benchmarks, k=10, indicated by a metrics of ndcg_at_10, meaning that we look at the top 10 items.           |
+|Precision      |Precision measures the model's ability to identify instances of a particular class correctly. Precision shows how often an ML model is correct when predicting the target class.           |
+|Spearman Correlation      | Spearman Correlation is based on cosine similarity. It is calculated by first computing the cosine similarity between variables, then ranking these scores and using the ranks to compute the Spearman Correlation.         |
+|V-measure      | V-measure is a metric used to evaluate the quality of clustering. It's calculated as a harmonic mean of homogeneity and completeness, ensuring a balance between the two for a meaningful score. Possible scores lie between 0 and 1, with 1 being perfectly complete labeling.    |
+
 ### How the scores are calculated
 
 The benchmark results originate from public datasets that are commonly used for language model evaluation. In most cases, the data is hosted in GitHub repositories maintained by the creators or curators of the data. Azure AI evaluation pipelines download data from their original sources, extract prompts from each example row, generate model responses, and then compute relevant accuracy metrics.
 
-Prompt construction follows best practice for each dataset, set forth by the paper introducing the dataset and industry standard. In most cases, each prompt contains several examples of complete questions and answers, or "shots," to prime the model for the task. The evaluation pipelines create shots by sampling questions and answers from a portion of the data that is held out from evaluation.
+Prompt construction follows best practices for each dataset, defined by the paper introducing the dataset and industry standard. In most cases, each prompt contains several examples of complete questions and answers, or "shots," to prime the model for the task. The evaluation pipelines create shots by sampling questions and answers from a portion of the data that is held out from evaluation.
 
 ### View options in the model benchmarks