How to run eval in Prod when AI tool is being used by users

[tgogoi]

I am wondering if you have an example showing how to use Microsoft.Extensions.AI.Evaluation in Prod (i.e. real time when agent is working or used by customer ). I am able to use the library successfully as unit tests and that is very helpful.

[shyamnamboodiripad]

Thanks for the feedback @tgogoi

While we don't have direct samples for this yet, it should be relatively straightforward to use Microsoft.Extensions.AI.Evaluation to orchestrate evals in production.

In production scenarios, the functionality in present in the Microsoft.Extensions.AI.Evaluation.Reporting package / layer may not be as relevant.

• For example, the response caching functionality in this package won't help much unless you are processing identical LLM requests repeatedly (in which case the caching could help provide speedup and cost savings).
• It may also be cost prohibitive to evaluate each and every unique LLM interaction involved in production. So, you may want to implement some kind of sampling logic to pick a random subset of interactions over time and evaluate just those. You could potentially implement your own wrapping IEvaluator (e.g., by wrapping CompositeEvaluator) that orchestrates the sampling logic only evaluate a subset of the requests it receives.
• Similarly, the report generation functionality that is used in the unit test scenario may not be as helpful in production unless you leverage external storage for capturing evaluation results (e.g., unless you use AzureStorageResultStore to store the results in an Azure Storage container or implement your own custom IResultStore to log the evaluation results to your preferred telemetry provider such as Azure Application Insights).

Based on the above, I would recommend starting with the samples included as part of the Evaluation API Examples which demonstrate how to orchestrate evals without leveraging the functionality in the reporting layer.

We have the following issue on our backlog to add some samples for this scenario - see dotnet/extensions#6343 which also contains some related ideas / discussion.