2025-06-02: Upgrade azure-ai-evaluation SDK dependency

What's Changed

• Update azure-ai-evaluation to 1.7.0 by Matt (@mattgotteiner) in #135
• Update python.yaml with ubuntu-latest instead of deprecated runner by Pamela Fox (@pamelafox) in #136
• Bump azure-ai-evaluation from 1.7.0 to 1.8.0 by Dependabot (@dependabot) in #138

New Contributors

• Matt (@mattgotteiner) made their first contribution in #135

Full Changelog: 2025-05-01...2025-06-02

2025-05-01: Upgrade azure-ai-evaluation dependency

What's Changed

• Bump azure-ai-evaluation from 1.2.0 to 1.5.0 by Dependabot (@dependabot) in #131
• Add missing encoding by Pamela Fox (@pamelafox) in #133

Full Changelog: 2025-02-06b...2025-05-01

2025-02-06b: Partial deprecation of generation commands

What's Changed

• Deprecate azure-ai-generative partially by Pamela Fox (@pamelafox) in #127

Full Changelog: 2025-02-07...2025-02-06b

Upgrade azure-ai-evaluation SDK

What's Changed

• Bump azure-ai-evaluation from 1.1.0 to 1.2.0 by Dependabot (@dependabot) in #125
• Bump Azure/setup-azd from 2.0.0 to 2.1.0 by Dependabot (@dependabot) in #124

Full Changelog: 2025-02-06...2025-02-07

2025-02-06: Installable package with more flexible configuration

What's Changed

• Make index citation field name configurable by Jake Wang (@jakeatmsft) in #96
• Updated example results by Pamela Fox (@pamelafox) in #101
• Bump promptflow-evals from 0.3.0 to 0.3.2 by Dependabot (@dependabot) in #98
• Prompt updates by Pamela Fox (@pamelafox) in #104
• Updates to example data by Pamela Fox (@pamelafox) in #105
• Update to azure-ai-evaluations by Pamela Fox (@pamelafox) in #106
• More generic, installable by Pamela Fox (@pamelafox) in #102
• Bump azure-ai-evaluation from 1.0.0b3 to 1.0.1 by Dependabot (@dependabot) in #112
• Don't misuse the model configuration to store extra keys by Pamela Fox (@pamelafox) in #115
• Bump Azure/setup-azd from 1.0.0 to 2.0.0 by Dependabot (@dependabot) in #117
• Bump azure-ai-evaluation from 1.0.1 to 1.1.0 by Dependabot (@dependabot) in #116
• Version upgrades by Pamela Fox (@pamelafox) in #119

New Contributors

• Jake Wang (@jakeatmsft) made their first contribution in #96

Full Changelog: 2024-06-05...2025-02-06

2024-06-05: Update to new AI Chat Protocol, increase flexibility

This release both updates the evaluator tool to assume that the chat backend conforms to the new Microsoft AI Chat Protocol but also adds two new properties to the config JSON to allow for use with backends using the older protocol or other protocols.

Just add these fields and customize the JMESPath expressions as needed:

    "target_response_answer_jmespath": "choices[0].message.content",
    "target_response_context_jmespath": "choices[0].context.data_points.text"

What's Changed

• Bump promptflow-evals from 0.2.0.dev0 to 0.3.0 by Dependabot (@dependabot) in #88
• Avoid rate-limiting, improve --changed by Pamela Fox (@pamelafox) in #91
• Update to new Protocol response format, use JMESPath expressions by Pamela Fox (@pamelafox) in #92

Full Changelog: 2024-05-13...2024-06-05

2024-05-13: Updated underlying SDK, new metrics

This release ports this tool to use the promptflow-evals SDK for the evaluation functionality, as the evaluate functionality is being deprecated in azure-ai-generative. The Q&A generation is still in azure-ai-generative for now.

Some user-facing changes:

• I renamed the custom metrics to "mygroundedness", "myrelevance", "mycoherence" to make it clear they're not the built-in metrics. If you previously generated custom metrics, you'll want to rename keys in evalresults.json to the keys above and rename requested metrics in your config.json file.
• I added more built-in metrics from the promptflow-evals SDK: fluency, similarity, f1score.

What's Changed

• Add citation_match metric, changed argument, answer reviewer by Pamela Fox (@pamelafox) in #57
• Fixing typo in readme by Matt Soucoup (@codemillmatt) in #59
• Improving OpenAI support by Pamela Fox (@pamelafox) in #62
• Pin promptflow by Pamela Fox (@pamelafox) in #65
• Bump azure-ai-generative[evaluate] from 1.0.0b7 to 1.0.0b8 by Dependabot (@dependabot) in #64
• Better debugging for evaluate command by Pamela Fox (@pamelafox) in #69
• Adding azd GitHub actions workflow by Pamela Fox (@pamelafox) in #70
• Fixes for running on GH by Pamela Fox (@pamelafox) in #71
• Add optional target URL to evaluate command and CI to test evaluate by Pamela Fox (@pamelafox) in #72
• Add more OSes to matrix by Pamela Fox (@pamelafox) in #73
• Fix target URL name by Pamela Fox (@pamelafox) in #74
• Add keyring setup to CI by Pamela Fox (@pamelafox) in #75
• Remove unused infrastructure, disable keyed access for OpenAI service by Pamela Fox (@pamelafox) in #83
• Port to new promptflow-evals SDK by Pamela Fox (@pamelafox) in #85

New Contributors

• Matt Soucoup (@codemillmatt) made their first contribution in #59

Full Changelog: 2024-03-05...2024-05-13

2024-03-05: Diff tool for single run

This PR makes the diff tool more flexible, so that you can now specify a single directory and it will diff against the ground truth answer in that case:

For example:

python -m review_tools diff example_results/baseline/

What's Changed

• Handle single directory for diff tool by Pamela Fox (@pamelafox) in #55

Full Changelog: 2024-03-04...2024-03-05

2024-03-04: Evaluate "I don't know" situations

The tools now support evaluating your app's ability to say "I don't know". See README:
https://github.com/Azure-Samples/ai-rag-chat-evaluator?tab=readme-ov-file#measuring-apps-ability-to-say-i-dont-know

There's also a new metric, citationmatch, to check whether an answer's citations contains the original citation from the ground truth answer.

What's Changed

• Handle failures with numeric ratings by Pamela Fox (@pamelafox) in #53
• Added citationmatch metric and tools for evaluating "I don't know" answers by Pamela Fox (@pamelafox) in #54

Full Changelog: 2024-02-15...2024-03-04

2024-02-15: Upgrade azure-ai-generative SDK, custom Prompt metrics

This week's release upgraded the azure-ai-generative SDK version, which introduced a regression that's now fixed.

The evaluator tool also now has the ability to run custom prompt metrics, which is particularly helpful if you need to localize the built-in prompts. See documentation here:
https://github.com/Azure-Samples/ai-rag-chat-evaluator/tree/main?tab=readme-ov-file#custom-metrics

What's Changed

• Bump actions/setup-python from 4 to 5 by Dependabot (@dependabot) in #1
• Fix env var to match service setup by Pamela Fox (@pamelafox) in #8
• Fix keys by Pamela Fox (@pamelafox) in #14
• Support for making a directory by Pamela Fox (@pamelafox) in #15
• Check for git directory before pre-commit by Pamela Fox (@pamelafox) in #16
• Remove pre-commit from devcontainer.json by Pamela Fox (@pamelafox) in #21
• Readme improvements by Pamela Fox (@pamelafox) in #20
• Update requirements by Pamela Fox (@pamelafox) in #22
• Pin AzureML metrics by Pamela Fox (@pamelafox) in #25
• Address feedback from Chris by Pamela Fox (@pamelafox) in #28
• Rename parameters json by Pamela Fox (@pamelafox) in #30
• Better error messages, tests, and encoding by Pamela Fox (@pamelafox) in #34
• Add english-only notes to readme by Pamela Fox (@pamelafox) in #36
• Make numquestions an optional argument by Pamela Fox (@pamelafox) in #39
• Fix numquestions correctly by Pamela Fox (@pamelafox) in #40
• Add test call to GPT deployment by Pamela Fox (@pamelafox) in #41
• Upgrade sdk, add latency by Pamela Fox (@pamelafox) in #45
• Make windows-friendly adjustments to README by Pamela Fox (@pamelafox) in #46
• Support/prioritize local prompt metrics by Pamela Fox (@pamelafox) in #50
• Fix Azure OpenAI bug with API key by Sofyan Ajridi (@sofyanajridi) in #51
• Fix data mapping to match new evaluate SDK expectations by Pamela Fox (@pamelafox) in #52

New Contributors

• Dependabot (@dependabot) made their first contribution in #1
• Pamela Fox (@pamelafox) made their first contribution in #8
• Sofyan Ajridi (@sofyanajridi) made their first contribution in #51

Full Changelog: https://github.com/Azure-Samples/ai-rag-chat-evaluator/commits/2024-02-15