Skip to content

Conversation

pamelafox
Copy link
Collaborator

Purpose

This PR updates the "baseline" folder to use embedding-3-large, since that's the new default for the repo. Along with that, this PR renames the old baseline to "gpt4omini-ada002" and renames other folders for consistency.

The other option is to not have a baseline folder, and just always use descriptive folder names. That wouldn't be compatible with the current evaluation workflow however, as it assumes there's always a baseline folder to diff against. I wish I could figure out how to symlink in git, but I haven't figured out a way, since then I would just symlink baseline to the most recent folder.

Does this introduce a breaking change?

When developers merge from main and run the server, azd up, or azd deploy, will this produce an error?
If you're not sure, try it out on an old environment.

[ ] Yes
[X] No

Does this require changes to learn.microsoft.com docs?

This repository is referenced by this tutorial
which includes deployment, settings and usage instructions. If text or screenshot need to change in the tutorial,
check the box below and notify the tutorial author. A Microsoft employee can do this for you if you're an external contributor.

[ ] Yes
[X] No

Type of change

[ ] Bugfix
[ ] Feature
[ ] Code style update (formatting, local variables)
[ ] Refactoring (no functional changes, no api changes)
[ ] Documentation content changes
[X] Other... Please describe:

Code quality checklist

See CONTRIBUTING.md for more details.

N/A

{
"testdata_path": "ground_truth.jsonl",
"results_dir": "results/gpt-4o-mini",
"results_dir": "results/experiment<TIMESTAMP>",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does this get replaced automatically?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh yes when I ran this eval, I didnt bother changing the results_dir, so thats why we see it changed in the config back to the default configuration. I renamed the folder after the fact instead. I could pretend that I had specified a folder name, but this config.json does actually reflect how I run it, so its true.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And yes, the evaluator replaces with current timestamp, so you dont accidentally override a past evaluation, if you forget to change the dir. I usually use that approach to be safe.

@pamelafox pamelafox merged commit 986009c into Azure-Samples:main May 12, 2025
25 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants