Merge branch 'main' into multi_chat_context

Author: avinash2692

CHANGELOG.md

@@ -1,3 +1,22 @@
+## [v0.2.1](https://github.com/generative-computing/mellea/releases/tag/v0.2.1) - 2025-12-10
+
+### Feature
+
+* Test-based Evaluation with LLM-as-a-judge ([#225](https://github.com/generative-computing/mellea/issues/225)) ([`0f1f0f8`](https://github.com/generative-computing/mellea/commit/0f1f0f8eb12e60f7940e3ad5b40ce91ded73fc88))
+* Add a `code_interpreter` tool ([#232](https://github.com/generative-computing/mellea/issues/232)) ([`b03c964`](https://github.com/generative-computing/mellea/commit/b03c96439501146965cd123ce5046f2c5907acfb))
+
+### Fix
+
+* Add simple lock to hf generation to prevent using incorrect weights ([#237](https://github.com/generative-computing/mellea/issues/237)) ([`6b2a527`](https://github.com/generative-computing/mellea/commit/6b2a5276a426be87ce02fa89f49818535f211fa6))
+* Collection of small fixes ([#238](https://github.com/generative-computing/mellea/issues/238)) ([`2120112`](https://github.com/generative-computing/mellea/commit/2120112ee807da4e980eabc0df54b3aae12d2cd2))
+* Fix unused litellm import ([#246](https://github.com/generative-computing/mellea/issues/246)) ([`633bfd7`](https://github.com/generative-computing/mellea/commit/633bfd7198eac45f8c163fefcc910d9bf8a76151))
+* Minor updates to answer relevance ([#245](https://github.com/generative-computing/mellea/issues/245)) ([`bde9b4d`](https://github.com/generative-computing/mellea/commit/bde9b4dd91ab92af0f7a661e321bb9d701da0589))
+* Pre-commit file selection ([#243](https://github.com/generative-computing/mellea/issues/243)) ([`e70d307`](https://github.com/generative-computing/mellea/commit/e70d3075f92152f8bb1a519687575fc7f30ffe1b))
+
+### Documentation
+
+* Fixed copyright in LICENSE ([#210](https://github.com/generative-computing/mellea/issues/210)) ([`3087051`](https://github.com/generative-computing/mellea/commit/3087051f5fc102bb8e0319af6baf9d7a0222e6ef))
+
 ## [v0.2.0](https://github.com/generative-computing/mellea/releases/tag/v0.2.0) - 2025-11-19
 
 ### Feature

cli/eval/__init__.py

@@ -0,0 +1 @@
+"""CLI for test-based evaluation"""

cli/eval/commands.py

@@ -0,0 +1,49 @@
+"""Use the eval command for LLM-as-a-judge evaluation, given a (set of) test file(s) consisting of prompts, instructions, and optionally, targets.
+Instantiate a generator model to produce candidate responses, and a judge model to determine whether the instructions have been followed."""
+
+import typer
+
+eval_app = typer.Typer(name="eval")
+
+
+def eval_run(
+    test_files: list[str] = typer.Argument(
+        ..., help="List of paths to json/jsonl files containing test cases"
+    ),
+    backend: str = typer.Option("ollama", "--backend", "-b", help="Generation backend"),
+    model: str = typer.Option(None, "--model", help="Generation model name"),
+    max_gen_tokens: int = typer.Option(
+        256, "--max-gen-tokens", help="Max tokens to generate for responses"
+    ),
+    judge_backend: str = typer.Option(
+        None, "--judge-backend", "-jb", help="Judge backend"
+    ),
+    judge_model: str = typer.Option(None, "--judge-model", help="Judge model name"),
+    max_judge_tokens: int = typer.Option(
+        256, "--max-judge-tokens", help="Max tokens for the judge model's judgement."
+    ),
+    output_path: str = typer.Option(
+        "eval_results", "--output-path", "-o", help="Output path for results"
+    ),
+    output_format: str = typer.Option(
+        "json", "--output-format", help="Either json or jsonl format for results"
+    ),
+    continue_on_error: bool = typer.Option(True, "--continue-on-error"),
+):
+    from cli.eval.runner import run_evaluations
+
+    run_evaluations(
+        test_files=test_files,
+        backend=backend,
+        model=model,
+        max_gen_tokens=max_gen_tokens,
+        judge_backend=judge_backend,
+        judge_model=judge_model,
+        max_judge_tokens=max_judge_tokens,
+        output_path=output_path,
+        output_format=output_format,
+        continue_on_error=continue_on_error,
+    )
+
+
+eval_app.command("run")(eval_run)