Create CLAUDE.md with LLM-as-a-judge evaluation prompt

Author: ftosoni

scripts/evaluation/CLAUDE.md

@@ -0,0 +1,68 @@
+# LLM-as-a-Judge Evaluation Prompt
+
+Use the prompt below to instruct Claude to adjudicate search results between Code2Code and BM25.
+
+---
+
+```markdown
+You are an expert software engineer and information retrieval scientist acting as an **LLM-as-a-judge**. 
+
+Your task is to adjudicate the search results of two code search engines: **Code2Code Semantic Search** and **BM25 Keyword Search**, and output a single JSON file called `judge_annotations.json` containing your graded relevance annotations.
+
+---
+
+### Input Data
+You will be provided with two JSON files containing the search results:
+1. `evaluation_results_code2codesearch_toolforge_org_search_7runs.json` (Code2Code results)
+2. `bm25_results_code2codesearch_toolforge_org_search_7runs.json` (BM25 results)
+
+Each file contains a list of benchmark queries (identified by IDs like `A1`, `A2`, ... `D4`). For each query, you are given:
+- The input code snippet (`code` of the query)
+- Up to 10 retrieved code snippets in the `results` list (ranked 1 to 10)
+
+---
+
+### Relevance Grading Scale
+For each retrieved code snippet (both in the BM25 and Code2Code results), assign a relevance score from the following scale:
+- **`1.0` (Relevant)**: Implements the exact same computational task or correct specific solution for the query intent. Differences in variable names or language do not affect relevance if the task is identical.
+- **`0.5` (Partially Relevant)**: Topically related; represents a partial realization of the query intent, a non-idiomatic implementation, or uses the target logic merely as a helper step in a larger function.
+- **`0.0` (Irrelevant)**: Does not solve the problem or is completely unrelated (e.g., matched on random variables/keywords like `mid` or `min` but does a completely different task).
+
+---
+
+### Metrics to Compute per Query
+For each query, calculate:
+1. **`bm25_p10`**: Fraction of the top-10 BM25 results with a score >= 0.5 (e.g., `0.7`).
+2. **`c2c_p10`**: Fraction of the top-10 Code2Code results with a score >= 0.5.
+3. **`bm25_p10_strict`**: Fraction of the top-10 BM25 results with a score equal to 1.0.
+4. **`c2c_p10_strict`**: Fraction of the top-10 Code2Code results with a score equal to 1.0.
+
+---
+
+### Output Format Requirement
+Your response must be **strictly a raw JSON block** matching this exact schema. Do not output any markdown text or conversational preamble outside of the JSON block.
+
+```json
+{
+  "metric": "P@10 = fraction of top-10 with graded relevance >= 0.5 (primary); strict = fraction == 1.0",
+  "judge": "Claude 3.5 Sonnet (LLM-as-a-judge, pooled BM25+Code2Code top-10)",
+  "per_query": {
+    "QUERY_ID": {
+      "bm25_scores": [
+        SCORE_RANK_1, SCORE_RANK_2, SCORE_RANK_3, SCORE_RANK_4, SCORE_RANK_5,
+        SCORE_RANK_6, SCORE_RANK_7, SCORE_RANK_8, SCORE_RANK_9, SCORE_RANK_10
+      ],
+      "c2c_scores": [
+        SCORE_RANK_1, SCORE_RANK_2, SCORE_RANK_3, SCORE_RANK_4, SCORE_RANK_5,
+        SCORE_RANK_6, SCORE_RANK_7, SCORE_RANK_8, SCORE_RANK_9, SCORE_RANK_10
+      ],
+      "bm25_p10": 0.0,
+      "c2c_p10": 0.0,
+      "bm25_p10_strict": 0.0,
+      "c2c_p10_strict": 0.0,
+      "comment": "Brief comment summarizing why the systems behaved the way they did. Mention specific ranks and tasks (e.g. 'BM25: r2 exact recursive gcd; r1 uses gcd as step. C2C: r1-7 real GCD impls')."
+    }
+  }
+}
+```
+```