Skip to content

Commit 8a6c821

Browse files
author
Mir Sameer
committed
Initial RAG evaluation workbench
0 parents  commit 8a6c821

32 files changed

Lines changed: 1748 additions & 0 deletions

‎.env.example‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
# The demo runs without secrets, network access, or paid API keys.
2+
RAG_EVAL_DOCS=examples/docs
3+
RAG_EVAL_QUESTIONS=examples/questions.json
4+
RAG_EVAL_OUTPUT=runs/demo
5+
RAG_EVAL_TOP_K=3
6+
RAG_EVAL_ENABLE_REMOTE=false

‎.github/workflows/ci.yml‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
name: CI
2+
3+
on:
4+
push:
5+
pull_request:
6+
7+
jobs:
8+
test:
9+
runs-on: ubuntu-latest
10+
strategy:
11+
matrix:
12+
python-version: ["3.10", "3.11", "3.12"]
13+
steps:
14+
- uses: actions/checkout@v4
15+
- uses: actions/setup-python@v5
16+
with:
17+
python-version: ${{ matrix.python-version }}
18+
- name: Run tests
19+
run: python -m unittest discover -s tests
20+
- name: Run offline demo
21+
run: python -m rag_eval_workbench demo --output runs/ci-demo

‎.gitignore‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
__pycache__/
2+
*.py[cod]
3+
*.egg-info/
4+
.pytest_cache/
5+
.mypy_cache/
6+
.ruff_cache/
7+
.coverage
8+
htmlcov/
9+
.venv/
10+
venv/
11+
env/
12+
.env
13+
runs/
14+
dist/
15+
build/
16+
.DS_Store

‎LICENSE‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
MIT License
2+
3+
Copyright (c) 2026 Sameeruddin Mir
4+
5+
Permission is hereby granted, free of charge, to any person obtaining a copy
6+
of this software and associated documentation files (the "Software"), to deal
7+
in the Software without restriction, including without limitation the rights
8+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9+
copies of the Software, and to permit persons to whom the Software is
10+
furnished to do so, subject to the following conditions:
11+
12+
The above copyright notice and this permission notice shall be included in all
13+
copies or substantial portions of the Software.
14+
15+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21+
SOFTWARE.

‎README.md‎

Lines changed: 165 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,165 @@
1+
# RAG Eval Workbench
2+
3+
Offline, standard-library RAG evaluation workbench for testing retrieval quality, answer grounding, latency, and hallucination risk without paid API keys.
4+
5+
![Dashboard preview](docs/artifacts/dashboard-preview.svg)
6+
7+
## Why This Project Matters
8+
9+
This repository is built as a portfolio-grade AI/ML/LLMOps project. It shows the practical engineering around RAG systems that hiring teams care about:
10+
11+
- deterministic document loading and chunking;
12+
- local vector-style retrieval with hashed TF-IDF and cosine similarity;
13+
- citation-first answer synthesis;
14+
- retrieval, grounding, hallucination, latency, and cost metrics;
15+
- trace files and reproducible run artifacts;
16+
- CI that runs tests and the demo without network services.
17+
18+
The core demo uses only Python 3 standard library modules. Optional integrations are documented for FastAPI, Streamlit, LangChain, and MLflow, but they are not required to run the project.
19+
20+
## Quickstart
21+
22+
```bash
23+
cd rag-eval-workbench
24+
python -m rag_eval_workbench demo --output runs/demo
25+
python -m unittest discover -s tests
26+
```
27+
28+
If your system only exposes Python as `python3`, use `python3 -m ...` for the same commands.
29+
30+
Open the generated dashboard:
31+
32+
```bash
33+
open runs/demo/dashboard.html
34+
```
35+
36+
The command works from a fresh checkout because the repo uses a `src/` package layout plus a thin top-level checkout shim for local execution. Installing the package is optional.
37+
38+
## Demo Output
39+
40+
The demo writes a portable run folder:
41+
42+
```text
43+
runs/demo/
44+
dashboard.html
45+
metrics.csv
46+
metrics.json
47+
results.json
48+
run_manifest.json
49+
summary.md
50+
traces.jsonl
51+
```
52+
53+
Example terminal output:
54+
55+
```text
56+
Run complete: runs/demo
57+
Dashboard: runs/demo/dashboard.html
58+
Metrics: retrieval@k=1.000 groundedness=0.900 risk=0.100 latency_ms=1.200 cost=$0.000000
59+
```
60+
61+
Actual latency varies by machine, but the corpus, benchmark questions, retrieval algorithm, and evaluation logic are deterministic.
62+
63+
## Metrics
64+
65+
| Metric | What It Answers | Why It Matters |
66+
| --- | --- | --- |
67+
| `retrieval@k` | Did the expected source document appear in the retrieved top-k contexts? | Catches retrieval and chunking regressions. |
68+
| `groundedness` | Are answer terms supported by retrieved context terms? | Flags unsupported generated claims. |
69+
| `answer_relevance` | Does the answer overlap with the reference answer? | Keeps responses aligned to expected facts. |
70+
| `hallucination_risk` | How risky is the answer based on grounding and retrieval quality? | Gives release owners a triage signal. |
71+
| `latency_ms` | How long did the local pipeline take per question? | Makes evaluation performance visible. |
72+
| `estimated_cost_usd` | What is the configured per-run token cost estimate? | Defaults to zero for the offline standard-library demo. |
73+
74+
## Architecture
75+
76+
```text
77+
examples/docs
78+
-> load markdown/text documents
79+
-> create overlapping word chunks
80+
-> build local hashed TF-IDF vectors
81+
-> retrieve top-k chunks by cosine similarity
82+
-> synthesize extractive cited answers
83+
-> compute heuristic metrics
84+
-> write traces, metrics, and static dashboard
85+
```
86+
87+
Key modules live under `src/rag_eval_workbench/`:
88+
89+
| Module | Role |
90+
| --- | --- |
91+
| `documents.py` | Document loading and deterministic chunking. |
92+
| `retrieval.py` | Local hash-vector retriever. |
93+
| `synthesis.py` | Extractive answer synthesis with citations. |
94+
| `evaluator.py` | Heuristic RAG metrics. |
95+
| `benchmark.py` | End-to-end benchmark orchestration. |
96+
| `tracing.py` | JSON, JSONL, CSV, and Markdown artifact writing. |
97+
| `dashboard.py` | Static HTML dashboard generation. |
98+
| `cli.py` | `python -m rag_eval_workbench` entrypoint. |
99+
100+
See [docs/architecture.md](docs/architecture.md) and [docs/eval-methodology.md](docs/eval-methodology.md) for more detail.
101+
102+
## CLI Usage
103+
104+
Run the bundled demo:
105+
106+
```bash
107+
python -m rag_eval_workbench demo --output runs/demo
108+
```
109+
110+
Run against your own local corpus:
111+
112+
```bash
113+
python -m rag_eval_workbench benchmark \
114+
--docs examples/docs \
115+
--questions examples/questions.json \
116+
--output runs/custom \
117+
--top-k 3
118+
```
119+
120+
Optional cost modeling:
121+
122+
```bash
123+
python -m rag_eval_workbench demo \
124+
--output runs/demo-costed \
125+
--cost-per-1k-tokens 0.002
126+
```
127+
128+
## Repository Layout
129+
130+
```text
131+
.
132+
.github/workflows/ci.yml
133+
docs/
134+
examples/
135+
src/rag_eval_workbench/
136+
tests/
137+
.env.example
138+
pyproject.toml
139+
```
140+
141+
## Resume Relevance
142+
143+
This project maps directly to AI engineer and LLMOps responsibilities:
144+
145+
- eval harness design for RAG quality gates;
146+
- deterministic local baseline before paid model integrations;
147+
- trace-first debugging for retrieval failures;
148+
- CI-friendly tests and demo artifacts;
149+
- static dashboard generation for stakeholder review;
150+
- clean extension path for model providers, vector stores, and experiment tracking.
151+
152+
## Optional Integrations
153+
154+
The standard demo intentionally avoids third-party packages. Suggested production wrappers are documented in [docs/optional-integrations.md](docs/optional-integrations.md):
155+
156+
- FastAPI for benchmark service endpoints;
157+
- Streamlit for analyst exploration;
158+
- LangChain for provider-backed retrievers;
159+
- MLflow for experiment tracking and artifact logging.
160+
161+
## Limitations
162+
163+
- The synthesizer is extractive and deterministic; it is not a generative LLM.
164+
- The evaluator is heuristic and transparent; it is not a replacement for human review or an LLM judge.
165+
- Hash-vector retrieval is a local baseline; production systems can swap in embeddings while keeping the same benchmark shape.

‎docs/architecture.md‎

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
# Architecture
2+
3+
`rag-eval-workbench` is intentionally small enough to run anywhere, but it mirrors production RAG evaluation systems:
4+
5+
```text
6+
examples/docs
7+
-> document loader
8+
-> word-window chunker
9+
-> hashed TF-IDF retriever
10+
-> extractive answer synthesizer
11+
-> heuristic evaluator
12+
-> trace and metrics writers
13+
-> static dashboard generator
14+
```
15+
16+
## Components
17+
18+
| Module | Responsibility |
19+
| --- | --- |
20+
| `documents.py` | Loads local markdown/text files and creates deterministic overlapping chunks. |
21+
| `retrieval.py` | Builds sparse hash vectors with TF-IDF weights and ranks chunks by cosine similarity. |
22+
| `synthesis.py` | Produces citation-first extractive answers from retrieved contexts. |
23+
| `evaluator.py` | Computes retrieval@k, groundedness, answer relevance, hallucination risk, latency, and estimated cost. |
24+
| `benchmark.py` | Runs end-to-end cases and aggregates metrics. |
25+
| `tracing.py` | Writes `metrics.json`, `results.json`, `traces.jsonl`, `metrics.csv`, and `summary.md`. |
26+
| `dashboard.py` | Generates a portable static `dashboard.html` report. |
27+
28+
## Why Hash Vectors
29+
30+
Production RAG systems often use embedding providers and vector databases. This project keeps the runnable core offline by using hashed TF-IDF vectors:
31+
32+
- no embedding model download;
33+
- no API key;
34+
- deterministic scoring;
35+
- enough retrieval behavior to evaluate chunking, top-k recall, answer grounding, and dashboard artifacts.
36+
37+
The interface is intentionally narrow so a real embedding backend can replace `HashVectorRetriever` later.
38+
39+
## Artifact Flow
40+
41+
Every run creates a directory like `runs/demo/`:
42+
43+
```text
44+
runs/demo/
45+
dashboard.html
46+
metrics.csv
47+
metrics.json
48+
results.json
49+
run_manifest.json
50+
summary.md
51+
traces.jsonl
52+
```
53+
54+
The manifest records that the run uses the Python standard library, requires no network, and requires no paid API keys.

‎docs/artifacts/dashboard-preview.svg‎

Lines changed: 58 additions & 0 deletions
Loading

‎docs/eval-methodology.md‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
# Evaluation Methodology
2+
3+
The demo uses a deterministic question set with expected source document ids. It is designed to demonstrate the mechanics of RAG evaluation rather than claim model-quality accuracy.
4+
5+
## Metrics
6+
7+
| Metric | Meaning | Implementation |
8+
| --- | --- | --- |
9+
| `retrieval@k` | Did the expected source appear in the top-k retrieved chunks? | Binary per case, averaged across the run. |
10+
| `groundedness` | How much of the answer is supported by retrieved context? | Ratio of answer terms that appear in retrieved context terms. |
11+
| `answer_relevance` | Does the answer overlap with the reference answer? | Ratio of expected answer terms found in the generated answer. |
12+
| `hallucination_risk` | How likely is the answer to include unsupported content? | Increases when groundedness is low, expected source is absent, or answer relevance is weak. |
13+
| `latency_ms` | End-to-end local processing time per case. | Measured with `time.perf_counter`. |
14+
| `estimated_cost_usd` | Cost model for the benchmark shape. | Defaults to zero because the standard-library demo is local. |
15+
16+
## Why Heuristic Judging
17+
18+
Many production eval stacks use an LLM judge. This repository deliberately avoids that dependency so the demo can run offline. The heuristic judge is transparent, deterministic, easy to test, and useful for catching broad regressions:
19+
20+
- expected source missing from retrieval;
21+
- answer not grounded in retrieved text;
22+
- required terms absent from the synthesized response;
23+
- latency or cost changes.
24+
25+
## Extending the Evaluator
26+
27+
Credible next steps for a production team:
28+
29+
- add exact-match or semantic answer checks for domain-specific facts;
30+
- compare hash retrieval against embedding retrieval;
31+
- track metric deltas across branches in CI;
32+
- export results to MLflow or a warehouse;
33+
- add human review labels for high-risk benchmark cases.

0 commit comments

Comments
 (0)