add docs skeleton

Author: saichandrapandraju

docs/index.md

@@ -0,0 +1,84 @@
+# vLLM Judge
+
+A lightweight library for LLM-as-a-Judge evaluations using vLLM hosted models.
+
+## Features
+
+- 🚀 **Simple Interface**: Single `evaluate()` method that adapts to any use case
+- 🎯 **Pre-built Metrics**: 20+ ready-to-use evaluation metrics
+- 🔧 **Template Support**: Dynamic evaluations with template variables
+- ⚡ **High Performance**: Optimized for vLLM with automatic batching
+- 🌐 **API Mode**: Run as a REST API service
+- 🔄 **Async Native**: Built for high-throughput evaluations
+
+## Installation
+
+```bash
+# Basic installation
+pip install vllm-judge
+
+# With API support
+pip install vllm-judge[api]
+
+# With Jinja2 template support
+pip install vllm-judge[jinja2]
+
+# Everything
+pip install vllm-judge[dev]
+```
+
+## Quick Start
+
+```python
+from vllm_judge import Judge
+
+# Initialize with vLLM url
+judge = await Judge.from_url("http://localhost:8000")
+
+# Simple evaluation
+result = await judge.evaluate(
+    response="The Earth orbits around the Sun.",
+    criteria="scientific accuracy"
+)
+print(f"Decision: {result.decision}")
+print(f"Reasoning: {result.reasoning}")
+
+# Using pre-built metrics
+from vllm_judge import CODE_QUALITY
+
+result = await judge.evaluate(
+    response="def add(a, b): return a + b",
+    metric=CODE_QUALITY
+)
+
+# With template variables
+result = await judge.evaluate(
+    response="Essay content here...",
+    criteria="Evaluate this {doc_type} for {audience}",
+    template_vars={
+        "doc_type": "essay",
+        "audience": "high school students"
+    }
+)
+```
+
+## API Server
+
+Run Judge as a REST API:
+
+```bash
+vllm-judge serve --base-url http://localhost:8000 --port 9090 --host localhost
+```
+
+Then use the HTTP API:
+
+```python
+from vllm_judge.api import JudgeClient
+
+client = JudgeClient("http://localhost:9090")
+result = await client.evaluate(
+    response="Python is great!",
+    criteria="technical accuracy"
+)
+```
+

mkdocs.yml

@@ -0,0 +1,90 @@
+site_name: vLLM Judge
+site_description: LLM-as-a-Judge evaluations for vLLM hosted models
+site_author: Sai Chandra Pandraju
+site_url: https://saichandrapandraju.github.io/vllm-judge
+repo_url: https://github.com/saichandrapandraju/vllm-judge
+
+
+theme:
+  name: material
+  palette:
+    - scheme: default
+      primary: teal
+      accent: teal
+      toggle:
+        icon: material/brightness-7
+        name: Switch to dark mode
+    - scheme: slate
+      primary: teal
+      accent: teal
+      toggle:
+        icon: material/brightness-4
+        name: Switch to light mode
+  features:
+    - navigation.tabs
+    - navigation.sections
+    - navigation.expand
+    - navigation.top
+    - search.suggest
+    - search.highlight
+    - content.tabs.link
+    - content.code.annotation
+    - content.code.copy
+  language: en
+  icon:
+    repo: fontawesome/brands/github
+
+plugins:
+  - search
+  - mkdocstrings:
+      handlers:
+        python:
+          options:
+            show_source: false
+
+markdown_extensions:
+  - pymdownx.highlight:
+      anchor_linenums: true
+  - pymdownx.inlinehilite
+  - pymdownx.snippets
+  - admonition
+  - pymdownx.arithmatex:
+      generic: true
+  - footnotes
+  - pymdownx.details
+  - pymdownx.superfences
+  - pymdownx.mark
+  - attr_list
+  - pymdownx.emoji:
+      emoji_index: !!python/name:material.extensions.emoji.twemoji
+      emoji_generator: !!python/name:material.extensions.emoji.to_svg
+  - pymdownx.tabbed:
+      alternate_style: true
+
+nav:
+  - Home: index.md
+  # - Getting Started:
+  #   - Installation: getting-started/installation.md
+  #   - Quick Start: getting-started/quickstart.md
+  #   - Core Concepts: getting-started/concepts.md
+  # - User Guide:
+  #   - Basic Evaluation: guide/basic-evaluation.md
+  #   - Using Metrics: guide/metrics.md
+  #   - Template Variables: guide/templates.md
+  #   - Batch Processing: guide/batch.md
+  #   - Advanced Usage: guide/advanced.md
+  # - API Server:
+  #   - Running the Server: api/server.md
+  #   - API Reference: api/reference.md
+  #   - Client Usage: api/client.md
+  #   - WebSocket API: api/websocket.md
+  # - Examples:
+  #   - Code Review: examples/code-review.md
+  #   - Content Moderation: examples/content-moderation.md
+  #   - Educational Assessment: examples/education.md
+  #   - A/B Testing: examples/ab-testing.md
+  # - Reference:
+  #   - Built-in Metrics: reference/metrics.md
+  #   - API Reference: reference/api.md
+  #   - Configuration: reference/config.md
+  #   - Exceptions: reference/exceptions.md

pyproject.toml

@@ -59,6 +59,7 @@ docs = [
     "mkdocs>=1.5.0",
     "mkdocs-material>=9.0.0",
     "mkdocstrings[python]>=0.24.0",
+    "mkdocs-material-extensions>=1.3.1"
 ]
 
 [project.scripts]