Initial release: LLM Safety Middleware v2.0.0

Production-grade async safety proxy for LLM APIs (Ollama, OpenAI-compatible,
and custom backends). Multi-layer pipeline: rate limiter, token filter, spaCy
rule engine, toxic-bert semantic classifier, post-generation check.

- FastAPI server with health/generate/check/statistics endpoints
- Async HTTP proxy via httpx with exponential-backoff retry
- 74-test suite (pytest-asyncio + respx for HTTP mocking)
- Docker + docker-compose, GitHub Actions CI (tests + docker build)
- Full documentation: README, QUICKSTART, DEPLOYMENT, PROJECT_OVERVIEW

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Author: 2 people

.env.example

@@ -0,0 +1,56 @@
+# LLM Safety Middleware — Environment Variables
+# ================================================
+# Copy this file to .env and fill in your values.
+# Never commit .env to git.
+#
+#   cp .env.example .env
+
+# ── LLM Backend ──────────────────────────────────────────────────────────────
+# Which backend protocol to use
+LLM_BACKEND_TYPE=ollama          # ollama | openai | custom
+
+# Remote LLM server URL
+LLM_BASE_URL=http://localhost:11434
+
+# Model name
+LLM_MODEL=llama2
+
+# Bearer token — required for OpenAI, Together AI, etc.
+# Leave blank for Ollama or unauthenticated backends.
+LLM_API_KEY=
+
+# Request timeout in seconds
+LLM_TIMEOUT_SECONDS=60
+
+# Retry attempts on transient server errors (5xx)
+LLM_MAX_RETRIES=3
+
+# Default generation parameters (can be overridden per-request)
+LLM_MAX_NEW_TOKENS=512
+LLM_TEMPERATURE=0.7
+LLM_TOP_P=0.95
+
+# System message prepended to every conversation (OpenAI mode only)
+LLM_SYSTEM_PROMPT=
+
+# ── Safety Middleware Server ──────────────────────────────────────────────────
+# Bind address
+HOST=0.0.0.0
+
+# Listen port
+PORT=8000
+
+# Number of uvicorn worker processes
+WORKERS=1
+
+# Path to the SafetyConfig JSON file
+CONFIG_PATH=config_production.json
+
+# Secret key for the POST /api/v1/statistics/reset admin endpoint.
+# Leave blank to disable that endpoint entirely.
+ADMIN_API_KEY=
+
+# Comma-separated list of allowed CORS origins.
+# Leave blank to disable cross-origin access (recommended for production).
+# Example: https://app.example.com,https://staging.example.com
+ALLOWED_ORIGINS=

.github/workflows/docker.yml

@@ -0,0 +1,40 @@
+name: Docker Build
+
+on:
+  push:
+    branches: [main, master]
+  pull_request:
+    branches: [main, master]
+
+jobs:
+  build:
+    name: Build Docker image
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@v3
+
+      - name: Build image
+        uses: docker/build-push-action@v5
+        with:
+          context: .
+          push: false
+          tags: llm-safety-middleware:ci
+          cache-from: type=gha
+          cache-to: type=gha,mode=max
+
+      - name: Smoke-test health endpoint
+        run: |
+          docker run -d --name smoke-test \
+            -e LLM_BACKEND_TYPE=ollama \
+            -e LLM_BASE_URL=http://localhost:99999 \
+            -e LLM_MODEL=test \
+            -p 8000:8000 \
+            llm-safety-middleware:ci
+          sleep 10
+          curl -f http://localhost:8000/health
+          docker stop smoke-test

.github/workflows/tests.yml

@@ -0,0 +1,81 @@
+name: Tests
+
+on:
+  push:
+    branches: [main, master, develop]
+  pull_request:
+    branches: [main, master]
+
+jobs:
+  test:
+    name: Python ${{ matrix.python-version }}
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        python-version: ["3.10", "3.11", "3.12"]
+
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Set up Python ${{ matrix.python-version }}
+        uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+          cache: pip
+
+      - name: Install uv
+        run: pip install uv
+
+      - name: Install dependencies
+        run: uv pip install --system -r requirements.txt
+
+      - name: Download spaCy model
+        run: python -m spacy download en_core_web_sm
+
+      - name: Run tests
+        run: pytest test_safety_pipeline.py -v --tb=short
+
+      - name: Upload coverage
+        if: matrix.python-version == '3.11'
+        run: |
+          uv pip install --system pytest-cov
+          pytest test_safety_pipeline.py --cov=llm_safety_pipeline --cov=api_server \
+            --cov-report=xml --tb=short -q
+
+      - name: Upload coverage to Codecov
+        if: matrix.python-version == '3.11'
+        uses: codecov/codecov-action@v4
+        with:
+          files: coverage.xml
+          fail_ci_if_error: false
+
+  lint:
+    name: Lint & Format
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+          cache: pip
+
+      - name: Install linters
+        run: pip install black isort flake8
+
+      - name: Check formatting (black)
+        run: black --check --diff .
+
+      - name: Check import order (isort)
+        run: isort --check --diff .
+
+      - name: Lint (flake8)
+        run: |
+          flake8 llm_safety_pipeline.py api_server.py \
+            --max-line-length=99 \
+            --extend-ignore=E203,W503

.gitignore

@@ -0,0 +1,89 @@
+# ── Python ────────────────────────────────────────────────────────────────
+__pycache__/
+*.py[cod]
+*$py.class
+*.so
+*.pyd
+
+# Distribution / packaging
+dist/
+build/
+*.egg-info/
+*.egg
+MANIFEST
+.eggs/
+wheels/
+
+# Virtual environments
+venv/
+.venv/
+env/
+.env/
+ENV/
+
+# ── Testing & coverage ────────────────────────────────────────────────────
+.pytest_cache/
+.coverage
+.coverage.*
+coverage.xml
+htmlcov/
+*.lcov
+nosetests.xml
+pytest_cache/
+
+# ── Type checkers ─────────────────────────────────────────────────────────
+.mypy_cache/
+.dmypy.json
+dmypy.json
+.pytype/
+.pyre/
+
+# ── Linters ───────────────────────────────────────────────────────────────
+.ruff_cache/
+
+# ── Secrets & environment ─────────────────────────────────────────────────
+.env
+.env.*
+!.env.example
+*.pem
+*.key
+
+# ── Runtime outputs ───────────────────────────────────────────────────────
+logs/
+*.log
+safety_reports/
+test_reports/
+
+# ── Demo artifacts ────────────────────────────────────────────────────────
+demo_safety_config.json
+
+# ── HuggingFace / model cache ─────────────────────────────────────────────
+*.bin
+*.safetensors
+models/
+model_cache/
+
+# ── Jupyter ───────────────────────────────────────────────────────────────
+.ipynb_checkpoints/
+*.ipynb
+
+# ── IDEs ──────────────────────────────────────────────────────────────────
+.idea/
+.vscode/
+*.swp
+*.swo
+*~
+
+# ── OS ────────────────────────────────────────────────────────────────────
+.DS_Store
+.DS_Store?
+Thumbs.db
+ehthumbs.db
+desktop.ini
+
+# ── Docker ────────────────────────────────────────────────────────────────
+# Keep Dockerfile and docker-compose.yml tracked; ignore runtime volumes
+docker-volumes/
+
+# ── Claude Code (internal) ────────────────────────────────────────────────
+.claude/

CHANGELOG.md

@@ -0,0 +1,89 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+---
+
+## [Unreleased]
+
+---
+
+## [2.0.0] — 2026-02-24
+
+### Breaking Changes
+
+- **Architecture change**: local GPT-2 generator completely removed; the pipeline
+  is now a pure **async HTTP proxy** to a remote LLM backend (Ollama,
+  OpenAI-compatible, or custom).
+- `SafetyConfig` — removed fields: `llm_model_name`, `max_new_tokens`,
+  `temperature`, `top_p`, `batch_size`, `max_cache_size`.  These are now
+  configured via `LLMBackendConfig` / environment variables.
+- `SafetyPipeline.process()` — `generate=` keyword argument removed.
+  `process()` is now input-only (sync, no LLM call).
+
+### Added
+
+- `LLMBackendConfig` dataclass with `from_env()` class method — supports
+  `ollama`, `openai`, and `custom` backend types.
+- `ExternalLLMClient` — async `httpx`-based HTTP client with exponential-backoff
+  retry (does not retry on 4xx errors).
+- `SafetyPipeline.async_process()` — full async pipeline:
+  input safety checks → remote LLM → output safety checks.
+- `SafetyPipeline.async_close()` — graceful shutdown of the underlying
+  `httpx.AsyncClient`.
+- `SafetyPipeline.backend_health()` — async probe of the remote LLM endpoint.
+- `RejectionReason.BACKEND_UNAVAILABLE` and `RejectionReason.INPUT_TOO_LONG`.
+- `SafetyConfig.max_prompt_length` (default 10 000 characters) — input length guard.
+- `GET /health/backend` API endpoint — probes the remote LLM reachability.
+- `POST /api/v1/statistics/reset` now requires `X-API-Key: <ADMIN_API_KEY>` header.
+- Full async test suite: 74 tests using `pytest-asyncio` and `respx` for httpx mocking.
+- `pyproject.toml`, `LICENSE`, `.gitignore`, `.env.example`, `CONTRIBUTING.md`,
+  `CHANGELOG.md`, GitHub Actions CI workflows.
+
+### Changed
+
+- CPU-bound PyTorch/spaCy inference offloaded to a thread-pool executor so the
+  asyncio event loop stays unblocked.
+- `SafetyReport` check-passed flags changed to `Optional[bool]` — `None` means
+  the check was not run (disabled or not reached), preventing false "passed" signals.
+- `get_statistics()` now returns a `copy.deepcopy()` to prevent callers from
+  mutating live counters.
+- `device` in `SafetyConfig` now uses `field(default_factory=...)` to detect
+  CUDA safely at construction time.
+- Safe-context regex no longer bypasses all downstream layers — it now defers to
+  the semantic classifier for a final verdict.
+- `SemanticSafetyClassifier` correctly handles `unitary/toxic-bert`'s multi-label
+  output (sigmoid + `probs.max()`) instead of binary softmax.
+- CORS: `allow_credentials=False`; origins opt-in via `ALLOWED_ORIGINS` env var.
+- FastAPI `@app.on_event` replaced with `@asynccontextmanager lifespan`.
+
+### Fixed
+
+- `_has_prohibited_entities` now checks both NER labels and plain text, fixing
+  false-negatives for entities spaCy did not label.
+- `_has_harmful_instruction_pattern` uses dependency-tree traversal instead of
+  a flat token scan, reducing false-positives.
+- `add_custom_filter` now properly raises on duplicate filter names.
+- `subprocess.run` in model download helper uses `sys.executable` so the correct
+  Python is invoked inside virtual environments.
+
+---
+
+## [1.0.0] — 2024-01-01
+
+### Added
+
+- Initial release: multi-layer LLM safety pipeline with local GPT-2 generation.
+- `TokenLevelFilter`, `RuleEngineFilter`, `SemanticSafetyClassifier`, `SafeLLMGenerator`.
+- `SafetyPipeline.process(prompt, generate=True/False)`.
+- FastAPI server with `/check`, `/generate`, `/statistics` endpoints.
+- Docker + docker-compose support.
+
+---
+
+[Unreleased]: https://github.com/SahilChachra/LLM-Safety-Middleware/compare/v2.0.0...HEAD
+[2.0.0]: https://github.com/SahilChachra/LLM-Safety-Middleware/compare/v1.0.0...v2.0.0
+[1.0.0]: https://github.com/SahilChachra/LLM-Safety-Middleware/releases/tag/v1.0.0