Real-time LLM hallucination guardrail — NLI + RAG fact-checking with token-level streaming halt
Director-AI sits between your LLM and the user. It scores every output for hallucination before it reaches anyone — and can halt generation mid-stream if coherence drops below threshold.
graph LR
LLM["LLM<br/>(any provider)"] --> D["Director-AI"]
D --> S["Scorer<br/>NLI + RAG"]
D --> K["StreamingKernel<br/>token-level halt"]
S --> V{Approved?}
K --> V
V -->|Yes| U["User"]
V -->|No| H["HALT + evidence"]
Four things make it different:
- Token-level streaming halt — not post-hoc review. Severs output the moment coherence degrades.
- Dual-entropy scoring — NLI contradiction detection (DeBERTa) + RAG fact-checking against your knowledge base.
- Server-level batching — FastAPI server with request queue, WebSocket streaming, and multi-tenant isolation.
- Your data, your rules — ingest your own documents. The scorer checks against your ground truth.
Pure Python core — no compiled extensions required. Optional Rust kernel (pip install director-ai[rust]) for SIMD-accelerated scoring. Works on any platform with Python 3.11+.
| Layer | Packages | Install |
|---|---|---|
| Core (zero heavy deps) | CoherenceScorer, StreamingKernel, GroundTruthStore, HaltMonitor |
pip install director-ai |
| NLI models | DeBERTa, FactCG, MiniCheck, ONNX Runtime | pip install director-ai[nli] |
| Vector DBs | ChromaDB ([vector]), Pinecone ([pinecone]), Weaviate ([weaviate]), Qdrant ([qdrant]) |
pip install director-ai[vector] |
| LLM judge | OpenAI, Anthropic escalation | pip install director-ai[openai] |
| Observability | OpenTelemetry spans | pip install director-ai[otel] |
| Server | FastAPI + Uvicorn | pip install director-ai[server] |
Duck-type detection for five SDK shapes: OpenAI-compatible (OpenAI, vLLM, Groq, LiteLLM, Ollama), Anthropic, AWS Bedrock, Google Gemini, and Cohere.
from director_ai import guard
from openai import OpenAI
client = guard(
OpenAI(),
facts={"refund_policy": "Refunds within 30 days only"},
)
response = client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": "What is the refund policy?"}],
)Score a single prompt/response pair without an SDK client:
from director_ai import score
cs = score("What is the refund policy?", response_text,
facts={"refund": "Refunds within 30 days only"},
threshold=0.3)
print(f"Coherence: {cs.score:.3f} Approved: {cs.approved}")Point any OpenAI-compatible client at the proxy:
pip install director-ai[server]
director-ai proxy --port 8080 --facts kb.txt --threshold 0.3Then set OPENAI_BASE_URL=http://localhost:8080/v1 in your app. Every response
gets scored; hallucinations are rejected (or flagged with --on-fail warn).
Guard your own API endpoints:
from director_ai.integrations.fastapi_guard import DirectorGuard
app.add_middleware(DirectorGuard,
facts={"policy": "Refunds within 30 days only"},
on_fail="reject",
)Responses on POST endpoints get X-Director-Score and X-Director-Approved
headers. Set paths=["/api/chat"] to limit which endpoints are scored.
pip install "director-ai[nli]" # recommended — NLI model scoring
pip install "director-ai[nli,vector,server]" # production stack with RAG + REST API
pip install director-ai # heuristic-only (limited accuracy)Privacy note: The optional LLM judge mode (
llm_judge_enabled=True) sends truncated prompt+response fragments (500 chars) to an external provider (OpenAI or Anthropic). Do not enable in privacy-sensitive deployments without user consent. The default NLI-only mode runs entirely locally with no external calls.
Extras: [vector] (ChromaDB), [finetune] (domain adaptation), [ingestion] (PDF/DOCX parsing), [colbert] (late-interaction retrieval).
Framework integrations: [langchain], [llamaindex], [langgraph], [haystack], [crewai].
Full installation guide: docs.
Dockerfile included for self-hosted builds. Pre-built images not yet published to a registry.
docker build -t director-ai . # build locally
docker run -p 8080:8080 director-ai # CPU
docker build -f Dockerfile.gpu -t director-ai:gpu . # GPU build
docker run --gpus all -p 8080:8080 director-ai:gpu # GPUScoring model: yaxili96/FactCG-DeBERTa-v3-Large (0.4B params, MIT license).
| Model | Balanced Acc | Params | Latency | Streaming |
|---|---|---|---|---|
| Bespoke-MiniCheck-7B | 77.4% | 7B | ~100 ms | No |
| Director-AI (FactCG) | 75.8% | 0.4B | 14.6 ms | Yes |
| MiniCheck-Flan-T5-L | 75.0% | 0.8B | ~120 ms | No |
| MiniCheck-DeBERTa-L | 72.6% | 0.4B | ~120 ms | No |
75.8% balanced accuracy comes from the FactCG-DeBERTa-v3-Large model (77.2% in the NAACL 2025 paper; our eval yields 75.86% due to threshold tuning and data split version). Latency: 14.6 ms/pair measured on GTX 1060 6GB with ONNX GPU batching (16-pair batch, 30 iterations, 5 warmup). Director-AI's unique value is the system: NLI + KB + streaming halt.
Full results: benchmarks/comparison/COMPETITOR_COMPARISON.md.
Performance trade-offs and E2E pipeline metrics: docs.
10 built-in profiles with preset thresholds (starting points — adjust for your data):
director-ai config --profile medical # threshold=0.30, NLI on, reranker on
director-ai config --profile finance # threshold=0.30, w_fact=0.6
director-ai config --profile legal # threshold=0.30, w_logic=0.6
director-ai config --profile creative # threshold=0.40, permissiveDomain-specific benchmark scripts exist but have not yet been validated with measured results. Run them yourself (requires GPU + HuggingFace datasets):
python -m benchmarks.medical_eval # MedNLI + PubMedQA
python -m benchmarks.legal_eval # ContractNLI + CUAD (RAGBench)
python -m benchmarks.finance_eval # FinanceBench + Financial PhraseBank- Heuristic fallback is weak: Without
[nli], scoring uses word-overlap heuristics (~55% accuracy). Usestrict_mode=Trueto reject (0.9) instead of guessing. - Summarisation FPR at 10.5%: Reduced from 95% via bidirectional NLI + baseline calibration (v3.5). AggreFact-CNN: 68.8%, ExpertQA: 59.1% (structurally expected at 0.4B params).
- ONNX CPU is slow: 383 ms/pair without GPU. Use
onnxruntime-gpufor production. - Weights are domain-dependent: Default
w_logic=0.6, w_fact=0.4suits general QA. Adjust for your domain or use a built-in profile. - LLM-as-judge sends data externally: When
llm_judge_enabled=True, truncated prompt+response (500 chars) are sent to the configured provider. Do not enable in privacy-sensitive deployments without user consent. - Threshold defaults differ by API surface:
guard()/score()default tothreshold=0.3(permissive).DirectorConfigdefaults tocoherence_threshold=0.6(conservative). Always set the threshold explicitly. - NLI-only scoring needs KB grounding: Without a knowledge base, PubMedQA F1=62.1%, FinanceBench 80%+ FPR. Load your domain facts into the vector store — that's where Director-AI's scoring discriminates well.
- Long documents need ≥16GB VRAM: Legal contracts and SEC filings exceed 6GB during chunked NLI inference.
@software{sotek2026director,
author = {Sotek, Miroslav},
title = {Director-AI: Real-time LLM Hallucination Guardrail},
year = {2026},
url = {https://github.com/anulum/director-ai},
version = {3.9.4},
license = {AGPL-3.0-or-later}
}Dual-licensed:
- Open-Source: GNU AGPL v3.0 — research, personal use, open-source projects.
- Commercial: Proprietary license — removes copyleft for closed-source and SaaS.
See Licensing for pricing tiers and FAQ.
Contact: anulum.li | director.class.ai@anulum.li
Join the Director-AI Discord for CI notifications, release announcements, and support. The Discord bot also provides /version, /docs, /install, /status, and /quickstart slash commands.
See CONTRIBUTING.md. By contributing, you agree to AGPL v3 terms.