Curated list of tools, frameworks, and resources for running, building, and deploying AI privately — on-prem, air-gapped, or self-hosted.
Private AI enables you to keep your data, models, and infrastructure under your control, avoiding unnecessary exposure to third parties. This list covers inference runtimes, model management, privacy tools, and more.
- Awesome Private AI
- Contents
- Inference Runtimes & Backends
- Model Management & Serving
- Fine-Tuning & Adapters
- Quantization & Compression
- Vector Databases & Embeddings
- Agents & Orchestration
- VS Code Plugins & Extensions
- Privacy, Security & Governance
- Observability & Evaluation
- Models for Private Deployment
- UI & Interaction Layers
- Image & Video Generation
- Speech & Audio
- Datasets & Data Prep
- Learning Resources & Research
- AI Routers & API Aggregators
- Contributing
- License
Engines and frameworks to run LLMs, vision, and multimodal models locally.
- vLLM - High-throughput, low-latency inference engine for LLMs.
- sglang - Fast serving engine for LLMs and vision-language models, with RadixAttention prefix caching and a structured generation language.
- mlx-lm - Fast, Apple Silicon-optimized LLM inference engine for running models locally and privately.
- oMLX - macOS-native MLX inference server with paged SSD KV caching and continuous batching. Serves LLM, VLM, embedding, and reranker models over OpenAI- and Anthropic-compatible endpoints for local coding agents on Apple Silicon.
- Jan - Privacy-first, offline AI assistant and LLM runtime for local, secure inference.
- LM Studio - Cross-platform desktop app for running local LLMs with an easy-to-use interface.
- Cherry Studio - Powerful and customizable cross-platform desktop app for LLM inference with built in web search, RAG, MCP support, and a quick assistant hotkey to summon your LLM from anywhere. Supports a wide variety of providers and OpenAI compatible endpoints for local inference.
- LLM-D - Privacy-first, distributed LLM inference engine for scalable, local deployments.
- Ollama - Local LLM runner with model packaging. Uses llama.cpp backend to serve cautious model defaults.
- llama.cpp - Portable, CPU/GPU-friendly LLM inference, good for GPU + CPU hybrid inference.
- ik_llama.cpp - Fork of llama.cpp with bleeding edge feature implementations and quantization improvements.
- llamafile - Distribute and run an entire LLM as a single executable file that works across six operating systems, with no install step.
- LocalAI - Drop-in OpenAI-compatible API for local inference across text, image, audio, and embedding models, with no GPU required.
- KTransformers - Optimized framework for running very large MoE models on limited hardware via GPU/CPU offloading and kernel injection.
- MLC LLM - Compiler and runtime that deploys LLMs natively to GPUs, phones, and browsers via machine learning compilation.
- RamaLama - Runs models as OCI containers, pulling from registries you control — a good fit for air-gapped and enterprise workflows.
- text-generation-inference - Optimized serving stack from Hugging Face.
- GPT4All - Local desktop model runner.
- exo - Run your own AI cluster at home with everyday devices. Dynamic model partitioning across multiple devices like iPhones, Macs, and Linux machines.
- exllama3 - An optimized quantization and inference library for running LLMs locally on modern consumer-class GPUs. Use TabbyAPI for an API server.
- tabbyAPI - Official API server for running exllamav2 and exllamav3 models. Aims to be a friendly backend with high customizablity and an idiotmatic OAI compatible API for users.
- YALS (Yet another llamacpp server) - TabbyAPI's sister project, adapted for llama.cpp and GGUF models. Built from the ground up using libllama instead of wrapping llama-server.
- llama-swap - Model swapping for llama.cpp (or any local OpenAPI compatible server).
Tools for hosting, scaling, and versioning AI models privately.
- Ray Serve - Scalable Python model serving.
- Seldon Core - Kubernetes-native model deployment.
- KServe - Serverless model inference on Kubernetes.
- BentoML - Model packaging & serving framework.
- Triton Inference Server - NVIDIA's multi-framework inference server, supporting TensorRT, PyTorch, ONNX, and vLLM backends behind one endpoint.
- Xinference - Serve and manage LLM, embedding, rerank, image, and audio models in one self-hosted cluster with an OpenAI-compatible API.
- vLLM Production Stack - End-to-end stack for deploying vLLM in production, including orchestration, monitoring, autoscaling, and best practices for private LLM serving.
- OME (Open Model Engine) - Unified, open-source engine for serving, managing, and scaling LLMs and multimodal models privately. Supports sglang, vLLM, and more.
Private workflows for adapting models to your needs.
- LoRA - Low-rank adaptation technique.
- PEFT - Parameter-efficient fine-tuning.
- QLoRA - Memory-efficient LoRA on quantized models.
- LLaMA-Factory - Unified fine-tuning for 100+ models with a web UI, covering SFT, DPO, PPO, and reward modelling on your own hardware.
- Unsloth - Fine-tune and reinforcement-train LLMs 2x faster with substantially less VRAM, on a single local GPU.
- Axolotl - YAML-driven post-training framework covering full fine-tunes, LoRA, QLoRA, and multi-GPU sharding.
- TRL - Hugging Face's library for SFT, DPO, GRPO, and reward-model training on top of transformers.
- ms-swift - Training and deployment toolkit covering 500+ LLMs and 200+ multimodal models, from PEFT through to quantized export.
- torchtune - Native PyTorch library for fine-tuning and experimenting with LLMs, with readable single-file recipes.
Shrink models to fit the hardware you actually own.
- llm-compressor - Apply GPTQ, SmoothQuant, SparseGPT, and FP8/INT4 weight-activation quantization, exporting straight to vLLM.
- bitsandbytes - 8-bit and 4-bit quantization primitives that underpin QLoRA and much of the local fine-tuning ecosystem.
- GPTQModel - Actively maintained GPTQ toolkit for producing and running 4-bit quantized models.
Private semantic search & retrieval-augmented generation.
- Milvus - Scalable vector database.
- Qdrant - High-performance Vector Database and Vector Search Engine.
- Weaviate - Open-source semantic search engine.
- Chroma - Local-first vector database.
- FAISS - Facebook AI Similarity Search.
- pgvector - Vector similarity search inside PostgreSQL, keeping embeddings in the database you already self-host.
- LanceDB - Embedded, serverless vector database that stores vectors and metadata as files on your own disk or object store.
- text-embeddings-inference - Fast local serving for embedding and reranker models, so retrieval never leaves your network.
Frameworks for chaining private AI tools & agents.
- AG2 - Open-source operating system for agentic AI with native Ollama support for local model deployment and multi-agent collaboration.
- agentgateway - Gateway for managing and orchestrating AI agents with support for local deployment.
- LangChain - Agent and LLM orchestration framework.
- Langflow - Visual workflow builder for creating and deploying AI-powered agents and workflows with built-in API servers.
- Haystack - End-to-end RAG pipelines.
- Flowise - No-code LangChain UI.
- LlamaIndex - Data framework for LLM apps.
- MetaGPT - Multi-agent framework for building collaborative AI systems with role-based agents that can work together on complex tasks.
- Trae Agent - Privacy-friendly agent framework for orchestrating LLMs and tools, designed for secure, local, and scalable AI workflows.
- Qwen-Agent - Open-source, privacy-friendly agent framework for orchestrating LLMs and tools, designed for secure, local, and scalable AI workflows.
- Herdr - Self-hosted runtime and terminal multiplexer for coding agents. Runs the agent CLIs on your own machine or a box you control, grouping them into workspaces you can detach from and reattach to over SSH.
- Crush - Privacy-first, open-source agentic coding and automation platform for local AI workflows.
- OpenCode AI - Open-source agentic coding platform for private, local, and secure AI-powered development workflows.
- Orkas - Open-source, local-first desktop AI workforce whose Commander coordinates specialist agents through one chat; model calls can use a compatible local endpoint.
- Goose - Local, extensible agent that runs on your machine and works against any LLM backend, including Ollama and other self-hosted endpoints.
- Aider - Terminal-based pair programming agent that edits code in your local git repo, with support for local models via Ollama and OpenAI-compatible servers.
- PydanticAI - Python agent framework by the Pydantic team, model-agnostic with Ollama support for local deployment.
- dspy - Modular, open-source agent framework for building composable, private LLM applications and workflows.
- CUA - enables AI agents to control full operating systems in virtual containers and deploy them locally or to the cloud.
- Bytebot - A desktop agent is an AI that has its own computer. Unlike browser-only agents or traditional RPA tools, Bytebot comes with a full virtual desktop.
- MFS - Exposes your code, docs, chat (Slack/Gmail/Jira), databases and object stores as one file-like, searchable namespace for agents (
ls/cat/grep+ semantic search); runs fully local with on-device ONNX embeddings on Milvus, no API key. - DeepCode - Open agentic coding framework that turns papers and specs into working code (Paper2Code, Text2Web, Text2Backend), running against local Ollama or vLLM backends.
- Skales - Source-available (BSL 1.1) local-first desktop AI agent that runs fully on-device, offline via Ollama or with 15+ providers; your files never leave your machine, no cloud required.
Privacy-first, open-source agentic coding plugins and extensions for VS Code and other editors.
- Roo Code - Privacy-first, open-source agentic coding platform for secure, local AI development (VS Code extension).
- cline - Privacy-first, open-source agentic coding platform for local AI workflows and automation (VS Code extension).
- Continue - Open-source autocomplete and chat assistant for VS Code and JetBrains, configurable against Ollama, llama.cpp, vLLM, and other local endpoints.
- Tabby - Self-hosted AI coding assistant with its own inference server, offering a private alternative to hosted completion services.
Keep AI deployments secure and compliant.
- BlindAI - Confidential AI inference using TEEs.
- OpenFL - Federated learning framework.
- Flower - Federated learning at scale.
- Concrete - Fully homomorphic encryption for AI.
- Presidio - Detect and de-identify PII in text and images before it ever reaches a model.
- garak - LLM vulnerability scanner that probes local models for prompt injection, jailbreaks, and data leakage.
- NeMo Guardrails - Add programmable topical and safety rails to LLM applications, running alongside self-hosted models.
- LLM Guard - Input and output scanning toolkit covering prompt injection, PII, toxicity, and secrets leakage.
Measure and monitor private deployments without shipping traces to a vendor.
- Langfuse - Self-hostable LLM observability, tracing, prompt management, and evaluation.
- promptfoo - Local-first evaluation and red-teaming for prompts and models, runnable entirely offline in CI.
- DeepEval - Unit-testing framework for LLM outputs, with metrics that can run against locally hosted judge models.
- lm-evaluation-harness - The standard harness for benchmarking language models, supporting local vLLM and Hugging Face backends.
- Phoenix - Self-hosted tracing, evaluation, and experiment tracking for LLM applications.
Open-weight models and model libraries you can self-host.
- LLaMA 3 - Meta’s open-weight language model.
- Mistral 7B - Dense 7B parameter model.
- Qwen 3 - A wide variety of general and specialized models in both dense and "Mixture of Experts" formats.
- Kimi K2 - Mixture-of-Experts model with 32 billion activated parameters and 1 trillion total parameters.
- Phi-4 - Small, high-quality models from Microsoft.
- Mixtral - Mixture-of-experts model.
- Falcon - Open-source model from TII.
- Gemma3 - Open source model from Google.
- MLX Community - Community-driven Hugging Face page for open MLX models, optimized for Apple Silicon and private deployment.
- Bielik - An open source project that provides data, tools and LLMs for the development of the Polish artificial intelligence landscape
Self-hosted chat & AI frontends.
- Chatbot UI - Open-source ChatGPT clone.
- LibreChat - Enhanced web UI for LLMs.
- AnythingLLM - Full-stack private LLM workspace.
- Open WebUI - Commonly recommended Web UI frontend which features built in search, web scrape, RAG, and optional user authentication.
- Lobe Chat - Modern self-hosted chat framework with plugin and multimodal support, deployable in one click against local backends.
- text-generation-webui - Long-standing Gradio web UI for local text generation, supporting llama.cpp, ExLlama, and transformers backends.
- SillyTavern - Self-hosted, highly customizable frontend for local models, with extensive character, prompt, and context management.
- Screenpipe - 24/7 local screen + microphone recording with OCR, audio transcription, and semantic search. Fully offline with Ollama or any local LLM. MCP server for Claude.
Run diffusion and video models on your own GPUs.
- ComfyUI - Node-based interface and backend for diffusion models, running image, video, and audio pipelines entirely locally.
- AUTOMATIC1111 Stable Diffusion WebUI - The most widely deployed self-hosted Stable Diffusion interface, with a large extension ecosystem.
- InvokeAI - Professional-grade local generative image toolkit with a unified canvas and workflow editor.
Private speech-to-text and text-to-speech.
- whisper.cpp - C++ port of OpenAI's Whisper automatic speech recognition model, optimized for local, CPU/GPU inference without internet connectivity.
- Whisper - The original open-weight speech recognition model, runnable fully offline.
- WhisperX - Whisper with word-level timestamps, speaker diarization, and much faster batched transcription.
- RealtimeSTT - Low-latency local speech-to-text with voice activity detection, for building private voice interfaces.
- F5-TTS - Local text-to-speech with voice cloning from a short reference sample.
- speaches - Self-hosted OpenAI-compatible server for transcription, translation, and speech generation.
Create and manage private training corpora.
- OpenWebText - Open dataset similar to GPT training data.
- RedPajama - Open LLM training dataset.
- Docling - Parse PDF, DOCX, PPTX, and HTML into structured formats for RAG and training, running entirely on your own hardware.
- MinerU - High-quality PDF-to-Markdown and JSON extraction, including formulas and tables, for building private corpora.
- Marker - Fast, accurate document-to-Markdown conversion across PDFs, images, and office formats.
- Unstructured - Ingestion and preprocessing library for turning messy documents into model-ready chunks.
- Label Studio - Self-hosted data labelling platform for text, image, audio, and video annotation.
- Argilla - Collaborative tool for curating, annotating, and quality-checking datasets for fine-tuning and evaluation.
Guides, papers, and tutorials on private AI.
- LLMs from Scratch - Build a GPT-style model step by step in PyTorch, on your own machine.
- ML Engineering Open Book - Field-tested notes on training and serving large models: hardware, parallelism, throughput, and debugging.
- LLM Course - Roadmap and notebooks covering LLM fundamentals, fine-tuning, quantization, and deployment.
- Smol Course - Hugging Face's practical course on aligning and fine-tuning small models that fit on local hardware.
Centralized routers and proxy layers for aggregating, governing, and securing your private AI stack. These tools simplify connections to multiple model servers, optimize LLM routing, and provide observability, security, and compliance.
- Nexus - Open-source AI router to aggregate Model Context Protocol (MCP) servers, intelligently route requests to the best LLMs, and provide security, governance, observability, and simplified architecture for private AI deployments. Blog
- LiteLLM - Self-hosted proxy exposing 100+ model backends — including Ollama, vLLM, and any OpenAI-compatible server — behind one API, with keys, budgets, routing, and logging.
Contributions welcome! See Contributing
Under CC0-1.0 license. see LICENSE