RFC: Dirty Workers - Keep ML Models Loaded in Memory

[benoitc]

🚀 Introducing Dirty Workers

I'm excited to share a new experimental feature: Dirty Workers - a way to keep stateful resources (like ML models) loaded in memory while your HTTP workers remain lightweight and responsive.

The Problem

Loading ML models is expensive - often taking several seconds. Traditional worker models either:

• Reload the model on every request (slow)
• Load models in each HTTP worker (memory wasteful)
• Require external services (complexity)

The Solution

Dirty workers run alongside your HTTP workers but maintain state between requests:

┌─────────────────┐     ┌──────────────────┐     ┌─────────────────────┐
│  HTTP Clients   │────►│  FastAPI (ASGI)  │────►│  DirtyWorker        │
│                 │     │  - Lightweight   │     │  - ML model loaded  │
│                 │◄────│  - Async I/O     │◄────│  - Stays in memory  │
└─────────────────┘     └──────────────────┘     └─────────────────────┘

Try It Out

You can find a concrete example: a text embedding service using sentence-transformers.

Quick start:

git clone https://github.com/benoitc/gunicorn.git
cd gunicorn
git checkout feature/dirty-arbiters

cd examples/embedding_service
docker compose up --build

Test it:

curl http://127.0.0.1:8000/health
curl -X POST http://127.0.0.1:8000/embed \\
  -H 'Content-Type: application/json' \\
  -d '{"texts": ["gunicorn is awesome", "I love Python"]}'

How It Works

1. DirtyApp - Define your stateful application:

from gunicorn.dirty.app import DirtyApp

class EmbeddingApp(DirtyApp):
    def init(self):
        from sentence_transformers import SentenceTransformer
        self.model = SentenceTransformer('all-MiniLM-L6-v2')

    def embed(self, texts):
        return self.model.encode(texts).tolist()

2. Configure - Add to your gunicorn config:

dirty_apps = ["myapp.embedding:EmbeddingApp"]
dirty_workers = 1
dirty_timeout = 60

3. Call - From your HTTP handlers:

from gunicorn.dirty.client import get_dirty_client

client = get_dirty_client()
result = client.execute("myapp.embedding:EmbeddingApp", "embed", texts)

Use Cases

• ML Inference - Keep models loaded (transformers, PyTorch, TensorFlow)
• Database Connections - Maintain connection pools
• Large Data Structures - Keep indexes/caches in memory
• GPU Resources - Manage CUDA contexts efficiently

Feedback Welcome

Please test it out and let me know!

Links

• 📁 Example code
• 📖 Documentation
• 🔀 Feature branch

---

cc @gunicorn/maintainers

[benoitc]

🎬 New: Streaming Chat Demo

Following feedback on the embedding service example, I've added a second demo showcasing streaming responses - essential for LLM applications where token-by-token output is expected.

What's New

The streaming API lets your DirtyApp yield chunks via generators:

class ChatApp(DirtyApp):
    def generate(self, prompt):
        for token in get_llm_response(prompt):
            yield token

From your HTTP handler, stream chunks as they arrive:

async def chat(request):
    client = await get_dirty_client_async()
    async for chunk in client.stream_async("app:ChatApp", "generate", prompt):
        yield f"data: {json.dumps({'token': chunk})}\n\n"

Try It

cd examples/streaming_chat
docker compose up --build
# Browser UI at http://localhost:8000

The demo simulates LLM token generation with realistic delays - no ML dependencies required.

Features Demonstrated

• Token-by-token SSE streaming
• Async generators via dirty workers
• Browser-based chat UI with cursor animation
• Both streaming and sync endpoints for comparison

📁 Streaming chat example

Feedback welcome!

[benoitc]

Branch has been merged to master.