How to Use TokenMizer with Claude & Other LLMs

Complete Practical Guide

What you're actually setting up

TokenMizer is a local proxy that runs on your computer. Every AI request goes through it first:

You → TokenMizer (localhost:8000) → Claude / GPT / Gemini / etc.

It compresses your input, caches repeated queries, builds graph memory of your session, and saves checkpoints. The AI never knows it's there.

Part 1: Install & Start (5 minutes)

Step 1 — Install

pip install "tokenmizer[anthropic,cache]"

No key yet? Use Ollama (completely free, runs locally):

# Install Ollama first: https://ollama.ai
ollama pull llama3
pip install tokenmizer

Step 2 — Configure

# Auto-setup (detects your provider)
curl -fsSL https://raw.githubusercontent.com/your-org/tokenmizer/main/scripts/setup.sh | bash

# Or manually — create tokenmizer.yaml:
cat > tokenmizer.yaml << 'EOF'
provider: anthropic
default_model: claude-sonnet-4-6

graph_checkpoint:
  enabled: true
  storage_dir: ./checkpoints

compression:
  enabled: true

cache:
  enabled: true
EOF

Set your API key (never put it in the yaml file):

export TOKENMIZER_ANTHROPIC_API_KEY=sk-ant-...
# or for OpenAI:
export TOKENMIZER_OPENAI_API_KEY=sk-...

Step 3 — Start

tokenmizer serve

You'll see:

🧠 TokenMizer
Proxy:     http://localhost:8000/v1/chat/completions
Dashboard: http://localhost:8000
API Docs:  http://localhost:8000/docs
Health:    http://localhost:8000/health

That's it. TokenMizer is running.

Part 2: Use with Claude.ai (Web Interface)

Claude.ai cannot be proxied directly — it goes through Anthropic's servers, not your local machine. But you can use TokenMizer WITH Claude.ai in two ways:

Method A — Copy-paste resume context

Run your coding/research session in Claude.ai normally
When the context fills up, ask Claude:

"Summarize what we've done, what's decided, and what's next in a structured list"
Copy that summary

In TokenMizer CLI:

# Start a new session with the summary as context
tokenmizer serve

In your next Claude.ai conversation, paste the summary at the start

This is manual but works with zero setup.

Method B — Use TokenMizer as your AI interface instead

Point your apps to TokenMizer instead of Claude directly. Same API, same models, just going through your proxy first.

Part 3: Use with Claude Code (Best Integration)

Claude Code's MCP config lives in ~/.claude/settings.json (user level, works across all projects) or .claude/settings.json in a project folder.

Setup — 3 steps

Step 1: Start TokenMizer

tokenmizer serve &
# runs in background on port 8000

Step 2: Add to Claude Code config

# User-level (works in every project):
claude mcp add tokenmizer --transport stdio \
  python3 -m tokenmizer.mcp.server

# Or edit ~/.claude/settings.json directly:

{
  "mcpServers": {
    "tokenmizer": {
      "command": "python3",
      "args": ["-m", "tokenmizer.mcp.server"],
      "env": {
        "TOKENMIZER_URL": "http://localhost:8000"
      }
    }
  }
}

Step 3: Restart Claude Code

claude  # restart — it auto-discovers new MCP servers

Now you can use slash commands inside Claude Code

/checkpoint my-project
→ TokenMizer saves everything to graph memory

/resume my-project
→ Returns ~300 token summary of what was done

/analyze /path/to/data.csv
→ Returns schema + stats + sample in 400 tokens
  instead of 400,000 tokens if you pasted the file

/analyze /path/to/report.pdf "what are the key risks"
→ Returns relevant pages + structure

What Claude Code does automatically (once MCP is connected)

When you say "let's continue working on my-project" → Claude Code calls resume_session and injects the context.

When you say "save our progress" → Claude Code calls checkpoint_session.

When you paste a file path → Claude Code calls analyze_file instead of reading the whole file.

Part 4: Use with Any OpenAI-Compatible App

Any app that uses the OpenAI SDK can point at TokenMizer. Just change the base_url. Nothing else changes.

Python apps

from openai import OpenAI

# Before: direct to Anthropic/OpenAI
# client = OpenAI(api_key="sk-ant-...")

# After: through TokenMizer
client = OpenAI(
    api_key="any-string-here",  # your real key is in tokenmizer.yaml
    base_url="http://localhost:8000/v1",
)

response = client.chat.completions.create(
    model="claude-sonnet-4-6",
    messages=[{"role": "user", "content": "Hello"}],
    extra_body={"session_id": "my-project"},  # enables graph memory
)

print(response.choices[0].message.content)

# Check what was saved
saved = response.model_extra["tokenmizer"]["total_saved"]
print(f"Tokens saved this request: {saved}")

Cursor IDE

Settings → Models → OpenAI API Base URL:

http://localhost:8000/v1

Continue.dev

.continue/config.json:

{
  "models": [{
    "title": "Claude via TokenMizer",
    "provider": "openai",
    "model": "claude-sonnet-4-6",
    "apiBase": "http://localhost:8000/v1",
    "apiKey": "any"
  }]
}

Any other tool

If it has an "OpenAI API endpoint" or "API base URL" setting: → Set it to http://localhost:8000/v1 → That's it.

Part 5: Session Resume — The Main Feature

This is what TokenMizer is actually for.

Scenario: You have a long coding session

Day 1 — Turn 1–50:
  - You build a FastAPI auth service
  - Make 10 architecture decisions
  - Modify 15 files
  - Fix 3 bugs
  - Context window hits 85%

Without TokenMizer:
  - Start new conversation
  - Spend 10 minutes re-explaining the project
  - Model doesn't know what was decided
  - Risk re-doing completed work

With TokenMizer:
  - Auto-checkpoint created at 85%
  - 300-token resume block saved
  
Day 2 — New conversation:

import httpx

# Get resume context
r = httpx.get("http://localhost:8000/api/resume/auth-service?level=standard")
resume = r.json()["resume_context"]

print(resume)
# Output:
# Goal: Build FastAPI auth service with JWT + PostgreSQL
# In progress: Refresh token rotation | Rate limiting
# Done: Project structure | User model | Auth endpoints | Fix 422 | Tests
# Decided: PostgreSQL (concurrent writes) | bcrypt | Redis for refresh tokens
# Files: api/auth.py, api/models.py, api/main.py, config.py
# Env: Python 3.12, FastAPI 0.111
# Continue from: Add rate limiting to auth endpoints

# Inject into new session
client = OpenAI(base_url="http://localhost:8000/v1", api_key="any")
response = client.chat.completions.create(
    model="claude-sonnet-4-6",
    messages=[
        {
            "role": "system",
            "content": f"[Previous session]\n{resume}\n\nContinue from here."
        },
        {
            "role": "user",
            "content": "Let's continue — implement the rate limiting"
        }
    ],
    extra_body={"session_id": "auth-service-day2"},
)

CLI shortcut

# Get resume context directly
tokenmizer resume auth-service

# Output:
# Goal: Build FastAPI auth service with JWT + PostgreSQL
# In progress: Refresh token rotation | Rate limiting
# Done: Project structure | User model | ...
# Decided: PostgreSQL | bcrypt | Redis for refresh tokens
# Continue from: Add rate limiting to auth endpoints

# Manually checkpoint
tokenmizer checkpoint auth-service

Part 6: Large File Analysis

Instead of pasting 50,000 rows into Claude:

# CLI
tokenmizer analyze /data/sales_2025.csv

# Or via API
curl -X POST http://localhost:8000/api/checkpoint \
  -H "Content-Type: application/json"

# Or in Python
import httpx
from tokenmizer.filters.file_intelligence import FileIntelligence

fi = FileIntelligence()
result = fi.process(
    open("/data/sales_2025.csv", "rb").read(),
    "sales_2025.csv",
    token_budget=500,
    query="which regions are underperforming",
)

print(f"Original: {result.original_tokens:,} tokens")
print(f"Extracted: {result.extracted_tokens} tokens")
print(f"Savings: {result.savings_pct:.0f}%")
print(result.content)

# Then send result.content to Claude — not the raw file

Real savings:

50,000-row CSV: 400,000 tokens → 450 tokens (99.9%)
200-page PDF: 150,000 tokens → 1,800 tokens (98.8%)
Excel 10 sheets: 300,000 tokens → 800 tokens (99.7%)

Part 7: Dashboard & Stats

Open http://localhost:8000 in your browser.

Shows:

Requests today / this week
Tokens saved and % reduction
Cache hit rate
Which layers are saving the most
Live graph node types

API stats (for your own tracking)

# Daily summary
curl http://localhost:8000/api/stats

# Cache stats
curl http://localhost:8000/api/cache/stats

# Session graph
curl http://localhost:8000/api/graph/my-project

# All checkpoints for a session
curl http://localhost:8000/api/checkpoints/my-project

Part 8: Run Benchmarks

These measure actual graph extraction quality on your machine.

# Install dev deps
pip install -e ".[dev]"

# Checkpoint accuracy benchmark
# Measures: task recall, decision recall, file accuracy, resume tokens
python -m benchmarks.checkpoint_accuracy.runner

# Output:
# Session: fastapi_auth_30turns (14 turns)
# ═══════════════════════════════════════════════
#   Nodes extracted:       12
#   Task Precision:        67%
#   Task Recall:           60%
#   Decision Recall:       67%
#   File Recall:           67%
#   Resume overhead:       187 tokens
#   Information Loss:      35%
#   Extraction time:       2.3ms

# Memory accuracy tests
pytest tests/memory_accuracy/ -v

# All tests
pytest tests/ -v

# With coverage
pytest tests/ --cov=tokenmizer --cov-report=term-missing

What the benchmark numbers mean

Metric	What it measures	Target
Task Precision	Of tasks in graph, % that are real tasks	≥ 70%
Task Recall	Of real tasks done, % captured in graph	≥ 60%
Decision Recall	Of real decisions, % captured	≥ 70%
File Recall	Of real files modified, % captured	≥ 80%
Resume Overhead	Tokens used for resume block	≤ 400
Information Loss	Overall % of session info lost	≤ 30%

Current heuristic extractor achieves roughly:

Task recall: 40–60% (improves with use_llm_extraction: true)
Decision recall: 40–67%
File recall: 60–80% (files are easiest — regex catches paths well)
Resume overhead: 150–300 tokens

With use_llm_extraction: true (uses cheap haiku model, ~$0.001/turn):

Task recall: ~75–85%
Decision recall: ~80–90%
Information loss: ~15–25%

Part 9: Enable LLM-Powered Extraction (Better Quality)

Edit tokenmizer.yaml:

graph_checkpoint:
  use_llm_extraction: true

This uses Claude Haiku (or gpt-4o-mini) to extract graph nodes from each conversation turn. Costs ~$0.001 per turn. Dramatically more accurate than the heuristic regex approach.

You need a key for this. Once you get your Anthropic key:

export TOKENMIZER_ANTHROPIC_API_KEY=sk-ant-...
tokenmizer serve

Part 10: Production Setup (When You're Ready)

Docker:

ANTHROPIC_API_KEY=sk-ant-... docker-compose up

All durable state (graph, checkpoints, session ownership) is SQLite in the checkpoints volume and survives restarts. There is no Redis — the state_backend setting is accepted but nothing reads it.

With authentication:

export TOKENMIZER_ENV=production          # refuse to start unauthenticated
export TOKENMIZER_API_KEY=your-strong-random-key
tokenmizer serve
# All endpoints now require: Authorization: Bearer your-key

Add more credentials to isolate sessions between callers — each key is its own principal, and a session belongs to the key that created it:

# tokenmizer.yaml
api_keys:
  - second-team-key

Multi-provider routing

Not implemented. The routing: block is accepted so old config files still load, but nothing reads it and savings.routing is always 0. Setting enabled: true logs a warning at startup and changes nothing.

Quick Reference

What you want	Command
Start proxy	`tokenmizer serve`
Save session	`tokenmizer checkpoint my-project`
Resume session	`tokenmizer resume my-project`
Analyze a file	`tokenmizer analyze /path/to/file.csv`
See stats	`tokenmizer stats`
Open dashboard	`http://localhost:8000`
Run benchmarks	`python -m benchmarks.checkpoint_accuracy.runner`
Run tests	`pytest tests/ -v`
API docs	`http://localhost:8000/docs`

Honest Limitations (Right Now)

Claude.ai web interface: Cannot be proxied. You'd need the API or Claude Code.

use_llm_extraction: false (default): Heuristic extraction. Works but misses some decisions and goals. Graph quality improves significantly with LLM extraction but that needs an API key.

No key yet: Use Ollama. Full pipeline works — compression, cache, graph memory, checkpoints, file intelligence. Only difference: your model is llama3/phi3 instead of Claude.

Graph memory is new: The validator (confidence scoring) and semantic edge linking are freshly written. Expect to tune the min_confidence threshold in tokenmizer.yaml for your use case. Default is 0.50.

FilesExpand file tree

USAGE.md

Latest commit

History