Skip to content

OBSERVABILITY.md

Latest commit

 

History

History
215 lines (141 loc) · 6.16 KB

OBSERVABILITY.md

File metadata and controls

215 lines (141 loc) · 6.16 KB

Observability

How to observe, intercept, and capture network traffic from the claw runtime using the Docker proxy setup.

Prerequisites

  • Docker or Podman with Compose support
  • An ANTHROPIC_API_KEY set in your environment or a .env file

Quick Start

cd docker
export ANTHROPIC_API_KEY=sk-ant-...
podman-compose up --build

This starts two services on a shared bridge network:

Service Port Purpose
mitmproxy 8080 HTTPS proxy (CONNECT-capable)
mitmproxy 8081 Web UI for inspecting flows
claw — The runtime, routing all traffic through the proxy

Viewing Traffic

mitmproxy Web UI

Open http://localhost:8081 in your browser. All intercepted HTTP/S requests and responses are displayed in real time.

mitmdump Console Output

podman logs -f docker_mitmproxy_1

This streams every request/response to your terminal in mitmdump format.

Wireshark / tcpdump

The claw-net bridge network is visible to the host. Capture on the bridge interface:

# Find the interface name
IFACE=$(ip link | grep -oP 'br-\w+' | head -1)

# Capture all traffic on the bridge
sudo tcpdump -i "$IFACE" -w claw-traffic.pcap

# Or open it live in Wireshark
sudo wireshark -k -i "$IFACE"

Note: traffic between claw and mitmproxy is unencrypted HTTP on the bridge (the TLS termination happens at mitmproxy). Traffic from mitmproxy to upstream servers is encrypted.

Decrypting Upstream TLS in Wireshark

To decrypt mitmproxy's outbound TLS in Wireshark, export the SSLKEYLOGFILE from inside the mitmproxy container:

# Start mitmproxy with key logging
podman-compose run -e SSLKEYLOGFILE=/capture/sslkeys.log mitmproxy

# In Wireshark: Edit > Preferences > Protocols > TLS > (Pre)-Master-Secret log filename
# Point it at: ./capture/sslkeys.log

pcap Capture

For offline analysis, enable built-in pcap capture:

ENABLE_PCAP=true podman-compose up

pcap files are written to the capture/ directory (or wherever CAPTURE_DIR points):

capture/claw-20260401-142756.pcap

Open these with Wireshark or tshark:

tshark -r capture/claw-20260401-142756.pcap
wireshark capture/claw-20260401-142756.pcap

TLS / Certificate Chain

mitmproxy generates a self-signed CA certificate on first start. The claw entrypoint automatically:

  1. Waits for the CA cert to appear at /mitmproxy-data/mitmproxy-ca-cert.pem
  2. Combines it with the system CA bundle
  3. Sets SSL_CERT_FILE, NODE_EXTRA_CA_CERTS, REQUESTS_CA_BUNDLE, CURL_CA_BUNDLE, CCR_CA_BUNDLE_PATH, and CCR_SYSTEM_CA_BUNDLE

The CA cert persists in the mitmproxy-data Docker volume across restarts.

Using Your Own CA

Mount your CA cert over the mitmproxy data directory:

# In docker-compose.yml, replace the mitmproxy-data volume with:
volumes:
  - ./my-certs:/root/.mitmproxy

Place your mitmproxy-ca-cert.pem and mitmproxy-ca.pem (private key) in ./my-certs/.

Environment Variables

Proxy routing (set automatically by entrypoint)

Variable Value Purpose
HTTPS_PROXY http://mitmproxy:8080 Route HTTPS through proxy
HTTP_PROXY http://mitmproxy:8080 Route HTTP through proxy
NO_PROXY localhost,127.0.0.1,::1 Bypass list (minimal)
CLAW_NO_PROXY_OVERRIDE localhost,127.0.0.1,::1 Overrides the hardcoded bypass list in the runtime so that anthropic.com, github.com, etc. are proxied

Transport debugging

Variable Value Purpose
CLAW_FORCE_HTTP_TRANSPORT true Downgrade WebSocket MCP transports to HTTP so they route through the proxy

Set this in the claw service environment if you need to inspect MCP WebSocket traffic:

environment:
  - CLAW_FORCE_HTTP_TRANSPORT=true

pcap capture

Variable Default Purpose
ENABLE_PCAP false Write pcap files inside the mitmproxy container
CAPTURE_DIR ./capture Host directory for pcap output

Troubleshooting

"mitmproxy CA cert not found after 15s"

The claw container starts before mitmproxy has finished initializing. This usually resolves by restarting:

podman-compose restart claw

If it persists, check that the mitmproxy-data volume is shared correctly:

podman volume inspect docker_mitmproxy-data

Traffic not appearing in mitmproxy

Check that CLAW_NO_PROXY_OVERRIDE is set. Without it, the runtime's hardcoded NO_PROXY list bypasses anthropic.com, github.com, and other common hosts.

WebSocket traffic not captured

WebSocket connections use CONNECT tunneling and upgrade to a persistent connection that mitmproxy can observe but not fully decode. Set CLAW_FORCE_HTTP_TRANSPORT=true to downgrade MCP WebSocket transports to HTTP for full request/response inspection.

LiteLLM Integration

The stack includes a LiteLLM proxy that sits between claw and upstream LLM providers. This lets you swap backends without changing claw's configuration.

claw  -->  litellm  -->  upstream provider (Anthropic, OpenAI, Ollama, etc.)
  \          |
   \         v
    +-->  mitmproxy  (observes litellm's outbound traffic)

Configuration

Edit docker/litellm-config.yaml to configure backends. The default config points at a local Ollama instance on the host.

To use a different backend, uncomment the relevant section and set the API key:

# Anthropic
ANTHROPIC_API_KEY=sk-ant-... podman-compose up

# GitHub Models
GITHUB_TOKEN=ghp_... podman-compose up

# Local Ollama (default, no key needed)
podman-compose up

Using a specific model

Pass the model name as defined in litellm-config.yaml:

podman-compose run claw claw --model qwen2.5:32b

LiteLLM admin UI

LiteLLM exposes its API on port 4000. You can check available models:

curl http://localhost:4000/v1/models -H "Authorization: Bearer sk-litellm-dev"

Traffic flow

  • claw -> litellm: Internal network, bypasses mitmproxy (litellm is in NO_PROXY)
  • litellm -> upstream: Outbound HTTPS, visible in mitmproxy if litellm is configured to use the proxy
  • claw -> mitmproxy: Only for claw's own non-LLM traffic (tool downloads, git, etc.)