Universal HTTP logging

.gitignore

@@ -9,4 +9,6 @@ venv
 .DS_Store
 .idea/
 .vscode/
-uv.lock
+uv.lock
+build/
+dist/

README.md

@@ -327,6 +327,34 @@ See also [the llm tag](https://simonwillison.net/tags/llm/) on my blog.
   * [Browsing logs using Datasette](https://llm.datasette.io/en/stable/logging.html#browsing-logs-using-datasette)
   * [Backing up your database](https://llm.datasette.io/en/stable/logging.html#backing-up-your-database)
   * [SQL schema](https://llm.datasette.io/en/stable/logging.html#sql-schema)
+* [HTTP Debugging and Logging](https://llm.datasette.io/en/stable/http-debugging.html)
+  * [Quick Start](https://llm.datasette.io/en/stable/http-debugging.html#quick-start)
+  * [Environment Variables](https://llm.datasette.io/en/stable/http-debugging.html#environment-variables)
+  * [CLI Flags](https://llm.datasette.io/en/stable/http-debugging.html#cli-flags)
+  * [What Gets Logged](https://llm.datasette.io/en/stable/http-debugging.html#what-gets-logged)
+    * [INFO Level (`LLM_HTTP_LOGGING=1`)](https://llm.datasette.io/en/stable/http-debugging.html#info-level-llm-http-logging-1)
+    * [DEBUG Level (`LLM_HTTP_DEBUG=1`)](https://llm.datasette.io/en/stable/http-debugging.html#debug-level-llm-http-debug-1)
+  * [Provider-Specific Debugging](https://llm.datasette.io/en/stable/http-debugging.html#provider-specific-debugging)
+    * [OpenAI Models (o1, o3, GPT-4, etc.)](https://llm.datasette.io/en/stable/http-debugging.html#openai-models-o1-o3-gpt-4-etc)
+    * [Gemini Models (including 2.5-pro reasoning)](https://llm.datasette.io/en/stable/http-debugging.html#gemini-models-including-2-5-pro-reasoning)
+    * [Anthropic Claude Models (including reasoning)](https://llm.datasette.io/en/stable/http-debugging.html#anthropic-claude-models-including-reasoning)
+  * [Real-World Use Cases](https://llm.datasette.io/en/stable/http-debugging.html#real-world-use-cases)
+    * [Debugging Reasoning Models](https://llm.datasette.io/en/stable/http-debugging.html#debugging-reasoning-models)
+    * [Investigating API Errors](https://llm.datasette.io/en/stable/http-debugging.html#investigating-api-errors)
+    * [Understanding Token Usage](https://llm.datasette.io/en/stable/http-debugging.html#understanding-token-usage)
+    * [Monitoring Rate Limits](https://llm.datasette.io/en/stable/http-debugging.html#monitoring-rate-limits)
+  * [Advanced Configuration](https://llm.datasette.io/en/stable/http-debugging.html#advanced-configuration)
+    * [Combining with LLM Logging](https://llm.datasette.io/en/stable/http-debugging.html#combining-with-llm-logging)
+    * [Filtering Logs](https://llm.datasette.io/en/stable/http-debugging.html#filtering-logs)
+    * [Performance Considerations](https://llm.datasette.io/en/stable/http-debugging.html#performance-considerations)
+  * [Troubleshooting](https://llm.datasette.io/en/stable/http-debugging.html#troubleshooting)
+    * [Common Issues](https://llm.datasette.io/en/stable/http-debugging.html#common-issues)
+    * [Integration with External Tools](https://llm.datasette.io/en/stable/http-debugging.html#integration-with-external-tools)
+  * [Security Considerations](https://llm.datasette.io/en/stable/http-debugging.html#security-considerations)
+  * [Backward Compatibility](https://llm.datasette.io/en/stable/http-debugging.html#backward-compatibility)
+    * [Legacy Support](https://llm.datasette.io/en/stable/http-debugging.html#legacy-support)
+    * [Migration Path](https://llm.datasette.io/en/stable/http-debugging.html#migration-path)
+  * [Related Documentation](https://llm.datasette.io/en/stable/http-debugging.html#related-documentation)
 * [Related tools](https://llm.datasette.io/en/stable/related-tools.html)
   * [strip-tags](https://llm.datasette.io/en/stable/related-tools.html#strip-tags)
   * [ttok](https://llm.datasette.io/en/stable/related-tools.html#ttok)

docs/help.md

@@ -63,9 +63,16 @@ Usage: llm [OPTIONS] COMMAND [ARGS]...
 
       llm prompt --help
 
+  Environment variables for HTTP debugging: - LLM_HTTP_LOGGING=1: Enable HTTP
+  request/response logging - LLM_HTTP_DEBUG=1: Enable verbose HTTP debugging -
+  LLM_OPENAI_SHOW_RESPONSES=1: Legacy OpenAI-only debugging
+
 Options:
-  --version   Show the version and exit.
-  -h, --help  Show this message and exit.
+  --version       Show the version and exit.
+  --http-logging  Enable HTTP request/response logging for debugging
+  --http-debug    Enable verbose HTTP debugging (includes connection details)
+  --no-color      Disable colored output in HTTP logging
+  -h, --help      Show this message and exit.
 
 Commands:
   prompt*       Execute a prompt
@@ -1082,4 +1089,4 @@ Options:
   --key TEXT  OpenAI API key
   -h, --help  Show this message and exit.
 ```
-<!-- [[[end]]] -->
+<!-- [[[end]]] -->

docs/http-debugging.md

@@ -0,0 +1,274 @@
+(http-debugging)=
+# HTTP Debugging and Logging
+
+LLM CLI provides comprehensive HTTP logging capabilities to debug requests and responses across all model providers. This feature enables you to see exactly what's being sent to and received from AI APIs, making it invaluable for troubleshooting and understanding model behavior.
+
+## Quick Start
+
+Enable HTTP logging for all providers:
+
+```bash
+# Basic HTTP logging (shows requests and responses)
+LLM_HTTP_LOGGING=1 llm -m gpt-4o "Hello world"
+
+# Verbose HTTP debugging (includes connection details)
+LLM_HTTP_DEBUG=1 llm -m gemini-2.5-pro "Reasoning task"
+
+# Using CLI flags (alternative to environment variables)
+llm --http-logging -m claude-4-sonnet "Debug this request"
+llm --http-debug -m o3 "Show all HTTP details"
+
+# For development/testing without global install
+LLM_HTTP_DEBUG=1 uv run llm -m gpt-4o "Test prompt"
+LLM_HTTP_DEBUG=1 python -m llm -m gpt-4o "Test prompt"  # if already installed
+```
+
+## Environment Variables
+
+| Variable | Level | Description |
+|----------|-------|-------------|
+| `LLM_HTTP_LOGGING=1` | INFO | Shows HTTP requests and responses |
+| `LLM_HTTP_DEBUG=1` | DEBUG | Shows detailed connection info and headers |
+| `LLM_HTTP_VERBOSE=1` | DEBUG | Alias for `LLM_HTTP_DEBUG` |
+| `LLM_OPENAI_SHOW_RESPONSES=1` | INFO | Legacy OpenAI-only debugging (still supported) |
+
+## CLI Flags
+
+```bash
+llm --http-logging [command]     # Enable INFO-level HTTP logging
+llm --http-debug [command]       # Enable DEBUG-level HTTP logging
+```
+
+## What Gets Logged
+
+### INFO Level (`LLM_HTTP_LOGGING=1`)
+
+Shows high-level HTTP request information:
+
+```
+httpx - INFO - HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
+httpx - INFO - HTTP Request: POST https://generativelanguage.googleapis.com/v1beta/models/gemini-2.5-pro:streamGenerateContent "HTTP/1.1 200 OK"
+```
+
+### DEBUG Level (`LLM_HTTP_DEBUG=1`)
+
+Shows detailed connection and protocol information:
+
+```
+httpcore.connection - DEBUG - connect_tcp.started host='api.openai.com' port=443
+httpcore.connection - DEBUG - connect_tcp.complete return_value=<httpcore._backends.sync.SyncStream object>
+httpcore.connection - DEBUG - start_tls.started ssl_context=<ssl.SSLContext object> server_hostname='api.openai.com'
+httpcore.http11 - DEBUG - send_request_headers.started request=<Request [b'POST']>
+httpcore.http11 - DEBUG - send_request_body.started request=<Request [b'POST']>
+httpcore.http11 - DEBUG - receive_response_headers.started request=<Request [b'POST']>
+httpcore.http11 - DEBUG - receive_response_headers.complete return_value=(b'HTTP/1.1', 200, b'OK', [...headers...])
+httpx - INFO - HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
+httpcore.http11 - DEBUG - receive_response_body.started request=<Request [b'POST']>
+httpcore.http11 - DEBUG - receive_response_body.complete
+```
+
+## Provider-Specific Debugging
+
+### OpenAI Models (o1, o3, GPT-4, etc.)
+
+**What you can see:**
+- Reasoning token usage: `reasoning_tokens`, `input_tokens`, `output_tokens`
+- Request parameters for reasoning models
+- Tool calling request/response cycles
+- Rate limiting headers
+
+**Example:**
+```bash
+export LLM_HTTP_DEBUG=1
+llm -m o3 "Solve this step by step: What is 15% of 240?"
+```
+Shows reasoning parameters and token usage in HTTP traffic.
+
+### Gemini Models (including 2.5-pro reasoning)
+
+**What you can see:**
+- Thinking budget configuration: `thinking_config: {"thinking_budget": 1000}`
+- Response with thinking tokens: `thoughtsTokenCount: 500`
+- Streaming response chunks
+- Model-specific parameters
+
+**Example:**
+```bash
+export LLM_HTTP_DEBUG=1
+llm -m gemini-2.5-pro "Think carefully about this complex problem"
+```
+Shows direct HTTP calls to Google's API with thinking parameters.
+
+### Anthropic Claude Models (including reasoning)
+
+**What you can see:**
+- Thinking configuration: `thinking: {"type": "enabled", "budget_tokens": 1000}`
+- Beta API usage indicators
+- Message structure and tool calls
+- Thinking parameters in requests
+
+**Example:**
+```bash
+export LLM_HTTP_DEBUG=1
+llm -m claude-4-sonnet "Analyze this problem methodically"
+```
+Shows Anthropic SDK's HTTP traffic including reasoning config.
+
+## Real-World Use Cases
+
+### Debugging Reasoning Models
+
+See what reasoning parameters are being sent:
+
+```bash
+export LLM_HTTP_DEBUG=1
+
+# OpenAI o3 reasoning
+llm -m o3 "Complex math problem" --max-tokens 1000
+
+# Gemini with thinking budget
+llm -m gemini-2.5-pro "Reasoning task" 
+
+# Claude with thinking enabled
+llm -m claude-4-sonnet "Analytical problem"
+```
+
+### Investigating API Errors
+
+Debug failed requests:
+
+```bash
+export LLM_HTTP_DEBUG=1
+llm -m gpt-4o "Test prompt"
+# Shows exact error responses, status codes, headers
+```
+
+### Understanding Token Usage
+
+See how tokens are calculated:
+
+```bash
+export LLM_HTTP_LOGGING=1
+llm -m o3-mini "Short prompt"
+# Response shows: reasoning_tokens, input_tokens, output_tokens
+```
+
+### Monitoring Rate Limits
+
+Track API usage and limits:
+
+```bash
+export LLM_HTTP_DEBUG=1
+llm -m gpt-4o "Test"
+# Shows rate limit headers in response
+```
+
+## Advanced Configuration
+
+### Combining with LLM Logging
+
+HTTP logging works alongside LLM's built-in SQLite logging:
+
+```bash
+# Enable both HTTP debugging and LLM logging
+export LLM_HTTP_LOGGING=1
+llm logs on
+llm -m gpt-4o "Test prompt"
+
+# View logged interactions
+llm logs list --json
+```
+
+### Filtering Logs
+
+Focus on specific providers or domains:
+
+```bash
+# Only log OpenAI requests
+export LLM_HTTP_DEBUG=1
+llm -m gpt-4o "Test" 2>&1 | grep "api.openai.com"
+
+# Only log Gemini requests  
+llm -m gemini-2.5-pro "Test" 2>&1 | grep "generativelanguage.googleapis.com"
+```
+
+### Performance Considerations
+
+HTTP logging adds overhead. For production use:
+
+```bash
+# Use INFO level for minimal overhead
+export LLM_HTTP_LOGGING=1
+
+# Avoid DEBUG level in production
+# export LLM_HTTP_DEBUG=1  # Don't use this in production
+```
+
+## Troubleshooting
+
+### Common Issues
+
+**No logs appearing:**
+- Verify environment variable is set: `echo $LLM_HTTP_LOGGING`
+- Check that the provider actually uses HTTP (not all providers do)
+- Some providers may use custom logging that bypasses this system
+
+**Too much output:**
+- Use `LLM_HTTP_LOGGING=1` instead of `LLM_HTTP_DEBUG=1`
+- Redirect stderr: `llm prompt "test" 2>/dev/null`
+
+**Missing reasoning details:**
+- Reasoning chains are generated server-side and not exposed in HTTP logs
+- You'll see token counts but not the actual reasoning text
+- This is a limitation of the provider APIs, not LLM CLI
+
+### Integration with External Tools
+
+**Using with jq for JSON parsing:**
+```bash
+export LLM_HTTP_DEBUG=1
+llm -m gpt-4o "test" 2>&1 | grep -A 10 "Request:" | jq '.'
+```
+
+**Saving logs to files:**
+```bash
+export LLM_HTTP_LOGGING=1
+llm -m gemini-2.5-pro "test" 2>debug.log
+```
+
+## Security Considerations
+
+⚠️ **Important Security Notes:**
+
+- HTTP logs may contain API keys in headers (these are automatically redacted)
+- Request/response bodies may contain sensitive data
+- Never commit log files containing real API interactions
+- Use environment variables instead of command-line flags in scripts to avoid shell history
+
+## Backward Compatibility
+
+The new HTTP logging system maintains full backward compatibility.
+
+### Legacy Support
+
+- `LLM_OPENAI_SHOW_RESPONSES=1` still works (OpenAI only)
+- New variables extend functionality to all providers
+- No breaking changes to existing workflows
+
+### Migration Path
+
+For users currently using OpenAI-specific debugging:
+```bash
+# Old way (still works)
+export LLM_OPENAI_SHOW_RESPONSES=1
+
+# New way (works for all providers)
+export LLM_HTTP_LOGGING=1
+```
+
+## Related Documentation
+
+- [LLM CLI Documentation](https://llm.datasette.io/)
+- [Logging and Storage](logging.md)
+- [Provider-Specific Setup](setup.md)
+- [Tools and Function Calling](tools.md)

docs/index.md

@@ -121,6 +121,7 @@ embeddings/index
 plugins/index
 python-api
 logging
+http-debugging
 related-tools
 help
 contributing

docs/logging.md

@@ -3,6 +3,8 @@
 
 `llm` defaults to logging all prompts and responses to a SQLite database.
 
+For debugging HTTP requests/responses to providers (OpenAI, Anthropic, Gemini, etc.), see {ref}`HTTP Debugging and Logging <http-debugging>`.
+
 You can find the location of that database using the `llm logs path` command:
 
 ```bash