Skip to content

Commit 8affd09

Browse files
haasonsaasclaude
haasonsaas
and
claude
committed
Major feature additions: Record/Replay, Cost Tracking, Error Mocking
Implemented critical missing features: - Full record/replay functionality for API interactions - Cost tracking showing money saved by using mocks - Error scenario mocking (rate limits, auth errors, timeouts) - Docker support for easy deployment - Integration tests for end-to-end validation - Request/response persistence system - Enhanced server with multiple modes New Components: - recorder.py: Records and replays API interactions - cost_tracker.py: Tracks API cost savings with detailed reports - Error mocking in core rules system - Docker/docker-compose configuration - Comprehensive integration tests This brings Mocktopus much closer to production readiness with key features needed for real-world usage. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
1 parent 3c96a23 commit 8affd09

File tree

11 files changed

+1479
-17
lines changed

11 files changed

+1479
-17
lines changed

‎Dockerfile‎

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
# Multi-stage build for smaller image
2+
FROM python:3.11-slim as builder
3+
4+
WORKDIR /app
5+
6+
# Install build dependencies
7+
RUN apt-get update && apt-get install -y \
8+
gcc \
9+
&& rm -rf /var/lib/apt/lists/*
10+
11+
# Copy requirements
12+
COPY pyproject.toml ./
13+
COPY src/ ./src/
14+
15+
# Install package
16+
RUN pip install --no-cache-dir -e .
17+
18+
# Runtime stage
19+
FROM python:3.11-slim
20+
21+
WORKDIR /app
22+
23+
# Copy installed package from builder
24+
COPY --from=builder /usr/local/lib/python3.11/site-packages /usr/local/lib/python3.11/site-packages
25+
COPY --from=builder /usr/local/bin/mocktopus /usr/local/bin/mocktopus
26+
27+
# Copy source code (for editable install)
28+
COPY src/ ./src/
29+
COPY pyproject.toml ./
30+
COPY examples/ ./examples/
31+
32+
# Create directories for recordings and scenarios
33+
RUN mkdir -p /data/recordings /data/scenarios
34+
35+
# Install runtime package
36+
RUN pip install --no-cache-dir -e .
37+
38+
# Expose default port
39+
EXPOSE 8080
40+
41+
# Health check
42+
HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
43+
CMD python -c "import urllib.request; urllib.request.urlopen('http://localhost:8080/health')" || exit 1
44+
45+
# Default scenario location
46+
ENV SCENARIO_PATH=/data/scenarios/scenario.yaml
47+
ENV RECORDINGS_DIR=/data/recordings
48+
49+
# Run server by default
50+
CMD ["mocktopus", "serve", "-s", "${SCENARIO_PATH}", "--host", "0.0.0.0"]

‎ROADMAP.md‎

Lines changed: 146 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,146 @@
1+
# 🗺️ Mocktopus Roadmap
2+
3+
## Current State (v0.1.0)
4+
✅ Basic HTTP server mimicking OpenAI/Anthropic APIs
5+
✅ YAML-based scenarios with pattern matching
6+
✅ Streaming support (SSE)
7+
✅ Tool/function calling
8+
✅ CLI with serve, validate, simulate commands
9+
10+
## Phase 1: Core Features (v0.2.0) - Q1 2024
11+
### Record & Replay
12+
- [ ] Implement request proxy to real APIs
13+
- [ ] Store interactions in SQLite/JSON
14+
- [ ] Intelligent replay matching
15+
- [ ] Sensitive data filtering
16+
- [ ] Compression for stored data
17+
18+
### Additional APIs
19+
- [ ] Embeddings API (/v1/embeddings)
20+
- [ ] Legacy Completions API (/v1/completions)
21+
- [ ] Models endpoint with proper data
22+
- [ ] Error response mocking
23+
24+
### Testing Improvements
25+
- [ ] Integration tests with OpenAI SDK
26+
- [ ] Integration tests with Anthropic SDK
27+
- [ ] Performance benchmarks
28+
- [ ] Load testing capabilities
29+
30+
## Phase 2: Intelligence (v0.3.0) - Q2 2024
31+
### Semantic Matching
32+
- [ ] Vector similarity matching using embeddings
33+
- [ ] Fuzzy matching with configurable thresholds
34+
- [ ] Intent-based routing
35+
- [ ] Context-aware responses
36+
37+
### Stateful Conversations
38+
- [ ] Conversation state tracking
39+
- [ ] Multi-turn dialogue support
40+
- [ ] Variable extraction and storage
41+
- [ ] Conditional response logic
42+
43+
### Response Templating
44+
- [ ] Jinja2-style templates
45+
- [ ] Dynamic variable injection
46+
- [ ] Helper functions (uuid, timestamp, random)
47+
- [ ] Request data access in templates
48+
49+
## Phase 3: Developer Experience (v0.4.0) - Q2 2024
50+
### Web Dashboard
51+
- [ ] Real-time request inspector
52+
- [ ] Visual scenario builder
53+
- [ ] Mock rule debugger
54+
- [ ] Performance metrics dashboard
55+
- [ ] Cost tracking visualization
56+
57+
### SDK Integrations
58+
- [ ] LangChain integration & examples
59+
- [ ] LlamaIndex integration & examples
60+
- [ ] Vercel AI SDK examples
61+
- [ ] Haystack integration
62+
- [ ] AutoGen examples
63+
64+
### Deployment
65+
- [ ] Docker image with multi-arch support
66+
- [ ] Kubernetes Helm chart
67+
- [ ] GitHub Action for CI integration
68+
- [ ] Cloud Run button
69+
- [ ] Railway/Render templates
70+
71+
## Phase 4: Advanced Features (v0.5.0) - Q3 2024
72+
### Assistants API
73+
- [ ] Full Assistants API support
74+
- [ ] Thread management
75+
- [ ] File handling
76+
- [ ] Code interpreter mocking
77+
- [ ] Function calling in assistants
78+
79+
### Vision & Audio
80+
- [ ] Image input support
81+
- [ ] Vision API mocking
82+
- [ ] Audio transcription mocking
83+
- [ ] TTS mocking
84+
85+
### Chaos Engineering
86+
- [ ] Random failure injection
87+
- [ ] Latency simulation
88+
- [ ] Partial failures
89+
- [ ] Network issues simulation
90+
- [ ] Rate limit simulation
91+
92+
## Phase 5: Enterprise (v1.0.0) - Q4 2024
93+
### Security & Compliance
94+
- [ ] Authentication (API keys, JWT)
95+
- [ ] Request filtering/whitelisting
96+
- [ ] Audit logging
97+
- [ ] PII detection and masking
98+
- [ ] Compliance reporting
99+
100+
### Scalability
101+
- [ ] Distributed mode with Redis
102+
- [ ] Horizontal scaling support
103+
- [ ] Connection pooling
104+
- [ ] Cache layer
105+
- [ ] Performance optimizations
106+
107+
### Observability
108+
- [ ] Prometheus metrics
109+
- [ ] OpenTelemetry support
110+
- [ ] Detailed logging
111+
- [ ] Health check endpoints
112+
- [ ] Performance profiling
113+
114+
## Future Ideas (v2.0+)
115+
- **WebSocket Support**: Real-time streaming applications
116+
- **GraphQL Mocking**: For GraphQL-based LLM APIs
117+
- **Plugin Marketplace**: Community-contributed plugins
118+
- **Cloud Service**: Hosted Mocktopus SaaS
119+
- **Test Generation**: Auto-generate test scenarios from production logs
120+
- **Smart Fuzzing**: Automatic edge case discovery
121+
- **Multi-Language SDKs**: Go, Rust, Java, Ruby clients
122+
- **OpenAPI Generator**: Generate mocks from OpenAPI specs
123+
- **Behavior Learning**: Learn patterns from real API usage
124+
- **Cost Optimization**: Suggest cheaper model alternatives
125+
126+
## Contributing
127+
Want to help? Check our [CONTRIBUTING.md](CONTRIBUTING.md) for:
128+
- 🐛 Bug fixes
129+
- ✨ Feature implementations
130+
- 📚 Documentation improvements
131+
- 🧪 Test coverage
132+
- 🎨 UI/UX improvements
133+
134+
## Metrics for Success
135+
- **Adoption**: 1000+ GitHub stars
136+
- **Usage**: 100+ companies using in CI/CD
137+
- **Performance**: <10ms response time for mocks
138+
- **Coverage**: 100% OpenAI API compatibility
139+
- **Reliability**: 99.9% uptime for hosted version
140+
- **Cost Savings**: $1M+ saved by users annually
141+
142+
## Get Involved
143+
- 💬 [Discord Community](https://discord.gg/mocktopus)
144+
- 🐦 [Twitter Updates](https://twitter.com/mocktopus)
145+
- 📧 [Newsletter](https://mocktopus.dev/newsletter)
146+
- 🎥 [YouTube Tutorials](https://youtube.com/@mocktopus)

‎docker-compose.yml‎

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
version: '3.8'
2+
3+
services:
4+
mocktopus:
5+
build: .
6+
image: mocktopus:latest
7+
container_name: mocktopus
8+
ports:
9+
- "8080:8080"
10+
volumes:
11+
# Mount scenarios directory
12+
- ./examples:/data/scenarios:ro
13+
# Mount recordings directory for record/replay
14+
- ./recordings:/data/recordings
15+
environment:
16+
# Default to mock mode
17+
- MODE=mock
18+
# Use chat-basic.yaml as default scenario
19+
- SCENARIO_PATH=/data/scenarios/chat-basic.yaml
20+
# Optional: Set API keys for record mode
21+
# - OPENAI_API_KEY=your-key-here
22+
# - ANTHROPIC_API_KEY=your-key-here
23+
command: >
24+
mocktopus serve
25+
-s ${SCENARIO_PATH}
26+
--mode ${MODE}
27+
--recordings-dir /data/recordings
28+
--host 0.0.0.0
29+
-v
30+
restart: unless-stopped
31+
networks:
32+
- mocktopus-net
33+
34+
# Optional: Add a test client service
35+
test-client:
36+
image: python:3.11-slim
37+
container_name: mocktopus-test
38+
depends_on:
39+
- mocktopus
40+
volumes:
41+
- ./tests:/tests
42+
environment:
43+
- MOCKTOPUS_URL=http://mocktopus:8080
44+
command: sleep infinity
45+
networks:
46+
- mocktopus-net
47+
48+
networks:
49+
mocktopus-net:
50+
driver: bridge
51+
52+
volumes:
53+
recordings:

‎examples/errors.yaml‎

Lines changed: 84 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,84 @@
1+
version: 1
2+
meta:
3+
description: Error scenario examples for testing error handling
4+
author: Mocktopus
5+
6+
rules:
7+
# Rate limit error
8+
- type: llm.openai
9+
when:
10+
messages_contains: "rate limit test"
11+
error:
12+
error_type: "rate_limit"
13+
message: "Rate limit exceeded. Please retry after some time."
14+
status_code: 429
15+
retry_after: 60
16+
delay_ms: 100
17+
18+
# Authentication error
19+
- type: llm.openai
20+
when:
21+
messages_contains: "auth test"
22+
error:
23+
error_type: "authentication"
24+
message: "Invalid API key provided"
25+
status_code: 401
26+
27+
# Invalid request error
28+
- type: llm.openai
29+
when:
30+
messages_contains: "invalid request"
31+
error:
32+
error_type: "invalid_request"
33+
message: "The model 'gpt-5' does not exist"
34+
status_code: 400
35+
36+
# Timeout simulation
37+
- type: llm.openai
38+
when:
39+
messages_contains: "timeout test"
40+
error:
41+
error_type: "timeout"
42+
message: "Request timed out after 30 seconds"
43+
status_code: 504
44+
delay_ms: 5000 # Wait 5 seconds then return timeout
45+
46+
# Server error
47+
- type: llm.openai
48+
when:
49+
messages_contains: "server error"
50+
error:
51+
error_type: "server_error"
52+
message: "Internal server error occurred"
53+
status_code: 500
54+
55+
# Intermittent error (fails 2 times, then works)
56+
- type: llm.openai
57+
when:
58+
messages_contains: "retry test"
59+
times: 2
60+
error:
61+
error_type: "server_error"
62+
message: "Temporary server issue"
63+
status_code: 503
64+
65+
# After errors exhausted, this rule handles success
66+
- type: llm.openai
67+
when:
68+
messages_contains: "retry test"
69+
respond:
70+
content: "Success after retries!"
71+
usage:
72+
input_tokens: 10
73+
output_tokens: 5
74+
75+
# Custom error
76+
- type: llm.openai
77+
when:
78+
messages_contains: "custom error"
79+
error:
80+
error_type: "content_filter"
81+
message: "Content was blocked by safety filters"
82+
code: "content_filtered"
83+
status_code: 422
84+
delay_ms: 200

‎pyproject.toml‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,5 +30,12 @@ packages = ["src/mocktopus"]
3030
line-length = 100
3131
target-version = "py39"
3232

33+
[project.optional-dependencies]
34+
test = [
35+
"pytest>=7.0",
36+
"pytest-asyncio>=0.21",
37+
]
38+
3339
[tool.pytest.ini_options]
3440
addopts = "-q"
41+
asyncio_mode = "auto"

‎src/mocktopus/core.py‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@ class Rule:
1616
when: Dict[str, Any]
1717
respond: Dict[str, Any]
1818
times: Optional[int] = None # None => unlimited
19+
error: Optional[Dict[str, Any]] = None # Error response config
1920
_remaining: Optional[int] = field(default=None, init=False, repr=False)
2021

2122
def ok_to_use(self) -> bool:
@@ -101,7 +102,9 @@ def _llm_rule_matches(self, rule: Rule, *, model: str, messages: List[Dict[str,
101102
def find_llm(self, *, model: str, messages: List[Dict[str, Any]]) -> Tuple[Optional[Rule], Optional[Dict[str, Any]]]:
102103
for rule in self.rules:
103104
if self._llm_rule_matches(rule, model=model, messages=messages):
104-
return rule, rule.respond
105+
# Return error config if present, otherwise normal response
106+
response_config = rule.error if rule.error else rule.respond
107+
return rule, response_config
105108
return None, None
106109

107110
# --- YAML I/O ---------------------------------------------------------
@@ -122,6 +125,7 @@ def from_yaml(cls, path: str) -> "Scenario":
122125
when=r.get("when", {}) or {},
123126
respond=r.get("respond", {}) or {},
124127
times=r.get("times"),
128+
error=r.get("error"),
125129
)
126130
)
127131
meta = data.get("meta", {})

0 commit comments

Comments
 (0)