Skip to content
This repository was archived by the owner on Jan 2, 2026. It is now read-only.

Commit 1d2e988

Browse files
zircoteclaude
andauthored
feat: Observability instrumentation and distributed tracing (#24)
* feat: initialize observability instrumentation implementation - Create PROGRESS.md with 29 tasks across 6 phases - Update README.md with started timestamp - Spec approved and ready for implementation Implements: SPEC-2025-12-25-001 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * feat(observability): implement Phase 1 core infrastructure Complete observability foundation with: Core Modules: - config.py: ObservabilityConfig with env-based configuration - metrics.py: Thread-safe MetricsCollector (counters, histograms, gauges) - tracing.py: Span context with contextvars-based propagation - session.py: SessionIdentifier with privacy-preserving hashes - decorators.py: measure_duration decorator (sync/async support) - logging.py: StructuredLogger with JSON/text formatters Exporters: - prometheus.py: Prometheus text format (no external deps) - json_exporter.py: Full JSON export for metrics and traces Key Features: - Lazy imports via __getattr__ for hook performance (<5ms import) - Thread-safe operations with threading.Lock - Privacy-preserving SHA256 hashes (8 char truncated) - Histogram buckets aligned with hook timeouts (2s, 5s, 15s, 30s) - Rolling window for bounded memory in histograms Tests: - 115 new tests across 8 test modules - Coverage: 87.76% (above 80% threshold) - All quality gates passing Closes Phase 1 of SPEC-2025-12-25-001 (Observability Instrumentation) Ref: #10 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * feat: Complete observability instrumentation (Phases 1-4) Implement comprehensive observability for git-notes-memory plugin: Phase 1 - Core Infrastructure: - MetricsCollector with counters, histograms, gauges (thread-safe) - SpanContext and trace_operation for distributed tracing - StructuredLogger with JSON/text formatters - SessionIdentifier for multi-tenant distinguishability - measure_duration decorator and timed_context Phase 2 - Service Instrumentation: - Instrument CaptureService, RecallService, EmbeddingService - Instrument IndexService, GitOps, Hook Handlers - Fix silent failure points with explicit logging and metrics - Add timed_hook_execution context manager Phase 3 - Structured Logging: - Add *args support to StructuredLogger for backwards compatibility - Migrate all 5 hook handlers to get_logger() - Migrate 7 hook support modules to structured logging Phase 4 - CLI & Export: - /memory:metrics command with text/json/prometheus formats - /memory:traces command with filtering options - /memory:health command with --timing for latency percentiles - PrometheusExporter class for Prometheus text format Phases 5-6 (OpenTelemetry, Docker stack) skipped as optional Tier 3. Closes #10 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: add exponential backoff and observability documentation MAXALL deep-clean remediation: - CRIT-001: Add exponential backoff with jitter to lock acquisition - Replaces fixed 100ms retry with 50ms→2s exponential backoff - Adds 0-10% jitter to prevent thundering herd - Reduces resource exhaustion risk under high concurrency - HIGH-004: Add composite index on (namespace, timestamp DESC) - Improves range query performance for get_by_namespace() - HIGH-014: Create docs/observability.md documentation - Metrics collection and Prometheus export - Tracing with span hierarchies - Structured logging configuration - Health checks and performance characteristics Code review artifacts in docs/code-review/2025/12/25/: - CODE_REVIEW.md: 90 findings from 9 specialist agents - REVIEW_SUMMARY.md: Executive summary - REMEDIATION_TASKS.md: Actionable checklist - REMEDIATION_REPORT.md: Final status with false positives All 1949 tests passing, 87.70% coverage. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * feat(observability): add Docker Compose local observability stack Phase 6 implementation (tasks 6.1-6.2): - docker-compose.yml with OTEL Collector, Prometheus, Tempo, Loki, Grafana - OpenTelemetry Collector config with OTLP receivers and exporters - Prometheus scrape configuration for OTEL metrics - Tempo distributed tracing backend configuration - Loki log aggregation configuration - Grafana datasource provisioning (Prometheus, Tempo, Loki) - Grafana dashboard provisioning with auto-discovery - memory-operations.json dashboard (captures, searches, latency) - hook-performance.json dashboard (executions, timeouts, p95 latency) Exposed ports: - 3000: Grafana UI - 9090: Prometheus UI - 4317/4318: OTEL gRPC/HTTP - 3100: Loki - 3200: Tempo Usage: cd docker && docker-compose up -d 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix(security): divest inline Python from command files into scripts Addresses PR #24 review comments: - SECURITY: Moved inline Python code from commands/*.md to scripts/*.py to prevent command injection via $ARGUMENTS string interpolation - scripts/metrics.py: Safe argument parsing via argparse/sys.argv - scripts/traces.py: Safe argument parsing via argparse/sys.argv - scripts/health.py: Safe argument parsing via argparse/sys.argv - Updated commands/metrics.md, traces.md, health.md to call scripts - Fixed exporters/__init__.py: explicit imports instead of lazy loading to resolve Copilot undefined export warnings The previous implementation embedded $ARGUMENTS directly into Python string literals, allowing code injection via crafted arguments. The new scripts pass arguments safely via command line. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * feat(observability): implement OTLP HTTP exporter for telemetry push Completes Phase 5 of the observability spec by adding OTLP HTTP export capability so telemetry can be pushed to OpenTelemetry Collector and the Docker Compose observability stack. Changes: - New `exporters/otlp.py` with OTLPExporter class using stdlib only - Converts internal Span/metrics to OTLP JSON format - Pushes to {endpoint}/v1/traces and {endpoint}/v1/metrics - Configurable via MEMORY_PLUGIN_OTLP_ENDPOINT env var - Updated Stop handler to flush telemetry at session end - Updated observability.md with OTLP configuration section - Added comprehensive tests for OTLP exporter - pyproject.toml: added S310 per-file-ignore for OTLP file The exporter uses urllib.request (stdlib) to avoid adding dependencies. When MEMORY_PLUGIN_OTLP_ENDPOINT is set, traces and metrics are automatically exported when the Claude Code session ends. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
1 parent c9f358e commit 1d2e988

69 files changed

Lines changed: 11147 additions & 1986 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

CLAUDE.md

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -286,6 +286,15 @@ LSP hooks are configured in `.claude/hooks.json` for immediate feedback on Pytho
286286

287287
## Completed Spec Projects
288288

289+
- `docs/spec/completed/2025-12-25-observability-instrumentation/` - Observability Instrumentation
290+
- Completed: 2025-12-26
291+
- Outcome: success
292+
- GitHub Issue: [#10](https://github.com/zircote/git-notes-memory/issues/10)
293+
- Features: Metrics collection, distributed tracing, structured logging, CLI commands (/metrics, /traces, /health)
294+
- Deliverables: 115+ tests, 4 phases completed (6 total, 2 optional skipped), 20 tasks, 11 ADRs
295+
- Note: Phases 5-6 (OpenTelemetry, Docker stack) skipped as optional Tier 3 enhancements
296+
- Key docs: REQUIREMENTS.md, ARCHITECTURE.md, IMPLEMENTATION_PLAN.md, DECISIONS.md, PROGRESS.md
297+
289298
- `docs/spec/completed/2025-12-25-fix-git-notes-fetch-refspec/` - Fix Git Notes Fetch Refspec
290299
- Completed: 2025-12-25
291300
- Outcome: success

commands/health.md

Lines changed: 135 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,135 @@
1+
---
2+
description: Health check with optional timing percentiles
3+
argument-hint: "[--timing] [--verbose]"
4+
allowed-tools: ["Bash", "Read"]
5+
---
6+
7+
<help_check>
8+
## Help Check
9+
10+
If `$ARGUMENTS` contains `--help` or `-h`:
11+
12+
**Output this help and HALT (do not proceed further):**
13+
14+
<help_output>
15+
```
16+
HEALTH(1) User Commands HEALTH(1)
17+
18+
NAME
19+
health - Health check with optional timing percentiles
20+
21+
SYNOPSIS
22+
/memory:health [--timing] [--verbose]
23+
24+
DESCRIPTION
25+
Perform a quick health check of the memory system. Shows component status
26+
and optionally includes latency percentiles from collected metrics.
27+
28+
OPTIONS
29+
--help, -h Show this help message
30+
--timing Include latency percentiles from metrics
31+
--verbose Show detailed component status
32+
33+
EXAMPLES
34+
/memory:health
35+
/memory:health --timing
36+
/memory:health --verbose
37+
/memory:health --timing --verbose
38+
/memory:health --help
39+
40+
SEE ALSO
41+
/memory:status for detailed system status
42+
/memory:metrics for all collected metrics
43+
/memory:traces for recent operation traces
44+
45+
HEALTH(1)
46+
```
47+
</help_output>
48+
49+
**After outputting help, HALT immediately. Do not proceed with command execution.**
50+
</help_check>
51+
52+
---
53+
54+
# /memory:health - System Health Check
55+
56+
Quick health check of the memory system with optional timing information.
57+
58+
## Your Task
59+
60+
You will check the health of the memory system components.
61+
62+
<step number="1" name="Parse Arguments">
63+
64+
**Arguments format**: `$ARGUMENTS`
65+
66+
Parse the following options:
67+
- `--timing` - Include latency percentiles
68+
- `--verbose` - Show detailed component status
69+
70+
</step>
71+
72+
<step number="2" name="Run Health Check">
73+
74+
**Execute the health check**:
75+
```bash
76+
PLUGIN_ROOT="${CLAUDE_PLUGIN_ROOT:-$(ls -d ~/.claude/plugins/cache/git-notes-memory/memory-capture/*/ 2>/dev/null | head -1)}"
77+
uv run --directory "$PLUGIN_ROOT" python3 "$PLUGIN_ROOT/scripts/health.py" $ARGUMENTS
78+
```
79+
80+
</step>
81+
82+
<step number="3" name="Provide Recommendations">
83+
84+
If issues are detected, show recommendations:
85+
86+
```
87+
### Recommendations
88+
89+
- **Index not initialized** - Run `/memory:sync` to create the index
90+
- **Embedding model not loaded** - First search may be slow while model loads
91+
- **Git notes not configured** - Run `/memory:capture` to create first memory
92+
- **High timeout rate** - Consider increasing hook timeouts in environment
93+
```
94+
95+
</step>
96+
97+
## Output Sections
98+
99+
| Section | Description |
100+
|---------|-------------|
101+
| Components | Status of each system component |
102+
| Latency Percentiles | Timing metrics (with --timing) |
103+
| Hook Timeout Rate | Percentage of timed-out hooks |
104+
| Component Details | Detailed stats (with --verbose) |
105+
106+
## Status Indicators
107+
108+
| Symbol | Meaning |
109+
|--------|---------|
110+
|| Healthy/OK |
111+
|| Error/Failed |
112+
|| Warning/Issues |
113+
114+
## Examples
115+
116+
**User**: `/memory:health`
117+
**Action**: Quick health check of all components
118+
119+
**User**: `/memory:health --timing`
120+
**Action**: Health check with latency percentiles
121+
122+
**User**: `/memory:health --verbose`
123+
**Action**: Health check with detailed component info
124+
125+
**User**: `/memory:health --timing --verbose`
126+
**Action**: Full health check with all details
127+
128+
## Related Commands
129+
130+
| Command | Description |
131+
|---------|-------------|
132+
| `/memory:status` | Detailed system status |
133+
| `/memory:metrics` | View all metrics |
134+
| `/memory:traces` | Recent operation traces |
135+
| `/memory:validate` | Full validation |

commands/metrics.md

Lines changed: 161 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,161 @@
1+
---
2+
description: Display observability metrics for the memory system
3+
argument-hint: "[--format=text|json|prometheus] [--filter=<pattern>]"
4+
allowed-tools: ["Bash", "Read"]
5+
---
6+
7+
<help_check>
8+
## Help Check
9+
10+
If `$ARGUMENTS` contains `--help` or `-h`:
11+
12+
**Output this help and HALT (do not proceed further):**
13+
14+
<help_output>
15+
```
16+
METRICS(1) User Commands METRICS(1)
17+
18+
NAME
19+
metrics - Display observability metrics for the memory system
20+
21+
SYNOPSIS
22+
/memory:metrics [--format=text|json|prometheus] [--filter=<pattern>]
23+
24+
DESCRIPTION
25+
Display collected observability metrics including counters, histograms, and gauges.
26+
Metrics track operation counts, durations, errors, and system health indicators.
27+
28+
OPTIONS
29+
--help, -h Show this help message
30+
--format=FORMAT Output format: text (default), json, prometheus
31+
--filter=PATTERN Filter metrics by name pattern (e.g., "capture", "hook")
32+
33+
EXAMPLES
34+
/memory:metrics
35+
/memory:metrics --format=json
36+
/memory:metrics --format=prometheus
37+
/memory:metrics --filter=hook
38+
/memory:metrics --format=json --filter=capture
39+
/memory:metrics --help
40+
41+
SEE ALSO
42+
/memory:status for system status
43+
/memory:health for health checks with timing
44+
45+
METRICS(1)
46+
```
47+
</help_output>
48+
49+
**After outputting help, HALT immediately. Do not proceed with command execution.**
50+
</help_check>
51+
52+
---
53+
54+
# /memory:metrics - Observability Metrics
55+
56+
Display collected observability metrics for the memory system.
57+
58+
## Your Task
59+
60+
You will display metrics collected by the observability system.
61+
62+
<step number="1" name="Parse Arguments">
63+
64+
**Arguments format**: `$ARGUMENTS`
65+
66+
Parse the following options:
67+
- `--format=text|json|prometheus` - Output format (default: text)
68+
- `--filter=<pattern>` - Filter metrics by name pattern
69+
70+
</step>
71+
72+
<step number="2" name="Collect and Display Metrics">
73+
74+
**Execute the metrics collection**:
75+
```bash
76+
PLUGIN_ROOT="${CLAUDE_PLUGIN_ROOT:-$(ls -d ~/.claude/plugins/cache/git-notes-memory/memory-capture/*/ 2>/dev/null | head -1)}"
77+
uv run --directory "$PLUGIN_ROOT" python3 "$PLUGIN_ROOT/scripts/metrics.py" $ARGUMENTS
78+
```
79+
80+
</step>
81+
82+
<step number="3" name="Explain Output">
83+
84+
After displaying metrics, provide context:
85+
86+
**For text format**:
87+
```
88+
### Metric Types
89+
90+
- **Counters**: Cumulative values that only increase (e.g., memories_captured_total)
91+
- **Histograms**: Distribution of values (e.g., capture_duration_ms)
92+
- **Gauges**: Current values that can go up or down (e.g., index_size_bytes)
93+
94+
Use `--format=prometheus` for Prometheus/Grafana scraping.
95+
```
96+
97+
**For JSON format**:
98+
```
99+
### JSON Structure
100+
101+
The output contains:
102+
- `counters`: Name → value pairs
103+
- `histograms`: Name → {count, sum, avg, p50, p95, p99}
104+
- `gauges`: Name → current value
105+
```
106+
107+
**For Prometheus format**:
108+
```
109+
### Prometheus Format
110+
111+
Ready for scraping by Prometheus. Each metric includes:
112+
- TYPE declaration (counter, histogram, gauge)
113+
- HELP description
114+
- Labels in {key="value"} format
115+
```
116+
117+
</step>
118+
119+
## Output Formats
120+
121+
| Format | Use Case |
122+
|--------|----------|
123+
| text | Human-readable output (default) |
124+
| json | Machine parsing, debugging |
125+
| prometheus | Prometheus/Grafana scraping |
126+
127+
## Available Metrics
128+
129+
| Metric | Type | Description |
130+
|--------|------|-------------|
131+
| memories_captured_total | counter | Total memories captured by namespace |
132+
| capture_duration_ms | histogram | Capture operation timing |
133+
| recall_duration_ms | histogram | Search/recall timing |
134+
| hook_execution_duration_ms | histogram | Hook handler timing |
135+
| hook_executions_total | counter | Hook invocations by name and status |
136+
| hook_timeouts_total | counter | Hook timeout events |
137+
| silent_failures_total | counter | Logged silent failures by location |
138+
| embedding_duration_ms | histogram | Embedding generation timing |
139+
| index_operations_total | counter | Index CRUD operations |
140+
141+
## Examples
142+
143+
**User**: `/memory:metrics`
144+
**Action**: Show all metrics in text format
145+
146+
**User**: `/memory:metrics --format=json`
147+
**Action**: Show all metrics as JSON
148+
149+
**User**: `/memory:metrics --filter=hook`
150+
**Action**: Show only hook-related metrics
151+
152+
**User**: `/memory:metrics --format=prometheus`
153+
**Action**: Show Prometheus exposition format for scraping
154+
155+
## Related Commands
156+
157+
| Command | Description |
158+
|---------|-------------|
159+
| `/memory:status` | System status and statistics |
160+
| `/memory:health` | Health checks with timing |
161+
| `/memory:validate` | Full system validation |

0 commit comments

Comments
 (0)