|
| 1 | +# CLAUDE.md |
| 2 | + |
| 3 | +This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. |
| 4 | + |
| 5 | +## Project Overview |
| 6 | + |
| 7 | +Service Quality Oracle is a Python-based Docker containerized service that: |
| 8 | +- Fetches indexer performance data from Google BigQuery daily at 10:00 UTC |
| 9 | +- Processes data to determine indexer issuance rewards eligibility based on threshold algorithms |
| 10 | +- Posts eligibility updates on-chain to the ServiceQualityOracle contract |
| 11 | +- Implements resilient RPC failover with circuit breaker pattern |
| 12 | +- Sends Slack notifications for monitoring |
| 13 | + |
| 14 | +## Key Commands |
| 15 | + |
| 16 | +### Development Setup |
| 17 | +```bash |
| 18 | +# Create and activate virtual environment |
| 19 | +python3 -m venv venv |
| 20 | +source venv/bin/activate |
| 21 | + |
| 22 | +# Install dependencies |
| 23 | +pip install -r requirements.txt |
| 24 | +``` |
| 25 | + |
| 26 | +### Linting and Formatting |
| 27 | +```bash |
| 28 | +# CRITICAL: Always run this before committing - CI enforces this |
| 29 | +./scripts/ruff_check_format_assets.sh |
| 30 | + |
| 31 | +# Optional additional checks |
| 32 | +mypy src/ --ignore-missing-imports |
| 33 | +bandit -r src/ |
| 34 | +``` |
| 35 | + |
| 36 | +### Testing |
| 37 | +```bash |
| 38 | +# Run all tests with coverage |
| 39 | +pytest |
| 40 | + |
| 41 | +# Run specific test file |
| 42 | +pytest tests/test_blockchain_client.py |
| 43 | + |
| 44 | +# Run tests matching pattern |
| 45 | +pytest -k "test_failover" |
| 46 | + |
| 47 | +# Run with verbose output |
| 48 | +pytest -v |
| 49 | +``` |
| 50 | + |
| 51 | +### Docker Operations |
| 52 | +```bash |
| 53 | +# Build and run with Docker Compose |
| 54 | +docker-compose up --build -d |
| 55 | + |
| 56 | +# Monitor logs |
| 57 | +docker-compose logs -f |
| 58 | + |
| 59 | +# Check container health |
| 60 | +docker-compose ps |
| 61 | +``` |
| 62 | + |
| 63 | +### Test Notifications |
| 64 | +```bash |
| 65 | +# Test Slack webhook |
| 66 | +export SLACK_WEBHOOK_URL="your_webhook_url" |
| 67 | +./scripts/test_slack_notifications.py |
| 68 | +``` |
| 69 | + |
| 70 | +## Architecture Overview |
| 71 | + |
| 72 | +The system follows a clear data pipeline with daily scheduled execution: |
| 73 | + |
| 74 | +1. **Scheduler (src/models/scheduler.py)**: Main entry point that runs daily at configured time, manages catch-up runs for missed days, and handles the application lifecycle. |
| 75 | + |
| 76 | +2. **Orchestrator (src/models/service_quality_oracle.py)**: Coordinates the end-to-end oracle run process by managing the flow between components. |
| 77 | + |
| 78 | +3. **BigQuery Provider (src/models/bigquery_provider.py)**: Executes SQL queries against BigQuery to fetch indexer performance metrics. |
| 79 | + |
| 80 | +4. **Eligibility Pipeline (src/models/eligibility_pipeline.py)**: Processes raw data, applies threshold algorithms per ELIGIBILITY_CRITERIA.md, generates CSV artifacts for auditing. |
| 81 | + |
| 82 | +5. **Blockchain Client (src/models/blockchain_client.py)**: Handles on-chain transactions with automatic RPC provider failover, transaction batching, and gas optimization. |
| 83 | + |
| 84 | +6. **Circuit Breaker (src/utils/circuit_breaker.py)**: Prevents infinite restart loops by tracking failure patterns and halting execution when threshold exceeded. |
| 85 | + |
| 86 | +7. **Slack Notifier (src/utils/slack_notifier.py)**: Sends operational notifications for success/failure monitoring. |
| 87 | + |
| 88 | +## Critical Implementation Details |
| 89 | + |
| 90 | +### RPC Failover Strategy |
| 91 | +- Multiple RPC providers configured with automatic rotation on failure |
| 92 | +- Each provider gets 5 retry attempts with exponential backoff |
| 93 | +- Circuit breaker prevents cascade failures |
| 94 | +- Slack notifications sent on provider rotation |
| 95 | + |
| 96 | +### Configuration Management |
| 97 | +- Primary config in `config.toml` (never commit actual values) |
| 98 | +- Sensitive data via environment variables (BLOCKCHAIN_PRIVATE_KEY, etc.) |
| 99 | +- Google Cloud auth via GOOGLE_APPLICATION_CREDENTIALS or credentials.json |
| 100 | + |
| 101 | +### Data Persistence |
| 102 | +- Last successful run date stored in `/app/data/last_run.txt` |
| 103 | +- CSV outputs saved to `/app/data/output/YYYY-MM-DD/` |
| 104 | +- Catch-up mechanism limits to 7 days of historical data to control BigQuery costs |
| 105 | + |
| 106 | +### Testing Patterns |
| 107 | +- Extensive use of mocks for external services (BigQuery, Web3, Slack) |
| 108 | +- Snapshot testing for SQL queries via pytest-snapshot |
| 109 | +- Parametrized tests for edge cases |
| 110 | +- All new features require corresponding tests |
| 111 | + |
| 112 | +### Error Handling |
| 113 | +- Retryable errors handled with exponential backoff via tenacity |
| 114 | +- Non-retryable errors trigger immediate failure and notifications |
| 115 | +- All exceptions logged with context before re-raising |
| 116 | +- System exits with code 1 on failure for Docker restart |
| 117 | + |
| 118 | +## Development Guidelines |
| 119 | + |
| 120 | +### Before Making Changes |
| 121 | +1. Read ELIGIBILITY_CRITERIA.md to understand current indexer requirements |
| 122 | +2. Check existing patterns in similar files |
| 123 | +3. Run tests to ensure baseline functionality |
| 124 | + |
| 125 | +### Code Style Enforcement |
| 126 | +- Ruff configuration in pyproject.toml enforces strict formatting |
| 127 | +- Custom formatter applies additional spacing rules |
| 128 | +- Line length limit: 115 characters |
| 129 | +- Import sorting via isort rules |
| 130 | + |
| 131 | +### Git Safety Guidelines |
| 132 | +- **NEVER use `git push --force`** - this can overwrite remote history and cause data loss |
| 133 | +- **Always use `git push --force-with-lease`** if you must force push - it checks that remote hasn't changed |
| 134 | +- Prefer rebasing and clean history over force pushing when possible |
| 135 | +- Always communicate with team before any history rewriting operations |
| 136 | + |
| 137 | +### PR Requirements |
| 138 | +- Must pass `./scripts/ruff_check_format_assets.sh` without changes |
| 139 | +- All tests must pass |
| 140 | +- Docker build must succeed |
| 141 | +- Non-empty PR title and description required |
| 142 | + |
| 143 | +## Security Guidelines |
| 144 | + |
| 145 | +### ⚠️ CRITICAL: This is a PUBLIC repository |
| 146 | + |
| 147 | +### Sensitive Information Handling |
| 148 | +- **NEVER commit secrets, keys, or credentials** to the repository |
| 149 | +- **NEVER commit actual config.toml files** - only commit config.toml.example with placeholder values |
| 150 | +- Always follow security best practices when handling private keys and API tokens |
| 151 | +- Never introduce code that exposes or logs secrets and keys |
| 152 | +- Use environment variables for sensitive configuration (see config.toml.example) |
| 153 | +- Verify that .gitignore properly excludes sensitive files before committing |
| 154 | + |
| 155 | +### Protected File Types (automatically ignored by .gitignore) |
| 156 | +- Environment files: `.env`, `.env.*` |
| 157 | +- Configuration: `config.toml`, `credentials.json` |
| 158 | +- Keys/Certificates: `*.key`, `*.pem`, `*.p12`, `*.pfx` |
| 159 | +- Secret directories: `**/secrets/`, `**/private/`, `.secrets` |
| 160 | + |
| 161 | +### Security Checklist Before Commits |
| 162 | +1. ✅ Run `git status` and verify no sensitive files are staged |
| 163 | +2. ✅ Review `git diff --cached` for any accidentally included secrets |
| 164 | +3. ✅ Ensure all examples use placeholder values (e.g., `"your_key_here"`) |
| 165 | +4. ✅ Never hardcode real addresses, URLs, or API endpoints |
| 166 | +5. ✅ Use descriptive placeholder text that shows expected format |
| 167 | + |
| 168 | +### Emergency: If Sensitive Data is Committed |
| 169 | +1. **DO NOT** just delete the file in a new commit - the data remains in Git history |
| 170 | +2. Immediately rotate/invalidate the exposed credentials |
| 171 | +3. Contact repository administrators to discuss history rewriting if necessary |
| 172 | +4. Consider the credential permanently compromised |
| 173 | + |
| 174 | +# Session Context Import |
| 175 | +@./SESSION_CONTEXT.md |
0 commit comments