From fd25c8122a5ce6252e7ef4a3558226e49b9f54a3 Mon Sep 17 00:00:00 2001 From: Christian Assing Date: Tue, 10 Mar 2026 15:01:35 +0100 Subject: [PATCH 1/2] Add migrate-integration and document-api-integration skills - New skill: migrate-integration - guides migration of reconcile/ integrations to qontract-api architecture (6 phases: discovery, utils, server-side, external endpoints, client-side, tests+docs) - New skill: document-api-integration - converted from command to skill for auto-triggering support - Simplified document-api-integration command to delegate to skill - New command: /migrate-integration as shortcut for the skill --- .claude/commands/document-api-integration.md | 178 +------ .claude/commands/migrate-integration.md | 6 + .../skills/document-api-integration/SKILL.md | 54 +++ .claude/skills/migrate-integration/SKILL.md | 458 ++++++++++++++++++ 4 files changed, 519 insertions(+), 177 deletions(-) create mode 100644 .claude/commands/migrate-integration.md create mode 100644 .claude/skills/document-api-integration/SKILL.md create mode 100644 .claude/skills/migrate-integration/SKILL.md diff --git a/.claude/commands/document-api-integration.md b/.claude/commands/document-api-integration.md index 86f30b4e86..f7705ce94b 100644 --- a/.claude/commands/document-api-integration.md +++ b/.claude/commands/document-api-integration.md @@ -3,180 +3,4 @@ description: Document an existing qontract-api integration scope: project --- -# Document qontract-api Integration - -You are helping document an existing qontract-api integration for the qontract-reconcile developer documentation. - -## Purpose - -This command analyzes an existing qontract-api integration and generates comprehensive documentation that will be added to `docs/integrations/`. The goal is to help new developers understand: - -- What the integration does -- How it works (architecture) -- What features it provides -- What limitations it has -- What components are required -- How to use it - -## Template - -Use the template from `docs/integrations/template.md` as the structure for all integration documentation. - -## Workflow - -1. **Ask for integration name** - - Prompt user for the integration name (e.g., "slack-usergroups") - - Validate that the integration exists in `qontract_api/qontract_api/integrations//` - -2. **Read the template** - - Read `docs/integrations/template.md` to understand the structure - - Use this template as the basis for the documentation - -3. **Analyze the integration** by reading: - - **Server-side code:** - - `qontract_api/qontract_api/integrations//models.py` - Request/Response/Action models - - `qontract_api/qontract_api/integrations//service.py` - Business logic - - `qontract_api/qontract_api/integrations//router.py` - API endpoints - - Other files in the integration directory (factory, client, etc.) - - **Client-side code:** - - `reconcile/_api.py` - Client integration - - **Related ADRs:** - - Check which ADRs are referenced in docstrings/comments - - Read relevant ADRs for architectural context - -4. **Extract information:** - - Extract all information needed to fill the template sections: - - **Features:** - - What does the integration do? (from docstrings, service logic) - - What actions can it perform? (from action models) - - What resources does it manage? (from models) - - **API Endpoints:** - - List all endpoints (POST, GET) with paths - - Document parameters and responses - - **Request/Response Models:** - - Document the Pydantic model structure - - Include key fields with descriptions - - Note validation rules (e.g., dry_run default=True) - - **Actions:** - - List all action types (from discriminated union models) - - Describe what each action does - - **Architecture:** - - Client-side responsibilities (GraphQL, desired state) - - Server-side responsibilities (current state, diff, actions) - - External API dependencies - - **Limits/Constraints:** - - Dry-run default (safety) - - Managed resources (e.g., managed_usergroups whitelist) - - Rate limits - - Cache TTLs - - Any other constraints - - **Required Components:** - - Vault token paths - - External API credentials - - Environment variables - - Cache backend - -5. **Generate documentation following the template:** - - Use `docs/integrations/template.md` as the exact structure - - Fill in all sections with information extracted from the code - - Keep the template structure intact - do not skip sections - - If a section is not applicable, note "[Not applicable for this integration]" - - Follow the exact markdown formatting from the template - -6. **Create integration documentation file:** - - Create directory if needed: `docs/integrations/` - - Create file: `docs/integrations/.md` - - Use the filled template - - Include all sections from the template - -7. **Update README.md:** - - Read and update `docs/integrations/README.md` accordingly - - Look for "## Integrations" section - - If exists: Add link to the new integration documentation - - If not exists: Create section after "## Documentation" section with: - - ```markdown - ## Integrations - - Available integrations: - - - [Integration Name](docs/integrations/.md) - Brief description - ``` - - - Add link in alphabetical order - - Keep existing content intact - -8. **Output:** - - Show a summary of the generated documentation to the user - - Ask for confirmation before writing files - - Write `docs/integrations/.md` - - Update `docs/integrations/README.md` with link to the new documentation - -## Important Guidelines - -- **Follow the template**: Use the exact structure from `docs/integrations/template.md` -- **Be comprehensive**: Include all important details a new developer needs -- **Be accurate**: Extract information from actual code, not assumptions -- **Be clear**: Use simple language, avoid jargon where possible -- **Be consistent**: Follow the template formatting exactly -- **Complete all sections**: Fill in every section from the template, or mark as "[Not applicable]" -- **Reference ADRs**: Link to relevant architectural decisions -- **Include examples**: Show practical usage with curl commands and CLI -- **Highlight safety**: Emphasize dry_run defaults and safety features -- **Update metadata**: Set Last Updated date and Maintainer information - -## Example Interaction - -User: `/document-api-integration` -Assistant: I'll help you document a qontract-api integration. Which integration would you like to document? - -User: slack-usergroups -Assistant: -[Reads template from docs/integrations/template.md] -[Reads integration files from qontract_api/qontract_api/integrations/slack-usergroups/] -[Reads client integration from reconcile/slack_usergroups_api.py] -[Analyzes code and extracts all required information] -[Fills template with extracted information] - -I've analyzed the slack-usergroups integration and generated comprehensive documentation following the standard template. - -**Documentation Summary:** - -- API Path: /api/v1/integrations/slack-usergroups -- Features: 5 features (usergroup management, user sync, channel sync, etc.) -- Actions: 3 action types (create, update_users, update_metadata) -- External API: Slack API -- Required: Vault tokens, Redis cache - -**Files to create/update:** - -1. `docs/integrations/slack-usergroups.md` (new file with full template) -2. `docs/integrations/README.md` (add link in Integrations section) - -Should I create these files? - -User: yes -Assistant: -[Creates directory docs/integrations/ if needed] -[Creates docs/integrations/slack-usergroups.md with filled template] -[Updates docs/integrations/README.md with link in Integrations section] - -Documentation created successfully! - -**Created:** - -- `docs/integrations/slack-usergroups.md` (comprehensive integration documentation) - -**Updated:** - -- `docs/integrations/README.md` (added link: `- [Slack Usergroups](docs/integrations/slack-usergroups.md) - Manage Slack usergroups via API`) - -The integration is now fully documented with all sections from the template. +Invoke the `document-api-integration` skill to document the specified integration (or ask the user which one). Follow the skill's workflow to analyze the integration code and generate documentation. diff --git a/.claude/commands/migrate-integration.md b/.claude/commands/migrate-integration.md new file mode 100644 index 0000000000..ae06e3bd42 --- /dev/null +++ b/.claude/commands/migrate-integration.md @@ -0,0 +1,6 @@ +--- +description: Migrate a reconcile/ integration to qontract-api +scope: project +--- + +Invoke the `migrate-integration` skill to migrate the specified integration (or ask the user which one) from reconcile/ to the qontract-api architecture. Follow the skill's phases: Discovery, Shared Utils, Server-Side, Client-Side, Tests, Documentation. diff --git a/.claude/skills/document-api-integration/SKILL.md b/.claude/skills/document-api-integration/SKILL.md new file mode 100644 index 0000000000..3512c43a9e --- /dev/null +++ b/.claude/skills/document-api-integration/SKILL.md @@ -0,0 +1,54 @@ +--- +name: document-api-integration +description: Generate comprehensive documentation for a qontract-api integration. Use this skill when a new integration has been created or migrated to qontract-api and needs documentation, or when someone asks to document an integration. Triggers on requests mentioning integration documentation, documenting API integrations, or after completing a migration with /migrate-integration. +--- + +# Document qontract-api Integration + +Analyze an existing qontract-api integration and generate documentation following the standard template at `docs/integrations/template.md`. + +## Input + +Integration name (e.g., "slack-usergroups"). If not provided, ask for it. + +Validate the integration exists at `qontract_api/qontract_api/integrations//` before proceeding. + +## Workflow + +1. **Read the template** from `docs/integrations/template.md` + +2. **Analyze the integration** by reading: + - `qontract_api/qontract_api/integrations//models.py` - Request/Response/Action models + - `qontract_api/qontract_api/integrations//service.py` - Business logic + - `qontract_api/qontract_api/integrations//router.py` - API endpoints + - `qontract_api/qontract_api/integrations//tasks.py` - Celery tasks + - Other files in the integration directory (factory, client, etc.) + - `reconcile/_api.py` or `reconcile/_api/` - Client integration + - Related ADRs referenced in docstrings/comments + +3. **Extract information** for each template section: + - **Features**: What it does, actions it can perform, resources it manages + - **API Endpoints**: POST/GET paths, parameters, responses + - **Models**: Pydantic structure, key fields, validation rules (dry_run default=True) + - **Actions**: All action types from discriminated union models + - **Architecture**: Client-side vs server-side responsibilities + - **Limits/Constraints**: Safety features, rate limits, cache TTLs, managed resources + - **Required Components**: Vault secrets, external APIs, environment variables + +4. **Generate documentation** using the template structure exactly. Fill all sections. Mark non-applicable sections as "[Not applicable for this integration]". + +5. **Show summary** to user before writing: + - API path, features count, action types, external APIs, requirements + +6. **Write files** after confirmation: + - Create `docs/integrations/.md` + - Update `docs/integrations/README.md` with link in alphabetical order + +## Guidelines + +- Follow the template structure from `docs/integrations/template.md` exactly +- Extract information from actual code, not assumptions +- Reference relevant ADRs +- Include practical usage examples (curl commands, CLI) +- Emphasize dry_run defaults and safety features +- Set Last Updated date to today diff --git a/.claude/skills/migrate-integration/SKILL.md b/.claude/skills/migrate-integration/SKILL.md new file mode 100644 index 0000000000..a5b0c3577d --- /dev/null +++ b/.claude/skills/migrate-integration/SKILL.md @@ -0,0 +1,458 @@ +--- +name: migrate-integration +description: Migrate a reconcile/ integration to the qontract-api architecture. Use this skill when someone wants to rewrite, migrate, or port an existing reconcile integration to the API-based pattern, or when they mention creating a new qontract-api integration based on an existing one. Also triggers on mentions of "migrate to api", "rewrite for qontract-api", "create api integration", or "port integration". This is the primary skill for any reconcile-to-api migration work. +--- + +# Migrate Integration to qontract-api + +Guide the migration of an existing `reconcile/` integration to the qontract-api architecture. This skill analyzes the existing integration, creates a migration plan, and generates all required code in phases. + +Two successful migrations serve as reference implementations: + +- `slack_usergroups` -> `slack_usergroups_api` +- `glitchtip_project_alerts` -> `glitchtip_project_alerts_api` + +## Input + +Integration name in kebab-case (e.g., `aws-account-manager`). If not provided, ask for it. + +## General + +The discovery phase is critical to ensure a smooth migration. Never skip it, and never start coding before the plan is confirmed by the user. + +## Migration Plans + +Migration plans are stored in `.claude/skills/migrate-integration/plans/.md`. These persist across sessions and allow resuming work after context clears. + +**Before starting any phase**, always read the migration plan for the integration to understand current status, decisions, and resumption context. Update the plan's status and checkboxes as you complete tasks. + +**Between phases**, the user may clear context (`/clear`). Each phase in the plan includes a "Resumption context" section that tells you what files to read to rebuild context for that phase. + +## Agent Teams + +Use agent teams throughout the migration to parallelize work, coordinate complex tasks, and enable deep thinking on architectural decisions. Don't hesitate to spin up multiple agents — they are cheap and fast. + +### When to use agent teams + +- **Phase 0 (Discovery)**: Always. Launch parallel agents to explore different aspects simultaneously: + - Agent 1: Deep dive into the existing integration code (all source files, data models, state management) + - Agent 2: Explore existing qontract-api patterns (reference implementations, task infrastructure, events) + - Agent 3: Explore ADRs, infrastructure, and existing utilities in `qontract_utils/` + - Agent 4 (deep thinker): If the integration has complex architectural challenges (state machines, multi-step workflows, novel patterns), dedicate an agent to think deeply about the design. Give it a `Plan` subagent type and let it explore the codebase AND reason about the solution. + +- **Implementation phases**: When a phase has independent sub-tasks (e.g., creating models.py, service.py, and router.py), launch agents in parallel for each file. One agent can write the service while another writes the router. + +- **Complex decisions**: When facing a non-trivial architectural decision, launch a dedicated `Plan` agent to think it through. Give it full context and ask it to explore alternatives, trade-offs, and propose a detailed design. + +### Agent team guidelines + +- **Background agents**: Use `run_in_background: true` for discovery/research agents so they work in parallel. Wait for all to complete before synthesizing. +- **Deep thinking agents**: Use `Plan` subagent type when you want an agent to reason deeply about architecture, not just search for code. +- **Explore agents**: Use `Explore` subagent type for thorough codebase searches. Specify thoroughness: "very thorough" for discovery phases. +- **Don't duplicate work**: If an agent is exploring a topic, don't search for the same things yourself. Trust the agent's results. +- **Synthesize results**: After all agents report back, compile their findings into a coherent plan. Ask the user questions interactively (one at a time, not as a text dump). + +## Stateful vs Stateless Integrations + +Some integrations are **stateless** (like slack-usergroups, glitchtip): each reconciliation run diffs desired vs current state and applies changes atomically. Others are **stateful** (like aws-account-manager): operations span multiple reconciliation runs with async external operations. + +### Identifying Stateful Integrations + +Look for these patterns in the existing integration: + +- S3/Redis/file-based state tracking across runs +- Async operations that require polling (e.g., AWS CreateAccount -> poll for completion) +- Multi-step sequential workflows (step N depends on step N-1's result) +- `AbortStateTransactionError` or similar "retry next run" patterns + +### Handling Stateful Integrations + +Stateful integrations use the **Workflow Framework** (`qontract_api/qontract_api/workflow/`). This provides: + +- **WorkflowStore**: Redis-backed persistence for workflow state (key: `workflow::`) +- **WorkflowExecutor**: Sequential step execution with resume-from-last-incomplete support +- **Management endpoints**: List, inspect, reset, and delete workflows via REST API +- **Step handlers**: Per-step functions returning `StepResult` with status + context updates + +Key patterns: + +- Steps return `StepStatus.IN_PROGRESS` for async operations (executor stops, resumes next run) +- Steps return `StepStatus.FAILED` for errors (operator can reset via API) +- `context` dict passes serializable data between steps (request IDs, account UIDs, etc.) +- Non-serializable deps (API clients) are injected via closures, NOT stored in context +- Distributed locking prevents concurrent modifications to the same workflow + +Stateful integrations typically need **two endpoints**: + +1. A stateful workflow endpoint (e.g., `/create`) for multi-step operations +2. A stateless diff endpoint (e.g., `/reconcile`) for ongoing reconciliation of existing resources + +## Phase 0: Discovery & Analysis + +1. **Find the existing integration.** Search for source files: + - `reconcile/.py` or `reconcile//` directory + - Related test files in `tests/` + - GraphQL queries in `reconcile/gql_definitions/` + - Any shared utilities the integration uses from `reconcile/utils/` + - **Existing API clients in `qontract_utils/qontract_utils/`** — search thoroughly for domain-related modules (e.g., `aws_api_typed/`, `slack/`, `glitchtip/`) + - **Existing domain layers in `qontract_api/qontract_api/`** — check if a domain layer already exists (e.g., `slack/`, `glitchtip/`) + - **Existing external endpoints in `qontract_api/qontract_api/external/`** + +2. **Show discovered files** and ask user to confirm or add missing ones. + +3. **Analyze the existing integration** to understand: + - What external APIs it calls (Slack, AWS, GitHub, PagerDuty, etc.) + - What the `run()` / `desired_state()` / `current_state()` functions do + - What reconciliation actions it performs (create, update, delete) + - What secrets/credentials it needs + - What data models it uses (dicts vs dataclasses vs pydantic) + - Whether it supports sharding, early-exit, or other patterns + - Whether it is stateful or stateless (see "Stateful vs Stateless Integrations" section) + - What shared utilities from `reconcile/utils/` it depends on (these can NOT be imported in qontract-api per ADR-007 — equivalent functionality must exist or be created in `qontract_utils/`) + - **What external data the client needs for desired state compilation** (e.g., PagerDuty schedules, VCS OWNERS files, AWS resource lists). This determines whether external endpoints are needed (Phase 3). + +4. **Save the migration plan** to `.claude/skills/migrate-integration/plans/.md`: + - All discovered source files and their purpose + - Key architectural decisions (action types, model structure, stateful/stateless, endpoint structure) + - Files to create per phase with status tracking (checkboxes) + - Each phase gets a "Resumption context" section explaining what to read after a `/clear` + - Phase dependency graph (which phases can run in parallel, which are prerequisites) + - What goes where: `qontract_utils/` vs `qontract_api//` vs `qontract_api/integrations/` vs `qontract_api/external/` + +5. **Present the plan to the user** and ask questions interactively (one at a time). Wait for user confirmation before proceeding. + +6. **Old integration**: Do NOT modify or delete the old integration in `reconcile/`. Inform the user that they can roll out the new `_api` integration via unleash feature toggles alongside the old one, and decommission the old one once the new one is verified in production. + +## Phase 1: Shared Utilities (qontract_utils/) + +Following ADR-007 (no reconcile/ imports in qontract-api) and ADR-014 (three-layer architecture). + +`qontract_utils/` contains **only** pure API client abstractions (Layer 1) and generic utilities. Everything here **must be synchronous** because it is used by Celery workers which run sync code. + +1. **IMPORTANT: Check for existing API clients first.** Before creating anything, thoroughly search `qontract_utils/qontract_utils/` for existing clients: + - Search for the domain name (e.g., `aws`, `slack`, `glitchtip`, `pagerduty`) + - Check both exact matches and related names (e.g., `aws_api_typed/` not just `aws/`) + - Use `Glob` on `qontract_utils/qontract_utils/**/*.py` and scan for relevant modules + - Existing clients to be aware of: + - `qontract_utils/qontract_utils/aws_api_typed/` - AWS APIs (Organizations, IAM, STS, S3, Support, Account, Service Quotas, etc.) + - `qontract_utils/qontract_utils/slack/` - Slack API + - `qontract_utils/qontract_utils/glitchtip/` - Glitchtip API + - `qontract_utils/qontract_utils/pagerduty/` - PagerDuty API + - If a client exists, check if it covers all needed methods. Only extend, never duplicate. + +2. **Layer 1 - Pure API Client** (`qontract_utils//api.py`): + - Thin synchronous wrapper around the external API (REST/GraphQL calls) + - No business logic, no caching, no state + - Uses `@with_hooks` and `@invoke_with_hooks()` decorators for metrics/retries (ADR-006) + - **All methods must be synchronous** - Celery workers are sync-only + - Example reference: `qontract_utils/slack/api.py`, `qontract_utils/glitchtip/api.py` + +3. **Create tests** for new API client classes in `tests/qontract_utils/`. + +**Important**: Workspace clients (Layer 2) do NOT go in `qontract_utils/`. They belong in `qontract_api/` (see Phase 2). + +## Phase 2: Server-Side Integration (qontract_api/) + +### Domain Layer (qontract_api/qontract_api//) + +**Check if a domain layer already exists** before creating a new one. Search `qontract_api/qontract_api/` for existing domain directories (e.g., `slack/`, `glitchtip/`). If one exists for your domain, extend it rather than creating a duplicate. + +Every integration gets its own domain layer in `qontract_api/`, even if only one integration uses it currently. This mirrors the existing pattern for slack and glitchtip. + +Create `qontract_api/qontract_api//`: + +- **`models.py`** - Shared domain models (Pydantic, frozen=True). These model the external system's concepts (workspaces, usergroups, instances, projects, etc.) +- **`_client_factory.py`** - Factory for creating workspace clients (ADR-017). Resolves secrets via SecretManager, creates API client + workspace client with proper configuration. +- **`workspace_client.py`** (Layer 2) - Caching layer on top of the pure API client: + - In-memory + Redis caching via `CacheBackend` + - Distributed locking for thread-safety + - Computed/derived data helpers + - **Synchronous** (runs in Celery worker context) + - Example reference: `qontract_api/qontract_api/slack/slack_workspace_client.py` + +Reference: `qontract_api/qontract_api/slack/`, `qontract_api/qontract_api/glitchtip/` + +### Integration Files (qontract_api/qontract_api/integrations//) + +#### models.py + +Following ADR-012 (typed Pydantic models): + +- **Request model**: `ReconcileRequest(BaseModel, frozen=True)` with `dry_run: bool = True` +- **Action models**: Discriminated union with `action_type` field as `Literal`. One model per action type (create, update, delete, etc.) +- **Task result**: `TaskResult(TaskResult, frozen=True)` with `actions: list[Action]` +- **Task response**: `TaskResponse(BaseModel, frozen=True)` with `id`, `status`, `status_url` +- All models `frozen=True` for immutability +- Sort list fields via `field_validator` for deterministic output + +Reference: `qontract_api/qontract_api/integrations/slack_usergroups/models.py` + +#### service.py + +Following ADR-011 (dependency injection) and ADR-014 (three-layer architecture): + +- **Class**: `Service` with constructor injection of `cache`, `secret_manager`, `settings`, and client factories +- **`reconcile()` method**: Main entry point accepting desired state + `dry_run` + 1. For each resource group: create client via factory, fetch current state, calculate diff + 2. Use `qontract_utils.differ.diff_iterables()` for diffing + 3. Generate typed action models + 4. Execute actions if `dry_run=False` (using `match/case` on action type) + 5. Return `TaskResult` with status, actions, applied_count, errors +- **Error handling**: Try/except per resource group and per action. Collect errors, continue processing. +- Static helper methods for `_calculate_actions()` and `_execute_action()` + +Reference: `qontract_api/qontract_api/integrations/slack_usergroups/service.py` + +#### router.py + +Following ADR-003 (async-only API with blocking GET): + +- **POST `/reconcile`** (HTTP 202 Accepted): + - Accepts `ReconcileRequest` + - Requires JWT auth (`UserDep`) + - Queues Celery task via `apply_async()` + - Returns `TaskResponse` with task_id and status_url + +- **GET `/reconcile/{task_id}`** (blocking/non-blocking): + - Optional `timeout` query param (1-300 seconds) + - Uses `wait_for_task_completion()` helper + - Returns `TaskResult` + +- Router prefix: `/` (e.g., `/aws-account-manager`) + +Reference: `qontract_api/qontract_api/integrations/slack_usergroups/router.py` + +#### tasks.py + +Following ADR-018 (event-driven communication): + +- **Celery task** with `@celery_app.task(bind=True, name=".reconcile", acks_late=True)` +- **Deduplication** via `@deduplicated_task(lock_key_fn=generate_lock_key, timeout=600)` +- Lock key from resource identifiers (workspace names, instance names, etc.) +- Create service with injected dependencies (`get_cache()`, `get_secret_manager()`, `get_event_manager()`) +- **Event publishing** for applied actions (non-dry-run): publish `Event` per action to Redis Streams +- Error handling: catch exceptions, return failed `TaskResult` + +Reference: `qontract_api/qontract_api/integrations/slack_usergroups/tasks.py` + +#### `__init__.py` + +Empty file or re-exports. + +### Infrastructure Registration + +1. **Register router** in `qontract_api/qontract_api/routers/integrations.py`: + + ```python + integrations_router.include_router(_router.router) + ``` + +2. **Register Celery task** in `qontract_api/qontract_api/tasks/__init__.py`: + Add module path to `include` list in `Celery()` config. + +3. **Add settings** to `qontract_api/qontract_api/config.py` if needed (cache TTLs, timeouts, etc.) + +4. **Regenerate the API client** after creating server-side routers: + + ```bash + cd qontract_api && make generate-openapi-spec + cd qontract_api_client && make generate-client + ``` + + Do this after Phase 2 AND again after Phase 3 (if external endpoints are added). + +## Phase 3: External Endpoints (if needed) + +**This phase is required when the client-side integration needs data from external services to compile its desired state.** For example, `slack_usergroups_api` needs PagerDuty schedule users and VCS repo OWNERS to build the complete desired state before sending it to the reconciliation endpoint. + +Following ADR-013 (centralize external API calls): the client MUST NOT call external APIs directly. Instead, qontract-api provides external endpoints that the client calls. + +Check if the old integration fetches data from external services during desired state compilation. Common patterns: + +- PagerDuty schedules/escalation policies for on-call users +- VCS/GitHub/GitLab for OWNERS file data +- AWS for resource listings +- Any other external API calls in the old `desired_state()` / `get_desired_state()` / `run()` + +If external endpoints are needed, create `qontract_api/qontract_api/external//`: + +- **`models.py`** - Request/response models for the external endpoint +- **`router.py`** - FastAPI endpoint (typically GET with query params for secret references) +- **`_workspace_client.py`** - Caching wrapper for the external API client +- **`_factory.py`** - Factory to create workspace clients + +Register external routers in `qontract_api/qontract_api/routers/external.py`. + +**Check if external endpoints already exist** before creating new ones. Existing externals: + +- `qontract_api/qontract_api/external/pagerduty/` - PagerDuty schedule/escalation policy users +- `qontract_api/qontract_api/external/vcs/` - VCS repo OWNERS +- `qontract_api/qontract_api/external/slack/` - Slack API proxying + +Reference: `qontract_api/qontract_api/external/pagerduty/`, `qontract_api/qontract_api/external/vcs/` + +### Auto-Generated Client + +After creating server-side routers (integration + external), regenerate the API client: + +```bash +cd qontract_api && make generate-openapi-spec +cd qontract_api_client && make generate-client +``` + +This creates typed Python client functions matching all new endpoints. + +## Phase 4: Client-Side Integration (reconcile/) + +Following ADR-008 (QontractReconcileApiIntegration pattern). + +The client-side integration is responsible for **all desired state computation**. It queries App-Interface via GraphQL, enriches the data with external service data (via qontract-api external endpoints from Phase 3), and sends the complete desired state to the reconciliation endpoint. + +Create `reconcile/_api.py` (single file) or `reconcile/_api/` (package with `integration.py`): + +- **Class**: `Integration(QontractReconcileApiIntegration[IntegrationParams])` +- **Params**: `IntegrationParams(PydanticRunParams)` with optional filter parameters +- **`async_run(dry_run: bool)`**: Main entry point (async, not sync) + +### Desired State Compilation (client-side responsibility) + +The client compiles the complete desired state. This typically involves: + +1. **Query App-Interface GraphQL** for configuration data (permissions, roles, resources, clusters, users, etc.) +2. **Enrich with external data** (if needed) by calling qontract-api external endpoints: + - PagerDuty users: `get_pagerduty_schedule_users()`, `get_pagerduty_escalation_policy_users()` + - VCS OWNERS: `get_repo_owners()` + - Use `asyncio.gather()` for parallel external calls +3. **Compile** the desired state from all sources into the request model +4. **Send** to qontract-api reconciliation endpoint + +Reference for complex desired state: `reconcile/slack_usergroups_api.py` - compiles users from 5 sources (roles, schedules, git OWNERS, PagerDuty, cluster access), all happening client-side before calling the API. + +### Task Handling + +- Call qontract-api via auto-generated client: `reconcile_(client=self.qontract_api_client, body=request)` +- **Dry-run**: wait for task completion via `_task_status(client, task_id, timeout=300)` +- **Non-dry-run**: fire-and-forget (task completes in background, events published via events framework) +- Log actions using `match/case` on action types +- Exit with error if task result contains errors + +Reference: `reconcile/slack_usergroups_api.py`, `reconcile/glitchtip_project_alerts_api/integration.py` + +### Integration Registration + +The new `_api` integration must be registered in `reconcile/cli.py`. Search for existing `_api` integrations (e.g., `slack_usergroups_api` or `glitchtip_project_alerts_api`) in that file and follow the same pattern. + +## Phase 5: Tests + +Create comprehensive tests following existing patterns: + +1. **Server-side tests** in `tests/qontract_api/integrations//`: + - `test_models.py` - Model validation, serialization, frozen behavior + - `test_service.py` - Reconciliation logic, diff calculation, action generation, error handling + - `test_router.py` - Endpoint behavior, auth, task queuing, status retrieval + - `test_tasks.py` - Task execution, deduplication, event publishing + +2. **Client-side tests** in `tests/test__api.py` or `tests/_api/`: + - Desired state compilation from all sources + - External API call handling + - API request construction + - Task result handling + +3. **Utility tests** in `tests/qontract_utils//`: + - API client methods (Layer 1) + +4. **Domain tests** in `tests/qontract_api//`: + - Workspace client caching behavior (Layer 2) + - Factory tests + +Use `pytest`, `@pytest.fixture`, `@pytest.mark.parametrize`. Mock external API calls. Test both dry-run and non-dry-run paths. + +## Phase 6: Documentation & Skill Update + +### Integration Documentation + +After all code is generated, invoke the `document-api-integration` skill to create `docs/integrations/.md` following the standard template. + +### Update This Skill + +After completing the migration, update **this SKILL.md** with any new knowledge gained during the process. This keeps the skill accurate for future migrations. Specifically check and update: + +- **Existing API clients list** (Phase 1): If new API clients were created or existing ones extended in `qontract_utils/`, add them to the known clients list. +- **Existing external endpoints list** (Phase 3): If new external endpoints were created, add them to the known externals list. +- **New ADRs**: If new ADRs were written (e.g., for workflow framework, new patterns), reference them in the relevant phase descriptions. +- **New architectural patterns**: If the migration introduced new patterns (e.g., workflow framework for stateful integrations), document them in the appropriate section. +- **Reference implementations**: Add the completed migration as a reference implementation alongside slack_usergroups and glitchtip at the top of this file. +- **Gotchas and lessons learned**: If any unexpected issues came up, add them to the Key Rules section or relevant phase. + +## Output Format + +Show progress after each phase: + +``` +## Migration: + +Phase 0: Discovery & Analysis + Found 5 source files, 3 action types, 2 external APIs + +Phase 1: Shared Utilities (qontract_utils/) + Created qontract_utils//api.py + +Phase 2: Server-Side (qontract_api/) + Created qontract_api//models.py, workspace_client.py, factory.py + Created qontract_api/integrations//models.py + Created qontract_api/integrations//service.py + Created qontract_api/integrations//router.py + Created qontract_api/integrations//tasks.py + +Phase 3: External Endpoints + Reusing existing pagerduty external endpoints + Created qontract_api/external//router.py + +Phase 4: Client-Side (reconcile/) + Created reconcile/_api.py + +Phase 5: Tests + Created 10 test files + +Phase 6: Documentation + Created docs/integrations/.md +``` + +## Verification + +After completing each implementation phase, run verification before moving on: + +```bash +make format # Auto-format code +make linter-test # Lint checks +make types-test # MyPy strict mode +make unittest # Unit tests (or pytest on specific test files) +``` + +Fix any issues before proceeding to the next phase. Update the migration plan with the phase status. + +## Phase Dependencies + +Phases have dependencies. Document these in the migration plan so phases can be parallelized where possible: + +- **Phase 1** (shared utils) is always a prerequisite for Phase 2 (server-side) +- **Phase 2** (server-side) is a prerequisite for Phase 3 (external endpoints) and Phase 4 (client-side) +- **Phase 5** (tests) can partially overlap with implementation phases (write tests as you go) +- **Sub-phases** (e.g., 1a, 1b, 1.5) may or may not depend on each other — analyze per migration and document in the plan + +## Key Rules + +- **Never import from `reconcile/` in qontract-api code** (ADR-007) +- **`qontract_utils/` is sync-only** - used by Celery workers. Only Layer 1 API clients go here. +- **Workspace clients (Layer 2) belong in `qontract_api/`**, not `qontract_utils/` +- **Domain models + factories always in `qontract_api//`**, even if only one integration uses them +- **All models are Pydantic with `frozen=True`** (ADR-012) +- **All dependencies injected via constructor** (ADR-011) +- **`dry_run` always defaults to `True`** - safety first +- **Use `match/case`** for action dispatch +- **Use `qontract_utils.differ.diff_iterables()`** for diffing +- **Client compiles ALL desired state** - GraphQL + external data enrichment happens in reconcile client +- **External API calls only via qontract-api external endpoints** (ADR-013) - client calls these, never external APIs directly +- **Don't touch the old integration** - leave `reconcile//` as-is. The user can roll out the new `_api` integration via unleash feature toggles and decommission the old one later. +- Read existing reference implementations before generating code - adapt patterns, don't copy blindly +- Read all relevant ADRs from `docs/adr/` before starting From 0e7761fe6bb1a42cc5edba49d5a71b2de2b4f3e5 Mon Sep 17 00:00:00 2001 From: Christian Assing Date: Wed, 11 Mar 2026 07:55:28 +0100 Subject: [PATCH 2/2] remove slash cmds --- .claude/commands/document-api-integration.md | 6 ------ .claude/commands/migrate-integration.md | 6 ------ 2 files changed, 12 deletions(-) delete mode 100644 .claude/commands/document-api-integration.md delete mode 100644 .claude/commands/migrate-integration.md diff --git a/.claude/commands/document-api-integration.md b/.claude/commands/document-api-integration.md deleted file mode 100644 index f7705ce94b..0000000000 --- a/.claude/commands/document-api-integration.md +++ /dev/null @@ -1,6 +0,0 @@ ---- -description: Document an existing qontract-api integration -scope: project ---- - -Invoke the `document-api-integration` skill to document the specified integration (or ask the user which one). Follow the skill's workflow to analyze the integration code and generate documentation. diff --git a/.claude/commands/migrate-integration.md b/.claude/commands/migrate-integration.md deleted file mode 100644 index ae06e3bd42..0000000000 --- a/.claude/commands/migrate-integration.md +++ /dev/null @@ -1,6 +0,0 @@ ---- -description: Migrate a reconcile/ integration to qontract-api -scope: project ---- - -Invoke the `migrate-integration` skill to migrate the specified integration (or ask the user which one) from reconcile/ to the qontract-api architecture. Follow the skill's phases: Discovery, Shared Utils, Server-Side, Client-Side, Tests, Documentation.