diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md new file mode 120000 index 0000000..3395d56 --- /dev/null +++ b/.github/copilot-instructions.md @@ -0,0 +1 @@ +../LLMS.md \ No newline at end of file diff --git a/LLMS.md b/LLMS.md new file mode 100644 index 0000000..45d3eec --- /dev/null +++ b/LLMS.md @@ -0,0 +1,274 @@ + +# LLM Context Guide for `infrahub-sync` + +`infrahub-sync` synchronizes data between infra sources and destinations (Infrahub, NetBox, Nautobot, etc.). It uses Poetry for packaging, a Typer CLI, and Invoke tasks for linting and docs. Examples live in `examples/`. + +## Agent Operating Principles + +1. **Plan → Ask → Act → Verify → Record** + Plan briefly, ask for missing context, act with the smallest change, verify locally, then record with a concise commit or PR note. + +2. **Default to read-only and dry runs** + Prefer `list`, `diff`, and `generate` before `sync`. Write/apply only with explicit instruction and human approval. + +3. **Be specific and reversible** + Use small, scoped commits. Do not mix large refactors with behavior changes in the same PR. + +4. **Match existing patterns** + Keep CLI, adapters, examples, and directory structure consistent with the codebase. + +5. **Idempotency and safety** + Favor operations that are safe to re-run. Use dry runs. Never print or guess secrets. Handle timeouts, auth, and network errors explicitly. + +## Required Development Workflow + +Run these in order before committing. + +```bash +poetry install +poetry run invoke format +poetry run invoke lint +poetry run mypy infrahub_sync/ --ignore-missing-imports +``` + +**Policy:** + +- New or changed code is Ruff-clean and typed where touched (docstrings, specific exceptions). +- Do not increase existing mypy debt. If needed, use targeted `# type: ignore[]` with a short TODO. +- If you add tests, run `poetry run pytest -q`. + +**CLI sanity after changes:** + +```bash +poetry run infrahub-sync --help +poetry run infrahub-sync list --directory examples/ +poetry run infrahub-sync generate --name from-netbox --directory examples/ +``` + +**Docs:** (only if user-facing changes) + +```bash +poetry run invoke docs.generate +poetry run invoke docs.docusaurus +``` + +## Repository Structure + +```text +infrahub-sync/ +├─ infrahub_sync/ # Source +│ ├─ cli.py # Typer entrypoint +│ ├─ __init__.py # Public API +│ ├─ utils.py # Utilities +│ ├─ potenda/ # Core sync engine +│ └─ adapters/ # NetBox/Nautobot/Infrahub adapters +├─ examples/ # Example sync configs +├─ tasks/ # Invoke task definitions +├─ docs/ # Docusaurus (npm project) +├─ tests/ # Scaffolding (no tests yet) +├─ pyproject.toml # Poetry + tool configs +└─ .github/workflows/ # CI +``` + +## Core Surfaces + +- **Adapters** (`infrahub_sync/adapters/`): per-system connectors. Use existing ones as patterns. +- **Engine** (`infrahub_sync/potenda/`): orchestrates `list`, `diff`, `generate`, and `sync`. +- **Examples** (`examples/`): runnable configs and templates. + +**CLI commands:** + +- `infrahub-sync list` — show available sync projects. +- `infrahub-sync diff` — compute differences (safe). +- `infrahub-sync generate` — generate Python from YAML config (servers required). +- `infrahub-sync sync` — perform synchronization (servers and approval required). + +## Configuration and Examples + +- YAML config keys: `name`, `source`, `destination`, `order`. +- `source` and `destination` specify adapter names and connection settings. +- `order` defines the sync sequence of object types. +- Defaults often target `localhost`; adjust for real deployments. +- Credentials must come from environment or a secret manager. Never commit secrets. + +## Code Standards + +### Python (3.10–3.12) + +- Prefer explicit types on new or changed code. +- Ruff: formatted and lint-clean. Honor `pyproject.toml`. +- Pylint: fix actionable issues in touched code; some warnings are expected. +- Mypy: run with `--ignore-missing-imports`; do not increase the error count. +- Public functions and classes require concise docstrings. +- Raise specific exceptions; avoid broad `except Exception:`. + +### CLI and UX + +- Predictable, idempotent commands with clear validation and errors. +- No secrets in logs or tracebacks. +- Prefer explicit flags over implicit behavior. + +## Testing + +Current state: `tests/` exists but has no active tests. + +If you introduce features or bug fixes, add targeted tests. + +- Unit tests for `utils` and adapter edge cases (timeouts, 401/403, empty pages). +- Parametrized tests for config parsing. +- Mark network or integration tests and keep them opt-in (for example, `-m integration`). +- Keep tests atomic and single-purpose. Use parametrization rather than loops. + +Run: + +```bash +poetry run pytest -q +``` + +## Documentation + +- Update `docs/` for any user-visible changes (flags, config, adapters). +- Generate CLI docs: + +```bash +poetry run invoke docs.generate +``` + +- Build site (ensure `cd docs && npm install` once): + +```bash +poetry run invoke docs.docusaurus +``` + +- Keep examples minimal, accurate, and redacted. + +### Linting documentation (markdownlint) + +Use `markdownlint` and `markdownlint-cli` for Markdown and MDX files. + +```bash +# Check and fix Markdown and MDX in docs +npx markdownlint-cli "docs/docs/**/*.{md,mdx}" +npx markdownlint-cli --fix "docs/docs/**/*.{md,mdx}" +``` + +## Invoke Tasks (reference) + +```bash +poetry run invoke --list +# linter.format-ruff Format Python code with ruff +# linter.lint-ruff Lint Python code with ruff +# linter.lint-pylint Lint Python code with pylint +# linter.lint-yaml Lint YAML files with yamllint +# docs.generate Generate CLI documentation +# docs.docusaurus Build documentation website +# format Alias for ruff format (if defined) +# lint Run all linters +``` + +## Known Issues and Limitations + +- Optional dependencies (for example, `pynetbox`, `pynautobot`) may be missing, producing import warnings. +- `generate` and `sync` require running servers (Infrahub, NetBox, Nautobot). +- Existing mypy debt exists; do not increase it and type the code you touch. +- Docs npm audit may flag dev-only vulnerabilities; they do not affect the Python package. + +## Development Rules + +### Git and CI + +- Do not force-push on shared branches. +- Do not amend to hide pre-commit fixes; use a follow-up commit. +- Apply PR labels: `bugs`, `breaking`, `enhancements`, `features` (default to `enhancements`). +- Always run the required workflow (format → lint → mypy → CLI sanity) before a PR. + +### Commit and PR Messages + +- Agents must identify themselves (for example, `🤖 Generated with Copilot`). +- Commit subject: imperative “what changed.” Rationale goes in the PR body. +- PR body includes: + - Problem or tension and the solution in one to two short paragraphs. + - Minimal code example or before/after snippet. + - Note any user-visible changes (CLI flags, config keys). + +## Review Process + +- Read surrounding code and examples. Align with established patterns. +- Verify claims via the smallest reproduction (CLI or unit). +- Consider edge cases: auth failures, empty inputs, pagination, rate limits, timeouts. +- Provide specific, actionable feedback. + +**Approval checklist:** + +- [ ] Format and lint clean on changed areas. +- [ ] No increase in mypy errors; new code typed. +- [ ] CLI behaviors validated (`--help`, `list`, targeted `generate`). +- [ ] Docs updated if flags or config changed. +- [ ] Error handling uses specific exception types and clear messages. + +## Operational and Safety Guidelines + +- Prefer dry runs (`diff`, `list`, `generate`) and include outputs in PRs when helpful. +- Least privilege: only touch minimal required resources. +- Idempotency: ensure safe re-runs and guard against partial failures. +- Observability: contextual logging without secrets (request IDs, endpoints, object counts). +- Concurrency: avoid collisions with live migrations or active syncs. Coordinate via PRs. + +If unsure, stop and ask with a concrete question. + +## Security and Secrets + +- Configure credentials via environment variables or secret managers. +- Never print tokens or keys in logs, exceptions, or PRs. Redact examples and tests. +- Keep example configs authentic but sanitized. + +## Platform-Specific Notes + +Mirror these principles to: + +- `CLAUDE.md` +- `.github/copilot-instructions.md` +- `GEMINI.md` +- `GPT.md` +- `.cursor/rules/dev-standard.mdc` + +Each should include the “Required Development Workflow” block and the “Approval checklist” verbatim. + +## Quickstart + +```bash +# Setup +pyenv local 3.12.x || use system Python 3.9–3.12 +pip install poetry +poetry install + +# Validate dev environment +poetry run infrahub-sync --help +poetry run infrahub-sync list --directory examples/ + +# Make a change, then: +poetry run invoke format +poetry run invoke lint +poetry run mypy infrahub_sync/ --ignore-missing-imports +poetry run infrahub-sync list --directory examples/ + +# If docs/CLI changed: +poetry run invoke docs.generate +poetry run invoke docs.docusaurus +``` + +## Adding a New Adapter + +1. Create `infrahub_sync/adapters/.py` following existing adapter patterns. +2. Add connection config schema and an example under `examples/`. +3. Provide `list` and `diff` pathways before enabling `sync`. +4. Document required environment variables and expected error cases. +5. Create a documentation page for the adapter in `docs/docs/adapters/`. + - Include overview, configuration keys, environment variables, example YAML, and common errors. + - Add it to the sidebar or navigation as needed. + - Validate with markdownlint: + + ```bash + npx markdownlint-cli "docs/docs/adapters/**/*.{md,mdx}" + npx markdownlint-cli --fix "docs/docs/adapters/**/*.{md,mdx}" + ``` diff --git a/README.md b/README.md index 768f01e..349aec4 100644 --- a/README.md +++ b/README.md @@ -17,3 +17,26 @@ Infrahub Sync is a versatile Python package that synchronizes data between a sou ## Using Infrahub sync Documentation for using Infrahub Sync is available [here](https://docs.infrahub.app/sync/) + +### Python API + +Infrahub Sync also provides a Python API for programmatic access. You can use sync operations directly in your Python code: + +```python +from infrahub_sync.api import sync, diff, list_projects + +# List available sync projects +projects = list_projects() + +# Perform a diff +result = diff(name="my-sync-project") +if result.success and result.changes_detected: + print(f"Changes detected: {result.message}") + +# Perform a sync +result = sync(name="my-sync-project") +if result.success: + print(f"Sync completed in {result.duration:.2f} seconds") +``` + +For detailed documentation on the Python API, see [docs/python-api.md](docs/python-api.md). diff --git a/docs/python-api.md b/docs/python-api.md new file mode 100644 index 0000000..d3845c3 --- /dev/null +++ b/docs/python-api.md @@ -0,0 +1,303 @@ +# Infrahub Sync Python API + +This document describes how to use Infrahub Sync programmatically via Python instead of the command line interface. + +## Overview + +The Python API provides the same functionality as the CLI but allows you to integrate sync operations directly into your Python applications. This is useful for: + +- Automating sync operations within larger Python workflows +- Building custom integrations and tools +- Implementing conditional sync logic +- Creating monitoring and alerting systems around sync operations + +## Installation + +The Python API is available when you install infrahub-sync: + +```bash +pip install infrahub-sync +``` + +## Quick Start + +### Basic Usage + +```python +from infrahub_sync.api import sync, diff, list_projects + +# List all available sync projects +projects = list_projects() +for project in projects: + print(f"{project.name}: {project.source.name} -> {project.destination.name}") + +# Perform a diff operation +result = diff(name="my-sync-project") +if result.success and result.changes_detected: + print(f"Changes detected: {result.message}") + +# Perform a sync operation +result = sync(name="my-sync-project") +if result.success: + print(f"Sync completed in {result.duration:.2f} seconds") +``` + +### Using Configuration Files + +```python +from infrahub_sync.api import sync, diff + +# Use a specific configuration file +result = diff(config_file="/path/to/config.yml") + +# Use a configuration file with custom directory +result = sync( + config_file="config.yml", + directory="/path/to/sync/configs" +) +``` + +### Advanced Usage with Potenda + +For fine-grained control over the sync process: + +```python +from infrahub_sync.api import create_potenda + +# Create a Potenda instance +ptd = create_potenda(name="my-sync-project") + +# Load data from source and destination separately +ptd.source_load() +ptd.destination_load() + +# Calculate differences +diff_result = ptd.diff() + +# Only sync if there are changes +if diff_result.has_diffs(): + print("Changes detected, syncing...") + ptd.sync(diff=diff_result) +else: + print("No changes to sync") +``` + +## API Reference + +### Functions + +#### `list_projects(directory=None)` + +List all available sync projects. + +**Parameters:** +- `directory` (str, optional): Base directory to search for sync configurations + +**Returns:** +- `List[SyncInstance]`: List of available sync configurations + +**Example:** +```python +projects = list_projects() +projects_in_custom_dir = list_projects(directory="/custom/sync/configs") +``` + +#### `diff(name=None, config_file=None, directory=None, branch=None, show_progress=True)` + +Calculate differences between source and destination systems. + +**Parameters:** +- `name` (str, optional): Name of the sync project (mutually exclusive with config_file) +- `config_file` (str, optional): Path to sync configuration YAML file +- `directory` (str, optional): Base directory to search for sync configurations +- `branch` (str, optional): Branch to use for the diff +- `show_progress` (bool): Show progress bar during operation + +**Returns:** +- `SyncResult`: Result object with success status, message, duration, and change detection + +**Raises:** +- `SyncError`: When parameters are invalid or operation fails + +**Example:** +```python +result = diff(name="my-sync") +if result.success: + if result.changes_detected: + print(f"Changes found: {result.message}") + else: + print("No changes detected") +``` + +#### `sync(name=None, config_file=None, directory=None, branch=None, diff_first=True, show_progress=True)` + +Synchronize data between source and destination systems. + +**Parameters:** +- `name` (str, optional): Name of the sync project (mutually exclusive with config_file) +- `config_file` (str, optional): Path to sync configuration YAML file +- `directory` (str, optional): Base directory to search for sync configurations +- `branch` (str, optional): Branch to use for the sync +- `diff_first` (bool): Calculate and show differences before syncing +- `show_progress` (bool): Show progress bar during operation + +**Returns:** +- `SyncResult`: Result object with success status, message, and timing information + +**Raises:** +- `SyncError`: When parameters are invalid or operation fails + +**Example:** +```python +result = sync(name="my-sync", show_progress=True) +if result.success: + print(f"Sync completed in {result.duration:.2f} seconds") + print(f"Changes synced: {result.changes_detected}") +else: + print(f"Sync failed: {result.error}") +``` + +#### `create_potenda(name=None, config_file=None, directory=None, branch=None, show_progress=True)` + +Create a Potenda instance for advanced programmatic control. + +**Parameters:** +- `name` (str, optional): Name of the sync project (mutually exclusive with config_file) +- `config_file` (str, optional): Path to sync configuration YAML file +- `directory` (str, optional): Base directory to search for sync configurations +- `branch` (str, optional): Branch to use for operations +- `show_progress` (bool): Show progress bar during operations + +**Returns:** +- `Potenda`: Potenda instance for direct method calls + +**Raises:** +- `SyncError`: When parameters are invalid or initialization fails + +**Example:** +```python +ptd = create_potenda(name="my-sync") +ptd.source_load() +ptd.destination_load() +diff_result = ptd.diff() +if diff_result.has_diffs(): + ptd.sync(diff=diff_result) +``` + +### Classes + +#### `SyncResult` + +Result object returned by sync operations. + +**Attributes:** +- `success` (bool): Whether the operation succeeded +- `message` (str): Description of the result +- `duration` (float): Time taken for the operation in seconds +- `changes_detected` (bool): Whether changes were found/synced +- `error` (Exception): Exception object if operation failed + +#### `SyncError` + +Exception raised when sync operations fail. + +Inherits from `Exception` and is raised when: +- Invalid parameters are provided +- Sync configuration cannot be loaded +- Source or destination loading fails +- Sync operations encounter errors + +## Error Handling + +Always wrap API calls in try-catch blocks to handle potential errors: + +```python +from infrahub_sync.api import sync, SyncError + +try: + result = sync(name="my-sync") + if result.success: + print("Sync completed successfully") + else: + print(f"Sync failed: {result.error}") +except SyncError as e: + print(f"Sync configuration or setup error: {e}") +except Exception as e: + print(f"Unexpected error: {e}") +``` + +## Migration from CLI + +Here's how common CLI commands translate to API calls: + +| CLI Command | Python API Equivalent | +|-------------|----------------------| +| `infrahub-sync list` | `list_projects()` | +| `infrahub-sync diff --name my-sync` | `diff(name="my-sync")` | +| `infrahub-sync sync --name my-sync` | `sync(name="my-sync")` | +| `infrahub-sync diff --config-file config.yml` | `diff(config_file="config.yml")` | +| `infrahub-sync sync --branch feature --name my-sync` | `sync(name="my-sync", branch="feature")` | + +## Examples + +See the `examples/python_api_example.py` file in the repository for a complete working example that demonstrates all API functions. + +## Integration Tips + +### In Automation Scripts + +```python +#!/usr/bin/env python3 +from infrahub_sync.api import sync, SyncError +import sys + +def main(): + try: + result = sync(name="production-sync", show_progress=False) + if result.success: + if result.changes_detected: + print(f"✅ Sync completed: {result.changes_detected} changes applied") + else: + print("✅ No changes needed") + return 0 + else: + print(f"❌ Sync failed: {result.error}") + return 1 + except SyncError as e: + print(f"❌ Configuration error: {e}") + return 1 + +if __name__ == "__main__": + sys.exit(main()) +``` + +### In Monitoring Systems + +```python +from infrahub_sync.api import diff, SyncError +import time + +def check_sync_drift(sync_name: str) -> dict: + """Check if sync has drift and return monitoring data.""" + start_time = time.time() + + try: + result = diff(name=sync_name, show_progress=False) + return { + "success": result.success, + "drift_detected": result.changes_detected, + "check_duration": time.time() - start_time, + "message": result.message if result.changes_detected else "No drift detected" + } + except SyncError as e: + return { + "success": False, + "error": str(e), + "check_duration": time.time() - start_time + } + +# Use in monitoring loop +monitoring_data = check_sync_drift("critical-system-sync") +``` + +This Python API provides full programmatic access to all Infrahub Sync functionality while maintaining the same robustness and error handling as the CLI interface. \ No newline at end of file diff --git a/examples/device42_to_infrahub/config.yml b/examples/device42_to_infrahub/config.yml new file mode 100644 index 0000000..bc43bca --- /dev/null +++ b/examples/device42_to_infrahub/config.yml @@ -0,0 +1,54 @@ +--- +name: from-device42 + +source: + name: genericrestapi + settings: + url: "http://swaggerdemo.device42.com" + auth_method: "basic" + username: "guest" + password: "device42_rocks!" + api_endpoint: "/api/1.0" + response_key_pattern: "objects" + +destination: + name: infrahub + settings: + url: "http://localhost:8000" + +order: [ + "BuiltinTag", + "OrganizationTenant", + "LocationSite", +] + +schema_mapping: + # Builtin Tags (Device42 tags) + - name: BuiltinTag + mapping: tags + identifiers: ["name"] + fields: + - name: name + mapping: name + + # Organizations (Customers in Device42) + - name: OrganizationTenant + mapping: customers + identifiers: ["name"] + fields: + - name: name + mapping: name + - name: description + mapping: notes + + # Locations (Buildings in Device42) + - name: LocationSite + mapping: buildings + identifiers: ["name"] + fields: + - name: name + mapping: name + - name: tags + mapping: tags + reference: BuiltinTag + diff --git a/examples/peeringdb_to_infrahub/genenericrestapi/__init__.py b/examples/device42_to_infrahub/genericrestapi/__init__.py similarity index 100% rename from examples/peeringdb_to_infrahub/genenericrestapi/__init__.py rename to examples/device42_to_infrahub/genericrestapi/__init__.py diff --git a/examples/device42_to_infrahub/genericrestapi/sync_adapter.py b/examples/device42_to_infrahub/genericrestapi/sync_adapter.py new file mode 100644 index 0000000..37df9d5 --- /dev/null +++ b/examples/device42_to_infrahub/genericrestapi/sync_adapter.py @@ -0,0 +1,18 @@ +from infrahub_sync.adapters.genericrestapi import GenericrestapiAdapter + +from .sync_models import ( + BuiltinTag, + LocationSite, + OrganizationTenant, +) + + +# ------------------------------------------------------- +# AUTO-GENERATED FILE, DO NOT MODIFY +# This file has been generated with the command `infrahub-sync generate` +# All modifications will be lost the next time you reexecute this command +# ------------------------------------------------------- +class GenericrestapiSync(GenericrestapiAdapter): + BuiltinTag = BuiltinTag + LocationSite = LocationSite + OrganizationTenant = OrganizationTenant diff --git a/examples/device42_to_infrahub/genericrestapi/sync_models.py b/examples/device42_to_infrahub/genericrestapi/sync_models.py new file mode 100644 index 0000000..41e7cda --- /dev/null +++ b/examples/device42_to_infrahub/genericrestapi/sync_models.py @@ -0,0 +1,42 @@ +from __future__ import annotations + +from typing import Any + +from infrahub_sync.adapters.genericrestapi import GenericrestapiModel + + +# ------------------------------------------------------- +# AUTO-GENERATED FILE, DO NOT MODIFY +# This file has been generated with the command `infrahub-sync generate` +# All modifications will be lost the next time you reexecute this command +# ------------------------------------------------------- +class BuiltinTag(GenericrestapiModel): + _modelname = "BuiltinTag" + _identifiers = ("name",) + _attributes = () + name: str + + local_id: str | None = None + local_data: Any | None = None + + +class LocationSite(GenericrestapiModel): + _modelname = "LocationSite" + _identifiers = ("name",) + _attributes = ("tags",) + name: str + tags: list[str] | None = [] + + local_id: str | None = None + local_data: Any | None = None + + +class OrganizationTenant(GenericrestapiModel): + _modelname = "OrganizationTenant" + _identifiers = ("name",) + _attributes = ("description",) + description: str | None = None + name: str + + local_id: str | None = None + local_data: Any | None = None diff --git a/examples/peeringdb_to_infrahub/generic_rest_api/__init__.py b/examples/device42_to_infrahub/infrahub/__init__.py similarity index 100% rename from examples/peeringdb_to_infrahub/generic_rest_api/__init__.py rename to examples/device42_to_infrahub/infrahub/__init__.py diff --git a/examples/device42_to_infrahub/infrahub/sync_adapter.py b/examples/device42_to_infrahub/infrahub/sync_adapter.py new file mode 100644 index 0000000..a1210b3 --- /dev/null +++ b/examples/device42_to_infrahub/infrahub/sync_adapter.py @@ -0,0 +1,18 @@ +from infrahub_sync.adapters.infrahub import InfrahubAdapter + +from .sync_models import ( + BuiltinTag, + LocationSite, + OrganizationTenant, +) + + +# ------------------------------------------------------- +# AUTO-GENERATED FILE, DO NOT MODIFY +# This file has been generated with the command `infrahub-sync generate` +# All modifications will be lost the next time you reexecute this command +# ------------------------------------------------------- +class InfrahubSync(InfrahubAdapter): + BuiltinTag = BuiltinTag + LocationSite = LocationSite + OrganizationTenant = OrganizationTenant diff --git a/examples/device42_to_infrahub/infrahub/sync_models.py b/examples/device42_to_infrahub/infrahub/sync_models.py new file mode 100644 index 0000000..d8e14ef --- /dev/null +++ b/examples/device42_to_infrahub/infrahub/sync_models.py @@ -0,0 +1,42 @@ +from __future__ import annotations + +from typing import Any + +from infrahub_sync.adapters.infrahub import InfrahubModel + + +# ------------------------------------------------------- +# AUTO-GENERATED FILE, DO NOT MODIFY +# This file has been generated with the command `infrahub-sync generate` +# All modifications will be lost the next time you reexecute this command +# ------------------------------------------------------- +class BuiltinTag(InfrahubModel): + _modelname = "BuiltinTag" + _identifiers = ("name",) + _attributes = () + name: str + + local_id: str | None = None + local_data: Any | None = None + + +class LocationSite(InfrahubModel): + _modelname = "LocationSite" + _identifiers = ("name",) + _attributes = ("tags",) + name: str + tags: list[str] | None = [] + + local_id: str | None = None + local_data: Any | None = None + + +class OrganizationTenant(InfrahubModel): + _modelname = "OrganizationTenant" + _identifiers = ("name",) + _attributes = ("description",) + description: str | None = None + name: str + + local_id: str | None = None + local_data: Any | None = None diff --git a/examples/peeringdb_to_infrahub/genenericrestapi/sync_adapter.py b/examples/peeringdb_to_infrahub/genenericrestapi/sync_adapter.py deleted file mode 100644 index 616c643..0000000 --- a/examples/peeringdb_to_infrahub/genenericrestapi/sync_adapter.py +++ /dev/null @@ -1,28 +0,0 @@ -from infrahub_sync.adapters.genenericrestapi import GenenericrestapiAdapter - -from .sync_models import ( - InfraAutonomousSystem, - InfraBGPCommunity, - InfraBGPPeerGroup, - InfraBGPRoutingPolicy, - InfraIXP, - InfraIXPConnection, - IpamIPAddress, - OrganizationProvider, -) - - -# ------------------------------------------------------- -# AUTO-GENERATED FILE, DO NOT MODIFY -# This file has been generated with the command `infrahub-sync generate` -# All modifications will be lost the next time you reexecute this command -# ------------------------------------------------------- -class GenenericrestapiSync(GenenericrestapiAdapter): - InfraAutonomousSystem = InfraAutonomousSystem - InfraBGPPeerGroup = InfraBGPPeerGroup - IpamIPAddress = IpamIPAddress - OrganizationProvider = OrganizationProvider - InfraBGPCommunity = InfraBGPCommunity - InfraBGPRoutingPolicy = InfraBGPRoutingPolicy - InfraIXP = InfraIXP - InfraIXPConnection = InfraIXPConnection diff --git a/examples/peeringdb_to_infrahub/genenericrestapi/sync_models.py b/examples/peeringdb_to_infrahub/genenericrestapi/sync_models.py deleted file mode 100644 index 3e095d1..0000000 --- a/examples/peeringdb_to_infrahub/genenericrestapi/sync_models.py +++ /dev/null @@ -1,141 +0,0 @@ -from __future__ import annotations - -from typing import Any - -from infrahub_sync.adapters.genenericrestapi import GenenericrestapiModel - - -# ------------------------------------------------------- -# AUTO-GENERATED FILE, DO NOT MODIFY -# This file has been generated with the command `infrahub-sync generate` -# All modifications will be lost the next time you reexecute this command -# ------------------------------------------------------- -class InfraAutonomousSystem(GenenericrestapiModel): - _modelname = "InfraAutonomousSystem" - _identifiers = ("asn",) - _attributes = ( - "organization", - "affiliated", - "irr_as_set", - "name", - "ipv4_max_prefixes", - "description", - "ipv6_max_prefixes", - ) - asn: int - affiliated: bool | None = None - irr_as_set: str | None = None - name: str - ipv4_max_prefixes: int | None = None - description: str | None = None - ipv6_max_prefixes: int | None = None - organization: str | None = None - - local_id: str | None = None - local_data: Any | None = None - - -class InfraBGPPeerGroup(GenenericrestapiModel): - _modelname = "InfraBGPPeerGroup" - _identifiers = ("name",) - _attributes = ("bgp_communities", "import_policies", "export_policies", "description", "status") - name: str - description: str | None = None - status: str | None = None - bgp_communities: list[str] | None = [] - import_policies: list[str] | None = [] - export_policies: list[str] | None = [] - - local_id: str | None = None - local_data: Any | None = None - - -class IpamIPAddress(GenenericrestapiModel): - _modelname = "IpamIPAddress" - _identifiers = ("address",) - _attributes = ("description",) - description: str | None = None - address: str - - local_id: str | None = None - local_data: Any | None = None - - -class OrganizationProvider(GenenericrestapiModel): - _modelname = "OrganizationProvider" - _identifiers = ("name",) - _attributes = () - name: str - - local_id: str | None = None - local_data: Any | None = None - - -class InfraBGPCommunity(GenenericrestapiModel): - _modelname = "InfraBGPCommunity" - _identifiers = ("name",) - _attributes = ("description", "label", "community_type", "value") - name: str - description: str | None = None - label: str | None = None - community_type: str | None = None - value: str - - local_id: str | None = None - local_data: Any | None = None - - -class InfraBGPRoutingPolicy(GenenericrestapiModel): - _modelname = "InfraBGPRoutingPolicy" - _identifiers = ("name",) - _attributes = ("bgp_communities", "address_family", "policy_type", "label", "weight", "description") - address_family: int - policy_type: str - label: str | None = None - weight: int | None = 1000 - name: str - description: str | None = None - bgp_communities: list[str] | None = [] - - local_id: str | None = None - local_data: Any | None = None - - -class InfraIXP(GenenericrestapiModel): - _modelname = "InfraIXP" - _identifiers = ("name",) - _attributes = ("export_policies", "bgp_communities", "import_policies", "status", "description") - status: str | None = "enabled" - name: str - description: str | None = None - export_policies: list[str] | None = [] - bgp_communities: list[str] | None = [] - import_policies: list[str] | None = [] - - local_id: str | None = None - local_data: Any | None = None - - -class InfraIXPConnection(GenenericrestapiModel): - _modelname = "InfraIXPConnection" - _identifiers = ("name",) - _attributes = ( - "internet_exchange_point", - "ipv4_address", - "ipv6_address", - "status", - "peeringdb_netixlan", - "vlan", - "description", - ) - name: str - status: str | None = "enabled" - peeringdb_netixlan: int | None = None - vlan: int | None = None - description: str | None = None - internet_exchange_point: str - ipv4_address: str | None = None - ipv6_address: str | None = None - - local_id: str | None = None - local_data: Any | None = None diff --git a/examples/peeringdb_to_infrahub/generic_rest_api/sync_adapter.py b/examples/peeringdb_to_infrahub/generic_rest_api/sync_adapter.py deleted file mode 100644 index 376377d..0000000 --- a/examples/peeringdb_to_infrahub/generic_rest_api/sync_adapter.py +++ /dev/null @@ -1,28 +0,0 @@ -from infrahub_sync.adapters.generic_rest_api import Generic_Rest_ApiAdapter - -from .sync_models import ( - InfraAutonomousSystem, - InfraBGPCommunity, - InfraBGPPeerGroup, - InfraBGPRoutingPolicy, - InfraIXP, - InfraIXPConnection, - IpamIPAddress, - OrganizationProvider, -) - - -# ------------------------------------------------------- -# AUTO-GENERATED FILE, DO NOT MODIFY -# This file has been generated with the command `infrahub-sync generate` -# All modifications will be lost the next time you reexecute this command -# ------------------------------------------------------- -class Generic_Rest_ApiSync(Generic_Rest_ApiAdapter): - InfraAutonomousSystem = InfraAutonomousSystem - InfraBGPPeerGroup = InfraBGPPeerGroup - IpamIPAddress = IpamIPAddress - OrganizationProvider = OrganizationProvider - InfraBGPRoutingPolicy = InfraBGPRoutingPolicy - InfraBGPCommunity = InfraBGPCommunity - InfraIXP = InfraIXP - InfraIXPConnection = InfraIXPConnection diff --git a/examples/peeringdb_to_infrahub/generic_rest_api/sync_models.py b/examples/peeringdb_to_infrahub/generic_rest_api/sync_models.py deleted file mode 100644 index abe1283..0000000 --- a/examples/peeringdb_to_infrahub/generic_rest_api/sync_models.py +++ /dev/null @@ -1,141 +0,0 @@ -from __future__ import annotations - -from typing import Any - -from infrahub_sync.adapters.generic_rest_api import Generic_Rest_ApiModel - - -# ------------------------------------------------------- -# AUTO-GENERATED FILE, DO NOT MODIFY -# This file has been generated with the command `infrahub-sync generate` -# All modifications will be lost the next time you reexecute this command -# ------------------------------------------------------- -class InfraAutonomousSystem(Generic_Rest_ApiModel): - _modelname = "InfraAutonomousSystem" - _identifiers = ("asn",) - _attributes = ( - "organization", - "affiliated", - "irr_as_set", - "name", - "ipv4_max_prefixes", - "description", - "ipv6_max_prefixes", - ) - asn: int - affiliated: bool | None = None - irr_as_set: str | None = None - name: str - ipv4_max_prefixes: int | None = None - description: str | None = None - ipv6_max_prefixes: int | None = None - organization: str | None = None - - local_id: str | None = None - local_data: Any | None = None - - -class InfraBGPPeerGroup(Generic_Rest_ApiModel): - _modelname = "InfraBGPPeerGroup" - _identifiers = ("name",) - _attributes = ("import_policies", "export_policies", "bgp_communities", "description", "status") - name: str - description: str | None = None - status: str | None = None - import_policies: list[str] | None = [] - export_policies: list[str] | None = [] - bgp_communities: list[str] | None = [] - - local_id: str | None = None - local_data: Any | None = None - - -class IpamIPAddress(Generic_Rest_ApiModel): - _modelname = "IpamIPAddress" - _identifiers = ("address",) - _attributes = ("description",) - description: str | None = None - address: str - - local_id: str | None = None - local_data: Any | None = None - - -class OrganizationProvider(Generic_Rest_ApiModel): - _modelname = "OrganizationProvider" - _identifiers = ("name",) - _attributes = () - name: str - - local_id: str | None = None - local_data: Any | None = None - - -class InfraBGPRoutingPolicy(Generic_Rest_ApiModel): - _modelname = "InfraBGPRoutingPolicy" - _identifiers = ("name",) - _attributes = ("bgp_communities", "label", "description", "policy_type", "weight", "address_family") - name: str - label: str | None = None - description: str | None = None - policy_type: str - weight: int | None = 1000 - address_family: int - bgp_communities: list[str] | None = [] - - local_id: str | None = None - local_data: Any | None = None - - -class InfraBGPCommunity(Generic_Rest_ApiModel): - _modelname = "InfraBGPCommunity" - _identifiers = ("name",) - _attributes = ("description", "label", "community_type", "value") - name: str - description: str | None = None - label: str | None = None - community_type: str | None = None - value: str - - local_id: str | None = None - local_data: Any | None = None - - -class InfraIXP(Generic_Rest_ApiModel): - _modelname = "InfraIXP" - _identifiers = ("name",) - _attributes = ("export_policies", "bgp_communities", "import_policies", "description", "status") - name: str - description: str | None = None - status: str | None = "enabled" - export_policies: list[str] | None = [] - bgp_communities: list[str] | None = [] - import_policies: list[str] | None = [] - - local_id: str | None = None - local_data: Any | None = None - - -class InfraIXPConnection(Generic_Rest_ApiModel): - _modelname = "InfraIXPConnection" - _identifiers = ("name",) - _attributes = ( - "internet_exchange_point", - "ipv4_address", - "ipv6_address", - "status", - "peeringdb_netixlan", - "vlan", - "description", - ) - name: str - status: str | None = "enabled" - peeringdb_netixlan: int | None = None - vlan: int | None = None - description: str | None = None - internet_exchange_point: str - ipv4_address: str | None = None - ipv6_address: str | None = None - - local_id: str | None = None - local_data: Any | None = None diff --git a/examples/peeringdb_to_infrahub/genericrestapi/sync_models.py b/examples/peeringdb_to_infrahub/genericrestapi/sync_models.py index f07111e..a7eec18 100644 --- a/examples/peeringdb_to_infrahub/genericrestapi/sync_models.py +++ b/examples/peeringdb_to_infrahub/genericrestapi/sync_models.py @@ -13,12 +13,12 @@ class InfraAutonomousSystem(GenericrestapiModel): _modelname = "InfraAutonomousSystem" _identifiers = ("asn",) - _attributes = ("irr_as_set", "name", "ipv4_max_prefixes", "ipv6_max_prefixes") - asn: int - irr_as_set: str | None = None + _attributes = ("ipv6_max_prefixes", "name", "ipv4_max_prefixes", "irr_as_set") + ipv6_max_prefixes: int | None = None name: str ipv4_max_prefixes: int | None = None - ipv6_max_prefixes: int | None = None + asn: int + irr_as_set: str | None = None local_id: str | None = None local_data: Any | None = None diff --git a/examples/peeringdb_to_infrahub/infrahub/sync_models.py b/examples/peeringdb_to_infrahub/infrahub/sync_models.py index dd1b07c..7d43792 100644 --- a/examples/peeringdb_to_infrahub/infrahub/sync_models.py +++ b/examples/peeringdb_to_infrahub/infrahub/sync_models.py @@ -13,12 +13,12 @@ class InfraAutonomousSystem(InfrahubModel): _modelname = "InfraAutonomousSystem" _identifiers = ("asn",) - _attributes = ("irr_as_set", "name", "ipv4_max_prefixes", "ipv6_max_prefixes") - asn: int - irr_as_set: str | None = None + _attributes = ("ipv6_max_prefixes", "name", "ipv4_max_prefixes", "irr_as_set") + ipv6_max_prefixes: int | None = None name: str ipv4_max_prefixes: int | None = None - ipv6_max_prefixes: int | None = None + asn: int + irr_as_set: str | None = None local_id: str | None = None local_data: Any | None = None diff --git a/examples/prometheus_to_infrahub (node_exporter)/infrahub/sync_adapter.py b/examples/prometheus_to_infrahub (node_exporter)/infrahub/sync_adapter.py index d91c544..5cdfe8d 100644 --- a/examples/prometheus_to_infrahub (node_exporter)/infrahub/sync_adapter.py +++ b/examples/prometheus_to_infrahub (node_exporter)/infrahub/sync_adapter.py @@ -1,10 +1,10 @@ from infrahub_sync.adapters.infrahub import InfrahubAdapter from .sync_models import ( + VirtualizationVirtualMachine, VirtualizationVMDisk, VirtualizationVMFilesystem, VirtualizationVMNetworkInterface, - VirtualizationVirtualMachine, ) diff --git a/examples/prometheus_to_infrahub (node_exporter)/infrahub/sync_models.py b/examples/prometheus_to_infrahub (node_exporter)/infrahub/sync_models.py index 79fbe14..17dd6c3 100644 --- a/examples/prometheus_to_infrahub (node_exporter)/infrahub/sync_models.py +++ b/examples/prometheus_to_infrahub (node_exporter)/infrahub/sync_models.py @@ -1,9 +1,10 @@ from __future__ import annotations -from typing import Any, List +from typing import Any from infrahub_sync.adapters.infrahub import InfrahubModel + # ------------------------------------------------------- # AUTO-GENERATED FILE, DO NOT MODIFY # This file has been generated with the command `infrahub-sync generate` @@ -20,6 +21,7 @@ class VirtualizationVMDisk(InfrahubModel): local_id: str | None = None local_data: Any | None = None + class VirtualizationVMFilesystem(InfrahubModel): _modelname = "VirtualizationVMFilesystem" _identifiers = ("virtual_machine", "mountpoint") @@ -35,6 +37,7 @@ class VirtualizationVMFilesystem(InfrahubModel): local_id: str | None = None local_data: Any | None = None + class VirtualizationVMNetworkInterface(InfrahubModel): _modelname = "VirtualizationVMNetworkInterface" _identifiers = ("virtual_machine", "name") @@ -51,10 +54,19 @@ class VirtualizationVMNetworkInterface(InfrahubModel): local_id: str | None = None local_data: Any | None = None + class VirtualizationVirtualMachine(InfrahubModel): _modelname = "VirtualizationVirtualMachine" _identifiers = ("name",) - _attributes = ("os_name", "ip_forwarding_enabled", "os_kernel", "status", "architecture", "conntrack_limit", "mem_total_bytes") + _attributes = ( + "os_name", + "ip_forwarding_enabled", + "os_kernel", + "status", + "architecture", + "conntrack_limit", + "mem_total_bytes", + ) os_name: str | None = None ip_forwarding_enabled: bool | None = None os_kernel: str | None = None diff --git a/examples/prometheus_to_infrahub (node_exporter)/prometheus/sync_adapter.py b/examples/prometheus_to_infrahub (node_exporter)/prometheus/sync_adapter.py index 9967a03..c1bd69f 100644 --- a/examples/prometheus_to_infrahub (node_exporter)/prometheus/sync_adapter.py +++ b/examples/prometheus_to_infrahub (node_exporter)/prometheus/sync_adapter.py @@ -1,10 +1,10 @@ from infrahub_sync.adapters.prometheus import PrometheusAdapter from .sync_models import ( + VirtualizationVirtualMachine, VirtualizationVMDisk, VirtualizationVMFilesystem, VirtualizationVMNetworkInterface, - VirtualizationVirtualMachine, ) diff --git a/examples/prometheus_to_infrahub (node_exporter)/prometheus/sync_models.py b/examples/prometheus_to_infrahub (node_exporter)/prometheus/sync_models.py index 11523da..573ba10 100644 --- a/examples/prometheus_to_infrahub (node_exporter)/prometheus/sync_models.py +++ b/examples/prometheus_to_infrahub (node_exporter)/prometheus/sync_models.py @@ -1,9 +1,10 @@ from __future__ import annotations -from typing import Any, List +from typing import Any from infrahub_sync.adapters.prometheus import PrometheusModel + # ------------------------------------------------------- # AUTO-GENERATED FILE, DO NOT MODIFY # This file has been generated with the command `infrahub-sync generate` @@ -20,6 +21,7 @@ class VirtualizationVMDisk(PrometheusModel): local_id: str | None = None local_data: Any | None = None + class VirtualizationVMFilesystem(PrometheusModel): _modelname = "VirtualizationVMFilesystem" _identifiers = ("virtual_machine", "mountpoint") @@ -35,6 +37,7 @@ class VirtualizationVMFilesystem(PrometheusModel): local_id: str | None = None local_data: Any | None = None + class VirtualizationVMNetworkInterface(PrometheusModel): _modelname = "VirtualizationVMNetworkInterface" _identifiers = ("virtual_machine", "name") @@ -51,10 +54,19 @@ class VirtualizationVMNetworkInterface(PrometheusModel): local_id: str | None = None local_data: Any | None = None + class VirtualizationVirtualMachine(PrometheusModel): _modelname = "VirtualizationVirtualMachine" _identifiers = ("name",) - _attributes = ("os_name", "ip_forwarding_enabled", "os_kernel", "status", "architecture", "conntrack_limit", "mem_total_bytes") + _attributes = ( + "os_name", + "ip_forwarding_enabled", + "os_kernel", + "status", + "architecture", + "conntrack_limit", + "mem_total_bytes", + ) os_name: str | None = None ip_forwarding_enabled: bool | None = None os_kernel: str | None = None diff --git a/examples/python_api_example.py b/examples/python_api_example.py new file mode 100644 index 0000000..b672187 --- /dev/null +++ b/examples/python_api_example.py @@ -0,0 +1,125 @@ +#!/usr/bin/env python3 +"""Example usage of the Infrahub Sync Python API. + +This script demonstrates how to use the Python API for Infrahub Sync +instead of the command line interface. +""" + +from infrahub_sync.api import ( + SyncError, + list_projects, + diff, + sync, + create_potenda, +) + + +def main(): + """Main example demonstrating API usage.""" + print("=== Infrahub Sync Python API Example ===\n") + + # Example 1: List all available sync projects + print("1. Listing all available sync projects:") + try: + projects = list_projects() + if projects: + for project in projects: + print(f" - {project.name}: {project.source.name} -> {project.destination.name}") + else: + print(" No sync projects found.") + except Exception as e: + print(f" Error listing projects: {e}") + print() + + # Example 2: Perform a diff operation (commented out as we don't have real config) + print("2. Example diff operation (requires actual sync configuration):") + print(""" + try: + result = diff(name="my-sync", show_progress=True) + if result.success: + print(f" Diff completed in {result.duration:.2f} seconds") + if result.changes_detected: + print(f" Changes detected:\\n{result.message}") + else: + print(" No changes detected") + else: + print(f" Diff failed: {result.error}") + except SyncError as e: + print(f" Sync error: {e}") + """) + + # Example 3: Perform a sync operation (commented out as we don't have real config) + print("3. Example sync operation (requires actual sync configuration):") + print(""" + try: + result = sync(name="my-sync", diff_first=True, show_progress=True) + if result.success: + print(f" Sync completed in {result.duration:.2f} seconds") + print(f" Result: {result.message}") + else: + print(f" Sync failed: {result.error}") + except SyncError as e: + print(f" Sync error: {e}") + """) + + # Example 4: Advanced usage with Potenda object + print("4. Example advanced usage with Potenda object:") + print(""" + try: + # Create a Potenda instance for fine-grained control + ptd = create_potenda(name="my-sync", show_progress=True) + + # Load data from source and destination + ptd.source_load() + ptd.destination_load() + + # Calculate differences + diff_result = ptd.diff() + + if diff_result.has_diffs(): + print(" Changes detected, proceeding with sync...") + ptd.sync(diff=diff_result) + print(" Sync completed successfully!") + else: + print(" No changes to sync.") + + except SyncError as e: + print(f" Sync error: {e}") + """) + + # Example 5: Error handling + print("5. Example error handling:") + print(""" + try: + # This will fail because we're not providing name or config_file + result = diff() + except SyncError as e: + print(f" Expected error: {e}") + """) + + print("\n=== API Documentation ===") + print(""" +Available functions in infrahub_sync.api: + +1. list_projects(directory=None) -> List[SyncInstance] + - Lists all available sync configurations + +2. diff(name=None, config_file=None, directory=None, branch=None, show_progress=True) -> SyncResult + - Calculates differences between source and destination + +3. sync(name=None, config_file=None, directory=None, branch=None, diff_first=True, show_progress=True) -> SyncResult + - Synchronizes data from source to destination + +4. create_potenda(name=None, config_file=None, directory=None, branch=None, show_progress=True) -> Potenda + - Creates a Potenda instance for advanced programmatic control + +Classes: +- SyncError: Exception raised when sync operations fail +- SyncResult: Result object returned by sync operations with success status, messages, timing, etc. + +For more information, see the docstrings in infrahub_sync.api module. +""") + + +if __name__ == "__main__": + main() \ No newline at end of file diff --git a/infrahub_sync/adapters/rest_api_client.py b/infrahub_sync/adapters/rest_api_client.py index 573ab40..65dcdcb 100644 --- a/infrahub_sync/adapters/rest_api_client.py +++ b/infrahub_sync/adapters/rest_api_client.py @@ -25,7 +25,7 @@ def __init__( # Determine authentication method, some are use by more than one API. # Example: - # -> Peering Manager + # -> Peering Manager & Device42 if auth_method == "token" and api_token: self.headers["Authorization"] = f"Token {api_token}" # -> LibreNMS @@ -37,7 +37,7 @@ def __init__( # -> RIPE API elif auth_method == "key" and api_token: self.headers["Authorization"] = f"Key {api_token}" - # -> Observium + # -> Observium & Device42 elif auth_method == "basic" and username and password: self.auth = (username, password) elif auth_method == "none": diff --git a/infrahub_sync/api.py b/infrahub_sync/api.py new file mode 100644 index 0000000..f97f860 --- /dev/null +++ b/infrahub_sync/api.py @@ -0,0 +1,273 @@ +"""Python API for Infrahub Sync. + +This module provides a Python API to perform sync operations +programmatically without using the CLI. +""" + +from __future__ import annotations + +from timeit import default_timer as timer +from typing import TYPE_CHECKING, Any + +from rich.console import Console + +from infrahub_sync.utils import ( + get_all_sync, + get_instance, + get_potenda_from_instance, +) + +if TYPE_CHECKING: + from infrahub_sync import SyncInstance + from infrahub_sync.potenda import Potenda + +console = Console() + + +class SyncError(Exception): + """Raised when sync operations fail.""" + + pass + + +class SyncResult: + """Result object returned by sync operations.""" + + def __init__( + self, + success: bool, + message: str | None = None, + duration: float | None = None, + changes_detected: bool | None = None, + error: Exception | None = None, + ): + self.success = success + self.message = message + self.duration = duration + self.changes_detected = changes_detected + self.error = error + + def __repr__(self) -> str: + return f"SyncResult(success={self.success}, message='{self.message}', duration={self.duration})" + + +def list_projects(directory: str | None = None) -> list[SyncInstance]: + """List all available sync projects. + + Args: + directory: Base directory to search for sync configurations. + If None, searches from the package directory. + + Returns: + List of SyncInstance objects representing available sync projects. + + Example: + >>> from infrahub_sync.api import list_projects + >>> projects = list_projects() + >>> for project in projects: + ... print(f"{project.name}: {project.source.name} -> {project.destination.name}") + """ + return get_all_sync(directory=directory) + + +def diff( + name: str | None = None, + config_file: str | None = None, + directory: str | None = None, + branch: str | None = None, + show_progress: bool = True, +) -> SyncResult: + """Calculate and return the differences between source and destination systems. + + Args: + name: Name of the sync to use (mutually exclusive with config_file). + config_file: File path to the sync configuration YAML file. + directory: Base directory to search for sync configurations. + branch: Branch to use for the diff. + show_progress: Show a progress bar during diff. + + Returns: + SyncResult object containing diff information and success status. + + Raises: + SyncError: When exactly one of name or config_file is not specified, + or when sync initialization/loading fails. + + Example: + >>> from infrahub_sync.api import diff + >>> result = diff(name="my-sync") + >>> if result.success: + ... print(f"Diff completed: {result.message}") + >>> else: + ... print(f"Diff failed: {result.error}") + """ + if sum([bool(name), bool(config_file)]) != 1: + raise SyncError("Please specify exactly one of 'name' or 'config_file'.") + + sync_instance = get_instance(name=name, config_file=config_file, directory=directory) + if not sync_instance: + raise SyncError("Failed to load sync instance.") + + try: + ptd = get_potenda_from_instance(sync_instance=sync_instance, branch=branch, show_progress=show_progress) + except ValueError as exc: + raise SyncError(f"Failed to initialize the Sync Instance: {exc}") from exc + + try: + ptd.source_load() + ptd.destination_load() + except ValueError as exc: + raise SyncError(f"Failed to load data: {exc}") from exc + + start_time = timer() + mydiff = ptd.diff() + end_time = timer() + + changes_detected = mydiff.has_diffs() + diff_str = mydiff.str() if changes_detected else "No differences found" + + return SyncResult( + success=True, + message=diff_str, + duration=end_time - start_time, + changes_detected=changes_detected, + ) + + +def sync( + name: str | None = None, + config_file: str | None = None, + directory: str | None = None, + branch: str | None = None, + diff_first: bool = True, + show_progress: bool = True, +) -> SyncResult: + """Synchronize data between source and destination systems. + + Args: + name: Name of the sync to use (mutually exclusive with config_file). + config_file: File path to the sync configuration YAML file. + directory: Base directory to search for sync configurations. + branch: Branch to use for the sync. + diff_first: Calculate and show differences before syncing. + show_progress: Show a progress bar during syncing. + + Returns: + SyncResult object containing sync status and information. + + Raises: + SyncError: When exactly one of name or config_file is not specified, + or when sync initialization/loading fails. + + Example: + >>> from infrahub_sync.api import sync + >>> result = sync(name="my-sync") + >>> if result.success: + ... print(f"Sync completed in {result.duration:.2f} seconds") + >>> else: + ... print(f"Sync failed: {result.error}") + """ + if sum([bool(name), bool(config_file)]) != 1: + raise SyncError("Please specify exactly one of 'name' or 'config_file'.") + + sync_instance = get_instance(name=name, config_file=config_file, directory=directory) + if not sync_instance: + raise SyncError("Failed to load sync instance.") + + try: + ptd = get_potenda_from_instance(sync_instance=sync_instance, branch=branch, show_progress=show_progress) + except ValueError as exc: + raise SyncError(f"Failed to initialize the Sync Instance: {exc}") from exc + + try: + ptd.source_load() + ptd.destination_load() + except ValueError as exc: + raise SyncError(f"Failed to load data: {exc}") from exc + + mydiff = ptd.diff() + + if mydiff.has_diffs(): + diff_message = f"Found differences to sync" + (f":\n{mydiff.str()}" if diff_first else "") + start_synctime = timer() + ptd.sync(diff=mydiff) + end_synctime = timer() + + return SyncResult( + success=True, + message=f"Sync completed. {diff_message}", + duration=end_synctime - start_synctime, + changes_detected=True, + ) + else: + return SyncResult( + success=True, + message="No differences found. Nothing to sync.", + duration=0.0, + changes_detected=False, + ) + + +def create_potenda( + name: str | None = None, + config_file: str | None = None, + directory: str | None = None, + branch: str | None = None, + show_progress: bool = True, +) -> Potenda: + """Create a Potenda instance for advanced programmatic control. + + This function provides direct access to the Potenda object for users who need + fine-grained control over the sync process, allowing them to call individual + methods like source_load(), destination_load(), diff(), and sync(). + + Args: + name: Name of the sync to use (mutually exclusive with config_file). + config_file: File path to the sync configuration YAML file. + directory: Base directory to search for sync configurations. + branch: Branch to use for the sync. + show_progress: Show a progress bar during operations. + + Returns: + Potenda instance configured with the specified sync configuration. + + Raises: + SyncError: When exactly one of name or config_file is not specified, + or when sync initialization fails. + + Example: + >>> from infrahub_sync.api import create_potenda + >>> ptd = create_potenda(name="my-sync") + >>> ptd.source_load() + >>> ptd.destination_load() + >>> diff = ptd.diff() + >>> if diff.has_diffs(): + ... ptd.sync(diff=diff) + """ + if sum([bool(name), bool(config_file)]) != 1: + raise SyncError("Please specify exactly one of 'name' or 'config_file'.") + + sync_instance = get_instance(name=name, config_file=config_file, directory=directory) + if not sync_instance: + raise SyncError("Failed to load sync instance.") + + try: + return get_potenda_from_instance(sync_instance=sync_instance, branch=branch, show_progress=show_progress) + except ValueError as exc: + raise SyncError(f"Failed to initialize the Sync Instance: {exc}") from exc + + +# Convenient aliases for common operations +diff_sync = diff # Alternative name for clarity +sync_data = sync # Alternative name for clarity + +__all__ = [ + "SyncError", + "SyncResult", + "list_projects", + "diff", + "sync", + "create_potenda", + "diff_sync", + "sync_data", +] \ No newline at end of file diff --git a/infrahub_sync/cli.py b/infrahub_sync/cli.py index c43b7e3..e34afef 100644 --- a/infrahub_sync/cli.py +++ b/infrahub_sync/cli.py @@ -1,5 +1,4 @@ import logging -from timeit import default_timer as timer from typing import TYPE_CHECKING import typer @@ -7,12 +6,11 @@ from infrahub_sdk.exceptions import ServerNotResponsiveError from rich.console import Console +from infrahub_sync.api import SyncError, diff as api_diff, list_projects as api_list_projects, sync as api_sync from infrahub_sync.utils import ( find_missing_schema_model, - get_all_sync, get_infrahub_config, get_instance, - get_potenda_from_instance, render_adapter, ) @@ -35,8 +33,12 @@ def list_projects( directory: str = typer.Option(default=None, help="Base directory to search for sync configurations"), ) -> None: """List all available SYNC projects.""" - for item in get_all_sync(directory=directory): - console.print(f"{item.name} | {item.source.name} >> {item.destination.name} | {item.directory}") + try: + projects = api_list_projects(directory=directory) + for project in projects: + console.print(f"{project.name} | {project.source.name} >> {project.destination.name} | {project.directory}") + except Exception as exc: + print_error_and_abort(f"Failed to list projects: {exc}") @app.command(name="diff") @@ -48,27 +50,21 @@ def diff_cmd( show_progress: bool = typer.Option(default=True, help="Show a progress bar during diff"), ) -> None: """Calculate and print the differences between the source and the destination systems for a given project.""" - if sum([bool(name), bool(config_file)]) != 1: - print_error_and_abort("Please specify exactly one of 'name' or 'config-file'.") - - sync_instance = get_instance(name=name, config_file=config_file, directory=directory) - if not sync_instance: - print_error_and_abort("Failed to load sync instance.") - - try: - ptd = get_potenda_from_instance(sync_instance=sync_instance, branch=branch, show_progress=show_progress) - except ValueError as exc: - print_error_and_abort(f"Failed to initialize the Sync Instance: {exc}") try: - ptd.source_load() - ptd.destination_load() - except ValueError as exc: + result = api_diff( + name=name, + config_file=config_file, + directory=directory, + branch=branch, + show_progress=show_progress, + ) + if result.success: + print(result.message) + else: + print_error_and_abort(f"Diff failed: {result.error}") + except SyncError as exc: print_error_and_abort(str(exc)) - mydiff = ptd.diff() - - print(mydiff.str()) - @app.command(name="sync") def sync_cmd( @@ -83,35 +79,25 @@ def sync_cmd( show_progress: bool = typer.Option(default=True, help="Show a progress bar during syncing"), ) -> None: """Synchronize the data between source and the destination systems for a given project or configuration file.""" - if sum([bool(name), bool(config_file)]) != 1: - print_error_and_abort("Please specify exactly one of 'name' or 'config-file'.") - - sync_instance = get_instance(name=name, config_file=config_file, directory=directory) - if not sync_instance: - print_error_and_abort("Failed to load sync instance.") - - try: - ptd = get_potenda_from_instance(sync_instance=sync_instance, branch=branch, show_progress=show_progress) - except ValueError as exc: - print_error_and_abort(f"Failed to initialize the Sync Instance: {exc}") try: - ptd.source_load() - ptd.destination_load() - except ValueError as exc: + result = api_sync( + name=name, + config_file=config_file, + directory=directory, + branch=branch, + diff_first=diff, + show_progress=show_progress, + ) + if result.success: + if result.changes_detected: + console.print(f"Sync: Completed in {result.duration} sec") + else: + console.print("No difference found. Nothing to sync") + else: + print_error_and_abort(f"Sync failed: {result.error}") + except SyncError as exc: print_error_and_abort(str(exc)) - mydiff = ptd.diff() - - if mydiff.has_diffs(): - if diff: - print(mydiff.str()) - start_synctime = timer() - ptd.sync(diff=mydiff) - end_synctime = timer() - console.print(f"Sync: Completed in {end_synctime - start_synctime} sec") - else: - console.print("No difference found. Nothing to sync") - @app.command(name="generate") def generate( diff --git a/infrahub_sync/utils.py b/infrahub_sync/utils.py index 9eaf1d0..24fa815 100644 --- a/infrahub_sync/utils.py +++ b/infrahub_sync/utils.py @@ -10,10 +10,12 @@ from diffsync.store.redis import RedisStore from infrahub_sdk import Config -from infrahub_sync import SyncAdapter, SyncConfig, SyncInstance from infrahub_sync.generator import render_template from infrahub_sync.potenda import Potenda +if TYPE_CHECKING: + from infrahub_sync import SyncAdapter, SyncConfig, SyncInstance + if TYPE_CHECKING: from collections.abc import MutableMapping @@ -24,6 +26,9 @@ def find_missing_schema_model( sync_instance: SyncInstance, schema: MutableMapping[str, Union[NodeSchema, GenericSchema]], ) -> list[str]: + # Import at runtime to avoid circular dependency + from infrahub_sync import SyncInstance + missing_schema_models = [] for item in sync_instance.schema_mapping: match_found = any(item.name == node.kind for node in schema.values()) @@ -38,6 +43,9 @@ def render_adapter( sync_instance: SyncInstance, schema: MutableMapping[str, Union[NodeSchema, GenericSchema]], ) -> list[tuple[str, str]]: + # Import at runtime to avoid circular dependency + from infrahub_sync import SyncInstance + files_to_render = ( ("diffsync_models.j2", "sync_models.py"), ("diffsync_adapter.j2", "sync_adapter.py"), @@ -66,6 +74,9 @@ def render_adapter( def import_adapter(sync_instance: SyncInstance, adapter: SyncAdapter): + # Import at runtime to avoid circular dependency + from infrahub_sync import SyncAdapter, SyncInstance + directory = Path(sync_instance.directory) sys.path.insert(0, str(directory)) adapter_file_path = directory / f"{adapter.name}" / "sync_adapter.py" @@ -90,6 +101,9 @@ def import_adapter(sync_instance: SyncInstance, adapter: SyncAdapter): def get_all_sync(directory: str | None = None) -> list[SyncInstance]: + # Import at runtime to avoid circular dependency + from infrahub_sync import SyncConfig, SyncInstance + results = [] search_directory = Path(directory) if directory else Path(__file__).parent config_files = search_directory.glob("**/config.yml") @@ -109,6 +123,9 @@ def get_instance( config_file: str | None = "config.yml", directory: str | None = None, ) -> SyncInstance | None: + # Import at runtime to avoid circular dependency + from infrahub_sync import SyncInstance + if name: all_sync_instances = get_all_sync(directory=directory) for item in all_sync_instances: @@ -141,6 +158,9 @@ def get_potenda_from_instance( branch: str | None = None, show_progress: bool | None = True, ) -> Potenda: + # Import at runtime to avoid circular dependency + from infrahub_sync import SyncInstance + source = import_adapter(sync_instance=sync_instance, adapter=sync_instance.source) destination = import_adapter(sync_instance=sync_instance, adapter=sync_instance.destination) diff --git a/tests/test_api.py b/tests/test_api.py new file mode 100644 index 0000000..35d890e --- /dev/null +++ b/tests/test_api.py @@ -0,0 +1,258 @@ +"""Tests for the Infrahub Sync Python API.""" + +import pytest +from unittest.mock import Mock, patch, MagicMock + +from infrahub_sync.api import ( + SyncError, + SyncResult, + list_projects, + diff, + sync, + create_potenda, +) +from infrahub_sync import SyncInstance, SyncAdapter + + +class TestPythonAPI: + """Test cases for the Python API.""" + + def test_sync_result_initialization(self): + """Test SyncResult object initialization.""" + result = SyncResult( + success=True, + message="Test completed", + duration=1.5, + changes_detected=True + ) + + assert result.success is True + assert result.message == "Test completed" + assert result.duration == 1.5 + assert result.changes_detected is True + assert result.error is None + + def test_sync_result_repr(self): + """Test SyncResult string representation.""" + result = SyncResult(success=False, message="Error occurred") + expected = "SyncResult(success=False, message='Error occurred', duration=None)" + assert repr(result) == expected + + @patch('infrahub_sync.api.get_all_sync') + def test_list_projects(self, mock_get_all_sync): + """Test list_projects function.""" + # Mock return value + mock_sync_instance = Mock(spec=SyncInstance) + mock_sync_instance.name = "test-sync" + mock_get_all_sync.return_value = [mock_sync_instance] + + # Call the function + result = list_projects() + + # Verify + mock_get_all_sync.assert_called_once_with(directory=None) + assert len(result) == 1 + assert result[0].name == "test-sync" + + @patch('infrahub_sync.api.get_all_sync') + def test_list_projects_with_directory(self, mock_get_all_sync): + """Test list_projects function with directory parameter.""" + mock_get_all_sync.return_value = [] + + result = list_projects(directory="/custom/path") + + mock_get_all_sync.assert_called_once_with(directory="/custom/path") + assert result == [] + + def test_diff_invalid_args(self): + """Test diff function with invalid arguments.""" + # Test with no arguments + with pytest.raises(SyncError, match="Please specify exactly one of 'name' or 'config_file'"): + diff() + + # Test with both arguments + with pytest.raises(SyncError, match="Please specify exactly one of 'name' or 'config_file'"): + diff(name="test", config_file="config.yml") + + @patch('infrahub_sync.api.get_instance') + def test_diff_no_instance(self, mock_get_instance): + """Test diff function when sync instance is not found.""" + mock_get_instance.return_value = None + + with pytest.raises(SyncError, match="Failed to load sync instance"): + diff(name="nonexistent") + + @patch('infrahub_sync.api.get_potenda_from_instance') + @patch('infrahub_sync.api.get_instance') + def test_diff_success(self, mock_get_instance, mock_get_potenda): + """Test successful diff operation.""" + # Setup mocks + mock_sync_instance = Mock(spec=SyncInstance) + mock_get_instance.return_value = mock_sync_instance + + mock_potenda = Mock() + mock_get_potenda.return_value = mock_potenda + + # Mock diff result + mock_diff = Mock() + mock_diff.has_diffs.return_value = True + mock_diff.str.return_value = "Changes detected" + mock_potenda.diff.return_value = mock_diff + + # Call function + result = diff(name="test-sync") + + # Verify calls + mock_get_instance.assert_called_once_with(name="test-sync", config_file=None, directory=None) + mock_get_potenda.assert_called_once_with( + sync_instance=mock_sync_instance, + branch=None, + show_progress=True + ) + mock_potenda.source_load.assert_called_once() + mock_potenda.destination_load.assert_called_once() + mock_potenda.diff.assert_called_once() + + # Verify result + assert result.success is True + assert result.message == "Changes detected" + assert result.changes_detected is True + assert result.duration is not None + + @patch('infrahub_sync.api.get_potenda_from_instance') + @patch('infrahub_sync.api.get_instance') + def test_diff_no_changes(self, mock_get_instance, mock_get_potenda): + """Test diff operation with no changes.""" + # Setup mocks + mock_sync_instance = Mock(spec=SyncInstance) + mock_get_instance.return_value = mock_sync_instance + + mock_potenda = Mock() + mock_get_potenda.return_value = mock_potenda + + # Mock diff result with no changes + mock_diff = Mock() + mock_diff.has_diffs.return_value = False + mock_potenda.diff.return_value = mock_diff + + # Call function + result = diff(name="test-sync") + + # Verify result + assert result.success is True + assert result.message == "No differences found" + assert result.changes_detected is False + + def test_sync_invalid_args(self): + """Test sync function with invalid arguments.""" + # Test with no arguments + with pytest.raises(SyncError, match="Please specify exactly one of 'name' or 'config_file'"): + sync() + + @patch('infrahub_sync.api.get_potenda_from_instance') + @patch('infrahub_sync.api.get_instance') + def test_sync_no_changes(self, mock_get_instance, mock_get_potenda): + """Test sync operation with no changes.""" + # Setup mocks + mock_sync_instance = Mock(spec=SyncInstance) + mock_get_instance.return_value = mock_sync_instance + + mock_potenda = Mock() + mock_get_potenda.return_value = mock_potenda + + # Mock diff result with no changes + mock_diff = Mock() + mock_diff.has_diffs.return_value = False + mock_potenda.diff.return_value = mock_diff + + # Call function + result = sync(name="test-sync") + + # Verify result + assert result.success is True + assert result.message == "No differences found. Nothing to sync." + assert result.changes_detected is False + assert result.duration == 0.0 + + # Verify sync was not called + mock_potenda.sync.assert_not_called() + + @patch('infrahub_sync.api.get_potenda_from_instance') + @patch('infrahub_sync.api.get_instance') + def test_sync_with_changes(self, mock_get_instance, mock_get_potenda): + """Test sync operation with changes.""" + # Setup mocks + mock_sync_instance = Mock(spec=SyncInstance) + mock_get_instance.return_value = mock_sync_instance + + mock_potenda = Mock() + mock_get_potenda.return_value = mock_potenda + + # Mock diff result with changes + mock_diff = Mock() + mock_diff.has_diffs.return_value = True + mock_diff.str.return_value = "Changes found" + mock_potenda.diff.return_value = mock_diff + + # Call function + result = sync(name="test-sync") + + # Verify sync was called + mock_potenda.sync.assert_called_once_with(diff=mock_diff) + + # Verify result + assert result.success is True + assert "Found differences to sync" in result.message + assert result.changes_detected is True + assert result.duration is not None + + @patch('infrahub_sync.api.get_potenda_from_instance') + @patch('infrahub_sync.api.get_instance') + def test_create_potenda_success(self, mock_get_instance, mock_get_potenda): + """Test successful creation of Potenda instance.""" + mock_sync_instance = Mock(spec=SyncInstance) + mock_get_instance.return_value = mock_sync_instance + + mock_potenda_instance = Mock() + mock_get_potenda.return_value = mock_potenda_instance + + result = create_potenda(name="test-sync") + + mock_get_instance.assert_called_once_with(name="test-sync", config_file=None, directory=None) + mock_get_potenda.assert_called_once_with( + sync_instance=mock_sync_instance, + branch=None, + show_progress=True + ) + assert result == mock_potenda_instance + + def test_create_potenda_invalid_args(self): + """Test create_potenda with invalid arguments.""" + with pytest.raises(SyncError, match="Please specify exactly one of 'name' or 'config_file'"): + create_potenda() + + @patch('infrahub_sync.api.get_potenda_from_instance') + @patch('infrahub_sync.api.get_instance') + def test_error_handling_load_failure(self, mock_get_instance, mock_get_potenda): + """Test error handling when loading fails.""" + mock_sync_instance = Mock(spec=SyncInstance) + mock_get_instance.return_value = mock_sync_instance + + mock_potenda = Mock() + mock_potenda.source_load.side_effect = ValueError("Load error") + mock_get_potenda.return_value = mock_potenda + + with pytest.raises(SyncError, match="Failed to load data: Load error"): + diff(name="test-sync") + + @patch('infrahub_sync.api.get_instance') + def test_error_handling_potenda_creation_failure(self, mock_get_instance): + """Test error handling when Potenda creation fails.""" + mock_sync_instance = Mock(spec=SyncInstance) + mock_get_instance.return_value = mock_sync_instance + + with patch('infrahub_sync.api.get_potenda_from_instance') as mock_get_potenda: + mock_get_potenda.side_effect = ValueError("Potenda creation failed") + + with pytest.raises(SyncError, match="Failed to initialize the Sync Instance: Potenda creation failed"): + diff(name="test-sync") \ No newline at end of file