Skip to content

feat: argo workflow orchestration and dependencies #2

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

feat: argo workflow orchestration and dependencies #2

wants to merge 1 commit into from

Conversation

wietzesuijker
Copy link
Collaborator

Argo Workflows orchestration, AMQP integration, and Python dependencies.

Stack: Part 1 of 5 (merge this first)

What's included

Argo Workflows (workflows/)

  • template.yaml: Main DAG pipeline (convert → register → augment)
  • sensor.yaml: AMQP event listener for RabbitMQ
  • eventsource.yaml: RabbitMQ connection configuration
  • rbac.yaml: Service account permissions
  • test-run.yaml, augment-only.yaml: Development/testing workflows
  • payload.json + examples: Test payloads

Dependencies

  • pyproject.toml: Python dependencies (eopf-geozarr, pystac-client, requests, tenacity)
  • uv.lock: Locked dependency versions
  • .pre-commit-config.yaml: Code quality hooks (ruff, mypy)

Architecture

RabbitMQ → EventSource → Sensor → Argo Workflows
                                   ├─ convert-geozarr
                                   ├─ register-stac
                                   └─ augment-stac

Review notes

This was referenced Oct 8, 2025
Open
Open
@wietzesuijker
feat: argo workflow orchestration and dependencies
c34d4c2 
Add complete Argo Workflows infrastructure for geozarr pipeline with automated AMQP event triggering.

Workflow pipeline:
- Convert: Sentinel-2 Zarr → GeoZarr (cloud-optimized)
- Register: Create STAC item with metadata
- Augment: Add visualization links (XYZ tiles, TileJSON)

Event-driven automation:
- AMQP EventSource subscribes to RabbitMQ queue
- Sensor triggers workflows on incoming messages
- RBAC configuration for secure execution

Configuration:
- Python dependencies (pyproject.toml, uv.lock)
- Pre-commit hooks (ruff, mypy, yaml validation)
- TTL cleanup (24h auto-delete completed workflows)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@wietzesuijker