Skip to content

Commit 56854e9

Browse files
adammcdonaghadammcdonagh
committed
Add docs
1 parent 586bb04 commit 56854e9

File tree

10 files changed

+869
-10
lines changed

10 files changed

+869
-10
lines changed

‎AGENTS.md‎

Lines changed: 202 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,202 @@
1+
# AGENTS.md — Open Task Framework (OTF)
2+
3+
This file is written for autonomous agents and maintainers who will modify, test, and extend the Open Task Framework codebase; it provides focused, actionable context so automated systems can make safe, verifiable changes.
4+
5+
## Table of contents
6+
7+
- [AGENTS.md — Open Task Framework (OTF)](#agentsmd--open-task-framework-otf)
8+
- [Table of contents](#table-of-contents)
9+
- [Quick scan](#quick-scan)
10+
- [High-level summary](#high-level-summary)
11+
- [Architecture and main components](#architecture-and-main-components)
12+
- [Contracts and data shapes — precise](#contracts-and-data-shapes--precise)
13+
- [Variable resolution \& templates](#variable-resolution--templates)
14+
- [Developer and agent workflow — run / test / iterate](#developer-and-agent-workflow--run--test--iterate)
15+
- [Concrete examples (copy-and-paste)](#concrete-examples-copy-and-paste)
16+
- [Tests, debugging, and logs](#tests-debugging-and-logs)
17+
- [CI and release pointers](#ci-and-release-pointers)
18+
- [Best practices for automated agents (rules)](#best-practices-for-automated-agents-rules)
19+
- [Where to look for related code](#where-to-look-for-related-code)
20+
- [Change summary and contact](#change-summary-and-contact)
21+
22+
## Quick scan
23+
24+
- Entry point(s): `src/opentaskpy/taskrun.py`, `src/opentaskpy/taskhandlers/taskhandler.py`
25+
- Schemas: `src/opentaskpy/config/schemas/` (validation source of truth)
26+
- Remote handlers: `src/opentaskpy/remotehandlers/` (SSH/SFTP/local/email/dummy)
27+
- Plugins: `src/opentaskpy/plugins/` (lookup family)
28+
- Run unit tests: `pytest tests/ -q`
29+
30+
## High-level summary
31+
32+
Open Task Framework is a modular Python framework that validates and runs tasks defined as JSON/YAML documents. Tasks describe either an execution (run a command) or a transfer (move files). The framework uses pluggable remote handlers (execution and transfer) to support protocols like SSH, SFTP, WinRM, and cloud storage services.
33+
34+
Key responsibilities:
35+
36+
- Validate task payloads against JSON schemas.
37+
- Orchestrate execution and transfer flows via `taskhandler` components.
38+
- Provide well-encapsulated protocol handlers: concrete classes implement a consistent handler interface so the taskhandler layer can be protocol-agnostic.
39+
- Provide test fixtures (unit and integration) to verify both logic and environment interactions.
40+
41+
## Architecture and main components
42+
43+
1. Core package: `src/opentaskpy`
44+
45+
- `taskhandler` — central orchestration logic: accepts a validated task, decides whether to call execution or transfer workflow, orchestrates staging and cleanup, and returns standardized results.
46+
- `remotehandlers` — contains abstract base classes and concrete implementations. Expect classes following the naming convention: `*Transfer` and `*Execution`.
47+
- `config/schemas` — JSON schemas that the code uses to validate task payloads. Schemas are authoritative; runtime assumes inputs match them.
48+
- `otflogging` — logging helpers used across the project for consistent log formatting and task-scoped contexts.
49+
50+
2. Tests and fixtures:
51+
52+
- `tests/` — pytest test suite with unit tests (fast) and integration tests (may require docker-compose fixtures).
53+
- `test/` — helper scripts and docker-compose configurations used to stand up test services (sshd, mock services). Look for `createTestFiles.sh`, `createTestDirectories.sh`, and `setupSSHKeys.sh`.
54+
55+
3. Addons: repository-level addons live in sibling repos. Each addon should follow the same shape: `remotehandlers` implementations, config schemas, tests, and an optional `AGENTS.md` describing the addon details (example: winrm addon).
56+
57+
## Contracts and data shapes — precise
58+
59+
Task manifest (canonical fields):
60+
61+
- `id` (string): unique task identifier
62+
- `type` (string): one of `transfer`, `execution`, `batch`
63+
- `source` / `destination` (objects): for transfers, each contains `hostname`, `directory`, `fileRegex`, and `protocol`
64+
- `hostname`, `directory`, `command` (for execution tasks)
65+
- `protocol` (object): minimally `{ "name": "<python-class-path>", "credentials": {...}, ... }`
66+
67+
Protocol object details:
68+
69+
- `name` (string): importable Python class path implementing a Transfer or Execution handler
70+
- `credentials` (object): fields are protocol-specific (e.g., `username`/`password`, `cert_pem`, `transport`)
71+
- `server_cert_validation` / `port` / `transport` are optional common fields used by multiple handlers
72+
73+
Handler interface expectations (implementations MUST):
74+
75+
- Transfer handlers expose: `list_files(spec)`, `pull_file(spec, dest)`, `push_file(spec, src)`, `move_file(spec)`, `delete_file(spec)`, plus bulk helpers like `pull_files_to_worker()` and `push_files_from_worker()`
76+
- Execution handlers expose: `execute(spec)` returning a controlled stream or result object, plus `kill(pid)` to request termination. Results must include `exit_code`, `stdout`, `stderr`. If a PID token is emitted by the remote, include `pid` in results.
77+
78+
Error model:
79+
80+
- Handlers should raise specific exceptions for common error classes (validation error, auth error, networkIO error). Taskhandler should catch and translate to standardized result objects for callers and tests.
81+
82+
If you change these shapes, update the JSON schemas in `src/opentaskpy/config/schemas/` and add/update tests in `tests/`.
83+
84+
## Variable resolution & templates
85+
86+
- File types: configuration and task payloads are JSON-based only. Files are either plain `.json` or Jinja2 templates with a `.json.j2` extension. YAML is not used for task/config payloads.
87+
- Pipeline: when a task/template file is loaded the system performs this pipeline:
88+
1. Read the `.json` or `.json.j2` file.
89+
90+
2. If it is a Jinja2 template (`.json.j2`), render it with the available context (variables, plugin helpers, and environment values).
91+
3. Parse the rendered output as JSON.
92+
4. Validate the parsed JSON against the appropriate schema in `src/opentaskpy/config/schemas/`.
93+
94+
- Template context and helpers: lookup plugins (see `src/opentaskpy/plugins/lookup`) and other small helpers/filters are available to templates to compute values at render time. Templates must render to valid JSON — agents should always validate rendered output before runtime.
95+
96+
- Guidance for agents:
97+
- When editing templates, ensure the rendered output is syntactically valid JSON (use a local render step in tests).
98+
- Do not introduce template constructs that rely on secrets stored in-repo; use environment variables or test fixtures for secret injection.
99+
- If new helpers/plugins are required by templates, add them under `src/opentaskpy/plugins/` and include unit tests that exercise rendering.
100+
101+
## Developer and agent workflow — run / test / iterate
102+
103+
Local dev quickstart (recommended):
104+
105+
1. Create and activate a virtual environment
106+
107+
```bash
108+
python -m venv .venv
109+
source .venv/bin/activate
110+
pip install -e .[test]
111+
```
112+
113+
2. Run a focused unit test
114+
115+
```bash
116+
pytest tests/test_file_helper.py::test_some_helper -q
117+
```
118+
119+
3. Run full unit test suite
120+
121+
```bash
122+
pytest tests/ -q
123+
```
124+
125+
Integration tests (requires docker):
126+
127+
```bash
128+
cd test
129+
./createTestDirectories.sh && ./createTestFiles.sh
130+
docker-compose up -d
131+
./setupSSHKeys.sh
132+
cd ..
133+
pytest tests/ -q
134+
```
135+
136+
CI notes:
137+
138+
- The project uses `pyproject.toml` for packaging and `pytest.ini` for test config. CI should install dependencies with `pip install -e .[test]` and run `pytest -q`.
139+
- Integration tests that depend on docker-compose should be gated behind a separate job that runs `cd test && docker-compose up -d` first.
140+
141+
## Concrete examples (copy-and-paste)
142+
143+
Example task manifest — execution
144+
145+
```json
146+
{
147+
"id": "task-123",
148+
"type": "execution",
149+
"hostname": "127.0.0.1",
150+
"directory": "/tmp",
151+
"command": "echo hello",
152+
"protocol": {
153+
"name": "ssh",
154+
"credentials": { "username": "test", "keyFile": "path/to/key" }
155+
}
156+
}
157+
```
158+
159+
Example transfer protocol snippet (schema-driven)
160+
161+
```json
162+
{
163+
"name": "sftp",
164+
"credentials": { "username": "user", "keyFile": "path/to/key" }
165+
}
166+
```
167+
168+
## Tests, debugging, and logs
169+
170+
- Test fixtures live in `tests/fixtures` or are defined in `tests/conftest.py`. Reuse existing fixtures whenever possible.
171+
- Integration test logs and artifacts created by `test/` helper scripts are placed under `test/testLogs/` for easy inspection.
172+
- Logging format: use `otflogging` helpers to include `task_id` and `hostname` in logs. New code should add context to loggers so tests can assert on log markers if needed.
173+
174+
Common debugging steps:
175+
176+
- Reproduce failing test locally with `-k <test_name>` and `-s` to see stdout/stderr streaming.
177+
- Inspect `test/testLogs/` for integration failures.
178+
- For networking/auth issues, replicate the protocol flow manually in a small script that uses the same handler class to connect and run a simple command.
179+
180+
## CI and release pointers
181+
182+
- Ensure `pyproject.toml` and `MANIFEST.in` contain any new package data files you add.
183+
- Bump versions according to semantic versioning and update `CHANGELOG.md` when releasing.
184+
- Unit tests should be quick; heavy integration tests should run in separate CI jobs that provision docker services.
185+
186+
## Best practices for automated agents (rules)
187+
188+
1. Run the unit tests that exercise your changed files before creating a PR. If you cannot reproduce remote integration locally, add/modify only unit tests or mock the remote layer.
189+
2. Never add secrets to the repo. Use environment variables or test fixtures that generate ephemeral keys.
190+
3. If changing a JSON schema, update the schema file and add at least one positive and one negative test case.
191+
4. Limit scope of edits in a single PR: small, focused changes are easier to review and test.
192+
193+
## Where to look for related code
194+
195+
- `src/opentaskpy/taskhandler.py` and `src/opentaskpy/taskhandler/*`
196+
- `src/opentaskpy/remotehandlers/` (SSH, SFTP, WinRM addons live in separate repos but follow the same interface)
197+
- `src/opentaskpy/config/schemas/` — JSON schemas (canonical)
198+
- `tests/`, `tests/conftest.py` and `test/` helper scripts
199+
200+
## Change summary and contact
201+
202+
This file was created to give automated agents a reliable starting point for code navigation, safe edits, and test execution. If you're an external maintainer, open issues or PRs in this repository; include failing test output and the `-k` test used to reproduce locally.

‎CHANGELOG.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,9 @@
11
# Changelog
22

3+
# No release
4+
5+
- Update docs
6+
37
# v25.37.0
48

59
- Add additional logging to `sftp` & `local` protocol.

0 commit comments

Comments
 (0)