Skip to content

Commit e28c9dc

Browse files
j3ssiej3ssie
committed
refactor: split RunID into RunUUID and ID, compute workspace dynamically, improve error handling
1 parent 96e9b4e commit e28c9dc

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

67 files changed

+3158
-443
lines changed

‎HACKING.md‎

Lines changed: 40 additions & 110 deletions
Original file line numberDiff line numberDiff line change
@@ -23,113 +23,43 @@ This document describes the technical architecture and development practices for
2323
## Project Structure
2424

2525
```
26-
osmedeus-ng/
27-
├── cmd/
28-
│ └── osmedeus/
29-
│ └── main.go # Application entry point
30-
├── internal/ # Private packages
31-
│ ├── config/ # Configuration management
32-
│ │ ├── config.go # Config loading and defaults
33-
│ │ ├── settings.go # Settings struct definitions
34-
│ │ └── secrets.go # Secret management
35-
│ ├── core/ # Core types and interfaces
36-
│ │ ├── workflow.go # Workflow and RunnerConfig structs
37-
│ │ ├── step.go # Step definition
38-
│ │ ├── trigger.go # Trigger types and events
39-
│ │ ├── types.go # Status types, constants
40-
│ │ ├── context.go # Execution context
41-
│ │ └── constants.go # Project metadata
42-
│ ├── parser/ # Workflow parsing
43-
│ │ ├── parser.go # YAML parser
44-
│ │ ├── loader.go # Workflow loader with caching
45-
│ │ └── validator.go # Workflow validation
46-
│ ├── executor/ # Workflow execution
47-
│ │ ├── executor.go # Main executor
48-
│ │ ├── dispatcher.go # Step dispatcher
49-
│ │ ├── bash_executor.go # Bash step handler
50-
│ │ ├── function_executor.go # Function step handler
51-
│ │ ├── foreach_executor.go # Foreach step handler
52-
│ │ ├── parallel_executor.go # Parallel step handler
53-
│ │ ├── remote_bash_executor.go # Remote-bash step handler
54-
│ │ ├── http_executor.go # HTTP request handler
55-
│ │ └── llm_executor.go # LLM step handler
56-
│ ├── runner/ # Execution environments
57-
│ │ ├── runner.go # Runner interface
58-
│ │ ├── host_runner.go # Local execution
59-
│ │ ├── docker_runner.go # Docker execution
60-
│ │ └── ssh_runner.go # SSH remote execution
61-
│ ├── template/ # Template rendering
62-
│ │ ├── engine.go # Template engine
63-
│ │ ├── generators.go # Value generators
64-
│ │ └── context.go # Template context
65-
│ ├── functions/ # Utility functions
66-
│ │ ├── registry.go # Function registry
67-
│ │ ├── goja_runtime.go # JavaScript runtime (Goja VM)
68-
│ │ ├── file_functions.go # File operations
69-
│ │ ├── string_functions.go # String operations
70-
│ │ ├── util_functions.go # Utility functions
71-
│ │ ├── event_functions.go # Event generation functions
72-
│ │ └── jq.go # JSON query functions
73-
│ ├── scheduler/ # Trigger scheduling
74-
│ │ └── scheduler.go # Cron, event, watch triggers
75-
│ ├── database/ # Data persistence
76-
│ │ ├── database.go # Connection management
77-
│ │ ├── models.go # Data models
78-
│ │ ├── jsonl.go # JSONL import/export
79-
│ │ └── repository/ # Data access layer
80-
│ ├── heuristics/ # Target analysis
81-
│ │ ├── heuristics.go # Target type detection
82-
│ │ ├── url.go # URL parsing
83-
│ │ └── domain.go # Domain analysis
84-
│ ├── linter/ # Workflow linting
85-
│ │ ├── linter.go # Main linter logic
86-
│ │ ├── rules.go # Linting rules (built-in variables, etc.)
87-
│ │ ├── formatter.go # Output formatters (pretty, JSON, GitHub)
88-
│ │ ├── ast.go # Workflow AST for position tracking
89-
│ │ └── types.go # Linter types and severity levels
90-
│ ├── workspace/ # Workspace management
91-
│ │ └── workspace.go # Workspace creation
92-
│ ├── snapshot/ # Workspace snapshots
93-
│ │ └── snapshot.go # Export/import ZIP archives
94-
│ ├── installer/ # Binary installation
95-
│ │ ├── installer.go # Core installer logic
96-
│ │ ├── nix.go # Nix package manager support
97-
│ │ └── registry.go # Binary registry management
98-
│ ├── state/ # Run state export
99-
│ │ ├── export.go # State export functionality
100-
│ │ └── types.go # State types
101-
│ ├── updater/ # Self-update functionality
102-
│ │ ├── updater.go # Update logic
103-
│ │ └── github_source.go # GitHub releases source
104-
│ ├── terminal/ # Terminal UI
105-
│ │ ├── printer.go # Output formatting
106-
│ │ ├── colors.go # ANSI colors
107-
│ │ ├── symbols.go # Unicode symbols
108-
│ │ ├── spinner.go # Loading spinners
109-
│ │ └── table.go # Table rendering
110-
│ └── logger/ # Structured logging
111-
│ └── logger.go # Zap logger wrapper
112-
├── pkg/ # Public packages
113-
│ ├── cli/ # CLI commands
114-
│ │ ├── root.go # Root command
115-
│ │ ├── run.go # Run command (scan)
116-
│ │ ├── workflow.go # Workflow command
117-
│ │ ├── function.go # Function command (with bulk processing)
118-
│ │ ├── db.go # Database CLI command
119-
│ │ └── server.go # Server command
120-
│ └── server/ # REST API
121-
│ ├── server.go # Server setup
122-
│ ├── handlers/ # Request handlers
123-
│ └── middleware/ # Auth middleware (JWT, API Key)
124-
├── test/ # Test suites
125-
│ ├── e2e/ # E2E CLI tests
126-
│ ├── integration/ # Integration tests
127-
│ └── testdata/
128-
│ ├── workflows/ # Test workflow fixtures
129-
│ └── complex-workflows/ # Complex workflow examples
130-
└── build/ # Build artifacts
131-
├── docker/ # Docker files
132-
└── DEPLOYMENT.md # Deployment guide
26+
osmedeus/
27+
├── cmd/osmedeus/ # Application entry point
28+
├── internal/ # Private packages
29+
│ ├── client/ # Remote API client
30+
│ ├── config/ # Configuration management
31+
│ ├── console/ # Console output capture
32+
│ ├── core/ # Core types (Workflow, Step, Trigger, etc.)
33+
│ ├── database/ # SQLite/PostgreSQL via Bun ORM
34+
│ ├── distributed/ # Distributed execution (master/worker)
35+
│ ├── executor/ # Workflow execution engine
36+
│ ├── functions/ # Utility functions (Goja JS runtime)
37+
│ ├── heuristics/ # Target type detection
38+
│ ├── installer/ # Binary installation (direct/Nix)
39+
│ ├── linter/ # Workflow linting and validation
40+
│ ├── logger/ # Structured logging (Zap)
41+
│ ├── parser/ # YAML parsing and caching
42+
│ ├── runner/ # Execution environments (host/docker/ssh)
43+
│ ├── scheduler/ # Trigger scheduling (cron/event/watch)
44+
│ ├── snapshot/ # Workspace export/import
45+
│ ├── state/ # Run state export
46+
│ ├── template/ # {{Variable}} interpolation engine
47+
│ ├── terminal/ # Terminal UI (colors, tables, spinners)
48+
│ ├── updater/ # Self-update via GitHub releases
49+
│ └── workspace/ # Workspace management
50+
├── lib/ # Shared library utilities
51+
├── pkg/ # Public packages
52+
│ ├── cli/ # Cobra CLI commands
53+
│ └── server/ # Fiber REST API server
54+
│ ├── handlers/ # Request handlers
55+
│ └── middleware/ # Auth middleware (JWT, API Key)
56+
├── public/ # Public assets (examples, presets, UI)
57+
├── test/ # Test suites
58+
│ ├── e2e/ # E2E CLI tests
59+
│ ├── integration/ # Integration tests
60+
│ └── testdata/ # Test workflow fixtures
61+
├── docs/ # API documentation
62+
└── build/ # Build artifacts and Docker files
13363
```
13464

13565
## Architecture Overview
@@ -574,7 +504,7 @@ func (e *Executor) injectBuiltinVariables(cfg *config.Config, params map[string]
574504
execCtx.SetVariable("Target", params["target"])
575505
execCtx.SetVariable("Output", filepath.Join(workspacesPath, targetSpace))
576506
execCtx.SetVariable("threads", threads)
577-
execCtx.SetVariable("TaskID", execCtx.RunID)
507+
execCtx.SetVariable("RunUUID", execCtx.RunUUID)
578508
// ... more variables
579509
}
580510
```
@@ -846,7 +776,7 @@ The linter recognizes all runtime-injected variables to avoid false positives. T
846776
847777
**Output Variables**: `Output`, `output`, `Workspace`, `workspace`
848778
849-
**Metadata Variables**: `Version`, `TaskID`, `TaskDate`, `TimeStamp`, `Today`, `RandomString`
779+
**Metadata Variables**: `Version`, `RunUUID`, `TaskDate`, `TimeStamp`, `Today`, `RandomString`
850780
851781
**Heuristic Variables**: `TargetType`, `TargetRootDomain`, `TargetTLD`, `Org`, `TargetHost`, `TargetPort`, etc.
852782
@@ -935,7 +865,7 @@ type Run struct {
935865
Target string
936866
Params map[string]string
937867
Status string // "pending", "running", "completed", "failed"
938-
WorkspacePath string
868+
Workspace string // Logical workspace name (same as TargetSpace)
939869
StartedAt time.Time
940870
CompletedAt time.Time
941871
ErrorMessage string

‎docs/api/README.md‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,11 @@ The Osmedeus API provides a RESTful interface for managing security automation w
1010

1111
## Authentication
1212

13-
Most API endpoints require JWT authentication. First, obtain a token via the login endpoint, then include it in subsequent requests using the `Authorization: Bearer <token>` header.
13+
Most API endpoints require authentication. Two methods are supported:
14+
15+
1. **JWT Token**: Obtain a token via the login endpoint, then include it in requests using the `Authorization: Bearer <token>` header.
16+
17+
2. **API Key**: Use a static API key via the `x-osm-api-key` header. Configure in `~/osmedeus-base/osm-settings.yaml` under `server.auth_api_key`.
1418

1519
See [Authentication](authentication.md) for details.
1620

@@ -28,6 +32,7 @@ See [Authentication](authentication.md) for details.
2832
| [Assets](assets.md) | View discovered assets |
2933
| [Vulnerabilities](vulnerabilities.md) | View and manage vulnerabilities |
3034
| [Event Logs](event-logs.md) | View execution event logs |
35+
| [Step Results](steps.md) | Query step execution results |
3136
| [Functions](functions.md) | Execute and list utility functions |
3237
| [System Statistics](system.md) | Get aggregated system stats |
3338
| [Settings](settings.md) | Manage server configuration |

‎docs/api/authentication.md‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -97,6 +97,53 @@ curl http://localhost:8002/osm/api/workflows \
9797
}
9898
```
9999

100+
## API Key Authentication
101+
102+
As an alternative to JWT tokens, you can authenticate using a static API key via the `x-osm-api-key` header. This is useful for scripts, CI/CD pipelines, or integrations where managing JWT token refresh is impractical.
103+
104+
### Configuration
105+
106+
API key authentication is configured in `~/osmedeus-base/osm-settings.yaml`:
107+
108+
```yaml
109+
server:
110+
# Enable API key authentication (default: true)
111+
enabled_auth_api: true
112+
# API key for x-osm-api-key header authentication
113+
# A random 32-character key is generated on first run
114+
auth_api_key: "your-api-key-here"
115+
```
116+
117+
### Using the API Key
118+
119+
Include the API key in requests using the `x-osm-api-key` header:
120+
121+
```bash
122+
# Store API key in environment variable
123+
export OSM_API_KEY="your-api-key-here"
124+
125+
# Use in API requests
126+
curl http://localhost:8002/osm/api/workflows \
127+
-H "x-osm-api-key: $OSM_API_KEY"
128+
```
129+
130+
### Error Response
131+
132+
**401 Unauthorized** - Invalid or missing API key:
133+
```json
134+
{
135+
"error": true,
136+
"message": "Invalid or missing API key"
137+
}
138+
```
139+
140+
### Notes
141+
142+
- API key authentication takes priority over JWT when enabled
143+
- A random 32-character API key is automatically generated on first server start
144+
- The API key is stored in plain text in the settings file; ensure appropriate file permissions
145+
- Empty, whitespace-only, or placeholder values (`null`, `undefined`, `nil`) are rejected
146+
100147
## Disabling Authentication
101148

102149
Authentication can be disabled by starting the server with the `--no-auth` flag:

‎docs/api/steps.md‎

Lines changed: 138 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,138 @@
1+
# Step Results
2+
3+
## List Step Results
4+
5+
Get a paginated list of step execution results with optional filtering.
6+
7+
**List all step results:**
8+
```bash
9+
curl http://localhost:8002/osm/api/step-results \
10+
-H "Authorization: Bearer $TOKEN"
11+
```
12+
13+
**With pagination:**
14+
```bash
15+
curl "http://localhost:8002/osm/api/step-results?offset=0&limit=50" \
16+
-H "Authorization: Bearer $TOKEN"
17+
```
18+
19+
**Filter by workspace:**
20+
```bash
21+
curl "http://localhost:8002/osm/api/step-results?workspace=example_com" \
22+
-H "Authorization: Bearer $TOKEN"
23+
```
24+
25+
**Filter by status:**
26+
```bash
27+
curl "http://localhost:8002/osm/api/step-results?status=completed" \
28+
-H "Authorization: Bearer $TOKEN"
29+
```
30+
31+
**Filter by step type:**
32+
```bash
33+
curl "http://localhost:8002/osm/api/step-results?step_type=bash" \
34+
-H "Authorization: Bearer $TOKEN"
35+
```
36+
37+
**Filter by run ID:**
38+
```bash
39+
curl "http://localhost:8002/osm/api/step-results?run_id=123" \
40+
-H "Authorization: Bearer $TOKEN"
41+
```
42+
43+
**Multiple filters:**
44+
```bash
45+
curl "http://localhost:8002/osm/api/step-results?workspace=example_com&status=completed&limit=100" \
46+
-H "Authorization: Bearer $TOKEN"
47+
```
48+
49+
**Query Parameters:**
50+
51+
| Parameter | Type | Description |
52+
|-----------|------|-------------|
53+
| `workspace` | string | Filter by workspace name |
54+
| `status` | string | Filter by status (pending, running, completed, failed) |
55+
| `step_type` | string | Filter by step type (bash, function, foreach, parallel-steps, remote-bash, http, llm) |
56+
| `run_id` | int | Filter by run ID |
57+
| `offset` | int | Pagination offset (default: 0) |
58+
| `limit` | int | Maximum records to return (default: 20, max: 10000) |
59+
60+
**Response:**
61+
```json
62+
{
63+
"data": [
64+
{
65+
"id": 1,
66+
"run_id": 123,
67+
"step_name": "run-subfinder",
68+
"step_type": "bash",
69+
"status": "completed",
70+
"command": "subfinder -d example.com -o {{Output}}/subdomains.txt",
71+
"output": "Found 150 subdomains",
72+
"error": "",
73+
"started_at": "2025-01-15T10:00:00Z",
74+
"completed_at": "2025-01-15T10:01:30Z",
75+
"duration_ms": 90000,
76+
"created_at": "2025-01-15T10:00:00Z",
77+
"updated_at": "2025-01-15T10:01:30Z"
78+
},
79+
{
80+
"id": 2,
81+
"run_id": 123,
82+
"step_name": "run-httpx",
83+
"step_type": "bash",
84+
"status": "completed",
85+
"command": "httpx -l {{Output}}/subdomains.txt -o {{Output}}/httpx.txt",
86+
"output": "Probed 150 hosts, 120 alive",
87+
"error": "",
88+
"started_at": "2025-01-15T10:01:30Z",
89+
"completed_at": "2025-01-15T10:03:00Z",
90+
"duration_ms": 90000,
91+
"created_at": "2025-01-15T10:01:30Z",
92+
"updated_at": "2025-01-15T10:03:00Z"
93+
},
94+
{
95+
"id": 3,
96+
"run_id": 124,
97+
"step_name": "process-results",
98+
"step_type": "function",
99+
"status": "completed",
100+
"command": "",
101+
"output": "true",
102+
"error": "",
103+
"started_at": "2025-01-15T11:00:00Z",
104+
"completed_at": "2025-01-15T11:00:01Z",
105+
"duration_ms": 1000,
106+
"created_at": "2025-01-15T11:00:00Z",
107+
"updated_at": "2025-01-15T11:00:01Z"
108+
}
109+
],
110+
"pagination": {
111+
"total": 250,
112+
"offset": 0,
113+
"limit": 20
114+
}
115+
}
116+
```
117+
118+
**Step Status Values:**
119+
120+
| Status | Description |
121+
|--------|-------------|
122+
| `pending` | Step is queued but not yet started |
123+
| `running` | Step is currently executing |
124+
| `completed` | Step finished successfully |
125+
| `failed` | Step failed with an error |
126+
| `skipped` | Step was skipped (pre_condition not met) |
127+
128+
**Step Types:**
129+
130+
| Type | Description |
131+
|------|-------------|
132+
| `bash` | Shell command execution |
133+
| `function` | Utility function execution |
134+
| `foreach` | Loop over input items |
135+
| `parallel-steps` | Execute steps in parallel |
136+
| `remote-bash` | Remote command execution (Docker/SSH) |
137+
| `http` | HTTP request step |
138+
| `llm` | LLM/AI step |

0 commit comments

Comments
 (0)