Documentation rework

docs/reference/cli-reference.md → docs/agent/cli-reference.md

@@ -16,7 +16,7 @@ The CLI is organised into the following command groups:
 
 ## Global Options
 
-All commands support the following verbosity options:
+The following verbosity options are supported by the `watch` command:
 
 | Option | Short | Type | Default | Description |
 |--------|-------|------|---------|-------------|
@@ -27,6 +27,8 @@ All commands support the following verbosity options:
 - **1 (-v)**: INFO level - General information and warnings
 - **2 (-vv)**: DEBUG level - Detailed debugging information
 
+> **Note**: The `-v`/`--verbose` flag currently only works with the `watch` command. Parse and validate commands use default logging levels. This is a known limitation.
+
 ## Parse Commands
 
 Parse commands extract and process data from EPU files without persisting to the backend API. These are primarily used for development, debugging, and data validation purposes.
@@ -44,7 +46,6 @@ python -m smartem_agent parse dir [OPTIONS] EPU_OUTPUT_DIR
 | Parameter | Type | Required | Description |
 |-----------|------|----------|-------------|
 | `epu_output_dir` | str | Yes | Path to EPU output directory containing multiple grid directories |
-| `--verbose` | count | No | Verbosity level (see [Global Options](#global-options)) |
 
 **Example:**
 ```bash
@@ -69,7 +70,6 @@ python -m smartem_agent parse grid [OPTIONS] GRID_DATA_DIR
 | Parameter | Type | Required | Description |
 |-----------|------|----------|-------------|
 | `grid_data_dir` | str | Yes | Path to individual grid data directory |
-| `--verbose` | count | No | Verbosity level (see [Global Options](#global-options)) |
 
 **Example:**
 ```bash
@@ -94,7 +94,6 @@ python -m smartem_agent parse session [OPTIONS] PATH
 | Parameter | Type | Required | Description |
 |-----------|------|----------|-------------|
 | `path` | str | Yes | Path to EPU session manifest file |
-| `--verbose` | count | No | Verbosity level (see [Global Options](#global-options)) |
 
 **Example:**
 ```bash
@@ -116,7 +115,6 @@ python -m smartem_agent parse atlas [OPTIONS] PATH
 | Parameter | Type | Required | Description |
 |-----------|------|----------|-------------|
 | `path` | str | Yes | Path to atlas manifest file |
-| `--verbose` | count | No | Verbosity level (see [Global Options](#global-options)) |
 
 **Example:**
 ```bash
@@ -138,7 +136,6 @@ python -m smartem_agent parse gridsquare-metadata [OPTIONS] PATH
 | Parameter | Type | Required | Description |
 |-----------|------|----------|-------------|
 | `path` | str | Yes | Path to grid square metadata file |
-| `--verbose` | count | No | Verbosity level (see [Global Options](#global-options)) |
 
 **Example:**
 ```bash
@@ -160,7 +157,6 @@ python -m smartem_agent parse gridsquare [OPTIONS] PATH
 | Parameter | Type | Required | Description |
 |-----------|------|----------|-------------|
 | `path` | str | Yes | Path to grid square manifest file |
-| `--verbose` | count | No | Verbosity level (see [Global Options](#global-options)) |
 
 **Example:**
 ```bash
@@ -182,7 +178,6 @@ python -m smartem_agent parse foilhole [OPTIONS] PATH
 | Parameter | Type | Required | Description |
 |-----------|------|----------|-------------|
 | `path` | str | Yes | Path to foil hole manifest file |
-| `--verbose` | count | No | Verbosity level (see [Global Options](#global-options)) |
 
 **Example:**
 ```bash
@@ -204,7 +199,6 @@ python -m smartem_agent parse micrograph [OPTIONS] PATH
 | Parameter | Type | Required | Description |
 |-----------|------|----------|-------------|
 | `path` | str | Yes | Path to micrograph manifest file |
-| `--verbose` | count | No | Verbosity level (see [Global Options](#global-options)) |
 
 **Example:**
 ```bash
@@ -226,7 +220,6 @@ python -m smartem_agent validate [OPTIONS] PATH
 | Parameter | Type | Required | Default | Description |
 |-----------|------|----------|---------|-------------|
 | `path` | str | Yes | - | Path to EPU project directory to validate |
-| `--verbose` | count | No | 0 | Verbosity level (see [Global Options](#global-options)) |
 
 **Exit Codes:**
 - **0**: Directory structure is valid
@@ -487,4 +480,4 @@ The watch command requires a compatible SmartEM backend API with the following e
 - Consider `--heartbeat-interval` tuning based on network stability
 - Use `--dry-run` for local testing and development
 
-This CLI reference provides comprehensive coverage of all SmartEM Agent command-line functionality. For additional help with specific use cases or troubleshooting, refer to the [CLI Troubleshooting Guide](troubleshoot-cli.md).
+This CLI reference provides comprehensive coverage of all SmartEM Agent command-line functionality. For additional help with specific use cases or troubleshooting, refer to the [CLI Troubleshooting Guide](troubleshooting.md).

docs/how-to/run-agent.md → docs/agent/deployment.md

@@ -27,7 +27,7 @@ The agent operates in several modes depending on the timing of EPU data acquisit
 
 ## Quick Start
 
-For comprehensive parameter documentation, see the [CLI Reference](../reference/cli-reference.md). For troubleshooting, see the [CLI Troubleshooting Guide](troubleshoot-cli.md).
+For comprehensive parameter documentation, see the [CLI Reference](cli-reference.md). For troubleshooting, see the [CLI Troubleshooting Guide](troubleshooting.md).
 
 ### Basic Directory Monitoring
 

docs/agent/index.md

@@ -0,0 +1,22 @@
+# Agent
+
+Documentation for the SmartEM Agent - the EPU filesystem watcher that runs on Windows workstations near the microscope.
+
+```{toctree}
+:maxdepth: 1
+
+cli-reference
+deployment
+troubleshooting
+```
+
+## Topics
+
+### Usage
+
+- [CLI Reference](cli-reference.md) - Command-line interface documentation and options
+- [Troubleshooting](troubleshooting.md) - Common issues and solutions
+
+### Deployment
+
+- [Agent Deployment](deployment.md) - Running the agent on Windows workstations

docs/how-to/troubleshoot-cli.md → docs/agent/troubleshooting.md

@@ -1,6 +1,6 @@
 # SmartEM CLI Troubleshooting Guide
 
-This guide provides solutions for common issues encountered when using the SmartEM Agent command-line interface. For comprehensive parameter documentation, see the [CLI Reference](../reference/cli-reference.md).
+This guide provides solutions for common issues encountered when using the SmartEM Agent command-line interface. For comprehensive parameter documentation, see the [CLI Reference](cli-reference.md).
 
 ## Quick Diagnostics
 

docs/architecture/index.md

@@ -0,0 +1,18 @@
+# Architecture
+
+System design documentation and architectural decision records.
+
+## Decision Records
+
+Architectural Decision Records (ADRs) document significant technical decisions.
+
+- [Decision Records](../decision-records/index.md) - Browse all ADRs
+
+## System Design
+
+Architecture documentation is maintained in the main smartem-decisions repository:
+
+- System overview and component interactions
+- Backend-Agent communication protocols (SSE, REST)
+- Data flow and message queue patterns
+- Database schema design

docs/athena/index.md

@@ -0,0 +1,30 @@
+# Athena Integration
+
+Documentation for ThermoFisher Athena API integration.
+
+## API Reference
+
+The Athena Decision Service API specification is available in the API documentation:
+
+- [Athena API Docs](../api/athena/index.html) - Interactive Swagger UI
+- [API Documentation Guide](../backend/api-documentation.md) - How to use the interactive documentation
+
+## Mock Server
+
+For local development without access to a real Athena service:
+
+```bash
+# Install mock dependencies
+pip install -e ".[mock]"
+
+# Start the Athena mock server
+python -c "
+from athena_api.mock import AthenaAPIServer
+server = AthenaAPIServer()
+server.run()
+"
+
+# Visit http://localhost:8000/docs
+```
+
+See the [API Documentation](../backend/api-documentation.md#mock-server-for-development) guide for more details.

docs/how-to/run-backend.md → docs/backend/api-server.md

@@ -10,10 +10,10 @@ The core backend service providing HTTP API, database operations, and message qu
 
 # launch RabbitMQ worker (consumer)
 python -m smartem_backend.consumer              # ERROR level (default)
-python -m smartem_backend.consumer -v           # INFO level  
+python -m smartem_backend.consumer -v           # INFO level
 python -m smartem_backend.consumer -vv          # DEBUG level
 
-# simulating an system event: 
+# simulating an system event:
 python -m smartem_backend.simulate_msg --help # to see a list of options
 ./tools/simulate-messages.sh # run a simulation, triggering system events in sequence
 

docs/how-to/database-migrations.md → docs/backend/database.md

@@ -33,10 +33,10 @@ python -m alembic upgrade head
 
 This will:
 1. Create the Alembic version tracking table
-2. Apply migration 001: Create user preferences table
-3. Apply baseline migration (6e6302f1ccb6): Create all core SmartEM schema tables (acquisition, grid, gridsquare, micrograph, foilhole, atlas, quality prediction models, etc.)
-4. Apply migration 002: Add performance indexes for acquisition datetime queries
-5. Apply remaining migrations including agent communication tables
+2. Apply migration 001: Create core SmartEM schema baseline (acquisition, grid, gridsquare, micrograph, foilhole, atlas, etc.)
+3. Apply migration 002: Add performance indexes for acquisition datetime queries
+4. Apply migration 003: Add prediction model tables
+5. Apply remaining migrations for quality metrics, training tables, and schema fixes
 
 ### Running Migrations
 
@@ -140,7 +140,7 @@ def upgrade() -> None:
         column('preference_key', sa.String),
         column('preference_value', sa.JSON)
     )
-    
+
     op.bulk_insert(user_preferences, [
         {
             'user_id': 'system',
@@ -268,11 +268,17 @@ Ensure your environment variables are set:
 │   ├── env.py                              # Migration environment
 │   ├── script.py.mako                      # Migration template
 │   └── versions/                           # Individual migration files
-│       ├── 001_add_user_preferences.py
-│       ├── 002_add_acquisition_time_index.py
-│       └── 003_seed_system_config.py
+│       ├── 2025_09_18_1042-001_create_core_smartem_schema_baseline.py
+│       ├── 2025_01_11_1431-002_add_acquisition_time_index.py
+│       ├── 2025_09_18_1630-003_add_scifi_robot_prediction_models.py
+│       └── ...                              # Additional migrations
 ```
 
+**Migration file naming convention**: `YYYY_MM_DD_HHMM-NNN_description.py` where:
+- `YYYY_MM_DD_HHMM` is the creation timestamp
+- `NNN` is the sequential migration number
+- `description` describes the change
+
 ## Further Reading
 
 - [Alembic Documentation](https://alembic.sqlalchemy.org/)

docs/how-to/use-core-http-api-client.md → docs/backend/http-api-client.md

@@ -71,17 +71,16 @@ acquisitions = await client.aget_acquisitions()
 ### Creating an Acquisition
 
 ```python
-from smartem_agent.model.schemas import EpuSessionData
+from smartem_common.schemas import AcquisitionData
 from datetime import datetime
 
-# Create an acquisition from EPU session data
-session = EpuSessionData(
-    id="my-acquisition-id",
+# Create an acquisition from acquisition data
+acquisition_data = AcquisitionData(
     name="My Acquisition",
     start_time=datetime.now(),
     storage_path="/path/to/storage"
 )
-response = client.create_acquisition(session)
+response = client.create_acquisition(acquisition_data)
 print(f"Created acquisition with ID: {response.id}")
 
 # Alternatively, create directly with an API request model
@@ -138,29 +137,28 @@ The client supports the full hierarchy of entities:
 Here's an example of creating entities at each level:
 
 ```python
+from smartem_common.schemas import (
+    AcquisitionData, GridData, GridSquareData, FoilHoleData,
+    MicrographData, MicrographManifest
+)
+
 # Create an acquisition
-acquisition = client.create_acquisition(EpuSessionData(id="acq-1", name="Test Acquisition"))
+acquisition_data = AcquisitionData(name="Test Acquisition")
+acquisition = client.create_acquisition(acquisition_data)
 
 # Create a grid for the acquisition
-from smartem_agent.model.schemas import GridData
-
 grid = GridData(data_dir="/path/to/grid")
 grid_response = client.create_acquisition_grid(acquisition.id, grid)
 
 # Create a grid square for the grid
-from smartem_agent.model.schemas import GridSquareData
-
 gridsquare = GridSquareData(id="gs-1", data_dir="/path/to/gridsquare")
 gridsquare_response = client.create_grid_gridsquare(grid_response.id, gridsquare)
 
 # Create a foil hole for the grid square
-from smartem_agent.model.schemas import FoilHoleData
-
 foilhole = FoilHoleData(id="fh-1", gridsquare_id=gridsquare.id)
 foilhole_response = client.create_gridsquare_foilhole(gridsquare_response.id, foilhole)
 
 # Create a micrograph for the foil hole
-from smartem_agent.model.schemas import MicrographData, MicrographManifest
 
 manifest = MicrographManifest(
     unique_id="mic-1",

docs/backend/index.md

@@ -0,0 +1,24 @@
+# Backend
+
+Documentation for the SmartEM backend services: API server, database, and message queue.
+
+```{toctree}
+:maxdepth: 1
+
+api-server
+database
+api-documentation
+http-api-client
+```
+
+## Topics
+
+### Running the Backend
+
+- [API Server](api-server.md) - Running and configuring the FastAPI server locally
+- [Database](database.md) - PostgreSQL setup, Alembic migrations, schema management
+
+### API Usage
+
+- [API Documentation](api-documentation.md) - Interactive Swagger UI documentation for SmartEM and Athena APIs
+- [HTTP API Client](http-api-client.md) - Python client for programmatic API access