add EPIC.md

Author: wiwski

EPIC.md

@@ -1,54 +1,232 @@
+# [EPIC] Project data mover (Files ↔ Blob Cool) with AzCopy
+
 ## Context
-PRD: Hot → Cool Project Data Management (Immutable Cool)
 
-This epic implements the **project-level** data lifecycle described in the PRD.
-The lifecycle applies to the **entire project data folder**, including runs,
-documents, and any other project-scoped files.
+This EPIC is part of the **Hot → Cool Project Data Management (Immutable Cool)** initiative.
+
+After delivering the **lifecycle state machine, immutability rules, eligibility logic, and admin visibility** (EPIC 1), the system now needs a **reliable, auditable, and verifiable mechanism** to physically move project data between storage tiers.
+
+Project data transitions must be:
+
+* **explicitly triggered**
+* **long-running**
+* **idempotent**
+* **verifiable**
+* **fully observable**
+
+Actual data movement is delegated to **euphrosyne-tools-api**, using **AzCopy** for storage-native, high-throughput transfers.
 
 ---
 
 ## Goal
-Implement **project-level** lifecycle state machine, automation, immutability
-enforcement, and UI visibility.
 
-The lifecycle controls when a project’s data is stored in:
-- Azure Files (HOT workspace), or
-- Azure Blob Storage (COOL, immutable).
+Implement **project-level COOL and RESTORE operations** that:
+
+* move **entire project data** between:
+
+  * Azure Files (HOT)
+  * Azure Blob Storage (Cool tier)
+* are executed via **AzCopy**
+* are tracked as **long-running lifecycle operations**
+* are verified using **AzCopy transfer summaries**
+* drive project lifecycle state transitions in Euphrosyne
+
+This EPIC is the first one that **moves bytes**, not just states.
 
 ---
 
-## Success criteria
+## Scope
+
+### In scope
+
+* Project-level data copy:
+
+  * Files → Blob Cool (COOL)
+  * Blob Cool → Files (RESTORE)
+* Long-running operation tracking
+* Operation polling and status reconciliation
+* Verification based on:
+
+  * expected file count
+  * expected byte size
+* Automatic triggering for eligible projects
+* Manual triggering for restore
+* Full auditability
+
+### Out of scope (explicit)
+
+* Deletion of HOT data after cooling
+* Cold / Archive tier
+* Partial project tiering
+* Delta sync or incremental copy
+* Deduplication
+* Concurrent HOT + COOL writes
+
+---
+
+## High-level design
+
+### Source of truth
+
+* **Euphrosyne DB** is the authoritative source for:
+
+  * lifecycle state
+  * storage class
+  * eligibility
+  * expected bytes / files
+* **tools-api** is the executor and reporter of physical copy operations
+
+State transitions **never happen based on AzCopy alone** —
+they only occur after **verified success** is reported back.
+
+---
+
+### Lifecycle operations
+
+Each COOL or RESTORE is modeled as a **single lifecycle operation** with:
+
+* a unique `operation_id`
+* a fixed direction (`COOL` or `RESTORE`)
+* immutable expectations (bytes/files)
+* monotonic status progression:
+
+  ```
+  PENDING → RUNNING → SUCCEEDED | FAILED
+  ```
+
+Operations are:
+
+* idempotent per `(project_id, operation_id)`
+* never reused across retries
+* fully auditable
+
+---
+
+### Storage movement model
+
+**COOL**
+
+```
+Azure Files (project folder)
+        ↓
+Azure Blob Storage (Cool tier, project prefix)
+```
+
+**RESTORE**
+
+```
+Azure Blob Storage (Cool tier)
+        ↓
+Azure Files (project folder)
+```
 
-- Projects automatically become eligible for cooling based on activity:
-  - Initial eligibility: `project.created + 6 months`
-  - Updated eligibility each time a new run is planned:
-    `run.end_date + 6 months`
-- Entire project data (runs + documents) is cooled as a single unit.
-- Restore works on demand and returns the project to HOT.
-- Writes are blocked when the project is in `COOL` or `COOLING`:
-  - document uploads/edits/deletes
-  - run outputs
-  - any write under the project folder
-- **New runs cannot be created** when the project is `COOL` or `COOLING`.
-- Admins can see:
-  - lifecycle state
-  - cooling eligibility date
-  - last lifecycle operation and error (if any).
+Characteristics:
+
+* Entire project moves as a single unit
+* Directory structure is preserved
+* COOL storage is treated as **immutable**
+* HOT storage is treated as **ephemeral workspace**
 
 ---
 
-## Non-goals (explicit for this epic)
+## AzCopy integration model
+
+### Role of AzCopy
+
+AzCopy is used as the **only mechanism** for data transfer:
+
+* high throughput
+* resumable
+* storage-native verification
+
+tools-api is responsible for:
 
-- No cold/archive tier
-- No partial (per-run) cooling
-- No deletion of hot data after cooling
-- No detection of concurrent readers (existing VM mounts tolerated)
+* starting AzCopy jobs
+* polling job progress
+* parsing AzCopy summaries
+* translating AzCopy outcomes into lifecycle operation status
 
 ---
 
-## Notes
+### Verification contract
+
+A lifecycle operation is considered **successful** if and only if:
+
+* AzCopy job completes successfully
+* `files_copied == expected_files`
+* `bytes_copied == expected_bytes`
+
+Any mismatch or execution error results in **FAILED**.
+
+There is no partial success.
+
+---
+
+## Execution flow (nominal)
+
+### COOL (automatic)
+
+1. Project becomes eligible for cooling
+2. Euphrosyne starts a COOL operation
+3. Project enters `COOLING`
+4. tools-api launches AzCopy (Files → Blob Cool)
+5. AzCopy completes
+6. tools-api reports verified success + stats
+7. Euphrosyne marks project `COOL`
+
+---
+
+### RESTORE (manual)
+
+1. User or admin triggers restore
+2. Project enters `RESTORING`
+3. tools-api launches AzCopy (Blob Cool → Files)
+4. AzCopy completes
+5. tools-api reports verified success + stats
+6. Euphrosyne marks project `HOT`
+
+---
+
+## Failure model
+
+* Any failure during copy or verification:
+
+  * lifecycle operation → `FAILED`
+  * project lifecycle → `ERROR`
+* Errors are:
+
+  * persisted
+  * visible to admins
+  * retryable via a new operation
+
+Project state never flips on partial or unverified success.
+
+---
+
+## Observability & auditability
+
+For each operation, the system records:
+
+* type (COOL / RESTORE)
+* timestamps (start / finish)
+* status
+* expected vs actual bytes/files
+* error details (if any)
+
+Admins can:
+
+* inspect operation history
+* understand failures
+* retry safely
+
+---
+
+## Success criteria
+
+This EPIC is considered complete when:
 
-- Lifecycle state is tracked **at the project level**, not run level.
-- Euphrosyne is the source of truth for lifecycle state and eligibility.
-- Physical data movement is handled by euphrosyne-tools-api and is out of scope
-  for this epic.
+* COOL and RESTORE operations move full project data via AzCopy
+* Operations are fully tracked and observable
+* Verification gates lifecycle state transitions
+* Automatic cooling works end-to-end
+* Restore reliably returns projects to HOT