feat: Add comprehensive job archiving system and bump to v1.3.0

Add complete job archiving functionality with policy-driven retention,
compression, and web dashboard integration.

## Core Archive Features
- Policy-based job archival with configurable retention periods
- Payload compression using gzip with configurable levels
- Archive table schema with PostgreSQL and MySQL support
- Restore archived jobs back to pending status
- Purge old archived jobs for compliance requirements

## CLI Integration
- Archive command for policy-driven job archival
- Restore command for individual job restoration
- List-archived command with filtering and pagination
- Purge-archived command for compliance cleanup
- Archive statistics and monitoring

## Web Dashboard Integration
- Complete archive management UI with statistics cards
- Archive jobs table with filtering and pagination
- Archive modal for policy configuration and execution
- Archive statistics modal with detailed metrics
- Job restoration workflow with audit trails
- Real-time archive statistics and compression tracking

## Technical Implementation
- Extended DatabaseQueue trait with archive methods
- Database migrations (010) for archive table setup
- Comprehensive doctests and test coverage
- Archive API endpoints with REST interface
- Enhanced error handling and user notifications

## Version Updates
- Bump workspace version to 1.3.0
- Update README and CHANGELOG documentation
- Migrate hammerwork-web to workspace dependencies
- Add roadmap item for WebSocket archive events

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>

Author: CodingAnarchy

CHANGELOG.md

@@ -5,6 +5,34 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [1.3.0] - 2025-07-01
+
+### Added
+- **🗄️ Job Archiving & Retention System**
+  - Policy-driven job archival with configurable retention periods per job status
+  - Payload compression using gzip for efficient long-term storage
+  - Archive table (`hammerwork_jobs_archive`) with compressed payloads
+  - Restore archived jobs back to pending status when needed
+  - Purge old archived jobs for compliance requirements (GDPR, data retention)
+  - Comprehensive archival statistics and monitoring
+  - CLI commands for archive management in `cargo-hammerwork`
+
+### Enhanced
+- **⚡ Archive Management Features**
+  - Automatic archival based on job age and status (Completed, Failed, Dead, TimedOut)
+  - Configurable compression levels (0-9) with integrity verification
+  - Batch processing for efficient large-scale archival operations
+  - Archive metadata tracking (reason, timestamp, who initiated)
+  - List and search archived jobs with pagination support
+  - Database schema migration (010) for archive table setup
+
+### Technical Implementation
+- New `archive` module with `ArchivalPolicy`, `ArchivalConfig`, and `JobArchiver` types
+- Extended `DatabaseQueue` trait with archival methods for all database backends
+- PostgreSQL and MySQL implementations with optimized archive queries
+- Comprehensive doctests and unit tests for all archival functionality
+- Archive CLI commands: `archive`, `restore`, `list-archived`, `purge-archived`
+
 ## [1.2.2] - 2025-07-01
 
 ### Added

Cargo.toml

@@ -9,7 +9,7 @@ members = [
 resolver = "2"
 
 [workspace.package]
-version = "1.2.2"
+version = "1.3.0"
 edition = "2024"
 license = "MIT"
 repository = "https://github.com/CodingAnarchy/hammerwork"
@@ -19,7 +19,7 @@ documentation = "https://docs.rs/hammerwork"
 rust-version = "1.86"
 
 [workspace.dependencies]
-hammerwork = { version = "1.2.2", path = "." }
+hammerwork = { version = "1.3.0", path = "." }
 tokio = { version = "1.0", features = ["full"] }
 sqlx = { version = "0.8", features = ["runtime-tokio-rustls", "chrono", "uuid", "json"] }
 chrono = { version = "0.4", features = ["serde"] }
@@ -44,6 +44,11 @@ indicatif = "0.17"
 toml = "0.8"
 dirs = "5.0"
 tempfile = "3.8"
+flate2 = "1.0"
+bcrypt = "0.15"
+base64 = "0.22"
+tokio-tungstenite = "0.21"
+futures-util = "0.3"
 
 [package]
 name = "hammerwork"
@@ -82,6 +87,7 @@ opentelemetry = { version = "0.22", optional = true }
 opentelemetry_sdk = { version = "0.22", features = ["rt-tokio"], optional = true }
 opentelemetry-otlp = { version = "0.15", features = ["tokio"], optional = true }
 tracing-opentelemetry = { version = "0.23", optional = true }
+flate2 = { workspace = true }
 
 [features]
 default = ["metrics", "alerting"]

README.md

@@ -8,6 +8,7 @@ A high-performance, database-driven job queue for Rust with comprehensive featur
 - **🧪 TestQueue Framework**: Complete in-memory testing implementation with MockClock for deterministic testing of time-dependent features, workflows, and job processing
 - **🔍 Job Tracing & Correlation**: Comprehensive distributed tracing with OpenTelemetry integration, trace IDs, correlation IDs, and lifecycle event hooks
 - **🔗 Job Dependencies & Workflows**: Create complex data processing pipelines with job dependencies, sequential chains, and parallel processing with synchronization barriers
+- **🗄️ Job Archiving & Retention**: Policy-driven archival with configurable retention periods, payload compression, and automated cleanup for compliance and performance
 - **Multi-database support**: PostgreSQL and MySQL backends with optimized dependency queries
 - **Advanced retry strategies**: Exponential backoff, linear, Fibonacci, and custom retry patterns with jitter
 - **Job prioritization**: Five priority levels with weighted and strict scheduling algorithms
@@ -29,15 +30,15 @@ A high-performance, database-driven job queue for Rust with comprehensive featur
 ```toml
 [dependencies]
 # Default features include metrics and alerting
-hammerwork = { version = "1.2", features = ["postgres"] }
+hammerwork = { version = "1.3", features = ["postgres"] }
 # or
-hammerwork = { version = "1.2", features = ["mysql"] }
+hammerwork = { version = "1.3", features = ["mysql"] }
 
 # With distributed tracing
-hammerwork = { version = "1.2", features = ["postgres", "tracing"] }
+hammerwork = { version = "1.3", features = ["postgres", "tracing"] }
 
 # Minimal installation
-hammerwork = { version = "1.2", features = ["postgres"], default-features = false }
+hammerwork = { version = "1.3", features = ["postgres"], default-features = false }
 ```
 
 **Feature Flags**: `postgres`, `mysql`, `metrics` (default), `alerting` (default), `tracing` (optional), `test` (for TestQueue)
@@ -50,7 +51,7 @@ cargo install hammerwork-web --features postgres
 
 # Or add to your project
 [dependencies]
-hammerwork-web = { version = "1.2", features = ["postgres"] }
+hammerwork-web = { version = "1.3", features = ["postgres"] }
 ```
 
 Start the dashboard:
@@ -71,6 +72,7 @@ See the [Quick Start Guide](docs/quick-start.md) for complete examples with Post
 - **[Web Dashboard](hammerwork-web/README.md)** - Real-time web interface for queue monitoring and job management
 - **[Job Tracing & Correlation](docs/tracing.md)** - Distributed tracing, correlation IDs, and OpenTelemetry integration
 - **[Job Dependencies & Workflows](docs/workflows.md)** - Complex pipelines, job dependencies, and orchestration
+- **[Job Archiving & Retention](docs/archiving.md)** - Policy-driven archival, compression, and compliance management
 - **[Job Types & Configuration](docs/job-types.md)** - Job creation, priorities, timeouts, cron jobs
 - **[Worker Configuration](docs/worker-configuration.md)** - Worker setup, rate limiting, statistics
 - **[Cron Scheduling](docs/cron-scheduling.md)** - Recurring jobs with timezone support  
@@ -275,6 +277,65 @@ async fn test_delayed_job_processing() {
 
 The `TestQueue` provides complete compatibility with the `DatabaseQueue` trait while offering deterministic time control through `MockClock`, making it perfect for testing complex workflows, retry logic, and time-dependent job processing.
 
+## Job Archiving Example
+
+Configure automatic job archival for compliance and database performance:
+
+```rust
+use hammerwork::{
+    archive::{ArchivalPolicy, ArchivalConfig, ArchivalReason},
+    queue::DatabaseQueue
+};
+use chrono::Duration;
+
+// Configure archival policy
+let policy = ArchivalPolicy::new()
+    .archive_completed_after(Duration::days(7))      // Archive completed jobs after 7 days
+    .archive_failed_after(Duration::days(30))        // Keep failed jobs for 30 days
+    .archive_dead_after(Duration::days(14))         // Archive dead jobs after 14 days
+    .archive_timed_out_after(Duration::days(21))    // Archive timed out jobs after 21 days
+    .purge_archived_after(Duration::days(365))      // Purge archived jobs after 1 year
+    .compress_archived_payloads(true)               // Enable gzip compression
+    .with_batch_size(1000)                          // Process up to 1000 jobs per batch
+    .enabled(true);
+
+let config = ArchivalConfig::new()
+    .with_compression_level(6)                      // Balanced compression
+    .with_compression_verification(true);           // Verify compression integrity
+
+// Run archival (typically scheduled as a cron job)
+let stats = queue.archive_jobs(
+    Some("payment_queue"),                          // Optional: archive specific queue
+    &policy,
+    &config,
+    ArchivalReason::Automatic,                      // Automatic, Manual, Compliance, Maintenance
+    Some("scheduler")                               // Who initiated the archival
+).await?;
+
+println!("Archived {} jobs, saved {} bytes (compression ratio: {:.2})",
+    stats.jobs_archived,
+    stats.bytes_archived,
+    stats.compression_ratio
+);
+
+// Restore an archived job if needed
+let job = queue.restore_archived_job(job_id).await?;
+
+// List archived jobs with filtering
+let archived_jobs = queue.list_archived_jobs(
+    Some("payment_queue"),     // Optional queue filter
+    Some(100),                // Limit
+    Some(0)                   // Offset for pagination
+).await?;
+
+// Purge old archived jobs for GDPR compliance
+let purged = queue.purge_archived_jobs(
+    Utc::now() - Duration::days(730)  // Delete jobs archived over 2 years ago
+).await?;
+```
+
+Archival moves completed/failed jobs to a separate table with compressed payloads, reducing the main table size while maintaining compliance requirements.
+
 ## Web Dashboard
 
 Start the real-time web dashboard for monitoring and managing your job queues:
@@ -344,11 +405,12 @@ queue.enqueue(job).await?;
 
 Hammerwork uses optimized tables with comprehensive indexing:
 - **`hammerwork_jobs`** - Main job table with priorities, timeouts, cron scheduling, retry strategies, result storage, and distributed tracing fields
+- **`hammerwork_jobs_archive`** - Archive table for completed/failed jobs with compressed payloads (v1.3.0+)
 - **`hammerwork_batches`** - Batch metadata and tracking (v0.7.0+)
 - **`hammerwork_job_results`** - Job result storage with TTL and expiration (v0.8.0+)
 - **`hammerwork_migrations`** - Migration tracking for schema evolution
 
-The schema supports all features including job prioritization, advanced retry strategies, timeouts, cron scheduling, batch processing, result storage with TTL, distributed tracing with trace/correlation IDs, worker autoscaling, and comprehensive lifecycle tracking. See [Database Migrations](docs/migrations.md) for details.
+The schema supports all features including job prioritization, advanced retry strategies, timeouts, cron scheduling, batch processing, result storage with TTL, distributed tracing with trace/correlation IDs, worker autoscaling, job archival with compression, and comprehensive lifecycle tracking. See [Database Migrations](docs/migrations.md) for details.
 
 ## Development
 

ROADMAP.md

@@ -5,23 +5,6 @@ This roadmap outlines planned features for Hammerwork, prioritized by impact lev
 ## Phase 2: Medium Impact, Variable Complexity
 *Valuable features for specific use cases or operational efficiency*
 
-### 🗄️ Job Archiving & Retention
-**Impact: Medium** | **Complexity: Medium** | **Priority: Medium**
-
-Important for compliance and database performance in high-volume systems.
-
-```rust
-// Automatic job archiving
-let archival_policy = ArchivalPolicy::new()
-    .archive_completed_after(Duration::from_days(7))
-    .archive_failed_after(Duration::from_days(30))
-    .purge_archived_after(Duration::from_days(365))
-    .compress_archived_payloads(true);
-
-queue.set_archival_policy(archival_policy).await?;
-```
-
-
 ### ⚡ Dynamic Job Spawning
 **Impact: Medium** | **Complexity: Medium-High** | **Priority: Medium**
 
@@ -57,6 +40,37 @@ let event_stream = EventStream::new()
     .with_filtering(|event| event.priority >= JobPriority::High);
 ```
 
+### 📡 Real-time Archive WebSocket Events
+**Impact: Low-Medium** | **Complexity: Low** | **Priority: Low**
+
+Enhance the web dashboard with real-time updates for archive operations.
+
+```rust
+// WebSocket events for archive operations
+#[derive(Serialize)]
+enum ArchiveEvent {
+    JobArchived { job_id: JobId, queue: String, reason: ArchivalReason },
+    JobRestored { job_id: JobId, queue: String, restored_by: Option<String> },
+    BulkArchiveStarted { operation_id: String, estimated_jobs: u64 },
+    BulkArchiveProgress { operation_id: String, jobs_processed: u64, total: u64 },
+    BulkArchiveCompleted { operation_id: String, stats: ArchivalStats },
+    JobsPurged { count: u64, older_than: DateTime<Utc> },
+}
+
+// Real-time dashboard updates
+websocket.send_event(ArchiveEvent::JobArchived {
+    job_id: job.id,
+    queue: job.queue_name.clone(),
+    reason: ArchivalReason::Automatic,
+});
+```
+
+**Benefits:**
+- Live archive operation progress tracking
+- Instant UI updates when jobs are archived/restored
+- Real-time compression ratio and storage statistics
+- Enhanced user experience during bulk operations
+
 ## Phase 3: Specialized Features
 *Features for specific enterprise or compliance requirements*
 
@@ -147,22 +161,16 @@ let geo_config = GeoReplicationConfig::new()
 
 Features are ordered within each phase by priority and should generally be implemented in the following sequence:
 
-**Phase 1 (Advanced Features) - CURRENT PRIORITIES**
-1. ✅ Job Dependencies & Workflows (COMPLETED v1.1.0)
-2. 🚧 Admin Dashboard & CLI Tools (CLI completed v1.1.0, Web Dashboard remaining)
-
-**Phase 2 (Operational Features)**
-1. Job Archiving & Retention
-2. Job Testing & Simulation
-3. Dynamic Job Spawning
-4. Webhook & Event Streaming
+**Phase 1 (Operational Features)**
+1. Dynamic Job Spawning
+2. Webhook & Event Streaming
 
-**Phase 3 (Enterprise Features)**
+**Phase 2 (Enterprise Features)**
 1. Job Encryption & PII Protection
 2. Access Control & Auditing
 3. Message Queue Integration
 
-**Phase 4 (Scaling Features)**
+**Phase 3 (Scaling Features)**
 1. Zero-downtime Deployments
 2. Queue Partitioning & Sharding
 3. Multi-region Support