Skip to content

implementation_plan.md

Latest commit

 

History

History
472 lines (373 loc) · 12.7 KB

implementation_plan.md

File metadata and controls

472 lines (373 loc) · 12.7 KB

Pastebox Engine: Enterprise-Grade Encrypted File Vault System

A production-ready daemon-based virtualization engine with encryption, load balancing, and advanced security for collaborative file management via SSH/SFTP.

User Review Required

Important

Daemon-Based Architecture

Building a multi-tier daemon system with proper isolation and encryption:

  • Routing Daemon - Main process that accepts commands, load balancing, manages all pasteboxes
  • Pastebox Instances - Isolated processes, each managing their own files and encryption
  • Encryption Queue - Async encryption pipeline with custom file extensions
  • MongoDB - Local metadata storage per pastebox (DB per box)
  • Advanced Security - Encryption at rest, secure key management, isolated processes

This is a production-grade system like Docker daemon architecture.

Warning

Security & Encryption Model

  • Each Pastebox runs as isolated process with own MongoDB instance
  • Encryption queue: Upload → Encrypt → Store → Delete original
  • Custom encrypted format: .pbx extension with metadata
  • Files encrypted with AES-256-GCM + user-provided passphrase
  • Zero-knowledge: Routing daemon cannot decrypt files
  • Pasteboxes can store: text, code, secrets (passwords, API keys)
  • Kill switch: Users can instantly terminate and delete pasteboxes

Proposed Changes

System Architecture

graph TB
    User[User/API Client]
    Router[Routing Daemon<br/>Load Balancer]
    Queue[Encryption Queue<br/>Bull/Redis]
    
    PB1[Pastebox Instance 1<br/>Isolated Process]
    PB2[Pastebox Instance 2<br/>Isolated Process]
    PB3[Pastebox Instance 3<br/>Isolated Process]
    
    Mongo1[(MongoDB 1<br/>Box Metadata)]
    Mongo2[(MongoDB 2<br/>Box Metadata)]
    Mongo3[(MongoDB 3<br/>Box Metadata)]
    
    FS1[Encrypted Storage 1<br/>.pbx files]
    FS2[Encrypted Storage 2<br/>.pbx files]
    FS3[Encrypted Storage 3<br/>.pbx files]
    
    SSH[SSH Gateway<br/>Port 2222]
    
    User -->|Create/Kill/Status| Router
    Router -->|Manage| PB1
    Router -->|Manage| PB2
    Router -->|Manage| PB3
    Router -->|Load Balance| Queue
    
    Queue -->|Encrypt Files| PB1
    Queue -->|Encrypt Files| PB2
    Queue -->|Encrypt Files| PB3
    
    PB1 -->|Store Metadata| Mongo1
    PB2 -->|Store Metadata| Mongo2
    PB3 -->|Store Metadata| Mongo3
    
    PB1 -->|Encrypted Files| FS1
    PB2 -->|Encrypted Files| FS2
    PB3 -->|Encrypted Files| FS3
    
    User -->|SSH/SFTP| SSH
    SSH -->|Route to Box| Router
    Router -->|Proxy| PB1
    
    style Router fill:#E91E63
    style Queue fill:#FF9800
    style PB1 fill:#4CAF50
    style PB2 fill:#4CAF50
    style PB3 fill:#4CAF50
Loading

Component 1: Routing Daemon (Main Controller)

[NEW] router-daemon.js

Purpose: Main daemon that manages all pasteboxes, load balancing, and routing.

Key Features:

  • Process Management: Spawns/kills pastebox instances as child processes
  • Load Balancing: Distributes load across available pasteboxes
  • Health Monitoring: Tracks status of each pastebox (active/idle/crashed)
  • Command Interface: Accepts commands (create, delete, kill, status, list)
  • SSH Routing: Routes SSH connections to appropriate pastebox instance
  • Resource Limits: Enforces CPU/memory limits per pastebox

API:

// Create new pastebox
POST /api/pastebox/create
{
  "userId": "user-123",
  "encryption": true,
  "passphrase": "user-secret",
  "ttl": 604800  // 7 days in seconds
}

// Kill pastebox immediately
POST /api/pastebox/kill/:boxId

// Delete pastebox and all data
DELETE /api/pastebox/:boxId

// Get pastebox status
GET /api/pastebox/:boxId/status

// List all pasteboxes
GET /api/pasteboxes

Pastebox Lifecycle:

graph LR
    A[Create Request] -->|Spawn Process| B[Initializing]
    B -->|Setup MongoDB| C[Active]
    C -->|User Kill| D[Terminating]
    C -->|TTL Expired| D
    D -->|Cleanup| E[Deleted]
    C -->|Crash| F[Failed]
    F -->|Auto-restart| C
Loading

Component 2: Pastebox Instance (Isolated Process)

[NEW] pastebox-instance.js

Purpose: Individual pastebox running as isolated process with own MongoDB.

Key Features:

  • Isolated Process: Runs as separate Node.js process (child of router)
  • Own MongoDB: Each box has dedicated MongoDB database
  • File Tracking: Maintains index of all files in the box
  • Encryption Management: Handles encrypt/decrypt operations
  • SSH Server: Embedded SFTP server for this box only
  • Auto-cleanup: Self-destructs on expiry

MongoDB Schema (per box):

// Database: pastebox_{boxId}
{
  // Box metadata
  box: {
    id: "box-abc123",
    ownerId: "user-123",
    createdAt: ISODate(),
    expiresAt: ISODate(),
    encryptionEnabled: true,
    status: "active"
  },
  
  // Files collection
  files: [
    {
      id: "file-xyz",
      originalName: "config.json",
      encryptedName: "a3f2c1d4.pbx",
      size: 1024,
      encryptedSize: 1088,
      mimeType: "application/json",
      uploadedAt: ISODate(),
      uploadedBy: "alice",
      checksum: "sha256:abc123...",
      encrypted: true,
      versions: [
        {
          versionNum: 1,
          encryptedName: "a3f2c1d4_v1.pbx",
          createdAt: ISODate(),
          createdBy: "alice"
        }
      ]
    }
  ],
  
  // Collaborators
  collaborators: [
    {
      userId: "user-123",
      userName: "alice",
      sshKeyFingerprint: "SHA256:...",
      sshPublicKey: "ssh-rsa ...",
      permission: "owner",
      addedAt: ISODate()
    }
  ],
  
  // Audit log
  auditLog: [
    {
      timestamp: ISODate(),
      userId: "user-123",
      action: "file_upload",
      fileName: "config.json",
      ipAddress: "192.168.1.1"
    }
  ]
}

Component 3: Encryption Queue System

[NEW] encryption-queue.js

Purpose: Async encryption pipeline for uploaded files.

Technology: Bull (Redis-based queue) for job processing

Encryption Flow:

graph LR
    A[File Upload] -->|Add to Queue| B[Encryption Job]
    B -->|AES-256-GCM| C[Encrypt File]
    C -->|Save .pbx| D[Encrypted Storage]
    D -->|Update MongoDB| E[File Indexed]
    E -->|Delete Original| F[Cleanup]
    F -->|Notify User| G[Complete]
Loading

Encryption Format (.pbx):

[Header: 64 bytes]
- Magic: "PASTEBOX" (8 bytes)
- Version: 1 (4 bytes)
- Algorithm: "AES-256-GCM" (16 bytes)
- IV: Random (16 bytes)
- Auth Tag: (16 bytes)
- Reserved: (4 bytes)

[Metadata: Variable length, encrypted]
- Original filename
- MIME type
- Timestamps
- Checksum

[Content: Variable length, encrypted]
- Actual file data

Key Derivation:

// User passphrase → Encryption key
const key = crypto.pbkdf2Sync(
  passphrase,
  salt,  // Stored in MongoDB
  100000,  // iterations
  32,  // key length
  'sha256'
);

Component 4: File Manager

[NEW] file-manager.js

Purpose: Manages files within a pastebox instance.

Key Features:

  • Upload Handler: Accepts files via SFTP, queues for encryption
  • Download Handler: Decrypts files on-the-fly for authorized users
  • Version Control: Creates encrypted snapshots on every edit
  • File Index: Maintains searchable index in MongoDB
  • Deduplication: Content-addressable storage (same content = same hash)
  • Quota Management: Enforces storage limits per box

Operations:

// Upload file
await fileManager.upload(filename, buffer, userId);
// → Queue encryption → Save .pbx → Index in MongoDB

// Download file (auto-decrypt)
const decrypted = await fileManager.download(filename, passphrase);

// List files
const files = await fileManager.list();

// Delete file (secure wipe)
await fileManager.delete(filename);

// Get file versions
const versions = await fileManager.getVersions(filename);

// Rollback to version
await fileManager.rollback(filename, versionNum);

Component 5: SSH Gateway with Routing

[NEW] ssh-router.js

Purpose: SSH/SFTP gateway that routes connections to pastebox instances.

How It Works:

  1. User connects: ssh box-abc123@pastezen.com
  2. Gateway authenticates via routing daemon
  3. Routing daemon finds pastebox instance process
  4. Gateway proxies connection to that instance's SFTP server
  5. All file operations go through encryption/decryption

Custom Commands:

pastebox> ls
config.json (encrypted, 1.2 KB)
secrets.env (encrypted, 512 B)

pastebox> upload myfile.txt
Uploading... Encrypting... Done!

pastebox> download config.json
Enter passphrase: ****
Decrypting... Done!

pastebox> history config.json
v3  2026-01-02 19:00  alice  (encrypted)
v2  2026-01-02 18:30  bob    (encrypted)
v1  2026-01-02 18:00  alice  (encrypted)

pastebox> status
Box: box-abc123
Status: Active
Files: 3 (encrypted)
Storage: 2.1 MB / 100 MB
Expires: 2026-01-09 19:00

Component 6: Security Features

[NEW] security-manager.js

Advanced Security:

  1. Process Isolation

    • Each pastebox runs in separate process
    • Resource limits (CPU, memory, disk)
    • Crash isolation (one box crash doesn't affect others)

  2. Encryption

    • AES-256-GCM (authenticated encryption)
    • PBKDF2 key derivation (100k iterations)
    • Unique IV per file
    • Zero-knowledge (server never sees plaintext)

  3. Access Control

    • SSH public key authentication
    • Per-file permission checks
    • Audit logging of all operations
    • IP-based rate limiting

  4. Secure Deletion

    • NIST 800-88 compliant overwriting
    • Metadata scrubbing
    • MongoDB collection drop
    • Process termination

  5. Secrets Storage

    • Dedicated secrets manager
    • Environment variable encryption
    • API key vault
    • Password generator

Component 7: Load Balancer

[NEW] load-balancer.js

Purpose: Distributes pasteboxes across available resources.

Strategies:

  • Round Robin: Distribute evenly
  • Least Loaded: Assign to instance with lowest CPU/memory
  • Geographic: Route based on user location (future)
  • Affinity: Keep user's boxes on same instance (cache locality)

Health Checks:

setInterval(() => {
  instances.forEach(instance => {
    const health = {
      cpu: instance.getCpuUsage(),
      memory: instance.getMemoryUsage(),
      activeBoxes: instance.getBoxCount(),
      status: instance.isHealthy() ? 'healthy' : 'degraded'
    };
    
    if (!health.status === 'healthy') {
      // Migrate boxes to healthy instance
      migrateBoxes(instance);
    }
  });
}, 30000); // Every 30 seconds

Technology Stack Summary

Component Technology Why
Routing Daemon Node.js + Express API server, process management
Pastebox Instances Node.js child processes Isolation, independent lifecycle
Encryption Node.js crypto (AES-256-GCM) Industry standard, built-in
Queue System Bull + Redis Reliable async job processing
Database MongoDB (per box) Flexible schema, fast queries
SSH/SFTP ssh2 library Cross-platform SSH implementation
Load Balancing Custom round-robin Simple, effective
IPC Node.js IPC / Unix sockets Fast inter-process communication

Verification Plan

Automated Tests

# Unit tests
npm test

# Integration tests
npm run test:integration

# Load tests (1000 concurrent boxes)
npm run test:load

# Security audit
npm run test:security

Manual Testing

  1. Create encrypted pastebox
  2. Upload files via SFTP → verify encryption queue
  3. Download files → verify decryption works
  4. Multi-user collaboration → verify permissions
  5. Kill pastebox → verify immediate termination
  6. TTL expiry → verify auto-cleanup
  7. Load balancing → create 100 boxes, verify distribution

Next Steps

  1. ✅ Plan approved - Daemon-based architecture with encryption
  2. Build routing daemon - Process manager and API
  3. Build pastebox instance - Isolated process with MongoDB
  4. Implement encryption - Queue system and .pbx format
  5. Build file manager - Upload/download with encryption
  6. Integrate SSH gateway - Routing and proxying
  7. Add load balancer - Distribution and health checks
  8. Test & deploy - Security audit and production deployment