Refactor Ingest Architecture: Remove Dynamic Disks & Introduce Single Inbox Pipeline

[N3XT0R]

Summary

The current ingest workflow evolved from a CLI-based FTP import tool
into a SaaS platform with multiple upload sources (web upload, FTP,
background services).

Over time this introduced architectural complexity and reliability
issues: - Dynamic disks in the critical ingest path - Multiple entry
points triggering the same processing logic - Tight coupling between
upload events and ingest execution - Race conditions and inconsistent
states across different sources

This issue proposes a structural refactor to establish a single source
of truth for ingest processing.

---

Target Architecture

1. Single Storage Disk

All video-related operations must use a single configured disk:

        'video' => [
            'driver' => 'local',
            'root' => storage_path('app/video'),
            'url' => env('APP_URL').'/video',
            'visibility' => 'public',
            'serve' => true,
            'throw' => false,
            'report' => false,
            'permissions' => [
                'file' => [
                    'public' => 0775,
                    'private' => 0775,
                ],
                'dir' => [
                    'public' => 0775,
                    'private' => 0775,
                ],
            ],
        ],

No dynamic disk resolution in the ingest path.

All uploads (web, FTP, services) must normalize into this disk.

---

2. Inbox Normalization

Introduce an inbox structure inside the video disk:

videos/
 ├── inbox/
 ├── inbox/.tmp/
 ├── processed/

Rules:

• Files are written to inbox/.tmp/
• After upload completion → atomic rename to inbox/
• Only stable files in inbox/ are processed
• After successful ingest → move to processed/ or delete

---

3. Cron-Based Ingest Scanner

Introduce scheduled command:

php artisan ingest:scan-video-inbox

Responsibilities:

• Scan video/inbox/
• Ignore unstable files (mtime or size threshold)
• Acquire per-file lock
• Process file
• Update video progress status
• Move/delete file after success

Scheduler:

->withoutOverlapping()
->onOneServer()

---

4. Video Processing Status

Add explicit processing state to videos table:

Suggested column:

processing_status ENUM:
- pending
- processing
- completed
- failed

Optional: - processing_progress (integer 0--100) -
processing_error_message

This enables: - Reliable UI state - Retry logic - Observability - Clear
operational debugging

---

Benefits

• Single source of truth for ingest
• No dynamic disks in processing path
• Reduced race conditions
• Source-agnostic architecture
• Improved reliability
• Explicit video processing lifecycle
• Simplified debugging and maintenance

---

Migration Plan

1. use uploads disk as canonical storage
2. Implement inbox structure
3. Add cron-based ingest scanner
4. Add processing_status to videos
5. Remove dynamic disk resolution from ingest logic
6. Disable direct job dispatch from upload
7. migrate preview_path from url to disk/path

---

Non-Goals

• No immediate workflow engine rewrite
• No UI redesign

This issue focuses solely on stabilizing ingest and storage
architecture.