diff --git a/.AI_INSTRUCTIONS_SYNC.md b/.AI_INSTRUCTIONS_SYNC.md new file mode 100644 index 0000000000..b268064af0 --- /dev/null +++ b/.AI_INSTRUCTIONS_SYNC.md @@ -0,0 +1,41 @@ +# AI Instructions Synchronization Guide + +**This file has moved!** + +All AI documentation and synchronization guidelines are now in the `.ai/` directory. + +## New Locations + +- **Sync Guide**: [.ai/meta/sync-guide.md](.ai/meta/sync-guide.md) +- **Maintaining Docs**: [.ai/meta/maintaining-docs.md](.ai/meta/maintaining-docs.md) +- **Documentation Hub**: [.ai/README.md](.ai/README.md) + +## Quick Overview + +All AI instructions are now organized in `.ai/` directory: + +``` +.ai/ +├── README.md # Navigation hub +├── core/ # Project information +├── development/ # Dev workflows +├── patterns/ # Code patterns +└── meta/ # Documentation guides +``` + +### For AI Assistants + +- **Claude Code**: Use `CLAUDE.md` (references `.ai/` files) +- **Cursor IDE**: Use `.cursor/rules/coolify-ai-docs.mdc` (references `.ai/` files) +- **All Tools**: Browse `.ai/` directory for detailed documentation + +### Key Principles + +1. **Single Source of Truth**: Each piece of information exists in ONE file only +2. **Cross-Reference**: Other files reference the source, don't duplicate +3. **Organized by Topic**: Core, Development, Patterns, Meta +4. **Version Consistency**: All versions in `.ai/core/technology-stack.md` + +## For More Information + +See [.ai/meta/sync-guide.md](.ai/meta/sync-guide.md) for complete synchronization guidelines and [.ai/meta/maintaining-docs.md](.ai/meta/maintaining-docs.md) for documentation maintenance instructions. diff --git a/.ai/README.md b/.ai/README.md new file mode 100644 index 0000000000..ea7812496e --- /dev/null +++ b/.ai/README.md @@ -0,0 +1,148 @@ +# Coolify AI Documentation + +Welcome to the Coolify AI documentation hub. This directory contains all AI assistant instructions organized by topic for easy navigation and maintenance. + +## Quick Start + +- **For Claude Code**: Start with [CLAUDE.md in root directory](../CLAUDE.md) +- **For Cursor IDE**: Check `.cursor/rules/coolify-ai-docs.mdc` which references this directory +- **For Other AI Tools**: Continue reading below + +## Documentation Structure + +### 📚 Core Documentation +Essential project information and architecture: + +- **[Technology Stack](core/technology-stack.md)** - All versions, packages, and dependencies (Laravel 12.4.1, PHP 8.4.7, etc.) +- **[Project Overview](core/project-overview.md)** - What Coolify is and how it works +- **[Application Architecture](core/application-architecture.md)** - System design and component relationships +- **[Deployment Architecture](core/deployment-architecture.md)** - How deployments work end-to-end, including Coolify Docker Compose extensions (custom fields) + +### 💻 Development +Day-to-day development practices: + +- **[Workflow](development/development-workflow.md)** - Development setup, commands, and daily workflows +- **[Testing Patterns](development/testing-patterns.md)** - How to write and run tests (Unit vs Feature, Docker requirements) +- **[Laravel Boost](development/laravel-boost.md)** - Laravel-specific guidelines and best practices + +### 🎨 Patterns +Code patterns and best practices by domain: + +- **[Database Patterns](patterns/database-patterns.md)** - Eloquent, migrations, relationships +- **[Frontend Patterns](patterns/frontend-patterns.md)** - Livewire, Alpine.js, Tailwind CSS +- **[Security Patterns](patterns/security-patterns.md)** - Authentication, authorization, security best practices +- **[Form Components](patterns/form-components.md)** - Enhanced form components with authorization +- **[API & Routing](patterns/api-and-routing.md)** - API design, routing conventions, REST patterns + +### 📖 Meta +Documentation about documentation: + +- **[Maintaining Docs](meta/maintaining-docs.md)** - How to update and improve this documentation +- **[Sync Guide](meta/sync-guide.md)** - Keeping documentation synchronized across tools + +## Quick Decision Tree + +**What do you need help with?** + +### Running Commands +→ [development/development-workflow.md](development/development-workflow.md) +- Frontend: `npm run dev`, `npm run build` +- Backend: `php artisan serve`, `php artisan migrate` +- Tests: Docker for Feature tests, mocking for Unit tests +- Code quality: `./vendor/bin/pint`, `./vendor/bin/phpstan` + +### Writing Tests +→ [development/testing-patterns.md](development/testing-patterns.md) +- **Unit tests**: No database, use mocking, run outside Docker +- **Feature tests**: Can use database, must run inside Docker +- Command: `docker exec coolify php artisan test` + +### Building UI +→ [patterns/frontend-patterns.md](patterns/frontend-patterns.md) or [patterns/form-components.md](patterns/form-components.md) +- Livewire components with server-side state +- Alpine.js for client-side interactivity +- Tailwind CSS 4.1.4 for styling +- Form components with built-in authorization + +### Database Work +→ [patterns/database-patterns.md](patterns/database-patterns.md) +- Eloquent ORM patterns +- Migration best practices +- Relationship definitions +- Query optimization + +### Security & Auth +→ [patterns/security-patterns.md](patterns/security-patterns.md) +- Team-based access control +- Policy and gate patterns +- Form authorization (canGate, canResource) +- API security + +### Laravel-Specific Questions +→ [development/laravel-boost.md](development/laravel-boost.md) +- Laravel 12 patterns +- Livewire 3 best practices +- Pest testing patterns +- Laravel conventions + +### Docker Compose Extensions +→ [core/deployment-architecture.md](core/deployment-architecture.md#coolify-docker-compose-extensions) +- Custom fields: `exclude_from_hc`, `content`, `isDirectory` +- How to use inline file content +- Health check exclusion patterns +- Volume creation control + +### Version Numbers +→ [core/technology-stack.md](core/technology-stack.md) +- **Single source of truth** for all version numbers +- Don't duplicate versions elsewhere, reference this file + +## Navigation Tips + +1. **Start broad**: Begin with project-overview or ../CLAUDE.md +2. **Get specific**: Navigate to topic-specific files for details +3. **Cross-reference**: Files link to related topics +4. **Single source**: Version numbers and critical data exist in ONE place only + +## For AI Assistants + +### Important Patterns to Follow + +**Testing Commands:** +- Unit tests: `./vendor/bin/pest tests/Unit` (no database, outside Docker) +- Feature tests: `docker exec coolify php artisan test` (requires database, inside Docker) +- NEVER run Feature tests outside Docker - they will fail with database connection errors + +**Version Numbers:** +- Always use exact versions from [technology-stack.md](core/technology-stack.md) +- Laravel 12.4.1, PHP 8.4.7, Tailwind 4.1.4 +- Don't use "v12" or "8.4" - be precise + +**Form Authorization:** +- ALWAYS include `canGate` and `:canResource` on form components +- See [form-components.md](patterns/form-components.md) for examples + +**Livewire Components:** +- MUST have exactly ONE root element +- See [frontend-patterns.md](patterns/frontend-patterns.md) for details + +**Code Style:** +- Run `./vendor/bin/pint` before finalizing changes +- Follow PSR-12 standards +- Use PHP 8.4 features (constructor promotion, typed properties, etc.) + +## Contributing + +When updating documentation: +1. Read [meta/maintaining-docs.md](meta/maintaining-docs.md) +2. Follow the single source of truth principle +3. Update cross-references when moving content +4. Test all links work +5. Run Pint on markdown files if applicable + +## Questions? + +- **Claude Code users**: Check [../CLAUDE.md](../CLAUDE.md) first +- **Cursor IDE users**: Check `.cursor/rules/coolify-ai-docs.mdc` +- **Documentation issues**: See [meta/maintaining-docs.md](meta/maintaining-docs.md) +- **Sync issues**: See [meta/sync-guide.md](meta/sync-guide.md) diff --git a/.ai/core/application-architecture.md b/.ai/core/application-architecture.md new file mode 100644 index 0000000000..c1fe7c470e --- /dev/null +++ b/.ai/core/application-architecture.md @@ -0,0 +1,612 @@ +# Coolify Application Architecture + +## Laravel Project Structure + +### **Core Application Directory** ([app/](mdc:app)) + +``` +app/ +├── Actions/ # Business logic actions (Action pattern) +├── Console/ # Artisan commands +├── Contracts/ # Interface definitions +├── Data/ # Data Transfer Objects (Spatie Laravel Data) +├── Enums/ # Enumeration classes +├── Events/ # Event classes +├── Exceptions/ # Custom exception classes +├── Helpers/ # Utility helper classes +├── Http/ # HTTP layer (Controllers, Middleware, Requests) +├── Jobs/ # Background job classes +├── Listeners/ # Event listeners +├── Livewire/ # Livewire components (Frontend) +├── Models/ # Eloquent models (Domain entities) +├── Notifications/ # Notification classes +├── Policies/ # Authorization policies +├── Providers/ # Service providers +├── Repositories/ # Repository pattern implementations +├── Services/ # Service layer classes +├── Traits/ # Reusable trait classes +└── View/ # View composers and creators +``` + +## Core Domain Models + +### **Infrastructure Management** + +#### **[Server.php](mdc:app/Models/Server.php)** (46KB, 1343 lines) +- **Purpose**: Physical/virtual server management +- **Key Relationships**: + - `hasMany(Application::class)` - Deployed applications + - `hasMany(StandalonePostgresql::class)` - Database instances + - `belongsTo(Team::class)` - Team ownership +- **Key Features**: + - SSH connection management + - Resource monitoring + - Proxy configuration (Traefik/Caddy) + - Docker daemon interaction + +#### **[Application.php](mdc:app/Models/Application.php)** (74KB, 1734 lines) +- **Purpose**: Application deployment and management +- **Key Relationships**: + - `belongsTo(Server::class)` - Deployment target + - `belongsTo(Environment::class)` - Environment context + - `hasMany(ApplicationDeploymentQueue::class)` - Deployment history +- **Key Features**: + - Git repository integration + - Docker build and deployment + - Environment variable management + - SSL certificate handling + +#### **[Service.php](mdc:app/Models/Service.php)** (58KB, 1325 lines) +- **Purpose**: Multi-container service orchestration +- **Key Relationships**: + - `hasMany(ServiceApplication::class)` - Service components + - `hasMany(ServiceDatabase::class)` - Service databases + - `belongsTo(Environment::class)` - Environment context +- **Key Features**: + - Docker Compose generation + - Service dependency management + - Health check configuration + +### **Team & Project Organization** + +#### **[Team.php](mdc:app/Models/Team.php)** (8.9KB, 308 lines) +- **Purpose**: Multi-tenant team management +- **Key Relationships**: + - `hasMany(User::class)` - Team members + - `hasMany(Project::class)` - Team projects + - `hasMany(Server::class)` - Team servers +- **Key Features**: + - Resource limits and quotas + - Team-based access control + - Subscription management + +#### **[Project.php](mdc:app/Models/Project.php)** (4.3KB, 156 lines) +- **Purpose**: Project organization and grouping +- **Key Relationships**: + - `hasMany(Environment::class)` - Project environments + - `belongsTo(Team::class)` - Team ownership +- **Key Features**: + - Environment isolation + - Resource organization + +#### **[Environment.php](mdc:app/Models/Environment.php)** +- **Purpose**: Environment-specific configuration +- **Key Relationships**: + - `hasMany(Application::class)` - Environment applications + - `hasMany(Service::class)` - Environment services + - `belongsTo(Project::class)` - Project context + +### **Database Management Models** + +#### **Standalone Database Models** +- **[StandalonePostgresql.php](mdc:app/Models/StandalonePostgresql.php)** (11KB, 351 lines) +- **[StandaloneMysql.php](mdc:app/Models/StandaloneMysql.php)** (11KB, 351 lines) +- **[StandaloneMariadb.php](mdc:app/Models/StandaloneMariadb.php)** (10KB, 337 lines) +- **[StandaloneMongodb.php](mdc:app/Models/StandaloneMongodb.php)** (12KB, 370 lines) +- **[StandaloneRedis.php](mdc:app/Models/StandaloneRedis.php)** (12KB, 394 lines) +- **[StandaloneKeydb.php](mdc:app/Models/StandaloneKeydb.php)** (11KB, 347 lines) +- **[StandaloneDragonfly.php](mdc:app/Models/StandaloneDragonfly.php)** (11KB, 347 lines) +- **[StandaloneClickhouse.php](mdc:app/Models/StandaloneClickhouse.php)** (10KB, 336 lines) + +**Common Features**: +- Database configuration management +- Backup scheduling and execution +- Connection string generation +- Health monitoring + +### **Configuration & Settings** + +#### **[EnvironmentVariable.php](mdc:app/Models/EnvironmentVariable.php)** (7.6KB, 219 lines) +- **Purpose**: Application environment variable management +- **Key Features**: + - Encrypted value storage + - Build-time vs runtime variables + - Shared variable inheritance + +#### **[InstanceSettings.php](mdc:app/Models/InstanceSettings.php)** (3.2KB, 124 lines) +- **Purpose**: Global Coolify instance configuration +- **Key Features**: + - FQDN and port configuration + - Auto-update settings + - Security configurations + +## Architectural Patterns + +### **Action Pattern** ([app/Actions/](mdc:app/Actions)) + +Using [lorisleiva/laravel-actions](mdc:composer.json) for business logic encapsulation: + +```php +// Example Action structure +class DeployApplication extends Action +{ + public function handle(Application $application): void + { + // Business logic for deployment + } + + public function asJob(Application $application): void + { + // Queue job implementation + } +} +``` + +**Key Action Categories**: +- **Application/**: Deployment and management actions +- **Database/**: Database operations +- **Server/**: Server management actions +- **Service/**: Service orchestration actions + +### **Repository Pattern** ([app/Repositories/](mdc:app/Repositories)) + +Data access abstraction layer: +- Encapsulates database queries +- Provides testable data layer +- Abstracts complex query logic + +### **Service Layer** ([app/Services/](mdc:app/Services)) + +Business logic services: +- External API integrations +- Complex business operations +- Cross-cutting concerns + +## Data Flow Architecture + +### **Request Lifecycle** + +1. **HTTP Request** → [routes/web.php](mdc:routes/web.php) +2. **Middleware** → Authentication, authorization +3. **Livewire Component** → [app/Livewire/](mdc:app/Livewire) +4. **Action/Service** → Business logic execution +5. **Model/Repository** → Data persistence +6. **Response** → Livewire reactive update + +### **Background Processing** + +1. **Job Dispatch** → Queue system (Redis) +2. **Job Processing** → [app/Jobs/](mdc:app/Jobs) +3. **Action Execution** → Business logic +4. **Event Broadcasting** → Real-time updates +5. **Notification** → User feedback + +## Security Architecture + +### **Multi-Tenant Isolation** + +```php +// Team-based query scoping +class Application extends Model +{ + public function scopeOwnedByCurrentTeam($query) + { + return $query->whereHas('environment.project.team', function ($q) { + $q->where('id', currentTeam()->id); + }); + } +} +``` + +### **Authorization Layers** + +1. **Team Membership** → User belongs to team +2. **Resource Ownership** → Resource belongs to team +3. **Policy Authorization** → [app/Policies/](mdc:app/Policies) +4. **Environment Isolation** → Project/environment boundaries + +### **Data Protection** + +- **Environment Variables**: Encrypted at rest +- **SSH Keys**: Secure storage and transmission +- **API Tokens**: Sanctum-based authentication +- **Audit Logging**: [spatie/laravel-activitylog](mdc:composer.json) + +## Configuration Hierarchy + +### **Global Configuration** +- **[InstanceSettings](mdc:app/Models/InstanceSettings.php)**: System-wide settings +- **[config/](mdc:config)**: Laravel configuration files + +### **Team Configuration** +- **[Team](mdc:app/Models/Team.php)**: Team-specific settings +- **[ServerSetting](mdc:app/Models/ServerSetting.php)**: Server configurations + +### **Project Configuration** +- **[ProjectSetting](mdc:app/Models/ProjectSetting.php)**: Project settings +- **[Environment](mdc:app/Models/Environment.php)**: Environment variables + +### **Application Configuration** +- **[ApplicationSetting](mdc:app/Models/ApplicationSetting.php)**: App-specific settings +- **[EnvironmentVariable](mdc:app/Models/EnvironmentVariable.php)**: Runtime configuration + +## Event-Driven Architecture + +### **Event Broadcasting** ([app/Events/](mdc:app/Events)) + +Real-time updates using Laravel Echo and WebSockets: + +```php +// Example event structure +class ApplicationDeploymentStarted implements ShouldBroadcast +{ + public function broadcastOn(): array + { + return [ + new PrivateChannel("team.{$this->application->team->id}"), + ]; + } +} +``` + +### **Event Listeners** ([app/Listeners/](mdc:app/Listeners)) + +- Deployment status updates +- Resource monitoring alerts +- Notification dispatching +- Audit log creation + +## Database Design Patterns + +### **Polymorphic Relationships** + +```php +// Environment variables can belong to multiple resource types +class EnvironmentVariable extends Model +{ + public function resource(): MorphTo + { + return $this->morphTo(); + } +} +``` + +### **Team-Based Soft Scoping** + +All major resources include team-based query scoping with request-level caching: + +```php +// ✅ CORRECT - Use cached methods (request-level cache via once()) +$applications = Application::ownedByCurrentTeamCached(); +$servers = Server::ownedByCurrentTeamCached(); + +// ✅ CORRECT - Filter cached collection in memory +$activeServers = Server::ownedByCurrentTeamCached()->where('is_active', true); + +// Only use query builder when you need eager loading or fresh data +$projects = Project::ownedByCurrentTeam()->with('environments')->get(); +``` + +See [Database Patterns](.ai/patterns/database-patterns.md#request-level-caching-with-ownedbycurrentteamcached) for full documentation. + +### **Configuration Inheritance** + +Environment variables cascade from: +1. **Shared Variables** → Team-wide defaults +2. **Project Variables** → Project-specific overrides +3. **Application Variables** → Application-specific values + +## Integration Patterns + +### **Git Provider Integration** + +Abstracted git operations supporting: +- **GitHub**: [app/Models/GithubApp.php](mdc:app/Models/GithubApp.php) +- **GitLab**: [app/Models/GitlabApp.php](mdc:app/Models/GitlabApp.php) +- **Bitbucket**: Webhook integration +- **Gitea**: Self-hosted Git support + +### **Docker Integration** + +- **Container Management**: Direct Docker API communication +- **Image Building**: Dockerfile and Buildpack support +- **Network Management**: Custom Docker networks +- **Volume Management**: Persistent storage handling + +### **SSH Communication** + +- **[phpseclib/phpseclib](mdc:composer.json)**: Secure SSH connections +- **Multiplexing**: Connection pooling for efficiency +- **Key Management**: [PrivateKey](mdc:app/Models/PrivateKey.php) model + +## Testing Architecture + +### **Test Structure** ([tests/](mdc:tests)) + +``` +tests/ +├── Feature/ # Integration tests +├── Unit/ # Unit tests +├── Browser/ # Dusk browser tests +├── Traits/ # Test helper traits +├── Pest.php # Pest configuration +└── TestCase.php # Base test case +``` + +### **Testing Patterns** + +- **Feature Tests**: Full request lifecycle testing +- **Unit Tests**: Individual class/method testing +- **Browser Tests**: End-to-end user workflows +- **Database Testing**: Factories and seeders + +## Performance Considerations + +### **Query Optimization** + +- **Eager Loading**: Prevent N+1 queries +- **Query Scoping**: Team-based filtering +- **Database Indexing**: Optimized for common queries + +### **Caching Strategy** + +- **Redis**: Session and cache storage +- **Model Caching**: Frequently accessed data +- **Query Caching**: Expensive query results + +### **Background Processing** + +- **Queue Workers**: Horizon-managed job processing +- **Job Batching**: Related job grouping +- **Failed Job Handling**: Automatic retry logic + +## Container Status Monitoring System + +### **Overview** + +Container health status is monitored and updated through **multiple independent paths**. When modifying status logic, **ALL paths must be updated** to ensure consistency. + +### **Critical Implementation Locations** + +#### **1. SSH-Based Status Updates (Scheduled)** +**File**: [app/Actions/Docker/GetContainersStatus.php](mdc:app/Actions/Docker/GetContainersStatus.php) +**Method**: `aggregateApplicationStatus()` (lines 487-540) +**Trigger**: Scheduled job or manual refresh +**Frequency**: Every minute (via `ServerCheckJob`) + +**Status Aggregation Logic**: +```php +// Tracks multiple status flags +$hasRunning = false; +$hasRestarting = false; +$hasUnhealthy = false; +$hasUnknown = false; // ⚠️ CRITICAL: Must track unknown +$hasExited = false; +// ... more states + +// Priority: restarting > degraded > running (unhealthy > unknown > healthy) +if ($hasRunning) { + if ($hasUnhealthy) return 'running (unhealthy)'; + elseif ($hasUnknown) return 'running (unknown)'; + else return 'running (healthy)'; +} +``` + +#### **2. Sentinel-Based Status Updates (Real-time)** +**File**: [app/Jobs/PushServerUpdateJob.php](mdc:app/Jobs/PushServerUpdateJob.php) +**Method**: `aggregateMultiContainerStatuses()` (lines 269-298) +**Trigger**: Sentinel push updates from remote servers +**Frequency**: Every ~30 seconds (real-time) + +**Status Aggregation Logic**: +```php +// ⚠️ MUST match GetContainersStatus logic +$hasRunning = false; +$hasUnhealthy = false; +$hasUnknown = false; // ⚠️ CRITICAL: Added to fix bug + +foreach ($relevantStatuses as $status) { + if (str($status)->contains('running')) { + $hasRunning = true; + if (str($status)->contains('unhealthy')) $hasUnhealthy = true; + if (str($status)->contains('unknown')) $hasUnknown = true; // ⚠️ CRITICAL + } +} + +// Priority: unhealthy > unknown > healthy +if ($hasRunning) { + if ($hasUnhealthy) $aggregatedStatus = 'running (unhealthy)'; + elseif ($hasUnknown) $aggregatedStatus = 'running (unknown)'; + else $aggregatedStatus = 'running (healthy)'; +} +``` + +#### **3. Multi-Server Status Aggregation** +**File**: [app/Actions/Shared/ComplexStatusCheck.php](mdc:app/Actions/Shared/ComplexStatusCheck.php) +**Method**: `resource()` (lines 48-210) +**Purpose**: Aggregates status across multiple servers for applications +**Used by**: Applications with multiple destinations + +**Key Features**: +- Aggregates statuses from main + additional servers +- Handles excluded containers (`:excluded` suffix) +- Calculates overall application health from all containers + +**Status Format with Excluded Containers**: +```php +// When all containers excluded from health checks: +return 'running:unhealthy:excluded'; // Container running but unhealthy, monitoring disabled +return 'running:unknown:excluded'; // Container running, health unknown, monitoring disabled +return 'running:healthy:excluded'; // Container running and healthy, monitoring disabled +return 'degraded:excluded'; // Some containers down, monitoring disabled +return 'exited:excluded'; // All containers stopped, monitoring disabled +``` + +#### **4. Service-Level Status Aggregation** +**File**: [app/Models/Service.php](mdc:app/Models/Service.php) +**Method**: `complexStatus()` (lines 176-288) +**Purpose**: Aggregates status for multi-container services +**Used by**: Docker Compose services + +**Status Calculation**: +```php +// Aggregates status from all service applications and databases +// Handles excluded containers separately +// Returns status with :excluded suffix when all containers excluded +if (!$hasNonExcluded && $complexStatus === null && $complexHealth === null) { + // All services excluded - calculate from excluded containers + return "{$excludedStatus}:excluded"; +} +``` + +### **Status Flow Diagram** + +``` +┌─────────────────────────────────────────────────────────────┐ +│ Container Status Sources │ +└─────────────────────────────────────────────────────────────┘ + │ + ┌────────────────────┼────────────────────┐ + │ │ │ + ▼ ▼ ▼ +┌───────────────┐ ┌─────────────────┐ ┌──────────────┐ +│ SSH-Based │ │ Sentinel-Based │ │ Multi-Server │ +│ (Scheduled) │ │ (Real-time) │ │ Aggregation │ +├───────────────┤ ├─────────────────┤ ├──────────────┤ +│ ServerCheck │ │ PushServerUp- │ │ ComplexStatus│ +│ Job │ │ dateJob │ │ Check │ +│ │ │ │ │ │ +│ Every ~1min │ │ Every ~30sec │ │ On demand │ +└───────┬───────┘ └────────┬────────┘ └──────┬───────┘ + │ │ │ + └────────────────────┼────────────────────┘ + │ + ▼ + ┌───────────────────────┐ + │ Application/Service │ + │ Status Property │ + └───────────────────────┘ + │ + ▼ + ┌───────────────────────┐ + │ UI Display (Livewire) │ + └───────────────────────┘ +``` + +### **Status Priority System** + +All status aggregation locations **MUST** follow the same priority: + +**For Running Containers**: +1. **unhealthy** - Container has failing health checks +2. **unknown** - Container health status cannot be determined +3. **healthy** - Container is healthy + +**For Non-Running States**: +1. **restarting** → `degraded (unhealthy)` +2. **running + exited** → `degraded (unhealthy)` +3. **dead/removing** → `degraded (unhealthy)` +4. **paused** → `paused` +5. **created/starting** → `starting` +6. **exited** → `exited (unhealthy)` + +### **Excluded Containers** + +When containers have `exclude_from_hc: true` flag or `restart: no`: + +**Behavior**: +- Status is still calculated from container state +- `:excluded` suffix is appended to indicate monitoring disabled +- UI shows "(Monitoring Disabled)" badge +- Action buttons respect the actual container state + +**Format**: `{actual-status}:excluded` +**Examples**: `running:unknown:excluded`, `degraded:excluded`, `exited:excluded` + +**All-Excluded Scenario**: +When ALL containers are excluded from health checks: +- All three status update paths (PushServerUpdateJob, GetContainersStatus, ComplexStatusCheck) **MUST** calculate status from excluded containers +- Status is returned with `:excluded` suffix (e.g., `running:healthy:excluded`) +- **NEVER** skip status updates - always calculate from excluded containers +- This ensures consistent status regardless of which update mechanism runs +- Shared logic is in `app/Traits/CalculatesExcludedStatus.php` + +### **Important Notes for Developers** + +✅ **Container Status Aggregation Service**: + +The container status aggregation logic is centralized in `App\Services\ContainerStatusAggregator`. + +**Status Format Standard**: +- **Backend/Storage**: Colon format (`running:healthy`, `degraded:unhealthy`) +- **UI/Display**: Transform to human format (`Running (Healthy)`, `Degraded (Unhealthy)`) + +1. **Using the ContainerStatusAggregator Service**: + - Import `App\Services\ContainerStatusAggregator` in any class needing status aggregation + - Two methods available: + - `aggregateFromStrings(Collection $statusStrings, int $maxRestartCount = 0)` - For pre-formatted status strings + - `aggregateFromContainers(Collection $containers, int $maxRestartCount = 0)` - For raw Docker container objects + - Returns colon format: `running:healthy`, `degraded:unhealthy`, etc. + - Automatically handles crash loop detection via `$maxRestartCount` parameter + +2. **State Machine Priority** (handled by service): + - Restarting → `degraded:unhealthy` (highest priority) + - Crash loop (exited with restarts) → `degraded:unhealthy` + - Mixed state (running + exited) → `degraded:unhealthy` + - Running → `running:unhealthy` / `running:unknown` / `running:healthy` + - Dead/Removing → `degraded:unhealthy` + - Paused → `paused:unknown` + - Starting/Created → `starting:unknown` + - Exited → `exited:unhealthy` (lowest priority) + +3. **Test both update paths**: + - Run unit tests: `./vendor/bin/pest tests/Unit/ContainerStatusAggregatorTest.php` + - Run integration tests: `./vendor/bin/pest tests/Unit/` + - Test SSH updates (manual refresh) + - Test Sentinel updates (wait 30 seconds) + +4. **Handle excluded containers**: + - All containers excluded (`exclude_from_hc: true`) - Use `CalculatesExcludedStatus` trait + - Mixed excluded/non-excluded containers - Filter then use `ContainerStatusAggregator` + - Containers with `restart: no` - Treated same as `exclude_from_hc: true` + +5. **Use shared trait for excluded containers**: + - Import `App\Traits\CalculatesExcludedStatus` in status calculation classes + - Use `getExcludedContainersFromDockerCompose()` to parse exclusions + - Use `calculateExcludedStatus()` for full Docker inspect objects (ComplexStatusCheck) + - Use `calculateExcludedStatusFromStrings()` for status strings (PushServerUpdateJob, GetContainersStatus) + +### **Related Tests** + +- **[tests/Unit/ContainerStatusAggregatorTest.php](mdc:tests/Unit/ContainerStatusAggregatorTest.php)**: Core state machine logic (42 comprehensive tests) +- **[tests/Unit/ContainerHealthStatusTest.php](mdc:tests/Unit/ContainerHealthStatusTest.php)**: Health status aggregation integration +- **[tests/Unit/PushServerUpdateJobStatusAggregationTest.php](mdc:tests/Unit/PushServerUpdateJobStatusAggregationTest.php)**: Sentinel update logic +- **[tests/Unit/ExcludeFromHealthCheckTest.php](mdc:tests/Unit/ExcludeFromHealthCheckTest.php)**: Excluded container handling + +### **Common Bugs to Avoid** + +✅ **Prevented by ContainerStatusAggregator Service**: +- ❌ **Old Bug**: Forgetting to track `$hasUnknown` flag → ✅ Now centralized in service +- ❌ **Old Bug**: Inconsistent priority across paths → ✅ Single source of truth +- ❌ **Old Bug**: Forgetting to update all 4 locations → ✅ Only one location to update + +**Still Relevant**: + +❌ **Bug**: Forgetting to filter excluded containers before aggregation +✅ **Fix**: Always use `CalculatesExcludedStatus` trait to filter before calling `ContainerStatusAggregator` + +❌ **Bug**: Not passing `$maxRestartCount` for crash loop detection +✅ **Fix**: Calculate max restart count from containers and pass to `aggregateFromStrings()`/`aggregateFromContainers()` + +❌ **Bug**: Not handling excluded containers with `:excluded` suffix +✅ **Fix**: Check for `:excluded` suffix in UI logic and button visibility diff --git a/.ai/core/deployment-architecture.md b/.ai/core/deployment-architecture.md new file mode 100644 index 0000000000..927bdc8de9 --- /dev/null +++ b/.ai/core/deployment-architecture.md @@ -0,0 +1,666 @@ +# Coolify Deployment Architecture + +## Deployment Philosophy + +Coolify orchestrates **Docker-based deployments** across multiple servers with automated configuration generation, zero-downtime deployments, and comprehensive monitoring. + +## Core Deployment Components + +### Deployment Models +- **[Application.php](mdc:app/Models/Application.php)** - Main application entity with deployment configurations +- **[ApplicationDeploymentQueue.php](mdc:app/Models/ApplicationDeploymentQueue.php)** - Deployment job orchestration +- **[Service.php](mdc:app/Models/Service.php)** - Multi-container service definitions +- **[Server.php](mdc:app/Models/Server.php)** - Target deployment infrastructure + +### Infrastructure Management +- **[PrivateKey.php](mdc:app/Models/PrivateKey.php)** - SSH key management for secure server access +- **[StandaloneDocker.php](mdc:app/Models/StandaloneDocker.php)** - Single container deployments +- **[SwarmDocker.php](mdc:app/Models/SwarmDocker.php)** - Docker Swarm orchestration + +## Deployment Workflow + +### 1. Source Code Integration +``` +Git Repository → Webhook → Coolify → Build & Deploy +``` + +#### Source Control Models +- **[GithubApp.php](mdc:app/Models/GithubApp.php)** - GitHub integration and webhooks +- **[GitlabApp.php](mdc:app/Models/GitlabApp.php)** - GitLab CI/CD integration + +#### Deployment Triggers +- **Git push** to configured branches +- **Manual deployment** via UI +- **Scheduled deployments** via cron +- **API-triggered** deployments + +### 2. Build Process +``` +Source Code → Docker Build → Image Registry → Deployment +``` + +#### Build Configurations +- **Dockerfile detection** and custom Dockerfile support +- **Buildpack integration** for framework detection +- **Multi-stage builds** for optimization +- **Cache layer** management for faster builds + +### 3. Deployment Orchestration +``` +Queue Job → Configuration Generation → Container Deployment → Health Checks +``` + +## Deployment Actions + +### Location: [app/Actions/](mdc:app/Actions) + +#### Application Deployment Actions +- **Application/** - Core application deployment logic +- **Docker/** - Docker container management +- **Service/** - Multi-container service orchestration +- **Proxy/** - Reverse proxy configuration + +#### Database Actions +- **Database/** - Database deployment and management +- Automated backup scheduling +- Connection management and health checks + +#### Server Management Actions +- **Server/** - Server provisioning and configuration +- SSH connection establishment +- Docker daemon management + +## Configuration Generation + +### Dynamic Configuration +- **[ConfigurationGenerator.php](mdc:app/Services/ConfigurationGenerator.php)** - Generates deployment configurations +- **[ConfigurationRepository.php](mdc:app/Services/ConfigurationRepository.php)** - Configuration management + +### Generated Configurations +#### Docker Compose Files +```yaml +# Generated docker-compose.yml structure +version: '3.8' +services: + app: + image: ${APP_IMAGE} + environment: + - ${ENV_VARIABLES} + labels: + - traefik.enable=true + - traefik.http.routers.app.rule=Host(`${FQDN}`) + volumes: + - ${VOLUME_MAPPINGS} + networks: + - coolify +``` + +#### Nginx Configurations +- **Reverse proxy** setup +- **SSL termination** with automatic certificates +- **Load balancing** for multiple instances +- **Custom headers** and routing rules + +## Container Orchestration + +### Docker Integration +- **[DockerImageParser.php](mdc:app/Services/DockerImageParser.php)** - Parse and validate Docker images +- **Container lifecycle** management +- **Resource allocation** and limits +- **Network isolation** and communication + +### Volume Management +- **[LocalFileVolume.php](mdc:app/Models/LocalFileVolume.php)** - Persistent file storage +- **[LocalPersistentVolume.php](mdc:app/Models/LocalPersistentVolume.php)** - Data persistence +- **Backup integration** for volume data + +### Network Configuration +- **Custom Docker networks** for isolation +- **Service discovery** between containers +- **Port mapping** and exposure +- **SSL/TLS termination** + +## Environment Management + +### Environment Isolation +- **[Environment.php](mdc:app/Models/Environment.php)** - Development, staging, production environments +- **[EnvironmentVariable.php](mdc:app/Models/EnvironmentVariable.php)** - Application-specific variables +- **[SharedEnvironmentVariable.php](mdc:app/Models/SharedEnvironmentVariable.php)** - Cross-application variables + +### Configuration Hierarchy +``` +Instance Settings → Server Settings → Project Settings → Application Settings +``` + +## Preview Environments + +### Git-Based Previews +- **[ApplicationPreview.php](mdc:app/Models/ApplicationPreview.php)** - Preview environment management +- **Automatic PR/MR previews** for feature branches +- **Isolated environments** for testing +- **Automatic cleanup** after merge/close + +### Preview Workflow +``` +Feature Branch → Auto-Deploy → Preview URL → Review → Cleanup +``` + +## SSL & Security + +### Certificate Management +- **[SslCertificate.php](mdc:app/Models/SslCertificate.php)** - SSL certificate automation +- **Let's Encrypt** integration for free certificates +- **Custom certificate** upload support +- **Automatic renewal** and monitoring + +### Security Patterns +- **Private Docker networks** for container isolation +- **SSH key-based** server authentication +- **Environment variable** encryption +- **Access control** via team permissions + +## Backup & Recovery + +### Database Backups +- **[ScheduledDatabaseBackup.php](mdc:app/Models/ScheduledDatabaseBackup.php)** - Automated database backups +- **[ScheduledDatabaseBackupExecution.php](mdc:app/Models/ScheduledDatabaseBackupExecution.php)** - Backup execution tracking +- **S3-compatible storage** for backup destinations + +### Application Backups +- **Volume snapshots** for persistent data +- **Configuration export** for disaster recovery +- **Cross-region replication** for high availability + +## Monitoring & Logging + +### Real-Time Monitoring +- **[ActivityMonitor.php](mdc:app/Livewire/ActivityMonitor.php)** - Live deployment monitoring +- **WebSocket-based** log streaming +- **Container health checks** and alerts +- **Resource usage** tracking + +### Deployment Logs +- **Build process** logging +- **Container startup** logs +- **Application runtime** logs +- **Error tracking** and alerting + +## Queue System + +### Background Jobs +Location: [app/Jobs/](mdc:app/Jobs) +- **Deployment jobs** for async processing +- **Server monitoring** jobs +- **Backup scheduling** jobs +- **Notification delivery** jobs + +### Queue Processing +- **Redis-backed** job queues +- **Laravel Horizon** for queue monitoring +- **Failed job** retry mechanisms +- **Queue worker** auto-scaling + +## Multi-Server Deployment + +### Server Types +- **Standalone servers** - Single Docker host +- **Docker Swarm** - Multi-node orchestration +- **Remote servers** - SSH-based deployment +- **Local development** - Docker Desktop integration + +### Load Balancing +- **Traefik integration** for automatic load balancing +- **Health check** based routing +- **Blue-green deployments** for zero downtime +- **Rolling updates** with configurable strategies + +## Deployment Strategies + +### Zero-Downtime Deployment +``` +Old Container → New Container Build → Health Check → Traffic Switch → Old Container Cleanup +``` + +### Blue-Green Deployment +- **Parallel environments** for safe deployments +- **Instant rollback** capability +- **Database migration** handling +- **Configuration synchronization** + +### Rolling Updates +- **Gradual instance** replacement +- **Configurable update** strategy +- **Automatic rollback** on failure +- **Health check** validation + +## API Integration + +### Deployment API +Routes: [routes/api.php](mdc:routes/api.php) +- **RESTful endpoints** for deployment management +- **Webhook receivers** for CI/CD integration +- **Status reporting** endpoints +- **Deployment triggering** via API + +### Authentication +- **Laravel Sanctum** API tokens +- **Team-based** access control +- **Rate limiting** for API calls +- **Audit logging** for API usage + +## Error Handling & Recovery + +### Deployment Failure Recovery +- **Automatic rollback** on deployment failure +- **Health check** failure handling +- **Container crash** recovery +- **Resource exhaustion** protection + +### Monitoring & Alerting +- **Failed deployment** notifications +- **Resource threshold** alerts +- **SSL certificate** expiry warnings +- **Backup failure** notifications + +## Performance Optimization + +### Build Optimization +- **Docker layer** caching +- **Multi-stage builds** for smaller images +- **Build artifact** reuse +- **Parallel build** processing + +### Docker Build Cache Preservation + +Coolify provides settings to preserve Docker build cache across deployments, addressing cache invalidation issues. + +#### The Problem + +By default, Coolify injects `ARG` statements into user Dockerfiles for build-time variables. This breaks Docker's cache mechanism because: +1. **ARG declarations invalidate cache** - Any change in ARG values after the `ARG` instruction invalidates all subsequent layers +2. **SOURCE_COMMIT changes every commit** - Causes full rebuilds even when code changes are minimal + +#### Application Settings + +Two toggles in **Advanced Settings** control this behavior: + +| Setting | Default | Description | +|---------|---------|-------------| +| `inject_build_args_to_dockerfile` | `true` | Controls whether Coolify adds `ARG` statements to Dockerfile | +| `include_source_commit_in_build` | `false` | Controls whether `SOURCE_COMMIT` is included in build context | + +**Database columns:** `application_settings.inject_build_args_to_dockerfile`, `application_settings.include_source_commit_in_build` + +#### Buildpack Coverage + +| Build Pack | ARG Injection | Method | +|------------|---------------|--------| +| **Dockerfile** | ✅ Yes | `add_build_env_variables_to_dockerfile()` | +| **Docker Compose** (with `build:`) | ✅ Yes | `modify_dockerfiles_for_compose()` | +| **PR Deployments** (Dockerfile only) | ✅ Yes | `add_build_env_variables_to_dockerfile()` | +| **Nixpacks** | ❌ No | Generates its own Dockerfile internally | +| **Static** | ❌ No | Uses internal Dockerfile | +| **Docker Image** | ❌ No | No build phase | + +#### How It Works + +**When `inject_build_args_to_dockerfile` is enabled (default):** +```dockerfile +# Coolify modifies your Dockerfile to add: +FROM node:20 +ARG MY_VAR=value +ARG COOLIFY_URL=... +ARG SOURCE_COMMIT=abc123 # (if include_source_commit_in_build is true) +# ... rest of your Dockerfile +``` + +**When `inject_build_args_to_dockerfile` is disabled:** +- Coolify does NOT modify the Dockerfile +- `--build-arg` flags are still passed (harmless without matching `ARG` in Dockerfile) +- User must manually add `ARG` statements for any build-time variables they need + +**When `include_source_commit_in_build` is disabled (default):** +- `SOURCE_COMMIT` is NOT included in build-time variables +- `SOURCE_COMMIT` is still available at **runtime** (in container environment) +- Docker cache preserved across different commits + +#### Recommended Configuration + +| Use Case | inject_build_args | include_source_commit | Cache Behavior | +|----------|-------------------|----------------------|----------------| +| Maximum cache preservation | `false` | `false` | Best cache retention | +| Need build-time vars, no commit | `true` | `false` | Cache breaks on var changes | +| Need commit at build-time | `true` | `true` | Cache breaks every commit | +| Manual ARG management | `false` | `true` | Cache preserved (no ARG in Dockerfile) | + +#### Implementation Details + +**Files:** +- `app/Jobs/ApplicationDeploymentJob.php`: + - `set_coolify_variables()` - Conditionally adds SOURCE_COMMIT to Docker build context based on `include_source_commit_in_build` setting + - `generate_coolify_env_variables(bool $forBuildTime)` - Distinguishes build-time vs. runtime variables; excludes cache-busting variables like SOURCE_COMMIT from build context unless explicitly enabled + - `generate_env_variables()` - Populates `$this->env_args` with build-time ARG values, respecting `include_source_commit_in_build` toggle + - `add_build_env_variables_to_dockerfile()` - Injects ARG statements into Dockerfiles after FROM instructions; skips injection if `inject_build_args_to_dockerfile` is disabled + - `modify_dockerfiles_for_compose()` - Applies ARG injection to Docker Compose service Dockerfiles; respects `inject_build_args_to_dockerfile` toggle +- `app/Models/ApplicationSetting.php` - Defines `inject_build_args_to_dockerfile` and `include_source_commit_in_build` boolean properties +- `app/Livewire/Project/Application/Advanced.php` - Livewire component providing UI bindings for cache preservation toggles +- `resources/views/livewire/project/application/advanced.blade.php` - Checkbox UI elements for user-facing toggles + +**Note:** Docker Compose services without a `build:` section (image-only) are automatically skipped. + +### Runtime Optimization +- **Container resource** limits +- **Auto-scaling** based on metrics +- **Connection pooling** for databases +- **CDN integration** for static assets + +## Compliance & Governance + +### Audit Trail +- **Deployment history** tracking +- **Configuration changes** logging +- **User action** auditing +- **Resource access** monitoring + +### Backup Compliance +- **Retention policies** for backups +- **Encryption at rest** for sensitive data +- **Cross-region** backup replication +- **Recovery testing** automation + +## Integration Patterns + +### CI/CD Integration +- **GitHub Actions** compatibility +- **GitLab CI** pipeline integration +- **Custom webhook** endpoints +- **Build status** reporting + +### External Services +- **S3-compatible** storage integration +- **External database** connections +- **Third-party monitoring** tools +- **Custom notification** channels + +--- + +## Coolify Docker Compose Extensions + +Coolify extends standard Docker Compose with custom fields (often called "magic fields") that provide Coolify-specific functionality. These extensions are processed during deployment and stripped before sending the final compose file to Docker, maintaining full compatibility with Docker's compose specification. + +### Overview + +**Why Custom Fields?** +- Enable Coolify-specific features without breaking Docker Compose compatibility +- Simplify configuration by embedding content directly in compose files +- Allow fine-grained control over health check monitoring +- Reduce external file dependencies + +**Processing Flow:** +1. User defines compose file with custom fields +2. Coolify parses and processes custom fields (creates files, stores settings) +3. Custom fields are stripped from final compose sent to Docker +4. Docker receives standard, valid compose file + +### Service-Level Extensions + +#### `exclude_from_hc` + +**Type:** Boolean +**Default:** `false` +**Purpose:** Exclude specific services from health check monitoring while still showing their status + +**Example Usage:** +```yaml +services: + watchtower: + image: containrrr/watchtower + exclude_from_hc: true # Don't monitor this service's health + + backup: + image: postgres:16 + exclude_from_hc: true # Backup containers don't need monitoring + restart: always +``` + +**Behavior:** +- Container status is still calculated from Docker state (running, exited, etc.) +- Status displays with `:excluded` suffix (e.g., `running:healthy:excluded`) +- UI shows "Monitoring Disabled" indicator +- Functionally equivalent to `restart: no` for health check purposes +- See [Container Status with All Excluded](application-architecture.md#container-status-when-all-containers-excluded) for detailed status handling + +**Use Cases:** +- Sidecar containers (watchtower, log collectors) +- Backup/maintenance containers +- One-time initialization containers +- Containers that intentionally restart frequently + +**Implementation:** +- Parsed: `bootstrap/helpers/parsers.php` +- Status logic: `app/Traits/CalculatesExcludedStatus.php` +- Validation: `tests/Unit/ExcludeFromHealthCheckTest.php` + +### Volume-Level Extensions + +Volume extensions only work with **long syntax** (array/object format), not short syntax (string format). + +#### `content` + +**Type:** String (supports multiline with `|` or `>`) +**Purpose:** Embed file content directly in compose file for automatic creation during deployment + +**Example Usage:** +```yaml +services: + app: + image: node:20 + volumes: + # Inline entrypoint script + - type: bind + source: ./entrypoint.sh + target: /app/entrypoint.sh + content: | + #!/bin/sh + set -e + echo "Starting application..." + npm run migrate + exec "$@" + + # Configuration file with environment variables + - type: bind + source: ./config.xml + target: /etc/app/config.xml + content: | + + + + ${DB_HOST} + ${DB_PORT} + + +``` + +**Behavior:** +- Content is written to the host at `source` path before container starts +- File is created with mode `644` (readable by all, writable by owner) +- Environment variables in content are interpolated at deployment time +- Content is stored in `LocalFileVolume` model (encrypted at rest) +- Original `docker_compose_raw` retains content for editing + +**Use Cases:** +- Entrypoint scripts +- Configuration files +- Environment-specific settings +- Small initialization scripts +- Templates that require dynamic content + +**Limitations:** +- Not suitable for large files (use git repo or external storage instead) +- Binary files not supported +- Changes require redeployment + +**Real-World Examples:** +- `templates/compose/traccar.yaml` - XML configuration file +- `templates/compose/supabase.yaml` - Multiple config files +- `templates/compose/chaskiq.yaml` - Entrypoint script + +**Implementation:** +- Parsed: `bootstrap/helpers/parsers.php` in `parseCompose()` function (handles `content` field extraction) +- Storage: `app/Models/LocalFileVolume.php` +- Validation: `tests/Unit/StripCoolifyCustomFieldsTest.php` + +#### `is_directory` / `isDirectory` + +**Type:** Boolean +**Default:** `true` (if neither `content` nor explicit flag provided) +**Purpose:** Indicate whether bind mount source should be created as directory or file + +**Example Usage:** +```yaml +services: + app: + volumes: + # Explicit file + - type: bind + source: ./config.json + target: /app/config.json + is_directory: false # Create as file + + # Explicit directory + - type: bind + source: ./logs + target: /var/log/app + is_directory: true # Create as directory + + # Auto-detected as file (has content) + - type: bind + source: ./script.sh + target: /entrypoint.sh + content: | + #!/bin/sh + echo "Hello" + # is_directory: false implied by content presence +``` + +**Behavior:** +- If `is_directory: true` → Creates directory with `mkdir -p` +- If `is_directory: false` → Creates empty file with `touch` +- If `content` provided → Implies `is_directory: false` +- If neither specified → Defaults to `true` (directory) + +**Naming Conventions:** +- `is_directory` (snake_case) - **Preferred**, consistent with PHP/Laravel conventions +- `isDirectory` (camelCase) - **Legacy support**, both work identically + +**Use Cases:** +- Disambiguating files vs directories when no content provided +- Ensuring correct bind mount type for Docker +- Pre-creating mount points before container starts + +**Implementation:** +- Parsed: `bootstrap/helpers/parsers.php` in `parseCompose()` function (handles `is_directory`/`isDirectory` field extraction) +- Storage: `app/Models/LocalFileVolume.php` (`is_directory` column) +- Validation: `tests/Unit/StripCoolifyCustomFieldsTest.php` + +### Custom Field Stripping + +**Function:** `stripCoolifyCustomFields()` in `bootstrap/helpers/docker.php` + +All custom fields are removed before the compose file is sent to Docker. This happens in two contexts: + +**1. Validation (User-Triggered)** +```php +// In validateComposeFile() - Edit Docker Compose modal +$yaml_compose = Yaml::parse($compose); +$yaml_compose = stripCoolifyCustomFields($yaml_compose); // Strip custom fields +// Send to docker compose config for validation +``` + +**2. Deployment (Automatic)** +```php +// In Service::parse() - During deployment +$docker_compose = parseCompose($docker_compose_raw); +// Custom fields are processed and then stripped +// Final compose sent to Docker has no custom fields +``` + +**What Gets Stripped:** +- Service-level: `exclude_from_hc` +- Volume-level: `content`, `isDirectory`, `is_directory` + +**What's Preserved:** +- All standard Docker Compose fields +- Environment variables +- Standard volume definitions (after custom fields removed) + +### Important Notes + +#### Long vs Short Volume Syntax + +**✅ Long Syntax (Works with Custom Fields):** +```yaml +volumes: + - type: bind + source: ./data + target: /app/data + content: "Hello" # ✅ Custom fields work here +``` + +**❌ Short Syntax (Custom Fields Ignored):** +```yaml +volumes: + - "./data:/app/data" # ❌ Cannot add custom fields to strings +``` + +#### Docker Compose Compatibility + +Custom fields are **Coolify-specific** and won't work with standalone `docker compose` CLI: + +```bash +# ❌ Won't work - Docker doesn't recognize custom fields +docker compose -f compose.yaml up + +# ✅ Works - Use Coolify's deployment (strips custom fields first) +# Deploy through Coolify UI or API +``` + +#### Editing Custom Fields + +When editing in "Edit Docker Compose" modal: +- Custom fields are preserved in the editor +- "Validate" button strips them temporarily for Docker validation +- "Save" button preserves them in `docker_compose_raw` +- They're processed again on next deployment + +### Template Examples + +See these templates for real-world usage: + +**Service Exclusions:** +- `templates/compose/budibase.yaml` - Excludes watchtower from monitoring +- `templates/compose/pgbackweb.yaml` - Excludes backup service +- `templates/compose/elasticsearch-with-kibana.yaml` - Excludes elasticsearch + +**Inline Content:** +- `templates/compose/traccar.yaml` - XML configuration (multiline) +- `templates/compose/supabase.yaml` - Multiple config files +- `templates/compose/searxng.yaml` - Settings file +- `templates/compose/invoice-ninja.yaml` - Nginx config + +**Directory Flags:** +- `templates/compose/paperless.yaml` - Explicit directory creation + +### Testing + +**Unit Tests:** +- `tests/Unit/StripCoolifyCustomFieldsTest.php` - Custom field stripping logic +- `tests/Unit/ExcludeFromHealthCheckTest.php` - Health check exclusion behavior +- `tests/Unit/ContainerStatusAggregatorTest.php` - Status aggregation with exclusions + +**Test Coverage:** +- ✅ All custom fields (exclude_from_hc, content, isDirectory, is_directory) +- ✅ Multiline content (YAML `|` syntax) +- ✅ Short vs long volume syntax +- ✅ Field stripping without data loss +- ✅ Standard Docker Compose field preservation diff --git a/.ai/core/project-overview.md b/.ai/core/project-overview.md new file mode 100644 index 0000000000..59fda48686 --- /dev/null +++ b/.ai/core/project-overview.md @@ -0,0 +1,156 @@ +# Coolify Project Overview + +## What is Coolify? + +Coolify is an **open-source & self-hostable alternative to Heroku / Netlify / Vercel**. It's a comprehensive deployment platform that helps you manage servers, applications, and databases on your own hardware with just an SSH connection. + +## Core Mission + +**"Imagine having the ease of a cloud but with your own servers. That is Coolify."** + +- **No vendor lock-in** - All configurations saved to your servers +- **Self-hosted** - Complete control over your infrastructure +- **SSH-only requirement** - Works with VPS, Bare Metal, Raspberry PIs, anything +- **Docker-first** - Container-based deployment architecture + +## Key Features + +### 🚀 **Application Deployment** +- Git-based deployments (GitHub, GitLab, Bitbucket, Gitea) +- Docker & Docker Compose support +- Preview deployments for pull requests +- Zero-downtime deployments +- Build cache optimization + +### 🖥️ **Server Management** +- Multi-server orchestration +- Real-time monitoring and logs +- SSH key management +- Proxy configuration (Traefik/Caddy) +- Resource usage tracking + +### 🗄️ **Database Management** +- PostgreSQL, MySQL, MariaDB, MongoDB +- Redis, KeyDB, Dragonfly, ClickHouse +- Automated backups with S3 integration +- Database clustering support + +### 🔧 **Infrastructure as Code** +- Docker Compose generation +- Environment variable management +- SSL certificate automation +- Custom domain configuration + +### 👥 **Team Collaboration** +- Multi-tenant team organization +- Role-based access control +- Project and environment isolation +- Team-wide resource sharing + +### 📊 **Monitoring & Observability** +- Real-time application logs +- Server resource monitoring +- Deployment status tracking +- Webhook integrations +- Notification systems (Email, Discord, Slack, Telegram) + +## Target Users + +### **DevOps Engineers** +- Infrastructure automation +- Multi-environment management +- CI/CD pipeline integration + +### **Developers** +- Easy application deployment +- Development environment provisioning +- Preview deployments for testing + +### **Small to Medium Businesses** +- Cost-effective Heroku alternative +- Self-hosted control and privacy +- Scalable infrastructure management + +### **Agencies & Consultants** +- Client project isolation +- Multi-tenant management +- White-label deployment solutions + +## Business Model + +### **Open Source (Free)** +- Complete feature set +- Self-hosted deployment +- Community support +- No feature restrictions + +### **Cloud Version (Paid)** +- Managed Coolify instance +- High availability +- Premium support +- Email notifications included +- Same price as self-hosted server (~$4-5/month) + +## Architecture Philosophy + +### **Server-Side First** +- Laravel backend with Livewire frontend +- Minimal JavaScript footprint +- Real-time updates via WebSockets +- Progressive enhancement approach + +### **Docker-Native** +- Container-first deployment strategy +- Docker Compose orchestration +- Image building and registry integration +- Volume and network management + +### **Security-Focused** +- SSH-based server communication +- Environment variable encryption +- Team-based access isolation +- Audit logging and activity tracking + +## Project Structure + +``` +coolify/ +├── app/ # Laravel application core +│ ├── Models/ # Domain models (Application, Server, Service) +│ ├── Livewire/ # Frontend components +│ ├── Actions/ # Business logic actions +│ └── Jobs/ # Background job processing +├── resources/ # Frontend assets and views +├── database/ # Migrations and seeders +├── docker/ # Docker configuration +├── scripts/ # Installation and utility scripts +└── tests/ # Test suites (Pest, Dusk) +``` + +## Key Differentiators + +### **vs. Heroku** +- ✅ Self-hosted (no vendor lock-in) +- ✅ Multi-server support +- ✅ No usage-based pricing +- ✅ Full infrastructure control + +### **vs. Vercel/Netlify** +- ✅ Backend application support +- ✅ Database management included +- ✅ Multi-environment workflows +- ✅ Custom server infrastructure + +### **vs. Docker Swarm/Kubernetes** +- ✅ User-friendly web interface +- ✅ Git-based deployment workflows +- ✅ Integrated monitoring and logging +- ✅ No complex YAML configuration + +## Development Principles + +- **Simplicity over complexity** +- **Convention over configuration** +- **Security by default** +- **Developer experience focused** +- **Community-driven development** diff --git a/.ai/core/technology-stack.md b/.ai/core/technology-stack.md new file mode 100644 index 0000000000..b12534db70 --- /dev/null +++ b/.ai/core/technology-stack.md @@ -0,0 +1,245 @@ +# Coolify Technology Stack + +Complete technology stack, dependencies, and infrastructure components. + +## Backend Framework + +### **Laravel 12.4.1** (PHP Framework) +- **Purpose**: Core application framework +- **Key Features**: + - Eloquent ORM for database interactions + - Artisan CLI for development tasks + - Queue system for background jobs + - Event-driven architecture + +### **PHP 8.4.7** +- **Requirement**: `^8.4` in composer.json +- **Features Used**: + - Typed properties and return types + - Attributes for validation and configuration + - Match expressions + - Constructor property promotion + +## Frontend Stack + +### **Livewire 3.5.20** (Primary Frontend Framework) +- **Purpose**: Server-side rendering with reactive components +- **Location**: `app/Livewire/` +- **Key Components**: + - Dashboard - Main interface + - ActivityMonitor - Real-time monitoring + - MonacoEditor - Code editor + +### **Alpine.js** (Client-Side Interactivity) +- **Purpose**: Lightweight JavaScript for DOM manipulation +- **Integration**: Works seamlessly with Livewire components +- **Usage**: Declarative directives in Blade templates + +### **Tailwind CSS 4.1.4** (Styling Framework) +- **Configuration**: `postcss.config.cjs` +- **Extensions**: + - `@tailwindcss/forms` - Form styling + - `@tailwindcss/typography` - Content typography + - `tailwind-scrollbar` - Custom scrollbars + +### **Vue.js 3.5.13** (Component Framework) +- **Purpose**: Enhanced interactive components +- **Integration**: Used alongside Livewire for complex UI +- **Build Tool**: Vite with Vue plugin + +## Database & Caching + +### **PostgreSQL 15** (Primary Database) +- **Purpose**: Main application data storage +- **Features**: JSONB support, advanced indexing +- **Models**: `app/Models/` + +### **Redis 7** (Caching & Real-time) +- **Purpose**: + - Session storage + - Queue backend + - Real-time data caching + - WebSocket session management + +### **Supported Databases** (For User Applications) +- **PostgreSQL**: StandalonePostgresql +- **MySQL**: StandaloneMysql +- **MariaDB**: StandaloneMariadb +- **MongoDB**: StandaloneMongodb +- **Redis**: StandaloneRedis +- **KeyDB**: StandaloneKeydb +- **Dragonfly**: StandaloneDragonfly +- **ClickHouse**: StandaloneClickhouse + +## Authentication & Security + +### **Laravel Sanctum 4.0.8** +- **Purpose**: API token authentication +- **Usage**: Secure API access for external integrations + +### **Laravel Fortify 1.25.4** +- **Purpose**: Authentication scaffolding +- **Features**: Login, registration, password reset + +### **Laravel Socialite 5.18.0** +- **Purpose**: OAuth provider integration +- **Providers**: + - GitHub, GitLab, Google + - Microsoft Azure, Authentik, Discord, Clerk + - Custom OAuth implementations + +## Background Processing + +### **Laravel Horizon 5.30.3** +- **Purpose**: Queue monitoring and management +- **Features**: Real-time queue metrics, failed job handling + +### **Queue System** +- **Backend**: Redis-based queues +- **Jobs**: `app/Jobs/` +- **Processing**: Background deployment and monitoring tasks + +## Development Tools + +### **Build Tools** +- **Vite 6.2.6**: Modern build tool and dev server +- **Laravel Vite Plugin**: Laravel integration +- **PostCSS**: CSS processing pipeline + +### **Code Quality** +- **Laravel Pint**: PHP code style fixer +- **Rector**: PHP automated refactoring +- **PHPStan**: Static analysis tool + +### **Testing Framework** +- **Pest 3.8.0**: Modern PHP testing framework +- **Laravel Dusk**: Browser automation testing +- **PHPUnit**: Unit testing foundation + +## External Integrations + +### **Git Providers** +- **GitHub**: Repository integration and webhooks +- **GitLab**: Self-hosted and cloud GitLab support +- **Bitbucket**: Atlassian integration +- **Gitea**: Self-hosted Git service + +### **Cloud Storage** +- **AWS S3**: league/flysystem-aws-s3-v3 +- **SFTP**: league/flysystem-sftp-v3 +- **Local Storage**: File system integration + +### **Notification Services** +- **Email**: resend/resend-laravel +- **Discord**: Custom webhook integration +- **Slack**: Webhook notifications +- **Telegram**: Bot API integration +- **Pushover**: Push notifications + +### **Monitoring & Logging** +- **Sentry**: sentry/sentry-laravel - Error tracking +- **Laravel Ray**: spatie/laravel-ray - Debug tool +- **Activity Log**: spatie/laravel-activitylog + +## DevOps & Infrastructure + +### **Docker & Containerization** +- **Docker**: Container runtime +- **Docker Compose**: Multi-container orchestration +- **Docker Swarm**: Container clustering (optional) + +### **Web Servers & Proxies** +- **Nginx**: Primary web server +- **Traefik**: Reverse proxy and load balancer +- **Caddy**: Alternative reverse proxy + +### **Process Management** +- **S6 Overlay**: Process supervisor +- **Supervisor**: Alternative process manager + +### **SSL/TLS** +- **Let's Encrypt**: Automatic SSL certificates +- **Custom Certificates**: Manual SSL management + +## Terminal & Code Editing + +### **XTerm.js 5.5.0** +- **Purpose**: Web-based terminal emulator +- **Features**: SSH session management, real-time command execution +- **Addons**: Fit addon for responsive terminals + +### **Monaco Editor** +- **Purpose**: Code editor component +- **Features**: Syntax highlighting, auto-completion +- **Integration**: Environment variable editing, configuration files + +## API & Documentation + +### **OpenAPI/Swagger** +- **Documentation**: openapi.json (373KB) +- **Generator**: zircote/swagger-php +- **API Routes**: `routes/api.php` + +### **WebSocket Communication** +- **Laravel Echo**: Real-time event broadcasting +- **Pusher**: WebSocket service integration +- **Soketi**: Self-hosted WebSocket server + +## Package Management + +### **PHP Dependencies** (composer.json) +```json +{ + "require": { + "php": "^8.4", + "laravel/framework": "12.4.1", + "livewire/livewire": "^3.5.20", + "spatie/laravel-data": "^4.13.1", + "lorisleiva/laravel-actions": "^2.8.6" + } +} +``` + +### **JavaScript Dependencies** (package.json) +```json +{ + "devDependencies": { + "vite": "^6.2.6", + "tailwindcss": "^4.1.4", + "@vitejs/plugin-vue": "5.2.3" + }, + "dependencies": { + "@xterm/xterm": "^5.5.0", + "ioredis": "5.6.0" + } +} +``` + +## Configuration Files + +### **Build Configuration** +- **vite.config.js**: Frontend build setup +- **postcss.config.cjs**: CSS processing +- **rector.php**: PHP refactoring rules +- **pint.json**: Code style configuration + +### **Testing Configuration** +- **phpunit.xml**: Unit test configuration +- **phpunit.dusk.xml**: Browser test configuration +- **tests/Pest.php**: Pest testing setup + +## Version Requirements + +### **Minimum Requirements** +- **PHP**: 8.4+ +- **Node.js**: 18+ (for build tools) +- **PostgreSQL**: 15+ +- **Redis**: 7+ +- **Docker**: 20.10+ +- **Docker Compose**: 2.0+ + +### **Recommended Versions** +- **Ubuntu**: 22.04 LTS or 24.04 LTS +- **Memory**: 2GB+ RAM +- **Storage**: 20GB+ available space +- **Network**: Stable internet connection for deployments diff --git a/.ai/development/development-workflow.md b/.ai/development/development-workflow.md new file mode 100644 index 0000000000..4ee3766965 --- /dev/null +++ b/.ai/development/development-workflow.md @@ -0,0 +1,648 @@ +# Coolify Development Workflow + +## Development Environment Setup + +### Prerequisites +- **PHP 8.4+** - Latest PHP version for modern features +- **Node.js 18+** - For frontend asset compilation +- **Docker & Docker Compose** - Container orchestration +- **PostgreSQL 15** - Primary database +- **Redis 7** - Caching and queues + +### Local Development Setup + +#### Using Docker (Recommended) +```bash +# Clone the repository +git clone https://github.com/coollabsio/coolify.git +cd coolify + +# Copy environment configuration +cp .env.example .env + +# Start development environment +docker-compose -f docker-compose.dev.yml up -d + +# Install PHP dependencies +docker-compose exec app composer install + +# Install Node.js dependencies +docker-compose exec app npm install + +# Generate application key +docker-compose exec app php artisan key:generate + +# Run database migrations +docker-compose exec app php artisan migrate + +# Seed development data +docker-compose exec app php artisan db:seed +``` + +#### Native Development +```bash +# Install PHP dependencies +composer install + +# Install Node.js dependencies +npm install + +# Setup environment +cp .env.example .env +php artisan key:generate + +# Setup database +createdb coolify_dev +php artisan migrate +php artisan db:seed + +# Start development servers +php artisan serve & +npm run dev & +php artisan queue:work & +``` + +## Development Tools & Configuration + +### Code Quality Tools +- **[Laravel Pint](mdc:pint.json)** - PHP code style fixer +- **[Rector](mdc:rector.php)** - PHP automated refactoring (989B, 35 lines) +- **PHPStan** - Static analysis for type safety +- **ESLint** - JavaScript code quality + +### Development Configuration Files +- **[docker-compose.dev.yml](mdc:docker-compose.dev.yml)** - Development Docker setup (3.4KB, 126 lines) +- **[vite.config.js](mdc:vite.config.js)** - Frontend build configuration (1.0KB, 42 lines) +- **[.editorconfig](mdc:.editorconfig)** - Code formatting standards (258B, 19 lines) + +### Git Configuration +- **[.gitignore](mdc:.gitignore)** - Version control exclusions (522B, 40 lines) +- **[.gitattributes](mdc:.gitattributes)** - Git file handling (185B, 11 lines) + +## Development Workflow Process + +### 1. Feature Development +```bash +# Create feature branch +git checkout -b feature/new-deployment-strategy + +# Make changes following coding standards +# Run code quality checks +./vendor/bin/pint +./vendor/bin/rector process --dry-run +./vendor/bin/phpstan analyse + +# Run tests +./vendor/bin/pest +./vendor/bin/pest --coverage + +# Commit changes +git add . +git commit -m "feat: implement blue-green deployment strategy" +``` + +### 2. Code Review Process +```bash +# Push feature branch +git push origin feature/new-deployment-strategy + +# Create pull request with: +# - Clear description of changes +# - Screenshots for UI changes +# - Test coverage information +# - Breaking change documentation +``` + +### 3. Testing Requirements +- **Unit tests** for new models and services +- **Feature tests** for API endpoints +- **Browser tests** for UI changes +- **Integration tests** for deployment workflows + +## Coding Standards & Conventions + +### PHP Coding Standards +```php +// Follow PSR-12 coding standards +class ApplicationDeploymentService +{ + public function __construct( + private readonly DockerService $dockerService, + private readonly ConfigurationGenerator $configGenerator + ) {} + + public function deploy(Application $application): ApplicationDeploymentQueue + { + return DB::transaction(function () use ($application) { + $deployment = $application->deployments()->create([ + 'status' => 'queued', + 'commit_sha' => $application->getLatestCommitSha(), + ]); + + DeployApplicationJob::dispatch($deployment); + + return $deployment; + }); + } +} +``` + +### Laravel Best Practices +```php +// Use Laravel conventions +class Application extends Model +{ + // Mass assignment protection + protected $fillable = [ + 'name', 'git_repository', 'git_branch', 'fqdn' + ]; + + // Type casting + protected $casts = [ + 'environment_variables' => 'array', + 'build_pack' => BuildPack::class, + 'created_at' => 'datetime', + ]; + + // Relationships + public function server(): BelongsTo + { + return $this->belongsTo(Server::class); + } + + public function deployments(): HasMany + { + return $this->hasMany(ApplicationDeploymentQueue::class); + } +} +``` + +### Frontend Standards +```javascript +// Alpine.js component structure +document.addEventListener('alpine:init', () => { + Alpine.data('deploymentMonitor', () => ({ + status: 'idle', + logs: [], + + init() { + this.connectWebSocket(); + }, + + connectWebSocket() { + Echo.private(`application.${this.applicationId}`) + .listen('DeploymentStarted', (e) => { + this.status = 'deploying'; + }) + .listen('DeploymentCompleted', (e) => { + this.status = 'completed'; + }); + } + })); +}); +``` + +### CSS/Tailwind Standards +```html + +
+
+

+ Application Status +

+
+ +
+
+
+``` + +## Database Development + +### Migration Best Practices +```php +// Create descriptive migration files +class CreateApplicationDeploymentQueuesTable extends Migration +{ + public function up(): void + { + Schema::create('application_deployment_queues', function (Blueprint $table) { + $table->id(); + $table->foreignId('application_id')->constrained()->cascadeOnDelete(); + $table->string('status')->default('queued'); + $table->string('commit_sha')->nullable(); + $table->text('build_logs')->nullable(); + $table->text('deployment_logs')->nullable(); + $table->timestamp('started_at')->nullable(); + $table->timestamp('finished_at')->nullable(); + $table->timestamps(); + + $table->index(['application_id', 'status']); + $table->index('created_at'); + }); + } + + public function down(): void + { + Schema::dropIfExists('application_deployment_queues'); + } +} +``` + +### Model Factory Development +```php +// Create comprehensive factories for testing +class ApplicationFactory extends Factory +{ + protected $model = Application::class; + + public function definition(): array + { + return [ + 'name' => $this->faker->words(2, true), + 'fqdn' => $this->faker->domainName, + 'git_repository' => 'https://github.com/' . $this->faker->userName . '/' . $this->faker->word . '.git', + 'git_branch' => 'main', + 'build_pack' => BuildPack::NIXPACKS, + 'server_id' => Server::factory(), + 'environment_id' => Environment::factory(), + ]; + } + + public function withCustomDomain(): static + { + return $this->state(fn (array $attributes) => [ + 'fqdn' => $this->faker->domainName, + ]); + } +} +``` + +## API Development + +### Controller Standards +```php +class ApplicationController extends Controller +{ + public function __construct() + { + $this->middleware('auth:sanctum'); + $this->middleware('team.access'); + } + + public function index(Request $request): AnonymousResourceCollection + { + $applications = $request->user() + ->currentTeam + ->applications() + ->with(['server', 'environment', 'latestDeployment']) + ->paginate(); + + return ApplicationResource::collection($applications); + } + + public function store(StoreApplicationRequest $request): ApplicationResource + { + $application = $request->user() + ->currentTeam + ->applications() + ->create($request->validated()); + + return new ApplicationResource($application); + } + + public function deploy(Application $application): JsonResponse + { + $this->authorize('deploy', $application); + + $deployment = app(ApplicationDeploymentService::class) + ->deploy($application); + + return response()->json([ + 'message' => 'Deployment started successfully', + 'deployment_id' => $deployment->id, + ]); + } +} +``` + +### API Resource Development +```php +class ApplicationResource extends JsonResource +{ + public function toArray($request): array + { + return [ + 'id' => $this->id, + 'name' => $this->name, + 'fqdn' => $this->fqdn, + 'status' => $this->status, + 'git_repository' => $this->git_repository, + 'git_branch' => $this->git_branch, + 'build_pack' => $this->build_pack, + 'created_at' => $this->created_at, + 'updated_at' => $this->updated_at, + + // Conditional relationships + 'server' => new ServerResource($this->whenLoaded('server')), + 'environment' => new EnvironmentResource($this->whenLoaded('environment')), + 'latest_deployment' => new DeploymentResource($this->whenLoaded('latestDeployment')), + + // Computed attributes + 'deployment_url' => $this->getDeploymentUrl(), + 'can_deploy' => $this->canDeploy(), + ]; + } +} +``` + +## Livewire Component Development + +### Component Structure +```php +class ApplicationShow extends Component +{ + public Application $application; + public bool $showLogs = false; + + protected $listeners = [ + 'deployment.started' => 'refreshDeploymentStatus', + 'deployment.completed' => 'refreshDeploymentStatus', + ]; + + public function mount(Application $application): void + { + $this->authorize('view', $application); + $this->application = $application; + } + + public function deploy(): void + { + $this->authorize('deploy', $this->application); + + try { + app(ApplicationDeploymentService::class)->deploy($this->application); + + $this->dispatch('deployment.started', [ + 'application_id' => $this->application->id + ]); + + session()->flash('success', 'Deployment started successfully'); + } catch (Exception $e) { + session()->flash('error', 'Failed to start deployment: ' . $e->getMessage()); + } + } + + public function refreshDeploymentStatus(): void + { + $this->application->refresh(); + } + + public function render(): View + { + return view('livewire.application.show', [ + 'deployments' => $this->application + ->deployments() + ->latest() + ->limit(10) + ->get() + ]); + } +} +``` + +## Queue Job Development + +### Job Structure +```php +class DeployApplicationJob implements ShouldQueue +{ + use Dispatchable, InteractsWithQueue, Queueable, SerializesModels; + + public int $tries = 3; + public int $maxExceptions = 1; + + public function __construct( + public ApplicationDeploymentQueue $deployment + ) {} + + public function handle( + DockerService $dockerService, + ConfigurationGenerator $configGenerator + ): void { + $this->deployment->update(['status' => 'running', 'started_at' => now()]); + + try { + // Generate configuration + $config = $configGenerator->generateDockerCompose($this->deployment->application); + + // Build and deploy + $imageTag = $dockerService->buildImage($this->deployment->application); + $dockerService->deployContainer($this->deployment->application, $imageTag); + + $this->deployment->update([ + 'status' => 'success', + 'finished_at' => now() + ]); + + // Broadcast success + broadcast(new DeploymentCompleted($this->deployment)); + + } catch (Exception $e) { + $this->deployment->update([ + 'status' => 'failed', + 'error_message' => $e->getMessage(), + 'finished_at' => now() + ]); + + broadcast(new DeploymentFailed($this->deployment)); + + throw $e; + } + } + + public function backoff(): array + { + return [1, 5, 10]; + } + + public function failed(Throwable $exception): void + { + $this->deployment->update([ + 'status' => 'failed', + 'error_message' => $exception->getMessage(), + 'finished_at' => now() + ]); + } +} +``` + +## Testing Development + +### Test Structure +```php +// Feature test example +test('user can deploy application via API', function () { + $user = User::factory()->create(); + $application = Application::factory()->create([ + 'team_id' => $user->currentTeam->id + ]); + + // Mock external services + $this->mock(DockerService::class, function ($mock) { + $mock->shouldReceive('buildImage')->andReturn('app:latest'); + $mock->shouldReceive('deployContainer')->andReturn(true); + }); + + $response = $this->actingAs($user) + ->postJson("/api/v1/applications/{$application->id}/deploy"); + + $response->assertStatus(200) + ->assertJson([ + 'message' => 'Deployment started successfully' + ]); + + expect($application->deployments()->count())->toBe(1); + expect($application->deployments()->first()->status)->toBe('queued'); +}); +``` + +## Documentation Standards + +### Code Documentation +```php +/** + * Deploy an application to the specified server. + * + * This method creates a new deployment queue entry and dispatches + * a background job to handle the actual deployment process. + * + * @param Application $application The application to deploy + * @param array $options Additional deployment options + * @return ApplicationDeploymentQueue The created deployment queue entry + * + * @throws DeploymentException When deployment cannot be started + * @throws ServerConnectionException When server is unreachable + */ +public function deploy(Application $application, array $options = []): ApplicationDeploymentQueue +{ + // Implementation +} +``` + +### API Documentation +```php +/** + * @OA\Post( + * path="/api/v1/applications/{application}/deploy", + * summary="Deploy an application", + * description="Triggers a new deployment for the specified application", + * operationId="deployApplication", + * tags={"Applications"}, + * security={{"bearerAuth":{}}}, + * @OA\Parameter( + * name="application", + * in="path", + * required=true, + * @OA\Schema(type="integer"), + * description="Application ID" + * ), + * @OA\Response( + * response=200, + * description="Deployment started successfully", + * @OA\JsonContent( + * @OA\Property(property="message", type="string"), + * @OA\Property(property="deployment_id", type="integer") + * ) + * ) + * ) + */ +``` + +## Performance Optimization + +### Database Optimization +```php +// Use eager loading to prevent N+1 queries +$applications = Application::with([ + 'server:id,name,ip', + 'environment:id,name', + 'latestDeployment:id,application_id,status,created_at' +])->get(); + +// Use database transactions for consistency +DB::transaction(function () use ($application) { + $deployment = $application->deployments()->create(['status' => 'queued']); + $application->update(['last_deployment_at' => now()]); + DeployApplicationJob::dispatch($deployment); +}); +``` + +### Caching Strategies +```php +// Cache expensive operations +public function getServerMetrics(Server $server): array +{ + return Cache::remember( + "server.{$server->id}.metrics", + now()->addMinutes(5), + fn () => $this->fetchServerMetrics($server) + ); +} +``` + +## Deployment & Release Process + +### Version Management +- **[versions.json](mdc:versions.json)** - Version tracking (355B, 19 lines) +- **[CHANGELOG.md](mdc:CHANGELOG.md)** - Release notes (187KB, 7411 lines) +- **[cliff.toml](mdc:cliff.toml)** - Changelog generation (3.2KB, 85 lines) + +### Release Workflow +```bash +# Create release branch +git checkout -b release/v4.1.0 + +# Update version numbers +# Update CHANGELOG.md +# Run full test suite +./vendor/bin/pest +npm run test + +# Create release commit +git commit -m "chore: release v4.1.0" + +# Create and push tag +git tag v4.1.0 +git push origin v4.1.0 + +# Merge to main +git checkout main +git merge release/v4.1.0 +``` + +## Contributing Guidelines + +### Pull Request Process +1. **Fork** the repository +2. **Create** feature branch from `main` +3. **Implement** changes with tests +4. **Run** code quality checks +5. **Submit** pull request with clear description +6. **Address** review feedback +7. **Merge** after approval + +### Code Review Checklist +- [ ] Code follows project standards +- [ ] Tests cover new functionality +- [ ] Documentation is updated +- [ ] No breaking changes without migration +- [ ] Performance impact considered +- [ ] Security implications reviewed + +### Issue Reporting +- Use issue templates +- Provide reproduction steps +- Include environment details +- Add relevant logs/screenshots +- Label appropriately diff --git a/.ai/development/laravel-boost.md b/.ai/development/laravel-boost.md new file mode 100644 index 0000000000..7f5922d944 --- /dev/null +++ b/.ai/development/laravel-boost.md @@ -0,0 +1,402 @@ + +=== foundation rules === + +# Laravel Boost Guidelines + +The Laravel Boost guidelines are specifically curated by Laravel maintainers for this application. These guidelines should be followed closely to enhance the user's satisfaction building Laravel applications. + +## Foundational Context +This application is a Laravel application and its main Laravel ecosystems package & versions are below. You are an expert with them all. Ensure you abide by these specific packages & versions. + +- php - 8.4.7 +- laravel/fortify (FORTIFY) - v1 +- laravel/framework (LARAVEL) - v12 +- laravel/horizon (HORIZON) - v5 +- laravel/prompts (PROMPTS) - v0 +- laravel/sanctum (SANCTUM) - v4 +- laravel/socialite (SOCIALITE) - v5 +- livewire/livewire (LIVEWIRE) - v3 +- laravel/dusk (DUSK) - v8 +- laravel/pint (PINT) - v1 +- laravel/telescope (TELESCOPE) - v5 +- pestphp/pest (PEST) - v3 +- phpunit/phpunit (PHPUNIT) - v11 +- rector/rector (RECTOR) - v2 +- laravel-echo (ECHO) - v2 +- tailwindcss (TAILWINDCSS) - v4 +- vue (VUE) - v3 + + +## Conventions +- You must follow all existing code conventions used in this application. When creating or editing a file, check sibling files for the correct structure, approach, naming. +- Use descriptive names for variables and methods. For example, `isRegisteredForDiscounts`, not `discount()`. +- Check for existing components to reuse before writing a new one. + +## Verification Scripts +- Do not create verification scripts or tinker when tests cover that functionality and prove it works. Unit and feature tests are more important. + +## Application Structure & Architecture +- Stick to existing directory structure - don't create new base folders without approval. +- Do not change the application's dependencies without approval. + +## Frontend Bundling +- If the user doesn't see a frontend change reflected in the UI, it could mean they need to run `npm run build`, `npm run dev`, or `composer run dev`. Ask them. + +## Replies +- Be concise in your explanations - focus on what's important rather than explaining obvious details. + +## Documentation Files +- You must only create documentation files if explicitly requested by the user. + + +=== boost rules === + +## Laravel Boost +- Laravel Boost is an MCP server that comes with powerful tools designed specifically for this application. Use them. + +## Artisan +- Use the `list-artisan-commands` tool when you need to call an Artisan command to double check the available parameters. + +## URLs +- Whenever you share a project URL with the user you should use the `get-absolute-url` tool to ensure you're using the correct scheme, domain / IP, and port. + +## Tinker / Debugging +- You should use the `tinker` tool when you need to execute PHP to debug code or query Eloquent models directly. +- Use the `database-query` tool when you only need to read from the database. + +## Reading Browser Logs With the `browser-logs` Tool +- You can read browser logs, errors, and exceptions using the `browser-logs` tool from Boost. +- Only recent browser logs will be useful - ignore old logs. + +## Searching Documentation (Critically Important) +- Boost comes with a powerful `search-docs` tool you should use before any other approaches. This tool automatically passes a list of installed packages and their versions to the remote Boost API, so it returns only version-specific documentation specific for the user's circumstance. You should pass an array of packages to filter on if you know you need docs for particular packages. +- The 'search-docs' tool is perfect for all Laravel related packages, including Laravel, Inertia, Livewire, Filament, Tailwind, Pest, Nova, Nightwatch, etc. +- You must use this tool to search for Laravel-ecosystem documentation before falling back to other approaches. +- Search the documentation before making code changes to ensure we are taking the correct approach. +- Use multiple, broad, simple, topic based queries to start. For example: `['rate limiting', 'routing rate limiting', 'routing']`. +- Do not add package names to queries - package information is already shared. For example, use `test resource table`, not `filament 4 test resource table`. + +### Available Search Syntax +- You can and should pass multiple queries at once. The most relevant results will be returned first. + +1. Simple Word Searches with auto-stemming - query=authentication - finds 'authenticate' and 'auth' +2. Multiple Words (AND Logic) - query=rate limit - finds knowledge containing both "rate" AND "limit" +3. Quoted Phrases (Exact Position) - query="infinite scroll" - Words must be adjacent and in that order +4. Mixed Queries - query=middleware "rate limit" - "middleware" AND exact phrase "rate limit" +5. Multiple Queries - queries=["authentication", "middleware"] - ANY of these terms + + +=== php rules === + +## PHP + +- Always use curly braces for control structures, even if it has one line. + +### Constructors +- Use PHP 8 constructor property promotion in `__construct()`. + - public function __construct(public GitHub $github) { } +- Do not allow empty `__construct()` methods with zero parameters. + +### Type Declarations +- Always use explicit return type declarations for methods and functions. +- Use appropriate PHP type hints for method parameters. + + +protected function isAccessible(User $user, ?string $path = null): bool +{ + ... +} + + +## Comments +- Prefer PHPDoc blocks over comments. Never use comments within the code itself unless there is something _very_ complex going on. + +## PHPDoc Blocks +- Add useful array shape type definitions for arrays when appropriate. + +## Enums +- Typically, keys in an Enum should be TitleCase. For example: `FavoritePerson`, `BestLake`, `Monthly`. + + +=== laravel/core rules === + +## Do Things the Laravel Way + +- Use `php artisan make:` commands to create new files (i.e. migrations, controllers, models, etc.). You can list available Artisan commands using the `list-artisan-commands` tool. +- If you're creating a generic PHP class, use `artisan make:class`. +- Pass `--no-interaction` to all Artisan commands to ensure they work without user input. You should also pass the correct `--options` to ensure correct behavior. + +### Database +- Always use proper Eloquent relationship methods with return type hints. Prefer relationship methods over raw queries or manual joins. +- Use Eloquent models and relationships before suggesting raw database queries +- Avoid `DB::`; prefer `Model::query()`. Generate code that leverages Laravel's ORM capabilities rather than bypassing them. +- Generate code that prevents N+1 query problems by using eager loading. +- Use Laravel's query builder for very complex database operations. + +### Model Creation +- When creating new models, create useful factories and seeders for them too. Ask the user if they need any other things, using `list-artisan-commands` to check the available options to `php artisan make:model`. + +### APIs & Eloquent Resources +- For APIs, default to using Eloquent API Resources and API versioning unless existing API routes do not, then you should follow existing application convention. + +### Controllers & Validation +- Always create Form Request classes for validation rather than inline validation in controllers. Include both validation rules and custom error messages. +- Check sibling Form Requests to see if the application uses array or string based validation rules. + +### Queues +- Use queued jobs for time-consuming operations with the `ShouldQueue` interface. + +### Authentication & Authorization +- Use Laravel's built-in authentication and authorization features (gates, policies, Sanctum, etc.). + +### URL Generation +- When generating links to other pages, prefer named routes and the `route()` function. + +### Configuration +- Use environment variables only in configuration files - never use the `env()` function directly outside of config files. Always use `config('app.name')`, not `env('APP_NAME')`. + +### Testing +- When creating models for tests, use the factories for the models. Check if the factory has custom states that can be used before manually setting up the model. +- Faker: Use methods such as `$this->faker->word()` or `fake()->randomDigit()`. Follow existing conventions whether to use `$this->faker` or `fake()`. +- When creating tests, make use of `php artisan make:test [options] ` to create a feature test, and pass `--unit` to create a unit test. Most tests should be feature tests. + +### Vite Error +- If you receive an "Illuminate\Foundation\ViteException: Unable to locate file in Vite manifest" error, you can run `npm run build` or ask the user to run `npm run dev` or `composer run dev`. + + +=== laravel/v12 rules === + +## Laravel 12 + +- Use the `search-docs` tool to get version specific documentation. +- This project upgraded from Laravel 10 without migrating to the new streamlined Laravel file structure. +- This is **perfectly fine** and recommended by Laravel. Follow the existing structure from Laravel 10. We do not to need migrate to the new Laravel structure unless the user explicitly requests that. + +### Laravel 10 Structure +- Middleware typically lives in `app/Http/Middleware/` and service providers in `app/Providers/`. +- There is no `bootstrap/app.php` application configuration in a Laravel 10 structure: + - Middleware registration happens in `app/Http/Kernel.php` + - Exception handling is in `app/Exceptions/Handler.php` + - Console commands and schedule register in `app/Console/Kernel.php` + - Rate limits likely exist in `RouteServiceProvider` or `app/Http/Kernel.php` + +### Database +- When modifying a column, the migration must include all of the attributes that were previously defined on the column. Otherwise, they will be dropped and lost. +- Laravel 12 allows limiting eagerly loaded records natively, without external packages: `$query->latest()->limit(10);`. + +### Models +- Casts can and likely should be set in a `casts()` method on a model rather than the `$casts` property. Follow existing conventions from other models. + + +=== livewire/core rules === + +## Livewire Core +- Use the `search-docs` tool to find exact version specific documentation for how to write Livewire & Livewire tests. +- Use the `php artisan make:livewire [Posts\\CreatePost]` artisan command to create new components +- State should live on the server, with the UI reflecting it. +- All Livewire requests hit the Laravel backend, they're like regular HTTP requests. Always validate form data, and run authorization checks in Livewire actions. + +## Livewire Best Practices +- Livewire components require a single root element. +- Use `wire:loading` and `wire:dirty` for delightful loading states. +- Add `wire:key` in loops: + + ```blade + @foreach ($items as $item) +
+ {{ $item->name }} +
+ @endforeach + ``` + +- Prefer lifecycle hooks like `mount()`, `updatedFoo()`) for initialization and reactive side effects: + + + public function mount(User $user) { $this->user = $user; } + public function updatedSearch() { $this->resetPage(); } + + + +## Testing Livewire + + + Livewire::test(Counter::class) + ->assertSet('count', 0) + ->call('increment') + ->assertSet('count', 1) + ->assertSee(1) + ->assertStatus(200); + + + + + $this->get('/posts/create') + ->assertSeeLivewire(CreatePost::class); + + + +=== livewire/v3 rules === + +## Livewire 3 + +### Key Changes From Livewire 2 +- These things changed in Livewire 2, but may not have been updated in this application. Verify this application's setup to ensure you conform with application conventions. + - Use `wire:model.live` for real-time updates, `wire:model` is now deferred by default. + - Components now use the `App\Livewire` namespace (not `App\Http\Livewire`). + - Use `$this->dispatch()` to dispatch events (not `emit` or `dispatchBrowserEvent`). + - Use the `components.layouts.app` view as the typical layout path (not `layouts.app`). + +### New Directives +- `wire:show`, `wire:transition`, `wire:cloak`, `wire:offline`, `wire:target` are available for use. Use the documentation to find usage examples. + +### Alpine +- Alpine is now included with Livewire, don't manually include Alpine.js. +- Plugins included with Alpine: persist, intersect, collapse, and focus. + +### Lifecycle Hooks +- You can listen for `livewire:init` to hook into Livewire initialization, and `fail.status === 419` for the page expiring: + + +document.addEventListener('livewire:init', function () { + Livewire.hook('request', ({ fail }) => { + if (fail && fail.status === 419) { + alert('Your session expired'); + } + }); + + Livewire.hook('message.failed', (message, component) => { + console.error(message); + }); +}); + + + +=== pint/core rules === + +## Laravel Pint Code Formatter + +- You must run `vendor/bin/pint --dirty` before finalizing changes to ensure your code matches the project's expected style. +- Do not run `vendor/bin/pint --test`, simply run `vendor/bin/pint` to fix any formatting issues. + + +=== pest/core rules === + +## Pest + +### Testing +- If you need to verify a feature is working, write or update a Unit / Feature test. + +### Pest Tests +- All tests must be written using Pest. Use `php artisan make:test --pest `. +- You must not remove any tests or test files from the tests directory without approval. These are not temporary or helper files - these are core to the application. +- Tests should test all of the happy paths, failure paths, and weird paths. +- Tests live in the `tests/Feature` and `tests/Unit` directories. +- Pest tests look and behave like this: + +it('is true', function () { + expect(true)->toBeTrue(); +}); + + +### Running Tests +- Run the minimal number of tests using an appropriate filter before finalizing code edits. +- To run all tests: `php artisan test`. +- To run all tests in a file: `php artisan test tests/Feature/ExampleTest.php`. +- To filter on a particular test name: `php artisan test --filter=testName` (recommended after making a change to a related file). +- When the tests relating to your changes are passing, ask the user if they would like to run the entire test suite to ensure everything is still passing. + +### Pest Assertions +- When asserting status codes on a response, use the specific method like `assertForbidden` and `assertNotFound` instead of using `assertStatus(403)` or similar, e.g.: + +it('returns all', function () { + $response = $this->postJson('/api/docs', []); + + $response->assertSuccessful(); +}); + + +### Mocking +- Mocking can be very helpful when appropriate. +- When mocking, you can use the `Pest\Laravel\mock` Pest function, but always import it via `use function Pest\Laravel\mock;` before using it. Alternatively, you can use `$this->mock()` if existing tests do. +- You can also create partial mocks using the same import or self method. + +### Datasets +- Use datasets in Pest to simplify tests which have a lot of duplicated data. This is often the case when testing validation rules, so consider going with this solution when writing tests for validation rules. + + +it('has emails', function (string $email) { + expect($email)->not->toBeEmpty(); +})->with([ + 'james' => 'james@laravel.com', + 'taylor' => 'taylor@laravel.com', +]); + + + +=== tailwindcss/core rules === + +## Tailwind Core + +- Use Tailwind CSS classes to style HTML, check and use existing tailwind conventions within the project before writing your own. +- Offer to extract repeated patterns into components that match the project's conventions (i.e. Blade, JSX, Vue, etc..) +- Think through class placement, order, priority, and defaults - remove redundant classes, add classes to parent or child carefully to limit repetition, group elements logically +- You can use the `search-docs` tool to get exact examples from the official documentation when needed. + +### Spacing +- When listing items, use gap utilities for spacing, don't use margins. + + +
+
Superior
+
Michigan
+
Erie
+
+
+ + +### Dark Mode +- If existing pages and components support dark mode, new pages and components must support dark mode in a similar way, typically using `dark:`. + + +=== tailwindcss/v4 rules === + +## Tailwind 4 + +- Always use Tailwind CSS v4 - do not use the deprecated utilities. +- `corePlugins` is not supported in Tailwind v4. +- In Tailwind v4, you import Tailwind using a regular CSS `@import` statement, not using the `@tailwind` directives used in v3: + + + + +### Replaced Utilities +- Tailwind v4 removed deprecated utilities. Do not use the deprecated option - use the replacement. +- Opacity values are still numeric. + +| Deprecated | Replacement | +|------------+--------------| +| bg-opacity-* | bg-black/* | +| text-opacity-* | text-black/* | +| border-opacity-* | border-black/* | +| divide-opacity-* | divide-black/* | +| ring-opacity-* | ring-black/* | +| placeholder-opacity-* | placeholder-black/* | +| flex-shrink-* | shrink-* | +| flex-grow-* | grow-* | +| overflow-ellipsis | text-ellipsis | +| decoration-slice | box-decoration-slice | +| decoration-clone | box-decoration-clone | + + +=== tests rules === + +## Test Enforcement + +- Every change must be programmatically tested. Write a new test or update an existing test, then run the affected tests to make sure they pass. +- Run the minimum number of tests needed to ensure code quality and speed. Use `php artisan test` with a specific filename or filter. +
\ No newline at end of file diff --git a/.ai/development/testing-patterns.md b/.ai/development/testing-patterns.md new file mode 100644 index 0000000000..875de8b3be --- /dev/null +++ b/.ai/development/testing-patterns.md @@ -0,0 +1,648 @@ +# Coolify Testing Architecture & Patterns + +> **Cross-Reference**: These detailed testing patterns align with the testing guidelines in **[CLAUDE.md](mdc:CLAUDE.md)**. Both documents share the same core principles about Docker execution and mocking preferences. + +## Testing Philosophy + +Coolify employs **comprehensive testing strategies** using modern PHP testing frameworks to ensure reliability of deployment operations, infrastructure management, and user interactions. + +### Test Execution Rules + +**CRITICAL**: Tests are categorized by database dependency: + +#### Unit Tests (`tests/Unit/`) +- **MUST NOT** use database connections +- **MUST** use mocking for models and external dependencies +- **CAN** run outside Docker: `./vendor/bin/pest tests/Unit` +- Purpose: Test isolated logic, helper functions, and business rules + +#### Feature Tests (`tests/Feature/`) +- **MAY** use database connections (factories, migrations, models) +- **MUST** run inside Docker container: `docker exec coolify php artisan test` +- **MUST** use `RefreshDatabase` trait if touching database +- Purpose: Test API endpoints, workflows, and integration scenarios + +**Rule of thumb**: If your test needs `Server::factory()->create()` or any database operation, it's a Feature test and MUST run in Docker. + +### Prefer Mocking Over Database + +When writing tests, always prefer mocking over real database operations: + +```php +// ❌ BAD: Unit test using database +it('extracts custom commands', function () { + $server = Server::factory()->create(['ip' => '1.2.3.4']); + $commands = extract_custom_proxy_commands($server, $yaml); + expect($commands)->toBeArray(); +}); + +// ✅ GOOD: Unit test using mocking +it('extracts custom commands', function () { + $server = Mockery::mock('App\Models\Server'); + $server->shouldReceive('proxyType')->andReturn('traefik'); + $commands = extract_custom_proxy_commands($server, $yaml); + expect($commands)->toBeArray(); +}); +``` + +**Design principles for testable code:** +- Use dependency injection instead of global state +- Create interfaces for external dependencies (SSH, Docker, etc.) +- Separate business logic from data persistence +- Make functions accept interfaces instead of concrete models when possible + +## Testing Framework Stack + +### Core Testing Tools +- **Pest PHP 3.8+** - Primary testing framework with expressive syntax +- **Laravel Dusk** - Browser automation and end-to-end testing +- **PHPUnit** - Underlying unit testing framework +- **Mockery** - Mocking and stubbing for isolated tests + +### Testing Configuration +- **[tests/Pest.php](mdc:tests/Pest.php)** - Pest configuration and global setup (1.5KB, 45 lines) +- **[tests/TestCase.php](mdc:tests/TestCase.php)** - Base test case class (163B, 11 lines) +- **[tests/CreatesApplication.php](mdc:tests/CreatesApplication.php)** - Application factory trait (375B, 22 lines) +- **[tests/DuskTestCase.php](mdc:tests/DuskTestCase.php)** - Browser testing setup (1.4KB, 58 lines) + +## Test Directory Structure + +### Test Organization +- **[tests/Feature/](mdc:tests/Feature)** - Feature and integration tests +- **[tests/Unit/](mdc:tests/Unit)** - Unit tests for isolated components +- **[tests/Browser/](mdc:tests/Browser)** - Laravel Dusk browser tests +- **[tests/Traits/](mdc:tests/Traits)** - Shared testing utilities + +## Unit Testing Patterns + +### Model Testing +```php +// Testing Eloquent models +test('application model has correct relationships', function () { + $application = Application::factory()->create(); + + expect($application->server)->toBeInstanceOf(Server::class); + expect($application->environment)->toBeInstanceOf(Environment::class); + expect($application->deployments)->toBeInstanceOf(Collection::class); +}); + +test('application can generate deployment configuration', function () { + $application = Application::factory()->create([ + 'name' => 'test-app', + 'git_repository' => 'https://github.com/user/repo.git' + ]); + + $config = $application->generateDockerCompose(); + + expect($config)->toContain('test-app'); + expect($config)->toContain('image:'); + expect($config)->toContain('networks:'); +}); +``` + +### Service Layer Testing +```php +// Testing service classes +test('configuration generator creates valid docker compose', function () { + $generator = new ConfigurationGenerator(); + $application = Application::factory()->create(); + + $compose = $generator->generateDockerCompose($application); + + expect($compose)->toBeString(); + expect(yaml_parse($compose))->toBeArray(); + expect($compose)->toContain('version: "3.8"'); +}); + +test('docker image parser validates image names', function () { + $parser = new DockerImageParser(); + + expect($parser->isValid('nginx:latest'))->toBeTrue(); + expect($parser->isValid('invalid-image-name'))->toBeFalse(); + expect($parser->parse('nginx:1.21'))->toEqual([ + 'registry' => 'docker.io', + 'namespace' => 'library', + 'repository' => 'nginx', + 'tag' => '1.21' + ]); +}); +``` + +### Action Testing +```php +// Testing Laravel Actions +test('deploy application action creates deployment queue', function () { + $application = Application::factory()->create(); + $action = new DeployApplicationAction(); + + $deployment = $action->handle($application); + + expect($deployment)->toBeInstanceOf(ApplicationDeploymentQueue::class); + expect($deployment->status)->toBe('queued'); + expect($deployment->application_id)->toBe($application->id); +}); + +test('server validation action checks ssh connectivity', function () { + $server = Server::factory()->create([ + 'ip' => '192.168.1.100', + 'port' => 22 + ]); + + $action = new ValidateServerAction(); + + // Mock SSH connection + $this->mock(SshConnection::class, function ($mock) { + $mock->shouldReceive('connect')->andReturn(true); + $mock->shouldReceive('execute')->with('docker --version')->andReturn('Docker version 20.10.0'); + }); + + $result = $action->handle($server); + + expect($result['ssh_connection'])->toBeTrue(); + expect($result['docker_installed'])->toBeTrue(); +}); +``` + +## Feature Testing Patterns + +### API Testing +```php +// Testing API endpoints +test('authenticated user can list applications', function () { + $user = User::factory()->create(); + $team = Team::factory()->create(); + $user->teams()->attach($team); + + $applications = Application::factory(3)->create([ + 'team_id' => $team->id + ]); + + $response = $this->actingAs($user) + ->getJson('/api/v1/applications'); + + $response->assertStatus(200) + ->assertJsonCount(3, 'data') + ->assertJsonStructure([ + 'data' => [ + '*' => ['id', 'name', 'fqdn', 'status', 'created_at'] + ] + ]); +}); + +test('user cannot access applications from other teams', function () { + $user = User::factory()->create(); + $otherTeam = Team::factory()->create(); + + $application = Application::factory()->create([ + 'team_id' => $otherTeam->id + ]); + + $response = $this->actingAs($user) + ->getJson("/api/v1/applications/{$application->id}"); + + $response->assertStatus(403); +}); +``` + +### Deployment Testing +```php +// Testing deployment workflows +test('application deployment creates docker containers', function () { + $application = Application::factory()->create([ + 'git_repository' => 'https://github.com/laravel/laravel.git', + 'git_branch' => 'main' + ]); + + // Mock Docker operations + $this->mock(DockerService::class, function ($mock) { + $mock->shouldReceive('buildImage')->andReturn('app:latest'); + $mock->shouldReceive('createContainer')->andReturn('container_id'); + $mock->shouldReceive('startContainer')->andReturn(true); + }); + + $deployment = $application->deploy(); + + expect($deployment->status)->toBe('queued'); + + // Process the deployment job + $this->artisan('queue:work --once'); + + $deployment->refresh(); + expect($deployment->status)->toBe('success'); +}); + +test('failed deployment triggers rollback', function () { + $application = Application::factory()->create(); + + // Mock failed deployment + $this->mock(DockerService::class, function ($mock) { + $mock->shouldReceive('buildImage')->andThrow(new DeploymentException('Build failed')); + }); + + $deployment = $application->deploy(); + + $this->artisan('queue:work --once'); + + $deployment->refresh(); + expect($deployment->status)->toBe('failed'); + expect($deployment->error_message)->toContain('Build failed'); +}); +``` + +### Webhook Testing +```php +// Testing webhook endpoints +test('github webhook triggers deployment', function () { + $application = Application::factory()->create([ + 'git_repository' => 'https://github.com/user/repo.git', + 'git_branch' => 'main' + ]); + + $payload = [ + 'ref' => 'refs/heads/main', + 'repository' => [ + 'clone_url' => 'https://github.com/user/repo.git' + ], + 'head_commit' => [ + 'id' => 'abc123', + 'message' => 'Update application' + ] + ]; + + $response = $this->postJson("/webhooks/github/{$application->id}", $payload); + + $response->assertStatus(200); + + expect($application->deployments()->count())->toBe(1); + expect($application->deployments()->first()->commit_sha)->toBe('abc123'); +}); + +test('webhook validates payload signature', function () { + $application = Application::factory()->create(); + + $payload = ['invalid' => 'payload']; + + $response = $this->postJson("/webhooks/github/{$application->id}", $payload); + + $response->assertStatus(400); +}); +``` + +## Browser Testing (Laravel Dusk) + +### End-to-End Testing +```php +// Testing complete user workflows +test('user can create and deploy application', function () { + $user = User::factory()->create(); + $server = Server::factory()->create(['team_id' => $user->currentTeam->id]); + + $this->browse(function (Browser $browser) use ($user, $server) { + $browser->loginAs($user) + ->visit('/applications/create') + ->type('name', 'Test Application') + ->type('git_repository', 'https://github.com/laravel/laravel.git') + ->type('git_branch', 'main') + ->select('server_id', $server->id) + ->press('Create Application') + ->assertPathIs('/applications/*') + ->assertSee('Test Application') + ->press('Deploy') + ->waitForText('Deployment started', 10) + ->assertSee('Deployment started'); + }); +}); + +test('user can monitor deployment logs in real-time', function () { + $user = User::factory()->create(); + $application = Application::factory()->create(['team_id' => $user->currentTeam->id]); + + $this->browse(function (Browser $browser) use ($user, $application) { + $browser->loginAs($user) + ->visit("/applications/{$application->id}") + ->press('Deploy') + ->waitForText('Deployment started') + ->click('@logs-tab') + ->waitFor('@deployment-logs') + ->assertSee('Building Docker image') + ->waitForText('Deployment completed', 30); + }); +}); +``` + +### UI Component Testing +```php +// Testing Livewire components +test('server status component updates in real-time', function () { + $user = User::factory()->create(); + $server = Server::factory()->create(['team_id' => $user->currentTeam->id]); + + $this->browse(function (Browser $browser) use ($user, $server) { + $browser->loginAs($user) + ->visit("/servers/{$server->id}") + ->assertSee('Status: Online') + ->waitFor('@server-metrics') + ->assertSee('CPU Usage') + ->assertSee('Memory Usage') + ->assertSee('Disk Usage'); + + // Simulate server going offline + $server->update(['status' => 'offline']); + + $browser->waitForText('Status: Offline', 5) + ->assertSee('Status: Offline'); + }); +}); +``` + +## Database Testing Patterns + +### Migration Testing +```php +// Testing database migrations +test('applications table has correct structure', function () { + expect(Schema::hasTable('applications'))->toBeTrue(); + expect(Schema::hasColumns('applications', [ + 'id', 'name', 'fqdn', 'git_repository', 'git_branch', + 'server_id', 'environment_id', 'created_at', 'updated_at' + ]))->toBeTrue(); +}); + +test('foreign key constraints are properly set', function () { + $application = Application::factory()->create(); + + expect($application->server)->toBeInstanceOf(Server::class); + expect($application->environment)->toBeInstanceOf(Environment::class); + + // Test cascade deletion + $application->server->delete(); + expect(Application::find($application->id))->toBeNull(); +}); +``` + +### Factory Testing +```php +// Testing model factories +test('application factory creates valid models', function () { + $application = Application::factory()->create(); + + expect($application->name)->toBeString(); + expect($application->git_repository)->toStartWith('https://'); + expect($application->server_id)->toBeInt(); + expect($application->environment_id)->toBeInt(); +}); + +test('application factory can create with custom attributes', function () { + $application = Application::factory()->create([ + 'name' => 'Custom App', + 'git_branch' => 'develop' + ]); + + expect($application->name)->toBe('Custom App'); + expect($application->git_branch)->toBe('develop'); +}); +``` + +## Queue Testing + +### Job Testing +```php +// Testing background jobs +test('deploy application job processes successfully', function () { + $application = Application::factory()->create(); + $deployment = ApplicationDeploymentQueue::factory()->create([ + 'application_id' => $application->id, + 'status' => 'queued' + ]); + + $job = new DeployApplicationJob($deployment); + + // Mock external dependencies + $this->mock(DockerService::class, function ($mock) { + $mock->shouldReceive('buildImage')->andReturn('app:latest'); + $mock->shouldReceive('deployContainer')->andReturn(true); + }); + + $job->handle(); + + $deployment->refresh(); + expect($deployment->status)->toBe('success'); +}); + +test('failed job is retried with exponential backoff', function () { + $application = Application::factory()->create(); + $deployment = ApplicationDeploymentQueue::factory()->create([ + 'application_id' => $application->id + ]); + + $job = new DeployApplicationJob($deployment); + + // Mock failure + $this->mock(DockerService::class, function ($mock) { + $mock->shouldReceive('buildImage')->andThrow(new Exception('Network error')); + }); + + expect(fn() => $job->handle())->toThrow(Exception::class); + + // Job should be retried + expect($job->tries)->toBe(3); + expect($job->backoff())->toBe([1, 5, 10]); +}); +``` + +## Security Testing + +### Authentication Testing +```php +// Testing authentication and authorization +test('unauthenticated users cannot access protected routes', function () { + $response = $this->get('/dashboard'); + $response->assertRedirect('/login'); +}); + +test('users can only access their team resources', function () { + $user1 = User::factory()->create(); + $user2 = User::factory()->create(); + + $team1 = Team::factory()->create(); + $team2 = Team::factory()->create(); + + $user1->teams()->attach($team1); + $user2->teams()->attach($team2); + + $application = Application::factory()->create(['team_id' => $team1->id]); + + $response = $this->actingAs($user2) + ->get("/applications/{$application->id}"); + + $response->assertStatus(403); +}); +``` + +### Input Validation Testing +```php +// Testing input validation and sanitization +test('application creation validates required fields', function () { + $user = User::factory()->create(); + + $response = $this->actingAs($user) + ->postJson('/api/v1/applications', []); + + $response->assertStatus(422) + ->assertJsonValidationErrors(['name', 'git_repository', 'server_id']); +}); + +test('malicious input is properly sanitized', function () { + $user = User::factory()->create(); + + $response = $this->actingAs($user) + ->postJson('/api/v1/applications', [ + 'name' => '', + 'git_repository' => 'javascript:alert("xss")', + 'server_id' => 'invalid' + ]); + + $response->assertStatus(422); +}); +``` + +## Performance Testing + +### Load Testing +```php +// Testing application performance under load +test('application list endpoint handles concurrent requests', function () { + $user = User::factory()->create(); + $applications = Application::factory(100)->create(['team_id' => $user->currentTeam->id]); + + $startTime = microtime(true); + + $response = $this->actingAs($user) + ->getJson('/api/v1/applications'); + + $endTime = microtime(true); + $responseTime = ($endTime - $startTime) * 1000; // Convert to milliseconds + + $response->assertStatus(200); + expect($responseTime)->toBeLessThan(500); // Should respond within 500ms +}); +``` + +### Memory Usage Testing +```php +// Testing memory efficiency +test('deployment process does not exceed memory limits', function () { + $initialMemory = memory_get_usage(); + + $application = Application::factory()->create(); + $deployment = $application->deploy(); + + // Process deployment + $this->artisan('queue:work --once'); + + $finalMemory = memory_get_usage(); + $memoryIncrease = $finalMemory - $initialMemory; + + expect($memoryIncrease)->toBeLessThan(50 * 1024 * 1024); // Less than 50MB +}); +``` + +## Test Utilities and Helpers + +### Custom Assertions +```php +// Custom test assertions +expect()->extend('toBeValidDockerCompose', function () { + $yaml = yaml_parse($this->value); + + return $yaml !== false && + isset($yaml['version']) && + isset($yaml['services']) && + is_array($yaml['services']); +}); + +expect()->extend('toHaveValidSshConnection', function () { + $server = $this->value; + + try { + $connection = new SshConnection($server); + return $connection->test(); + } catch (Exception $e) { + return false; + } +}); +``` + +### Test Traits +```php +// Shared testing functionality +trait CreatesTestServers +{ + protected function createTestServer(array $attributes = []): Server + { + return Server::factory()->create(array_merge([ + 'name' => 'Test Server', + 'ip' => '127.0.0.1', + 'port' => 22, + 'team_id' => $this->user->currentTeam->id + ], $attributes)); + } +} + +trait MocksDockerOperations +{ + protected function mockDockerService(): void + { + $this->mock(DockerService::class, function ($mock) { + $mock->shouldReceive('buildImage')->andReturn('test:latest'); + $mock->shouldReceive('createContainer')->andReturn('container_123'); + $mock->shouldReceive('startContainer')->andReturn(true); + $mock->shouldReceive('stopContainer')->andReturn(true); + }); + } +} +``` + +## Continuous Integration Testing + +### GitHub Actions Integration +```yaml +# .github/workflows/tests.yml +name: Tests +on: [push, pull_request] +jobs: + test: + runs-on: ubuntu-latest + services: + postgres: + image: postgres:15 + env: + POSTGRES_PASSWORD: password + options: >- + --health-cmd pg_isready + --health-interval 10s + --health-timeout 5s + --health-retries 5 + steps: + - uses: actions/checkout@v3 + - name: Setup PHP + uses: shivammathur/setup-php@v2 + with: + php-version: 8.4 + - name: Install dependencies + run: composer install + - name: Run tests + run: ./vendor/bin/pest +``` + +### Test Coverage +```php +// Generate test coverage reports +test('application has adequate test coverage', function () { + $coverage = $this->getCoverageData(); + + expect($coverage['application'])->toBeGreaterThan(80); + expect($coverage['models'])->toBeGreaterThan(90); + expect($coverage['actions'])->toBeGreaterThan(85); +}); +``` diff --git a/.ai/meta/maintaining-docs.md b/.ai/meta/maintaining-docs.md new file mode 100644 index 0000000000..1a15523995 --- /dev/null +++ b/.ai/meta/maintaining-docs.md @@ -0,0 +1,172 @@ +# Maintaining AI Documentation + +Guidelines for creating and maintaining AI documentation to ensure consistency and effectiveness across all AI tools (Claude Code, Cursor IDE, etc.). + +## Documentation Structure + +All AI documentation lives in the `.ai/` directory with the following structure: + +``` +.ai/ +├── README.md # Navigation hub +├── core/ # Core project information +├── development/ # Development practices +├── patterns/ # Code patterns and best practices +└── meta/ # Documentation maintenance guides +``` + +> **Note**: `CLAUDE.md` is in the repository root, not in the `.ai/` directory. + +## Required File Structure + +When creating new documentation files: + +```markdown +# Title + +Brief description of what this document covers. + +## Section 1 + +- **Main Points in Bold** + - Sub-points with details + - Examples and explanations + +## Section 2 + +### Subsection + +Content with code examples: + +```language +// ✅ DO: Show good examples +const goodExample = true; + +// ❌ DON'T: Show anti-patterns +const badExample = false; +``` +``` + +## File References + +- Use relative paths: `See [technology-stack.md](../core/technology-stack.md)` +- For code references: `` `app/Models/Application.php` `` +- Keep links working across different tools + +## Content Guidelines + +### DO: +- Start with high-level overview +- Include specific, actionable requirements +- Show examples of correct implementation +- Reference existing code when possible +- Keep documentation DRY by cross-referencing +- Use bullet points for clarity +- Include both DO and DON'T examples + +### DON'T: +- Create theoretical examples when real code exists +- Duplicate content across multiple files +- Use tool-specific formatting that won't work elsewhere +- Make assumptions about versions - specify exact versions + +## Rule Improvement Triggers + +Update documentation when you notice: +- New code patterns not covered by existing docs +- Repeated similar implementations across files +- Common error patterns that could be prevented +- New libraries or tools being used consistently +- Emerging best practices in the codebase + +## Analysis Process + +When updating documentation: +1. Compare new code with existing rules +2. Identify patterns that should be standardized +3. Look for references to external documentation +4. Check for consistent error handling patterns +5. Monitor test patterns and coverage + +## Rule Updates + +### Add New Documentation When: +- A new technology/pattern is used in 3+ files +- Common bugs could be prevented by documentation +- Code reviews repeatedly mention the same feedback +- New security or performance patterns emerge + +### Modify Existing Documentation When: +- Better examples exist in the codebase +- Additional edge cases are discovered +- Related documentation has been updated +- Implementation details have changed + +## Quality Checks + +Before committing documentation changes: +- [ ] Documentation is actionable and specific +- [ ] Examples come from actual code +- [ ] References are up to date +- [ ] Patterns are consistently enforced +- [ ] Cross-references work correctly +- [ ] Version numbers are exact and current + +## Continuous Improvement + +- Monitor code review comments +- Track common development questions +- Update docs after major refactors +- Add links to relevant documentation +- Cross-reference related docs + +## Deprecation + +When patterns become outdated: +1. Mark outdated patterns as deprecated +2. Remove docs that no longer apply +3. Update references to deprecated patterns +4. Document migration paths for old patterns + +## Synchronization + +### Single Source of Truth +- Each piece of information should exist in exactly ONE location +- Other files should reference the source, not duplicate it +- Example: Version numbers live in `core/technology-stack.md`, other files reference it + +### Cross-Tool Compatibility +- **CLAUDE.md**: Main instructions for Claude Code users (references `.ai/` files) +- **.cursor/rules/**: Single master file pointing to `.ai/` documentation +- **Both tools**: Should get same information from `.ai/` directory + +### When to Update What + +**Version Changes** (Laravel, PHP, packages): +1. Update `core/technology-stack.md` (single source) +2. Verify CLAUDE.md references it correctly +3. No other files should duplicate version numbers + +**Workflow Changes** (commands, setup): +1. Update `development/workflow.md` +2. Ensure CLAUDE.md quick reference is updated +3. Verify all cross-references work + +**Pattern Changes** (how to write code): +1. Update appropriate file in `patterns/` +2. Add/update examples from real codebase +3. Cross-reference from related docs + +## Documentation Files + +Keep documentation files only when explicitly needed. Don't create docs that merely describe obvious functionality - the code itself should be clear. + +## Breaking Changes + +When making breaking changes to documentation structure: +1. Update this maintaining-docs.md file +2. Update `.ai/README.md` navigation +3. Update CLAUDE.md references +4. Update `.cursor/rules/coolify-ai-docs.mdc` +5. Test all cross-references still work +6. Document the changes in sync-guide.md diff --git a/.ai/meta/sync-guide.md b/.ai/meta/sync-guide.md new file mode 100644 index 0000000000..ab9a45d1ac --- /dev/null +++ b/.ai/meta/sync-guide.md @@ -0,0 +1,214 @@ +# AI Instructions Synchronization Guide + +This document explains how AI instructions are organized and synchronized across different AI tools used with Coolify. + +## Overview + +Coolify maintains AI instructions with a **single source of truth** approach: + +1. **CLAUDE.md** - Main entry point for Claude Code (references `.ai/` directory) +2. **.cursor/rules/coolify-ai-docs.mdc** - Master reference file for Cursor IDE (references `.ai/` directory) +3. **.ai/** - Single source of truth containing all detailed documentation + +All AI tools (Claude Code, Cursor IDE, etc.) reference the same `.ai/` directory to ensure consistency. + +## Structure + +### CLAUDE.md (Root Directory) +- **Purpose**: Entry point for Claude Code with quick-reference guide +- **Format**: Single markdown file +- **Includes**: + - Quick-reference development commands + - High-level architecture overview + - Essential patterns and guidelines + - References to detailed `.ai/` documentation + +### .cursor/rules/coolify-ai-docs.mdc +- **Purpose**: Master reference file for Cursor IDE +- **Format**: Single .mdc file with frontmatter +- **Content**: Quick decision tree and references to `.ai/` directory +- **Note**: Replaces all previous topic-specific .mdc files + +### .ai/ Directory (Single Source of Truth) +- **Purpose**: All detailed, topic-specific documentation +- **Format**: Organized markdown files by category +- **Structure**: + ``` + .ai/ + ├── README.md # Navigation hub + ├── core/ # Project information + │ ├── technology-stack.md # Version numbers (SINGLE SOURCE OF TRUTH) + │ ├── project-overview.md + │ ├── application-architecture.md + │ └── deployment-architecture.md + ├── development/ # Development practices + │ ├── development-workflow.md + │ ├── testing-patterns.md + │ └── laravel-boost.md + ├── patterns/ # Code patterns + │ ├── database-patterns.md + │ ├── frontend-patterns.md + │ ├── security-patterns.md + │ ├── form-components.md + │ └── api-and-routing.md + └── meta/ # Documentation guides + ├── maintaining-docs.md + └── sync-guide.md (this file) + ``` +- **Used by**: All AI tools through CLAUDE.md or coolify-ai-docs.mdc + +## Cross-References + +All systems reference the `.ai/` directory as the source of truth: + +- **CLAUDE.md** → references `.ai/` files for detailed documentation +- **.cursor/rules/coolify-ai-docs.mdc** → references `.ai/` files for detailed documentation +- **.ai/README.md** → provides navigation to all documentation + +## Maintaining Consistency + +### 1. Core Principles (MUST be consistent) + +These are defined ONCE in `.ai/core/technology-stack.md`: +- Laravel version (currently Laravel 12.4.1) +- PHP version (8.4.7) +- All package versions (Livewire 3.5.20, Tailwind 4.1.4, etc.) + +**Exception**: CLAUDE.md is permitted to show essential version numbers as a quick reference for convenience. These must stay synchronized with `technology-stack.md`. When updating versions, update both locations. + +Other critical patterns defined in `.ai/`: +- Testing execution rules (Docker for Feature tests, mocking for Unit tests) +- Security patterns and authorization requirements +- Code style requirements (Pint, PSR-12) + +### 2. Where to Make Changes + +**For version numbers** (Laravel, PHP, packages): +1. Update `.ai/core/technology-stack.md` (single source of truth) +2. Update CLAUDE.md quick reference section (essential versions only) +3. Verify both files stay synchronized +4. Never duplicate version numbers in other locations + +**For workflow changes** (how to run commands, development setup): +1. Update `.ai/development/development-workflow.md` +2. Update quick reference in CLAUDE.md if needed +3. Verify `.cursor/rules/coolify-ai-docs.mdc` references are correct + +**For architectural patterns** (how code should be structured): +1. Update appropriate file in `.ai/core/` +2. Add cross-references from related docs +3. Update CLAUDE.md if it needs to highlight this pattern + +**For code patterns** (how to write code): +1. Update appropriate file in `.ai/patterns/` +2. Add examples from real codebase +3. Cross-reference from related docs + +**For testing patterns**: +1. Update `.ai/development/testing-patterns.md` +2. Ensure CLAUDE.md testing section references it + +### 3. Update Checklist + +When making significant changes: + +- [ ] Identify if change affects core principles (version numbers, critical patterns) +- [ ] Update primary location in `.ai/` directory +- [ ] Check if CLAUDE.md needs quick-reference update +- [ ] Verify `.cursor/rules/coolify-ai-docs.mdc` references are still accurate +- [ ] Update cross-references in related `.ai/` files +- [ ] Verify all relative paths work correctly +- [ ] Test links in markdown files +- [ ] Run: `./vendor/bin/pint` on modified files (if applicable) + +### 4. Common Inconsistencies to Watch + +- **Version numbers**: Should ONLY exist in `.ai/core/technology-stack.md` +- **Testing instructions**: Docker execution requirements must be consistent +- **File paths**: Ensure relative paths work from their location +- **Command syntax**: Docker commands, artisan commands must be accurate +- **Cross-references**: Links must point to current file locations + +## File Organization + +``` +/ +├── CLAUDE.md # Claude Code entry point +├── .AI_INSTRUCTIONS_SYNC.md # Redirect to this file +├── .cursor/ +│ └── rules/ +│ └── coolify-ai-docs.mdc # Cursor IDE master reference +└── .ai/ # SINGLE SOURCE OF TRUTH + ├── README.md # Navigation hub + ├── core/ # Project information + ├── development/ # Development practices + ├── patterns/ # Code patterns + └── meta/ # Documentation guides +``` + +## Recent Updates + +### 2025-11-18 - Documentation Consolidation +- ✅ Consolidated all documentation into `.ai/` directory +- ✅ Created single source of truth for version numbers +- ✅ Reduced CLAUDE.md from 719 to 319 lines +- ✅ Replaced 11 .cursor/rules/*.mdc files with single coolify-ai-docs.mdc +- ✅ Organized by topic: core/, development/, patterns/, meta/ +- ✅ Standardized version numbers (Laravel 12.4.1, PHP 8.4.7, Tailwind 4.1.4) +- ✅ Created comprehensive navigation with .ai/README.md + +### 2025-10-07 +- ✅ Added cross-references between CLAUDE.md and .cursor/rules/ +- ✅ Synchronized Laravel version (12) across all files +- ✅ Added comprehensive testing execution rules (Docker for Feature tests) +- ✅ Added test design philosophy (prefer mocking over database) +- ✅ Fixed inconsistencies in testing documentation + +## Maintenance Commands + +```bash +# Check for version inconsistencies (should only be in technology-stack.md) +# Note: CLAUDE.md is allowed to show quick reference versions +grep -r "Laravel 12" .ai/ CLAUDE.md .cursor/rules/coolify-ai-docs.mdc +grep -r "PHP 8.4" .ai/ CLAUDE.md .cursor/rules/coolify-ai-docs.mdc + +# Check for broken cross-references to old .mdc files +grep -r "\.cursor/rules/.*\.mdc" .ai/ CLAUDE.md + +# Format all documentation +./vendor/bin/pint CLAUDE.md .ai/**/*.md + +# Search for specific patterns across all docs +grep -r "pattern_to_check" CLAUDE.md .ai/ .cursor/rules/ + +# Verify all markdown links work (from repository root) +find .ai -name "*.md" -exec grep -H "\[.*\](.*)" {} \; +``` + +## Contributing + +When contributing documentation: + +1. **Check `.ai/` directory** for existing documentation +2. **Update `.ai/` files** - this is the single source of truth +3. **Use cross-references** - never duplicate content +4. **Update CLAUDE.md** if adding critical quick-reference information +5. **Verify `.cursor/rules/coolify-ai-docs.mdc`** still references correctly +6. **Test all links** work from their respective locations +7. **Update this sync-guide.md** if changing organizational structure +8. **Verify consistency** before submitting PR + +## Questions? + +If unsure about where to document something: + +- **Version numbers** → `.ai/core/technology-stack.md` (ONLY location) +- **Quick reference / commands** → CLAUDE.md + `.ai/development/development-workflow.md` +- **Detailed patterns / examples** → `.ai/patterns/[topic].md` +- **Architecture / concepts** → `.ai/core/[topic].md` +- **Development practices** → `.ai/development/[topic].md` +- **Documentation guides** → `.ai/meta/[topic].md` + +**Golden Rule**: Each piece of information exists in ONE location in `.ai/`, other files reference it. + +When in doubt, prefer detailed documentation in `.ai/` and lightweight references in CLAUDE.md and coolify-ai-docs.mdc. diff --git a/.ai/patterns/api-and-routing.md b/.ai/patterns/api-and-routing.md new file mode 100644 index 0000000000..ceaadaad55 --- /dev/null +++ b/.ai/patterns/api-and-routing.md @@ -0,0 +1,469 @@ +# Coolify API & Routing Architecture + +## Routing Structure + +Coolify implements **multi-layered routing** with web interfaces, RESTful APIs, webhook endpoints, and real-time communication channels. + +## Route Files + +### Core Route Definitions +- **[routes/web.php](mdc:routes/web.php)** - Web application routes (21KB, 362 lines) +- **[routes/api.php](mdc:routes/api.php)** - RESTful API endpoints (13KB, 185 lines) +- **[routes/webhooks.php](mdc:routes/webhooks.php)** - Webhook receivers (815B, 22 lines) +- **[routes/channels.php](mdc:routes/channels.php)** - WebSocket channel definitions (829B, 33 lines) +- **[routes/console.php](mdc:routes/console.php)** - Artisan command routes (592B, 20 lines) + +## Web Application Routing + +### Authentication Routes +```php +// Laravel Fortify authentication +Route::middleware('guest')->group(function () { + Route::get('/login', [AuthController::class, 'login']); + Route::get('/register', [AuthController::class, 'register']); + Route::get('/forgot-password', [AuthController::class, 'forgotPassword']); +}); +``` + +### Dashboard & Core Features +```php +// Main application routes +Route::middleware(['auth', 'verified'])->group(function () { + Route::get('/dashboard', Dashboard::class)->name('dashboard'); + Route::get('/projects', ProjectIndex::class)->name('projects'); + Route::get('/servers', ServerIndex::class)->name('servers'); + Route::get('/teams', TeamIndex::class)->name('teams'); +}); +``` + +### Resource Management Routes +```php +// Server management +Route::prefix('servers')->group(function () { + Route::get('/{server}', ServerShow::class)->name('server.show'); + Route::get('/{server}/edit', ServerEdit::class)->name('server.edit'); + Route::get('/{server}/logs', ServerLogs::class)->name('server.logs'); +}); + +// Application management +Route::prefix('applications')->group(function () { + Route::get('/{application}', ApplicationShow::class)->name('application.show'); + Route::get('/{application}/deployments', ApplicationDeployments::class); + Route::get('/{application}/environment-variables', ApplicationEnvironmentVariables::class); + Route::get('/{application}/logs', ApplicationLogs::class); +}); +``` + +## RESTful API Architecture + +### API Versioning +```php +// API route structure +Route::prefix('v1')->group(function () { + // Application endpoints + Route::apiResource('applications', ApplicationController::class); + Route::apiResource('servers', ServerController::class); + Route::apiResource('teams', TeamController::class); +}); +``` + +### Authentication & Authorization +```php +// Sanctum API authentication +Route::middleware('auth:sanctum')->group(function () { + Route::get('/user', function (Request $request) { + return $request->user(); + }); + + // Team-scoped resources + Route::middleware('team.access')->group(function () { + Route::apiResource('applications', ApplicationController::class); + }); +}); +``` + +### Application Management API +```php +// Application CRUD operations +Route::prefix('applications')->group(function () { + Route::get('/', [ApplicationController::class, 'index']); + Route::post('/', [ApplicationController::class, 'store']); + Route::get('/{application}', [ApplicationController::class, 'show']); + Route::patch('/{application}', [ApplicationController::class, 'update']); + Route::delete('/{application}', [ApplicationController::class, 'destroy']); + + // Deployment operations + Route::post('/{application}/deploy', [ApplicationController::class, 'deploy']); + Route::post('/{application}/restart', [ApplicationController::class, 'restart']); + Route::post('/{application}/stop', [ApplicationController::class, 'stop']); + Route::get('/{application}/logs', [ApplicationController::class, 'logs']); +}); +``` + +### Server Management API +```php +// Server operations +Route::prefix('servers')->group(function () { + Route::get('/', [ServerController::class, 'index']); + Route::post('/', [ServerController::class, 'store']); + Route::get('/{server}', [ServerController::class, 'show']); + Route::patch('/{server}', [ServerController::class, 'update']); + Route::delete('/{server}', [ServerController::class, 'destroy']); + + // Server actions + Route::post('/{server}/validate', [ServerController::class, 'validate']); + Route::get('/{server}/usage', [ServerController::class, 'usage']); + Route::post('/{server}/cleanup', [ServerController::class, 'cleanup']); +}); +``` + +### Database Management API +```php +// Database operations +Route::prefix('databases')->group(function () { + Route::get('/', [DatabaseController::class, 'index']); + Route::post('/', [DatabaseController::class, 'store']); + Route::get('/{database}', [DatabaseController::class, 'show']); + Route::patch('/{database}', [DatabaseController::class, 'update']); + Route::delete('/{database}', [DatabaseController::class, 'destroy']); + + // Database actions + Route::post('/{database}/backup', [DatabaseController::class, 'backup']); + Route::post('/{database}/restore', [DatabaseController::class, 'restore']); + Route::get('/{database}/logs', [DatabaseController::class, 'logs']); +}); +``` + +## Webhook Architecture + +### Git Integration Webhooks +```php +// GitHub webhook endpoints +Route::post('/webhooks/github/{application}', [GitHubWebhookController::class, 'handle']) + ->name('webhooks.github'); + +// GitLab webhook endpoints +Route::post('/webhooks/gitlab/{application}', [GitLabWebhookController::class, 'handle']) + ->name('webhooks.gitlab'); + +// Generic Git webhooks +Route::post('/webhooks/git/{application}', [GitWebhookController::class, 'handle']) + ->name('webhooks.git'); +``` + +### Deployment Webhooks +```php +// Deployment status webhooks +Route::post('/webhooks/deployment/{deployment}/success', [DeploymentWebhookController::class, 'success']); +Route::post('/webhooks/deployment/{deployment}/failure', [DeploymentWebhookController::class, 'failure']); +Route::post('/webhooks/deployment/{deployment}/progress', [DeploymentWebhookController::class, 'progress']); +``` + +### Third-Party Integration Webhooks +```php +// Monitoring webhooks +Route::post('/webhooks/monitoring/{server}', [MonitoringWebhookController::class, 'handle']); + +// Backup status webhooks +Route::post('/webhooks/backup/{backup}', [BackupWebhookController::class, 'handle']); + +// SSL certificate webhooks +Route::post('/webhooks/ssl/{certificate}', [SslWebhookController::class, 'handle']); +``` + +## WebSocket Channel Definitions + +### Real-Time Channels +```php +// Private channels for team members +Broadcast::channel('team.{teamId}', function ($user, $teamId) { + return $user->teams->contains('id', $teamId); +}); + +// Application deployment channels +Broadcast::channel('application.{applicationId}', function ($user, $applicationId) { + return $user->hasAccessToApplication($applicationId); +}); + +// Server monitoring channels +Broadcast::channel('server.{serverId}', function ($user, $serverId) { + return $user->hasAccessToServer($serverId); +}); +``` + +### Presence Channels +```php +// Team collaboration presence +Broadcast::channel('team.{teamId}.presence', function ($user, $teamId) { + if ($user->teams->contains('id', $teamId)) { + return ['id' => $user->id, 'name' => $user->name]; + } +}); +``` + +## API Controllers + +### Location: [app/Http/Controllers/Api/](mdc:app/Http/Controllers) + +#### Resource Controllers +```php +class ApplicationController extends Controller +{ + public function index(Request $request) + { + return ApplicationResource::collection( + $request->user()->currentTeam->applications() + ->with(['server', 'environment']) + ->paginate() + ); + } + + public function store(StoreApplicationRequest $request) + { + $application = $request->user()->currentTeam + ->applications() + ->create($request->validated()); + + return new ApplicationResource($application); + } + + public function deploy(Application $application) + { + $deployment = $application->deploy(); + + return response()->json([ + 'message' => 'Deployment started', + 'deployment_id' => $deployment->id + ]); + } +} +``` + +### API Responses & Resources +```php +// API Resource classes +class ApplicationResource extends JsonResource +{ + public function toArray($request) + { + return [ + 'id' => $this->id, + 'name' => $this->name, + 'fqdn' => $this->fqdn, + 'status' => $this->status, + 'git_repository' => $this->git_repository, + 'git_branch' => $this->git_branch, + 'created_at' => $this->created_at, + 'updated_at' => $this->updated_at, + 'server' => new ServerResource($this->whenLoaded('server')), + 'environment' => new EnvironmentResource($this->whenLoaded('environment')), + ]; + } +} +``` + +## API Authentication + +### Sanctum Token Authentication +```php +// API token generation +Route::post('/auth/tokens', function (Request $request) { + $request->validate([ + 'name' => 'required|string', + 'abilities' => 'array' + ]); + + $token = $request->user()->createToken( + $request->name, + $request->abilities ?? [] + ); + + return response()->json([ + 'token' => $token->plainTextToken, + 'abilities' => $token->accessToken->abilities + ]); +}); +``` + +### Team-Based Authorization +```php +// Team access middleware +class EnsureTeamAccess +{ + public function handle($request, Closure $next) + { + $teamId = $request->route('team'); + + if (!$request->user()->teams->contains('id', $teamId)) { + abort(403, 'Access denied to team resources'); + } + + return $next($request); + } +} +``` + +## Rate Limiting + +### API Rate Limits +```php +// API throttling configuration +RateLimiter::for('api', function (Request $request) { + return Limit::perMinute(60)->by($request->user()?->id ?: $request->ip()); +}); + +// Deployment rate limiting +RateLimiter::for('deployments', function (Request $request) { + return Limit::perMinute(10)->by($request->user()->id); +}); +``` + +### Webhook Rate Limiting +```php +// Webhook throttling +RateLimiter::for('webhooks', function (Request $request) { + return Limit::perMinute(100)->by($request->ip()); +}); +``` + +## Route Model Binding + +### Custom Route Bindings +```php +// Custom model binding for applications +Route::bind('application', function ($value) { + return Application::where('uuid', $value) + ->orWhere('id', $value) + ->firstOrFail(); +}); + +// Team-scoped model binding +Route::bind('team_application', function ($value, $route) { + $teamId = $route->parameter('team'); + return Application::whereHas('environment.project', function ($query) use ($teamId) { + $query->where('team_id', $teamId); + })->findOrFail($value); +}); +``` + +## API Documentation + +### OpenAPI Specification +- **[openapi.json](mdc:openapi.json)** - API documentation (373KB, 8316 lines) +- **[openapi.yaml](mdc:openapi.yaml)** - YAML format documentation (184KB, 5579 lines) + +### Documentation Generation +```php +// Swagger/OpenAPI annotations +/** + * @OA\Get( + * path="/api/v1/applications", + * summary="List applications", + * tags={"Applications"}, + * security={{"bearerAuth":{}}}, + * @OA\Response( + * response=200, + * description="List of applications", + * @OA\JsonContent(type="array", @OA\Items(ref="#/components/schemas/Application")) + * ) + * ) + */ +``` + +## Error Handling + +### API Error Responses +```php +// Standardized error response format +class ApiExceptionHandler +{ + public function render($request, Throwable $exception) + { + if ($request->expectsJson()) { + return response()->json([ + 'message' => $exception->getMessage(), + 'error_code' => $this->getErrorCode($exception), + 'timestamp' => now()->toISOString() + ], $this->getStatusCode($exception)); + } + + return parent::render($request, $exception); + } +} +``` + +### Validation Error Handling +```php +// Form request validation +class StoreApplicationRequest extends FormRequest +{ + public function rules() + { + return [ + 'name' => 'required|string|max:255', + 'git_repository' => 'required|url', + 'git_branch' => 'required|string', + 'server_id' => 'required|exists:servers,id', + 'environment_id' => 'required|exists:environments,id' + ]; + } + + public function failedValidation(Validator $validator) + { + throw new HttpResponseException( + response()->json([ + 'message' => 'Validation failed', + 'errors' => $validator->errors() + ], 422) + ); + } +} +``` + +## Real-Time API Integration + +### WebSocket Events +```php +// Broadcasting deployment events +class DeploymentStarted implements ShouldBroadcast +{ + public $application; + public $deployment; + + public function broadcastOn() + { + return [ + new PrivateChannel("application.{$this->application->id}"), + new PrivateChannel("team.{$this->application->team->id}") + ]; + } + + public function broadcastWith() + { + return [ + 'deployment_id' => $this->deployment->id, + 'status' => 'started', + 'timestamp' => now() + ]; + } +} +``` + +### API Event Streaming +```php +// Server-Sent Events for real-time updates +Route::get('/api/v1/applications/{application}/events', function (Application $application) { + return response()->stream(function () use ($application) { + while (true) { + $events = $application->getRecentEvents(); + foreach ($events as $event) { + echo "data: " . json_encode($event) . "\n\n"; + } + usleep(1000000); // 1 second + } + }, 200, [ + 'Content-Type' => 'text/event-stream', + 'Cache-Control' => 'no-cache', + ]); +}); +``` diff --git a/.ai/patterns/database-patterns.md b/.ai/patterns/database-patterns.md new file mode 100644 index 0000000000..5a9d16f711 --- /dev/null +++ b/.ai/patterns/database-patterns.md @@ -0,0 +1,377 @@ +# Coolify Database Architecture & Patterns + +## Database Strategy + +Coolify uses **PostgreSQL 15** as the primary database with **Redis 7** for caching and real-time features. The architecture supports managing multiple external databases across different servers. + +## Primary Database (PostgreSQL) + +### Core Tables & Models + +#### User & Team Management +- **[User.php](mdc:app/Models/User.php)** - User authentication and profiles +- **[Team.php](mdc:app/Models/Team.php)** - Multi-tenant organization structure +- **[TeamInvitation.php](mdc:app/Models/TeamInvitation.php)** - Team collaboration invitations +- **[PersonalAccessToken.php](mdc:app/Models/PersonalAccessToken.php)** - API token management + +#### Infrastructure Management +- **[Server.php](mdc:app/Models/Server.php)** - Physical/virtual server definitions (46KB, complex) +- **[PrivateKey.php](mdc:app/Models/PrivateKey.php)** - SSH key management +- **[ServerSetting.php](mdc:app/Models/ServerSetting.php)** - Server-specific configurations + +#### Project Organization +- **[Project.php](mdc:app/Models/Project.php)** - Project containers for applications +- **[Environment.php](mdc:app/Models/Environment.php)** - Environment isolation (staging, production, etc.) +- **[ProjectSetting.php](mdc:app/Models/ProjectSetting.php)** - Project-specific settings + +#### Application Deployment +- **[Application.php](mdc:app/Models/Application.php)** - Main application entity (74KB, highly complex) +- **[ApplicationSetting.php](mdc:app/Models/ApplicationSetting.php)** - Application configurations +- **[ApplicationDeploymentQueue.php](mdc:app/Models/ApplicationDeploymentQueue.php)** - Deployment orchestration +- **[ApplicationPreview.php](mdc:app/Models/ApplicationPreview.php)** - Preview environment management + +#### Service Management +- **[Service.php](mdc:app/Models/Service.php)** - Service definitions (58KB, complex) +- **[ServiceApplication.php](mdc:app/Models/ServiceApplication.php)** - Service components +- **[ServiceDatabase.php](mdc:app/Models/ServiceDatabase.php)** - Service-attached databases + +## Database Type Support + +### Standalone Database Models +Each database type has its own dedicated model with specific configurations: + +#### SQL Databases +- **[StandalonePostgresql.php](mdc:app/Models/StandalonePostgresql.php)** - PostgreSQL instances +- **[StandaloneMysql.php](mdc:app/Models/StandaloneMysql.php)** - MySQL instances +- **[StandaloneMariadb.php](mdc:app/Models/StandaloneMariadb.php)** - MariaDB instances + +#### NoSQL & Analytics +- **[StandaloneMongodb.php](mdc:app/Models/StandaloneMongodb.php)** - MongoDB instances +- **[StandaloneClickhouse.php](mdc:app/Models/StandaloneClickhouse.php)** - ClickHouse analytics + +#### Caching & In-Memory +- **[StandaloneRedis.php](mdc:app/Models/StandaloneRedis.php)** - Redis instances +- **[StandaloneKeydb.php](mdc:app/Models/StandaloneKeydb.php)** - KeyDB instances +- **[StandaloneDragonfly.php](mdc:app/Models/StandaloneDragonfly.php)** - Dragonfly instances + +## Configuration Management + +### Environment Variables +- **[EnvironmentVariable.php](mdc:app/Models/EnvironmentVariable.php)** - Application-specific environment variables +- **[SharedEnvironmentVariable.php](mdc:app/Models/SharedEnvironmentVariable.php)** - Shared across applications + +### Settings Hierarchy +- **[InstanceSettings.php](mdc:app/Models/InstanceSettings.php)** - Global Coolify instance settings +- **[ServerSetting.php](mdc:app/Models/ServerSetting.php)** - Server-specific settings +- **[ProjectSetting.php](mdc:app/Models/ProjectSetting.php)** - Project-level settings +- **[ApplicationSetting.php](mdc:app/Models/ApplicationSetting.php)** - Application settings + +## Storage & Backup Systems + +### Storage Management +- **[S3Storage.php](mdc:app/Models/S3Storage.php)** - S3-compatible storage configurations +- **[LocalFileVolume.php](mdc:app/Models/LocalFileVolume.php)** - Local filesystem volumes +- **[LocalPersistentVolume.php](mdc:app/Models/LocalPersistentVolume.php)** - Persistent volume management + +### Backup Infrastructure +- **[ScheduledDatabaseBackup.php](mdc:app/Models/ScheduledDatabaseBackup.php)** - Automated backup scheduling +- **[ScheduledDatabaseBackupExecution.php](mdc:app/Models/ScheduledDatabaseBackupExecution.php)** - Backup execution tracking + +### Task Scheduling +- **[ScheduledTask.php](mdc:app/Models/ScheduledTask.php)** - Cron job management +- **[ScheduledTaskExecution.php](mdc:app/Models/ScheduledTaskExecution.php)** - Task execution history + +## Notification & Integration Models + +### Notification Channels +- **[EmailNotificationSettings.php](mdc:app/Models/EmailNotificationSettings.php)** - Email notifications +- **[DiscordNotificationSettings.php](mdc:app/Models/DiscordNotificationSettings.php)** - Discord integration +- **[SlackNotificationSettings.php](mdc:app/Models/SlackNotificationSettings.php)** - Slack integration +- **[TelegramNotificationSettings.php](mdc:app/Models/TelegramNotificationSettings.php)** - Telegram bot +- **[PushoverNotificationSettings.php](mdc:app/Models/PushoverNotificationSettings.php)** - Pushover notifications + +### Source Control Integration +- **[GithubApp.php](mdc:app/Models/GithubApp.php)** - GitHub App integration +- **[GitlabApp.php](mdc:app/Models/GitlabApp.php)** - GitLab integration + +### OAuth & Authentication +- **[OauthSetting.php](mdc:app/Models/OauthSetting.php)** - OAuth provider configurations + +## Docker & Container Management + +### Container Orchestration +- **[StandaloneDocker.php](mdc:app/Models/StandaloneDocker.php)** - Standalone Docker containers +- **[SwarmDocker.php](mdc:app/Models/SwarmDocker.php)** - Docker Swarm management + +### SSL & Security +- **[SslCertificate.php](mdc:app/Models/SslCertificate.php)** - SSL certificate management + +## Database Migration Strategy + +### Migration Location: [database/migrations/](mdc:database/migrations) + +#### Migration Patterns +```php +// Typical Coolify migration structure +Schema::create('applications', function (Blueprint $table) { + $table->id(); + $table->string('name'); + $table->string('fqdn')->nullable(); + $table->json('environment_variables')->nullable(); + $table->foreignId('destination_id'); + $table->foreignId('source_id'); + $table->timestamps(); +}); +``` + +### Schema Versioning +- **Incremental migrations** for database evolution +- **Data migrations** for complex transformations +- **Rollback support** for deployment safety + +## Eloquent Model Patterns + +### Base Model Structure +- **[BaseModel.php](mdc:app/Models/BaseModel.php)** - Common model functionality +- **UUID primary keys** for distributed systems +- **Soft deletes** for audit trails +- **Activity logging** with Spatie package + +### **CRITICAL: Mass Assignment Protection** +**When adding new database columns, you MUST update the model's `$fillable` array.** Without this, Laravel will silently ignore mass assignment operations like `Model::create()` or `$model->update()`. + +**Checklist for new columns:** +1. ✅ Create migration file +2. ✅ Run migration +3. ✅ **Add column to model's `$fillable` array** +4. ✅ Update any Livewire components that sync this property +5. ✅ Test that the column can be read and written + +**Example:** +```php +class Server extends BaseModel +{ + protected $fillable = [ + 'name', + 'ip', + 'port', + 'is_validating', // ← MUST add new columns here + ]; +} +``` + +### Relationship Patterns +```php +// Typical relationship structure in Application model +class Application extends Model +{ + public function server() + { + return $this->belongsTo(Server::class); + } + + public function environment() + { + return $this->belongsTo(Environment::class); + } + + public function deployments() + { + return $this->hasMany(ApplicationDeploymentQueue::class); + } + + public function environmentVariables() + { + return $this->hasMany(EnvironmentVariable::class); + } +} +``` + +### Model Traits +```php +// Common traits used across models +use SoftDeletes; +use LogsActivity; +use HasFactory; +use HasUuids; +``` + +## Caching Strategy (Redis) + +### Cache Usage Patterns +- **Session storage** - User authentication sessions +- **Queue backend** - Background job processing +- **Model caching** - Expensive query results +- **Real-time data** - WebSocket state management + +### Cache Keys Structure +``` +coolify:session:{session_id} +coolify:server:{server_id}:status +coolify:deployment:{deployment_id}:logs +coolify:user:{user_id}:teams +``` + +## Query Optimization Patterns + +### Eager Loading +```php +// Optimized queries with relationships +$applications = Application::with([ + 'server', + 'environment.project', + 'environmentVariables', + 'deployments' => function ($query) { + $query->latest()->limit(5); + } +])->get(); +``` + +### Chunking for Large Datasets +```php +// Processing large datasets efficiently +Server::chunk(100, function ($servers) { + foreach ($servers as $server) { + // Process server monitoring + } +}); +``` + +### Database Indexes +- **Primary keys** on all tables +- **Foreign key indexes** for relationships +- **Composite indexes** for common queries +- **Unique constraints** for business rules + +### Request-Level Caching with ownedByCurrentTeamCached() + +Many models have both `ownedByCurrentTeam()` (returns query builder) and `ownedByCurrentTeamCached()` (returns cached collection). **Always prefer the cached version** to avoid duplicate database queries within the same request. + +**Models with cached methods available:** +- `Server`, `PrivateKey`, `Project` +- `Application` +- `StandalonePostgresql`, `StandaloneMysql`, `StandaloneRedis`, `StandaloneMariadb`, `StandaloneMongodb`, `StandaloneKeydb`, `StandaloneDragonfly`, `StandaloneClickhouse` +- `Service`, `ServiceApplication`, `ServiceDatabase` + +**Usage patterns:** +```php +// ✅ CORRECT - Uses request-level cache (via Laravel's once() helper) +$servers = Server::ownedByCurrentTeamCached(); + +// ❌ AVOID - Makes a new database query each time +$servers = Server::ownedByCurrentTeam()->get(); + +// ✅ CORRECT - Filter cached collection in memory +$activeServers = Server::ownedByCurrentTeamCached()->where('is_active', true); +$server = Server::ownedByCurrentTeamCached()->firstWhere('id', $serverId); +$serverIds = Server::ownedByCurrentTeamCached()->pluck('id'); + +// ❌ AVOID - Making filtered database queries when data is already cached +$activeServers = Server::ownedByCurrentTeam()->where('is_active', true)->get(); +``` + +**When to use which:** +- `ownedByCurrentTeamCached()` - **Default choice** for reading team data +- `ownedByCurrentTeam()` - Only when you need to chain query builder methods that can't be done on collections (like `with()` for eager loading), or when you explicitly need a fresh database query + +**Implementation pattern for new models:** +```php +/** + * Get query builder for resources owned by current team. + * If you need all resources without further query chaining, use ownedByCurrentTeamCached() instead. + */ +public static function ownedByCurrentTeam() +{ + return self::whereTeamId(currentTeam()->id); +} + +/** + * Get all resources owned by current team (cached for request duration). + */ +public static function ownedByCurrentTeamCached() +{ + return once(function () { + return self::ownedByCurrentTeam()->get(); + }); +} +``` + +## Data Consistency Patterns + +### Database Transactions +```php +// Atomic operations for deployment +DB::transaction(function () { + $application = Application::create($data); + $application->environmentVariables()->createMany($envVars); + $application->deployments()->create(['status' => 'queued']); +}); +``` + +### Model Events +```php +// Automatic cleanup on model deletion +class Application extends Model +{ + protected static function booted() + { + static::deleting(function ($application) { + $application->environmentVariables()->delete(); + $application->deployments()->delete(); + }); + } +} +``` + +## Backup & Recovery + +### Database Backup Strategy +- **Automated PostgreSQL backups** via scheduled tasks +- **Point-in-time recovery** capability +- **Cross-region backup** replication +- **Backup verification** and testing + +### Data Export/Import +- **Application configurations** export/import +- **Environment variable** bulk operations +- **Server configurations** backup and restore + +## Performance Monitoring + +### Query Performance +- **Laravel Telescope** for development debugging +- **Slow query logging** in production +- **Database connection** pooling +- **Read replica** support for scaling + +### Metrics Collection +- **Database size** monitoring +- **Connection count** tracking +- **Query execution time** analysis +- **Cache hit rates** monitoring + +## Multi-Tenancy Pattern + +### Team-Based Isolation +```php +// Global scope for team-based filtering +class Application extends Model +{ + protected static function booted() + { + static::addGlobalScope('team', function (Builder $builder) { + if (auth()->user()) { + $builder->whereHas('environment.project', function ($query) { + $query->where('team_id', auth()->user()->currentTeam->id); + }); + } + }); + } +} +``` + +### Data Separation +- **Team-scoped queries** by default +- **Cross-team access** controls +- **Admin access** patterns +- **Data isolation** guarantees diff --git a/.ai/patterns/form-components.md b/.ai/patterns/form-components.md new file mode 100644 index 0000000000..3ff1d0f817 --- /dev/null +++ b/.ai/patterns/form-components.md @@ -0,0 +1,447 @@ + +# Enhanced Form Components with Authorization + +## Overview + +Coolify's form components now feature **built-in authorization** that automatically handles permission-based UI control, dramatically reducing code duplication and improving security consistency. + +## Enhanced Components + +All form components now support the `canGate` authorization system: + +- **[Input.php](mdc:app/View/Components/Forms/Input.php)** - Text, password, and other input fields +- **[Select.php](mdc:app/View/Components/Forms/Select.php)** - Dropdown selection components +- **[Textarea.php](mdc:app/View/Components/Forms/Textarea.php)** - Multi-line text areas +- **[Checkbox.php](mdc:app/View/Components/Forms/Checkbox.php)** - Boolean toggle components +- **[Button.php](mdc:app/View/Components/Forms/Button.php)** - Action buttons + +## Authorization Parameters + +### Core Parameters +```php +public ?string $canGate = null; // Gate name: 'update', 'view', 'deploy', 'delete' +public mixed $canResource = null; // Resource model instance to check against +public bool $autoDisable = true; // Automatically disable if no permission +``` + +### How It Works +```php +// Automatic authorization logic in each component +if ($this->canGate && $this->canResource && $this->autoDisable) { + $hasPermission = Gate::allows($this->canGate, $this->canResource); + + if (! $hasPermission) { + $this->disabled = true; + // For Checkbox: also sets $this->instantSave = false; + } +} +``` + +## Usage Patterns + +### ✅ Recommended: Single Line Pattern + +**Before (Verbose, 6+ lines per element):** +```html +@can('update', $application) + + + Save +@else + + +@endcan +``` + +**After (Clean, 1 line per element):** +```html + + +Save +``` + +**Result: 90% code reduction!** + +### Component-Specific Examples + +#### Input Fields +```html + + + + + + + + +``` + +#### Select Dropdowns +```html + + + + + + + + + + @foreach($servers as $server) + + @endforeach + +``` + +#### Checkboxes with InstantSave +```html + + + + + + + + +``` + +#### Textareas +```html + + + + + +``` + +#### Buttons +```html + + + Save Configuration + + + + + Deploy Application + + + + + Delete Application + +``` + +## Advanced Usage + +### Custom Authorization Logic +```html + + +``` + +### Multiple Permission Checks +```html + + +``` + +### Conditional Resources +```html + + + {{ $isEditing ? 'Save Changes' : 'View Details' }} + +``` + +## Supported Gates + +### Resource-Level Gates +- `view` - Read access to resource details +- `update` - Modify resource configuration and settings +- `deploy` - Deploy, restart, or manage resource state +- `delete` - Remove or destroy resource +- `clone` - Duplicate resource to another location + +### Global Gates +- `createAnyResource` - Create new resources of any type +- `manageTeam` - Team administration permissions +- `accessServer` - Server-level access permissions + +## Supported Resources + +### Primary Resources +- `$application` - Application instances and configurations +- `$service` - Docker Compose services and components +- `$database` - Database instances (PostgreSQL, MySQL, etc.) +- `$server` - Physical or virtual server instances + +### Container Resources +- `$project` - Project containers and environments +- `$environment` - Environment-specific configurations +- `$team` - Team and organization contexts + +### Infrastructure Resources +- `$privateKey` - SSH private keys and certificates +- `$source` - Git sources and repositories +- `$destination` - Deployment destinations and targets + +## Component Behavior + +### Input Components (Input, Select, Textarea) +When authorization fails: +- **disabled = true** - Field becomes non-editable +- **Visual styling** - Opacity reduction and disabled cursor +- **Form submission** - Values are ignored in forms +- **User feedback** - Clear visual indication of restricted access + +### Checkbox Components +When authorization fails: +- **disabled = true** - Checkbox becomes non-clickable +- **instantSave = false** - Automatic saving is disabled +- **State preservation** - Current value is maintained but read-only +- **Visual styling** - Disabled appearance with reduced opacity + +### Button Components +When authorization fails: +- **disabled = true** - Button becomes non-clickable +- **Event blocking** - Click handlers are ignored +- **Visual styling** - Disabled appearance and cursor +- **Loading states** - Loading indicators are disabled + +## Migration Guide + +### Converting Existing Forms + +**Old Pattern:** +```html +
+ @can('update', $application) + + ... + + Save + @else + + ... + + @endcan + +``` + +**New Pattern:** +```html +
+ + ... + + Save + +``` + +### Gradual Migration Strategy + +1. **Start with new forms** - Use the new pattern for all new components +2. **Convert high-traffic areas** - Migrate frequently used forms first +3. **Batch convert similar forms** - Group similar authorization patterns +4. **Test thoroughly** - Verify authorization behavior matches expectations +5. **Remove old patterns** - Clean up legacy @can/@else blocks + +## Testing Patterns + +### Component Authorization Tests +```php +// Test authorization integration in components +test('input component respects authorization', function () { + $user = User::factory()->member()->create(); + $application = Application::factory()->create(); + + // Member should see disabled input + $component = Livewire::actingAs($user) + ->test(TestComponent::class, [ + 'canGate' => 'update', + 'canResource' => $application + ]); + + expect($component->get('disabled'))->toBeTrue(); +}); + +test('checkbox disables instantSave for unauthorized users', function () { + $user = User::factory()->member()->create(); + $application = Application::factory()->create(); + + $component = Livewire::actingAs($user) + ->test(CheckboxComponent::class, [ + 'instantSave' => true, + 'canGate' => 'update', + 'canResource' => $application + ]); + + expect($component->get('disabled'))->toBeTrue(); + expect($component->get('instantSave'))->toBeFalse(); +}); +``` + +### Integration Tests +```php +// Test full form authorization behavior +test('application form respects member permissions', function () { + $member = User::factory()->member()->create(); + $application = Application::factory()->create(); + + $this->actingAs($member) + ->get(route('application.edit', $application)) + ->assertSee('disabled') + ->assertDontSee('Save Configuration'); +}); +``` + +## Best Practices + +### Consistent Gate Usage +- Use `update` for configuration changes +- Use `deploy` for operational actions +- Use `view` for read-only access +- Use `delete` for destructive actions + +### Resource Context +- Always pass the specific resource being acted upon +- Use team context for creation permissions +- Consider nested resource relationships + +### Error Handling +- Provide clear feedback for disabled components +- Use helper text to explain permission requirements +- Consider tooltips for disabled buttons + +### Performance +- Authorization checks are cached per request +- Use eager loading for resource relationships +- Consider query optimization for complex permissions + +## Common Patterns + +### Application Configuration Forms +```html + + +... + +Save +``` + +### Service Configuration Forms +```html + + + + +Save + + + + + +@can('update', $service) + +@endcan +``` + +### Server Management Forms +```html + + +... +Delete Server +``` + +### Resource Creation Forms +```html + + +... +Create Application +``` \ No newline at end of file diff --git a/.ai/patterns/frontend-patterns.md b/.ai/patterns/frontend-patterns.md new file mode 100644 index 0000000000..675881608c --- /dev/null +++ b/.ai/patterns/frontend-patterns.md @@ -0,0 +1,696 @@ +# Coolify Frontend Architecture & Patterns + +## Frontend Philosophy + +Coolify uses a **server-side first** approach with minimal JavaScript, leveraging Livewire for reactivity and Alpine.js for lightweight client-side interactions. + +## Core Frontend Stack + +### Livewire 3.5+ (Primary Framework) +- **Server-side rendering** with reactive components +- **Real-time updates** without page refreshes +- **State management** handled on the server +- **WebSocket integration** for live updates + +### Alpine.js (Client-Side Interactivity) +- **Lightweight JavaScript** for DOM manipulation +- **Declarative directives** in HTML +- **Component-like behavior** without build steps +- **Perfect companion** to Livewire + +### Tailwind CSS 4.1+ (Styling) +- **Utility-first** CSS framework +- **Custom design system** for deployment platform +- **Responsive design** built-in +- **Dark mode support** + +## Livewire Component Structure + +### Location: [app/Livewire/](mdc:app/Livewire) + +#### Core Application Components +- **[Dashboard.php](mdc:app/Livewire/Dashboard.php)** - Main dashboard interface +- **[ActivityMonitor.php](mdc:app/Livewire/ActivityMonitor.php)** - Real-time activity tracking +- **[MonacoEditor.php](mdc:app/Livewire/MonacoEditor.php)** - Code editor component + +#### Server Management +- **Server/** directory - Server configuration and monitoring +- Real-time server status updates +- SSH connection management +- Resource monitoring + +#### Project & Application Management +- **Project/** directory - Project organization +- Application deployment interfaces +- Environment variable management +- Service configuration + +#### Settings & Configuration +- **Settings/** directory - System configuration +- **[SettingsEmail.php](mdc:app/Livewire/SettingsEmail.php)** - Email notification setup +- **[SettingsOauth.php](mdc:app/Livewire/SettingsOauth.php)** - OAuth provider configuration +- **[SettingsBackup.php](mdc:app/Livewire/SettingsBackup.php)** - Backup configuration + +#### User & Team Management +- **Team/** directory - Team collaboration features +- **Profile/** directory - User profile management +- **Security/** directory - Security settings + +## Blade Template Organization + +### Location: [resources/views/](mdc:resources/views) + +#### Layout Structure +- **layouts/** - Base layout templates +- **components/** - Reusable UI components +- **livewire/** - Livewire component views + +#### Feature-Specific Views +- **server/** - Server management interfaces +- **auth/** - Authentication pages +- **emails/** - Email templates +- **errors/** - Error pages + +## Interactive Components + +### Monaco Editor Integration +- **Code editing** for configuration files +- **Syntax highlighting** for multiple languages +- **Live validation** and error detection +- **Integration** with deployment process + +### Terminal Emulation (XTerm.js) +- **Real-time terminal** access to servers +- **WebSocket-based** communication +- **Multi-session** support +- **Secure connection** through SSH + +### Real-Time Updates +- **WebSocket connections** via Laravel Echo +- **Live deployment logs** streaming +- **Server monitoring** with live metrics +- **Activity notifications** in real-time + +## Alpine.js Patterns + +### Common Directives Used +```html + +
+ + + +``` + +## Tailwind CSS Patterns + +### Design System +- **Consistent spacing** using Tailwind scale +- **Color palette** optimized for deployment platform +- **Typography** hierarchy for technical content +- **Component classes** for reusable elements + +### Responsive Design +```html + +
+ +
+``` + +### Dark Mode Support +```html + +
+ +
+``` + +## Build Process + +### Vite Configuration ([vite.config.js](mdc:vite.config.js)) +- **Fast development** with hot module replacement +- **Optimized production** builds +- **Asset versioning** for cache busting +- **CSS processing** with PostCSS + +### Asset Compilation +```bash +# Development +npm run dev + +# Production build +npm run build +``` + +## State Management Patterns + +### Server-Side State (Livewire) +- **Component properties** for persistent state +- **Session storage** for user preferences +- **Database models** for application state +- **Cache layer** for performance + +### Client-Side State (Alpine.js) +- **Local component state** for UI interactions +- **Form validation** and user feedback +- **Modal and dropdown** state management +- **Temporary UI states** (loading, hover, etc.) + +## Real-Time Features + +### WebSocket Integration +```php +// Livewire component with real-time updates +class ActivityMonitor extends Component +{ + public function getListeners() + { + return [ + 'deployment.started' => 'refresh', + 'deployment.finished' => 'refresh', + 'server.status.changed' => 'updateServerStatus', + ]; + } +} +``` + +### Event Broadcasting +- **Laravel Echo** for client-side WebSocket handling +- **Pusher protocol** for real-time communication +- **Private channels** for user-specific events +- **Presence channels** for collaborative features + +## Performance Patterns + +### Lazy Loading +```php +// Livewire lazy loading +class ServerList extends Component +{ + public function placeholder() + { + return view('components.loading-skeleton'); + } +} +``` + +### Caching Strategies +- **Fragment caching** for expensive operations +- **Image optimization** with lazy loading +- **Asset bundling** and compression +- **CDN integration** for static assets + +## Enhanced Form Components + +### Built-in Authorization System +Coolify features **enhanced form components** with automatic authorization handling: + +```html + + + +Save + + +@can('update', $application) + +@else + +@endcan +``` + +### Authorization Parameters +```php +// Available on all form components (Input, Select, Textarea, Checkbox, Button) +public ?string $canGate = null; // Gate name: 'update', 'view', 'deploy', 'delete' +public mixed $canResource = null; // Resource model instance to check against +public bool $autoDisable = true; // Automatically disable if no permission (default: true) +``` + +### Benefits +- **90% code reduction** for authorization-protected forms +- **Consistent security** across all form components +- **Automatic disabling** for unauthorized users +- **Smart behavior** (disables instantSave on checkboxes for unauthorized users) + +For complete documentation, see **[form-components.md](.ai/patterns/form-components.md)** + +## Form Handling Patterns + +### Livewire Component Data Synchronization Pattern + +**IMPORTANT**: All Livewire components must use the **manual `syncData()` pattern** for synchronizing component properties with Eloquent models. + +#### Property Naming Convention +- **Component properties**: Use camelCase (e.g., `$gitRepository`, `$isStatic`) +- **Database columns**: Use snake_case (e.g., `git_repository`, `is_static`) +- **View bindings**: Use camelCase matching component properties (e.g., `id="gitRepository"`) + +#### The syncData() Method Pattern + +```php +use Livewire\Attributes\Validate; +use Illuminate\Foundation\Auth\Access\AuthorizesRequests; + +class MyComponent extends Component +{ + use AuthorizesRequests; + + public Application $application; + + // Properties with validation attributes + #[Validate(['required'])] + public string $name; + + #[Validate(['string', 'nullable'])] + public ?string $description = null; + + #[Validate(['boolean', 'required'])] + public bool $isStatic = false; + + public function mount() + { + $this->authorize('view', $this->application); + $this->syncData(); // Load from model + } + + public function syncData(bool $toModel = false): void + { + if ($toModel) { + $this->validate(); + + // Sync TO model (camelCase → snake_case) + $this->application->name = $this->name; + $this->application->description = $this->description; + $this->application->is_static = $this->isStatic; + + $this->application->save(); + } else { + // Sync FROM model (snake_case → camelCase) + $this->name = $this->application->name; + $this->description = $this->application->description; + $this->isStatic = $this->application->is_static; + } + } + + public function submit() + { + $this->authorize('update', $this->application); + $this->syncData(toModel: true); // Save to model + $this->dispatch('success', 'Saved successfully.'); + } +} +``` + +#### Validation with #[Validate] Attributes + +All component properties should have `#[Validate]` attributes: + +```php +// Boolean properties +#[Validate(['boolean'])] +public bool $isEnabled = false; + +// Required strings +#[Validate(['string', 'required'])] +public string $name; + +// Nullable strings +#[Validate(['string', 'nullable'])] +public ?string $description = null; + +// With constraints +#[Validate(['integer', 'min:1'])] +public int $timeout; +``` + +#### Benefits of syncData() Pattern + +- **Explicit Control**: Clear visibility of what's being synchronized +- **Type Safety**: #[Validate] attributes provide compile-time validation info +- **Easy Debugging**: Single method to check for data flow issues +- **Maintainability**: All sync logic in one place +- **Flexibility**: Can add custom logic (encoding, transformations, etc.) + +#### Creating New Form Components with syncData() + +#### Step-by-Step Component Creation Guide + +**Step 1: Define properties in camelCase with #[Validate] attributes** +```php +use Livewire\Attributes\Validate; +use Illuminate\Foundation\Auth\Access\AuthorizesRequests; +use Livewire\Component; + +class MyFormComponent extends Component +{ + use AuthorizesRequests; + + // The model we're syncing with + public Application $application; + + // Component properties in camelCase with validation + #[Validate(['string', 'required'])] + public string $name; + + #[Validate(['string', 'nullable'])] + public ?string $gitRepository = null; + + #[Validate(['string', 'nullable'])] + public ?string $installCommand = null; + + #[Validate(['boolean'])] + public bool $isStatic = false; +} +``` + +**Step 2: Implement syncData() method** +```php +public function syncData(bool $toModel = false): void +{ + if ($toModel) { + $this->validate(); + + // Sync TO model (component camelCase → database snake_case) + $this->application->name = $this->name; + $this->application->git_repository = $this->gitRepository; + $this->application->install_command = $this->installCommand; + $this->application->is_static = $this->isStatic; + + $this->application->save(); + } else { + // Sync FROM model (database snake_case → component camelCase) + $this->name = $this->application->name; + $this->gitRepository = $this->application->git_repository; + $this->installCommand = $this->application->install_command; + $this->isStatic = $this->application->is_static; + } +} +``` + +**Step 3: Implement mount() to load initial data** +```php +public function mount() +{ + $this->authorize('view', $this->application); + $this->syncData(); // Load data from model to component properties +} +``` + +**Step 4: Implement action methods with authorization** +```php +public function instantSave() +{ + try { + $this->authorize('update', $this->application); + $this->syncData(toModel: true); // Save component properties to model + $this->dispatch('success', 'Settings saved.'); + } catch (\Throwable $e) { + return handleError($e, $this); + } +} + +public function submit() +{ + try { + $this->authorize('update', $this->application); + $this->syncData(toModel: true); // Save component properties to model + $this->dispatch('success', 'Changes saved successfully.'); + } catch (\Throwable $e) { + return handleError($e, $this); + } +} +``` + +**Step 5: Create Blade view with camelCase bindings** +```blade +
+
+ + + + + + + + + + Save Changes + + +
+``` + +**Key Points**: +- Use `wire:model="camelCase"` and `id="camelCase"` in Blade views +- Component properties are camelCase, database columns are snake_case +- Always include authorization checks (`authorize()`, `canGate`, `canResource`) +- Use `instantSave` for checkboxes that save immediately without form submission + +#### Special Patterns + +**Pattern 1: Related Models (e.g., Application → Settings)** +```php +public function syncData(bool $toModel = false): void +{ + if ($toModel) { + $this->validate(); + + // Sync main model + $this->application->name = $this->name; + $this->application->save(); + + // Sync related model + $this->application->settings->is_static = $this->isStatic; + $this->application->settings->save(); + } else { + // From main model + $this->name = $this->application->name; + + // From related model + $this->isStatic = $this->application->settings->is_static; + } +} +``` + +**Pattern 2: Custom Encoding/Decoding** +```php +public function syncData(bool $toModel = false): void +{ + if ($toModel) { + $this->validate(); + + // Encode before saving + $this->application->custom_labels = base64_encode($this->customLabels); + $this->application->save(); + } else { + // Decode when loading + $this->customLabels = $this->application->parseContainerLabels(); + } +} +``` + +**Pattern 3: Error Rollback** +```php +public function submit() +{ + $this->authorize('update', $this->resource); + $original = $this->model->getOriginal(); + + try { + $this->syncData(toModel: true); + $this->dispatch('success', 'Saved successfully.'); + } catch (\Throwable $e) { + // Rollback on error + $this->model->setRawAttributes($original); + $this->model->save(); + $this->syncData(); // Reload from model + return handleError($e, $this); + } +} +``` + +#### Property Type Patterns + +**Required Strings** +```php +#[Validate(['string', 'required'])] +public string $name; // No ?, no default, always has value +``` + +**Nullable Strings** +```php +#[Validate(['string', 'nullable'])] +public ?string $description = null; // ?, = null, can be empty +``` + +**Booleans** +```php +#[Validate(['boolean'])] +public bool $isEnabled = false; // Always has default value +``` + +**Integers with Constraints** +```php +#[Validate(['integer', 'min:1'])] +public int $timeout; // Required + +#[Validate(['integer', 'min:1', 'nullable'])] +public ?int $port = null; // Nullable +``` + +#### Testing Checklist + +After creating a new component with syncData(), verify: + +- [ ] All checkboxes save correctly (especially `instantSave` ones) +- [ ] All form inputs persist to database +- [ ] Custom encoded fields (like labels) display correctly if applicable +- [ ] Form validation works for all fields +- [ ] No console errors in browser +- [ ] Authorization checks work (`@can` directives and `authorize()` calls) +- [ ] Error rollback works if exceptions occur +- [ ] Related models save correctly if applicable (e.g., Application + ApplicationSetting) + +#### Common Pitfalls to Avoid + +1. **snake_case in component properties**: Always use camelCase for component properties (e.g., `$gitRepository` not `$git_repository`) +2. **Missing #[Validate] attributes**: Every property should have validation attributes for type safety +3. **Forgetting to call syncData()**: Must call `syncData()` in `mount()` to load initial data +4. **Missing authorization**: Always use `authorize()` in methods and `canGate`/`canResource` in views +5. **View binding mismatch**: Use camelCase in Blade (e.g., `id="gitRepository"` not `id="git_repository"`) +6. **wire:model vs wire:model.live**: Use `.live` for `instantSave` checkboxes to avoid timing issues +7. **Validation sync**: If using `rules()` method, keep it in sync with `#[Validate]` attributes +8. **Related models**: Don't forget to save both main and related models in syncData() method + +### Livewire Forms +```php +class ServerCreateForm extends Component +{ + public $name; + public $ip; + + protected $rules = [ + 'name' => 'required|min:3', + 'ip' => 'required|ip', + ]; + + public function save() + { + $this->validate(); + // Save logic + } +} +``` + +### Real-Time Validation +- **Live validation** as user types +- **Server-side validation** rules +- **Error message** display +- **Success feedback** patterns + +## Component Communication + +### Parent-Child Communication +```php +// Parent component +$this->emit('serverCreated', $server->id); + +// Child component +protected $listeners = ['serverCreated' => 'refresh']; +``` + +### Cross-Component Events +- **Global events** for application-wide updates +- **Scoped events** for feature-specific communication +- **Browser events** for JavaScript integration + +## Error Handling & UX + +### Loading States +- **Skeleton screens** during data loading +- **Progress indicators** for long operations +- **Optimistic updates** with rollback capability + +### Error Display +- **Toast notifications** for user feedback +- **Inline validation** errors +- **Global error** handling +- **Retry mechanisms** for failed operations + +## Accessibility Patterns + +### ARIA Labels and Roles +```html + +``` + +### Keyboard Navigation +- **Tab order** management +- **Keyboard shortcuts** for power users +- **Focus management** in modals and forms +- **Screen reader** compatibility + +## Mobile Optimization + +### Touch-Friendly Interface +- **Larger tap targets** for mobile devices +- **Swipe gestures** where appropriate +- **Mobile-optimized** forms and navigation + +### Progressive Enhancement +- **Core functionality** works without JavaScript +- **Enhanced experience** with JavaScript enabled +- **Offline capabilities** where possible diff --git a/.ai/patterns/security-patterns.md b/.ai/patterns/security-patterns.md new file mode 100644 index 0000000000..ac1470ac94 --- /dev/null +++ b/.ai/patterns/security-patterns.md @@ -0,0 +1,1100 @@ +# Coolify Security Architecture & Patterns + +## Security Philosophy + +Coolify implements **defense-in-depth security** with multiple layers of protection including authentication, authorization, encryption, network isolation, and secure deployment practices. + +## Authentication Architecture + +### Multi-Provider Authentication +- **[Laravel Fortify](mdc:config/fortify.php)** - Core authentication scaffolding (4.9KB, 149 lines) +- **[Laravel Sanctum](mdc:config/sanctum.php)** - API token authentication (2.4KB, 69 lines) +- **[Laravel Socialite](mdc:config/services.php)** - OAuth provider integration + +### OAuth Integration +- **[OauthSetting.php](mdc:app/Models/OauthSetting.php)** - OAuth provider configurations +- **Supported Providers**: + - Google OAuth + - Microsoft Azure AD + - Clerk + - Authentik + - Discord + - GitHub (via GitHub Apps) + - GitLab + +### Authentication Models +```php +// User authentication with team-based access +class User extends Authenticatable +{ + use HasApiTokens, HasFactory, Notifiable; + + protected $fillable = [ + 'name', 'email', 'password' + ]; + + protected $hidden = [ + 'password', 'remember_token' + ]; + + protected $casts = [ + 'email_verified_at' => 'datetime', + 'password' => 'hashed', + ]; + + public function teams(): BelongsToMany + { + return $this->belongsToMany(Team::class) + ->withPivot('role') + ->withTimestamps(); + } + + public function currentTeam(): BelongsTo + { + return $this->belongsTo(Team::class, 'current_team_id'); + } +} +``` + +## Authorization & Access Control + +### Enhanced Form Component Authorization System + +Coolify now features a **centralized authorization system** built into all form components (`Input`, `Select`, `Textarea`, `Checkbox`, `Button`) that automatically handles permission-based UI control. + +#### Component Authorization Parameters +```php +// Available on all form components +public ?string $canGate = null; // Gate name (e.g., 'update', 'view', 'delete') +public mixed $canResource = null; // Resource to check against (model instance) +public bool $autoDisable = true; // Auto-disable if no permission (default: true) +``` + +#### Smart Authorization Logic +```php +// Automatic authorization handling in component constructor +if ($this->canGate && $this->canResource && $this->autoDisable) { + $hasPermission = Gate::allows($this->canGate, $this->canResource); + + if (! $hasPermission) { + $this->disabled = true; + // For Checkbox: also disables instantSave + } +} +``` + +#### Usage Examples + +**✅ Recommended Pattern (Single Line):** +```html + + + + + + + + + + + + + + + Save Configuration + +``` + +**❌ Old Pattern (Verbose, Deprecated):** +```html + +@can('update', $application) + + Save +@else + +@endcan +``` + +#### Advanced Usage with Custom Control + +**Custom Authorization Logic:** +```html + + +``` + +**Multiple Permission Checks:** +```html + + +``` + +#### Supported Gates and Resources + +**Common Gates:** +- `view` - Read access to resource +- `update` - Modify resource configuration +- `deploy` - Deploy/restart resource +- `delete` - Remove resource +- `createAnyResource` - Create new resources + +**Resource Types:** +- `Application` - Application instances +- `Service` - Docker Compose services +- `Server` - Server instances +- `Project` - Project containers +- `Environment` - Environment contexts +- `Database` - Database instances + +#### Benefits + +**🔥 Massive Code Reduction:** +- **90% less code** for authorization-protected forms +- **Single line** instead of 6-12 lines per form element +- **No more @can/@else blocks** cluttering templates + +**🛡️ Consistent Security:** +- **Unified authorization logic** across all form components +- **Automatic disabling** for unauthorized users +- **Smart behavior** (like disabling instantSave on checkboxes) + +**🎨 Better UX:** +- **Consistent disabled styling** across all components +- **Proper visual feedback** for restricted access +- **Clean, professional interface** + +#### Implementation Details + +**Component Enhancement:** +```php +// Enhanced in all form components +use Illuminate\Support\Facades\Gate; + +public function __construct( + // ... existing parameters + public ?string $canGate = null, + public mixed $canResource = null, + public bool $autoDisable = true, +) { + // Handle authorization-based disabling + if ($this->canGate && $this->canResource && $this->autoDisable) { + $hasPermission = Gate::allows($this->canGate, $this->canResource); + + if (! $hasPermission) { + $this->disabled = true; + // For Checkbox: $this->instantSave = false; + } + } +} +``` + +**Backward Compatibility:** +- All existing form components continue to work unchanged +- New authorization parameters are optional +- Legacy @can/@else patterns still function but are discouraged + +### Custom Component Authorization Patterns + +When dealing with **custom Alpine.js components** or complex UI elements that don't use the standard `x-forms.*` components, manual authorization protection is required since the automatic `canGate` system only applies to enhanced form components. + +#### Common Custom Components Requiring Manual Protection + +**⚠️ Custom Components That Need Manual Authorization:** +- Custom dropdowns/selects with Alpine.js +- Complex form widgets with JavaScript interactions +- Multi-step wizards or dynamic forms +- Third-party component integrations +- Custom date/time pickers +- File upload components with drag-and-drop + +#### Manual Authorization Pattern + +**✅ Proper Manual Authorization:** +```html + +
+
+ + +
+ @can('update', $resource) + +
+ +
+ + +
+
+ @else + +
+ + + + +
+ @endcan +
+``` + +#### Implementation Checklist + +When implementing authorization for custom components: + +**🔍 1. Identify Custom Components:** +- Look for Alpine.js `x-data` declarations +- Find components not using `x-forms.*` prefix +- Check for JavaScript-heavy interactions +- Review complex form widgets + +**🛡️ 2. Wrap with Authorization:** +- Use `@can('gate', $resource)` / `@else` / `@endcan` structure +- Provide full functionality in the `@can` block +- Create disabled/readonly version in the `@else` block + +**🎨 3. Design Disabled State:** +- Apply `readonly disabled` attributes to inputs +- Add `opacity-50 cursor-not-allowed` classes for visual feedback +- Remove interactive JavaScript behaviors +- Show current value or appropriate placeholder + +**🔒 4. Backend Protection:** +- Ensure corresponding Livewire methods check authorization +- Add `$this->authorize('gate', $resource)` in relevant methods +- Validate permissions before processing any changes + +#### Real-World Examples + +**Custom Date Range Picker:** +```html +@can('update', $application) +
+ +
+@else +
+ + +
+@endcan +``` + +**Multi-Select Component:** +```html +@can('update', $server) +
+ +
+@else +
+ @foreach($selectedValues as $value) +
+ {{ $value }} +
+ @endforeach +
+@endcan +``` + +**File Upload Widget:** +```html +@can('update', $application) +
+ +
+@else +
+

File upload restricted

+ @if($currentFile) +

Current: {{ $currentFile }}

+ @endif +
+@endcan +``` + +#### Key Principles + +**🎯 Consistency:** +- Maintain similar visual styling between enabled/disabled states +- Use consistent disabled patterns across the application +- Apply the same opacity and cursor styling + +**🔐 Security First:** +- Always implement backend authorization checks +- Never rely solely on frontend hiding/disabling +- Validate permissions on every server action + +**💡 User Experience:** +- Show current values in disabled state when appropriate +- Provide clear visual feedback about restricted access +- Maintain layout stability between states + +**🚀 Performance:** +- Minimize Alpine.js initialization for disabled components +- Avoid loading unnecessary JavaScript for unauthorized users +- Use simple HTML structures for read-only states + +### Team-Based Multi-Tenancy +- **[Team.php](mdc:app/Models/Team.php)** - Multi-tenant organization structure (8.9KB, 308 lines) +- **[TeamInvitation.php](mdc:app/Models/TeamInvitation.php)** - Secure team collaboration +- **Role-based permissions** within teams +- **Resource isolation** by team ownership + +### Authorization Patterns +```php +// Team-scoped authorization middleware +class EnsureTeamAccess +{ + public function handle(Request $request, Closure $next): Response + { + $user = $request->user(); + $teamId = $request->route('team'); + + if (!$user->teams->contains('id', $teamId)) { + abort(403, 'Access denied to team resources'); + } + + // Set current team context + $user->switchTeam($teamId); + + return $next($request); + } +} + +// Resource-level authorization policies +class ApplicationPolicy +{ + public function view(User $user, Application $application): bool + { + return $user->teams->contains('id', $application->team_id); + } + + public function deploy(User $user, Application $application): bool + { + return $this->view($user, $application) && + $user->hasTeamPermission($application->team_id, 'deploy'); + } + + public function delete(User $user, Application $application): bool + { + return $this->view($user, $application) && + $user->hasTeamRole($application->team_id, 'admin'); + } +} +``` + +### Global Scopes for Data Isolation +```php +// Automatic team-based filtering +class Application extends Model +{ + protected static function booted(): void + { + static::addGlobalScope('team', function (Builder $builder) { + if (auth()->check() && auth()->user()->currentTeam) { + $builder->whereHas('environment.project', function ($query) { + $query->where('team_id', auth()->user()->currentTeam->id); + }); + } + }); + } +} +``` + +## API Security + +### Token-Based Authentication +```php +// Sanctum API token management +class PersonalAccessToken extends Model +{ + protected $fillable = [ + 'name', 'token', 'abilities', 'expires_at' + ]; + + protected $casts = [ + 'abilities' => 'array', + 'expires_at' => 'datetime', + 'last_used_at' => 'datetime', + ]; + + public function tokenable(): MorphTo + { + return $this->morphTo(); + } + + public function hasAbility(string $ability): bool + { + return in_array('*', $this->abilities) || + in_array($ability, $this->abilities); + } +} +``` + +### API Rate Limiting +```php +// Rate limiting configuration +RateLimiter::for('api', function (Request $request) { + return Limit::perMinute(60)->by($request->user()?->id ?: $request->ip()); +}); + +RateLimiter::for('deployments', function (Request $request) { + return Limit::perMinute(10)->by($request->user()->id); +}); + +RateLimiter::for('webhooks', function (Request $request) { + return Limit::perMinute(100)->by($request->ip()); +}); +``` + +### API Input Validation +```php +// Comprehensive input validation +class StoreApplicationRequest extends FormRequest +{ + public function authorize(): bool + { + return $this->user()->can('create', Application::class); + } + + public function rules(): array + { + return [ + 'name' => 'required|string|max:255|regex:/^[a-zA-Z0-9\-_]+$/', + 'git_repository' => 'required|url|starts_with:https://', + 'git_branch' => 'required|string|max:100|regex:/^[a-zA-Z0-9\-_\/]+$/', + 'server_id' => 'required|exists:servers,id', + 'environment_id' => 'required|exists:environments,id', + 'environment_variables' => 'array', + 'environment_variables.*' => 'string|max:1000', + ]; + } + + public function prepareForValidation(): void + { + $this->merge([ + 'name' => strip_tags($this->name), + 'git_repository' => filter_var($this->git_repository, FILTER_SANITIZE_URL), + ]); + } +} +``` + +## SSH Security + +### Private Key Management +- **[PrivateKey.php](mdc:app/Models/PrivateKey.php)** - Secure SSH key storage (6.5KB, 247 lines) +- **Encrypted key storage** in database +- **Key rotation** capabilities +- **Access logging** for key usage + +### SSH Connection Security +```php +class SshConnection +{ + private string $host; + private int $port; + private string $username; + private PrivateKey $privateKey; + + public function __construct(Server $server) + { + $this->host = $server->ip; + $this->port = $server->port; + $this->username = $server->user; + $this->privateKey = $server->privateKey; + } + + public function connect(): bool + { + $connection = ssh2_connect($this->host, $this->port); + + if (!$connection) { + throw new SshConnectionException('Failed to connect to server'); + } + + // Use private key authentication + $privateKeyContent = decrypt($this->privateKey->private_key); + $publicKeyContent = decrypt($this->privateKey->public_key); + + if (!ssh2_auth_pubkey_file($connection, $this->username, $publicKeyContent, $privateKeyContent)) { + throw new SshAuthenticationException('SSH authentication failed'); + } + + return true; + } + + public function execute(string $command): string + { + // Sanitize command to prevent injection + $command = escapeshellcmd($command); + + $stream = ssh2_exec($this->connection, $command); + + if (!$stream) { + throw new SshExecutionException('Failed to execute command'); + } + + return stream_get_contents($stream); + } +} +``` + +## Container Security + +### Docker Security Patterns +```php +class DockerSecurityService +{ + public function createSecureContainer(Application $application): array + { + return [ + 'image' => $this->validateImageName($application->docker_image), + 'user' => '1000:1000', // Non-root user + 'read_only' => true, + 'no_new_privileges' => true, + 'security_opt' => [ + 'no-new-privileges:true', + 'apparmor:docker-default' + ], + 'cap_drop' => ['ALL'], + 'cap_add' => ['CHOWN', 'SETUID', 'SETGID'], // Minimal capabilities + 'tmpfs' => [ + '/tmp' => 'rw,noexec,nosuid,size=100m', + '/var/tmp' => 'rw,noexec,nosuid,size=50m' + ], + 'ulimits' => [ + 'nproc' => 1024, + 'nofile' => 1024 + ] + ]; + } + + private function validateImageName(string $image): string + { + // Validate image name against allowed registries + $allowedRegistries = ['docker.io', 'ghcr.io', 'quay.io']; + + $parser = new DockerImageParser(); + $parsed = $parser->parse($image); + + if (!in_array($parsed['registry'], $allowedRegistries)) { + throw new SecurityException('Image registry not allowed'); + } + + return $image; + } +} +``` + +### Network Isolation +```yaml +# Docker Compose security configuration +version: '3.8' +services: + app: + image: ${APP_IMAGE} + networks: + - app-network + security_opt: + - no-new-privileges:true + - apparmor:docker-default + read_only: true + tmpfs: + - /tmp:rw,noexec,nosuid,size=100m + cap_drop: + - ALL + cap_add: + - CHOWN + - SETUID + - SETGID + +networks: + app-network: + driver: bridge + internal: true + ipam: + config: + - subnet: 172.20.0.0/16 +``` + +## SSL/TLS Security + +### Certificate Management +- **[SslCertificate.php](mdc:app/Models/SslCertificate.php)** - SSL certificate automation +- **Let's Encrypt** integration for free certificates +- **Automatic renewal** and monitoring +- **Custom certificate** upload support + +### SSL Configuration +```php +class SslCertificateService +{ + public function generateCertificate(Application $application): SslCertificate + { + $domains = $this->validateDomains($application->getAllDomains()); + + $certificate = SslCertificate::create([ + 'application_id' => $application->id, + 'domains' => $domains, + 'provider' => 'letsencrypt', + 'status' => 'pending' + ]); + + // Generate certificate using ACME protocol + $acmeClient = new AcmeClient(); + $certData = $acmeClient->generateCertificate($domains); + + $certificate->update([ + 'certificate' => encrypt($certData['certificate']), + 'private_key' => encrypt($certData['private_key']), + 'chain' => encrypt($certData['chain']), + 'expires_at' => $certData['expires_at'], + 'status' => 'active' + ]); + + return $certificate; + } + + private function validateDomains(array $domains): array + { + foreach ($domains as $domain) { + if (!filter_var($domain, FILTER_VALIDATE_DOMAIN)) { + throw new InvalidDomainException("Invalid domain: {$domain}"); + } + + // Check domain ownership + if (!$this->verifyDomainOwnership($domain)) { + throw new DomainOwnershipException("Domain ownership verification failed: {$domain}"); + } + } + + return $domains; + } +} +``` + +## Environment Variable Security + +### Secure Configuration Management +```php +class EnvironmentVariable extends Model +{ + protected $fillable = [ + 'key', 'value', 'is_secret', 'application_id' + ]; + + protected $casts = [ + 'is_secret' => 'boolean', + 'value' => 'encrypted' // Automatic encryption for sensitive values + ]; + + public function setValueAttribute($value): void + { + // Automatically encrypt sensitive environment variables + if ($this->isSensitiveKey($this->key)) { + $this->attributes['value'] = encrypt($value); + $this->attributes['is_secret'] = true; + } else { + $this->attributes['value'] = $value; + } + } + + public function getValueAttribute($value): string + { + if ($this->is_secret) { + return decrypt($value); + } + + return $value; + } + + private function isSensitiveKey(string $key): bool + { + $sensitivePatterns = [ + 'PASSWORD', 'SECRET', 'KEY', 'TOKEN', 'API_KEY', + 'DATABASE_URL', 'REDIS_URL', 'PRIVATE', 'CREDENTIAL', + 'AUTH', 'CERTIFICATE', 'ENCRYPTION', 'SALT', 'HASH', + 'OAUTH', 'JWT', 'BEARER', 'ACCESS', 'REFRESH' + ]; + + foreach ($sensitivePatterns as $pattern) { + if (str_contains(strtoupper($key), $pattern)) { + return true; + } + } + + return false; + } +} +``` + +## Webhook Security + +### Webhook Signature Verification +```php +class WebhookSecurityService +{ + public function verifyGitHubSignature(Request $request, string $secret): bool + { + $signature = $request->header('X-Hub-Signature-256'); + + if (!$signature) { + return false; + } + + $expectedSignature = 'sha256=' . hash_hmac('sha256', $request->getContent(), $secret); + + return hash_equals($expectedSignature, $signature); + } + + public function verifyGitLabSignature(Request $request, string $secret): bool + { + $signature = $request->header('X-Gitlab-Token'); + + return hash_equals($secret, $signature); + } + + public function validateWebhookPayload(array $payload): array + { + // Sanitize and validate webhook payload + $validator = Validator::make($payload, [ + 'repository.clone_url' => 'required|url|starts_with:https://', + 'ref' => 'required|string|max:255', + 'head_commit.id' => 'required|string|size:40', // Git SHA + 'head_commit.message' => 'required|string|max:1000' + ]); + + if ($validator->fails()) { + throw new InvalidWebhookPayloadException('Invalid webhook payload'); + } + + return $validator->validated(); + } +} +``` + +## Input Sanitization & Validation + +### XSS Prevention +```php +class SecurityMiddleware +{ + public function handle(Request $request, Closure $next): Response + { + // Sanitize input data + $input = $request->all(); + $sanitized = $this->sanitizeInput($input); + $request->merge($sanitized); + + return $next($request); + } + + private function sanitizeInput(array $input): array + { + foreach ($input as $key => $value) { + if (is_string($value)) { + // Remove potentially dangerous HTML tags + $input[$key] = strip_tags($value, '


'); + + // Escape special characters + $input[$key] = htmlspecialchars($input[$key], ENT_QUOTES, 'UTF-8'); + } elseif (is_array($value)) { + $input[$key] = $this->sanitizeInput($value); + } + } + + return $input; + } +} +``` + +### SQL Injection Prevention +```php +// Always use parameterized queries and Eloquent ORM +class ApplicationRepository +{ + public function findByName(string $name): ?Application + { + // Safe: Uses parameter binding + return Application::where('name', $name)->first(); + } + + public function searchApplications(string $query): Collection + { + // Safe: Eloquent handles escaping + return Application::where('name', 'LIKE', "%{$query}%") + ->orWhere('description', 'LIKE', "%{$query}%") + ->get(); + } + + // NEVER do this - vulnerable to SQL injection + // public function unsafeSearch(string $query): Collection + // { + // return DB::select("SELECT * FROM applications WHERE name LIKE '%{$query}%'"); + // } +} +``` + +## Audit Logging & Monitoring + +### Activity Logging +```php +// Using Spatie Activity Log package +class Application extends Model +{ + use LogsActivity; + + protected static $logAttributes = [ + 'name', 'git_repository', 'git_branch', 'fqdn' + ]; + + protected static $logOnlyDirty = true; + + public function getDescriptionForEvent(string $eventName): string + { + return "Application {$this->name} was {$eventName}"; + } +} + +// Custom security events +class SecurityEventLogger +{ + public function logFailedLogin(string $email, string $ip): void + { + activity('security') + ->withProperties([ + 'email' => $email, + 'ip' => $ip, + 'user_agent' => request()->userAgent() + ]) + ->log('Failed login attempt'); + } + + public function logSuspiciousActivity(User $user, string $activity): void + { + activity('security') + ->causedBy($user) + ->withProperties([ + 'activity' => $activity, + 'ip' => request()->ip(), + 'timestamp' => now() + ]) + ->log('Suspicious activity detected'); + } +} +``` + +### Security Monitoring +```php +class SecurityMonitoringService +{ + public function detectAnomalousActivity(User $user): bool + { + // Check for unusual login patterns + $recentLogins = $user->activities() + ->where('description', 'like', '%login%') + ->where('created_at', '>=', now()->subHours(24)) + ->get(); + + // Multiple failed attempts + $failedAttempts = $recentLogins->where('description', 'Failed login attempt')->count(); + if ($failedAttempts > 5) { + $this->triggerSecurityAlert($user, 'Multiple failed login attempts'); + return true; + } + + // Login from new location + $uniqueIps = $recentLogins->pluck('properties.ip')->unique(); + if ($uniqueIps->count() > 3) { + $this->triggerSecurityAlert($user, 'Login from multiple IP addresses'); + return true; + } + + return false; + } + + private function triggerSecurityAlert(User $user, string $reason): void + { + // Send security notification + $user->notify(new SecurityAlertNotification($reason)); + + // Log security event + activity('security') + ->causedBy($user) + ->withProperties(['reason' => $reason]) + ->log('Security alert triggered'); + } +} +``` + +## Backup Security + +### Encrypted Backups +```php +class SecureBackupService +{ + public function createEncryptedBackup(ScheduledDatabaseBackup $backup): void + { + $database = $backup->database; + $dumpPath = $this->createDatabaseDump($database); + + // Encrypt backup file + $encryptedPath = $this->encryptFile($dumpPath, $backup->encryption_key); + + // Upload to secure storage + $this->uploadToSecureStorage($encryptedPath, $backup->s3Storage); + + // Clean up local files + unlink($dumpPath); + unlink($encryptedPath); + } + + private function encryptFile(string $filePath, string $key): string + { + $data = file_get_contents($filePath); + $encryptedData = encrypt($data, $key); + + $encryptedPath = $filePath . '.encrypted'; + file_put_contents($encryptedPath, $encryptedData); + + return $encryptedPath; + } +} +``` + +## Security Headers & CORS + +### Security Headers Configuration +```php +// Security headers middleware +class SecurityHeadersMiddleware +{ + public function handle(Request $request, Closure $next): Response + { + $response = $next($request); + + $response->headers->set('X-Content-Type-Options', 'nosniff'); + $response->headers->set('X-Frame-Options', 'DENY'); + $response->headers->set('X-XSS-Protection', '1; mode=block'); + $response->headers->set('Referrer-Policy', 'strict-origin-when-cross-origin'); + $response->headers->set('Permissions-Policy', 'geolocation=(), microphone=(), camera=()'); + + if ($request->secure()) { + $response->headers->set('Strict-Transport-Security', 'max-age=31536000; includeSubDomains'); + } + + return $response; + } +} +``` + +### CORS Configuration +```php +// CORS configuration for API endpoints +return [ + 'paths' => ['api/*', 'webhooks/*'], + 'allowed_methods' => ['GET', 'POST', 'PUT', 'PATCH', 'DELETE'], + 'allowed_origins' => [ + 'https://app.coolify.io', + 'https://*.coolify.io' + ], + 'allowed_origins_patterns' => [], + 'allowed_headers' => ['*'], + 'exposed_headers' => [], + 'max_age' => 0, + 'supports_credentials' => true, +]; +``` + +## Security Testing + +### Security Test Patterns +```php +// Security-focused tests +test('prevents SQL injection in search', function () { + $user = User::factory()->create(); + $maliciousInput = "'; DROP TABLE applications; --"; + + $response = $this->actingAs($user) + ->getJson("/api/v1/applications?search={$maliciousInput}"); + + $response->assertStatus(200); + + // Verify applications table still exists + expect(Schema::hasTable('applications'))->toBeTrue(); +}); + +test('prevents XSS in application names', function () { + $user = User::factory()->create(); + $xssPayload = ''; + + $response = $this->actingAs($user) + ->postJson('/api/v1/applications', [ + 'name' => $xssPayload, + 'git_repository' => 'https://github.com/user/repo.git', + 'server_id' => Server::factory()->create()->id + ]); + + $response->assertStatus(422); +}); + +test('enforces team isolation', function () { + $user1 = User::factory()->create(); + $user2 = User::factory()->create(); + + $team1 = Team::factory()->create(); + $team2 = Team::factory()->create(); + + $user1->teams()->attach($team1); + $user2->teams()->attach($team2); + + $application = Application::factory()->create(['team_id' => $team1->id]); + + $response = $this->actingAs($user2) + ->getJson("/api/v1/applications/{$application->id}"); + + $response->assertStatus(403); +}); +``` diff --git a/app/Actions/Database/PgBackrestRestore.php b/app/Actions/Database/PgBackrestRestore.php new file mode 100644 index 0000000000..6081d6064c --- /dev/null +++ b/app/Actions/Database/PgBackrestRestore.php @@ -0,0 +1,44 @@ + (string) new Cuid2, + 'database_id' => $database->id, + 'database_type' => $database->getMorphClass(), + 'engine' => 'pgbackrest', + 'scheduled_database_backup_execution_id' => $execution?->id, + 'target_label' => $execution?->pgbackrest_label, + 'target_time' => $targetTime, + 'status' => 'pending', + ]); + + PgBackrestRestoreJob::dispatch($database, $restore, $execution, $targetTime); + + return $restore; + } + + public function rules(): array + { + return [ + 'database' => ['required'], + 'targetTime' => ['nullable', 'date'], + ]; + } +} diff --git a/app/Jobs/PgBackrestRestoreJob.php b/app/Jobs/PgBackrestRestoreJob.php new file mode 100644 index 0000000000..101caea23b --- /dev/null +++ b/app/Jobs/PgBackrestRestoreJob.php @@ -0,0 +1,445 @@ +onQueue('high'); + } + + public function handle(): void + { + $stanza = PgBackrestService::getStanzaName($this->database); + $backupTimestamp = time(); + + try { + $this->restore->updateStatus('running', 'Starting PgBackRest restore.'); + + $this->preflight($stanza); + + $this->restore->appendLog('Stopping PostgreSQL container.'); + StopDatabase::run($this->database, dockerCleanup: false); + + $this->restore->appendLog('Backing up current PGDATA before restore.'); + $backupPath = $this->backupCurrentPgData($backupTimestamp); + + try { + $this->restore->appendLog('Capturing PGDATA ownership.'); + $this->getPgDataOwnership(); // Capture ownership before wiping + + $this->restore->appendLog('Clearing PGDATA directory.'); + $this->clearPgData(); + + $this->restore->appendLog('Restoring via PgBackRest sidecar container.'); + $this->runRestoreSidecar($stanza); + + $this->restore->appendLog('Fixing PGDATA ownership.'); + $this->fixPgDataOwnership(); // Restore ownership after sidecar writes + + $this->restore->appendLog('Verifying restored database.'); + $this->verifyRestore(); + + $this->restore->appendLog('Starting PostgreSQL after restore.'); + StartPostgresql::run($this->database); + + if ($backupPath) { + $this->restore->appendLog('Removing backup of previous database.'); + $this->removePgDataBackup($backupPath); + } + + $this->restore->updateStatus('success', 'PgBackRest restore completed successfully.'); + + $team = $this->database->team(); + if ($team) { + $team->notify(new DatabaseRestoreSuccess( + $this->database, + $this->restore->target_label, + $this->targetTime + )); + } + } catch (Throwable $restoreError) { + $this->restore->appendLog('Restore failed, attempting to recover from backup: ' . $restoreError->getMessage()); + if ($backupPath) { + $this->recoverFromBackup($backupPath); + } + throw $restoreError; + } + } catch (Throwable $e) { + Log::error('PgBackRest restore failed', [ + 'database' => $this->database->uuid, + 'restore_id' => $this->restore->uuid, + 'error' => $e->getMessage(), + 'trace' => $e->getTraceAsString(), + ]); + + $this->restore->updateStatus('failed', 'Restore failed: ' . $e->getMessage()); + + $team = $this->database->team(); + if ($team) { + $team->notify(new DatabaseRestoreFailed( + $this->database, + $e->getMessage(), + $this->restore->target_label + )); + } + + try { + $this->restore->appendLog('Attempting to restart PostgreSQL after failed restore.'); + StartPostgresql::run($this->database); + } catch (Throwable $restartError) { + $this->restore->appendLog('Failed to restart PostgreSQL: ' . $restartError->getMessage()); + } + + throw $e; + } + } + + private function preflight(string $stanza): void + { + $this->restore->appendLog('Running pre-flight validation.'); + + if (!$this->database->hasPgBackrestBackups()) { + throw new RuntimeException('No PgBackRest backup configuration found for this database.'); + } + + $backup = $this->database->pgbackrestBackups()->where('enabled', true)->first(); + if (!$backup) { + throw new RuntimeException('No enabled PgBackRest backup configuration found.'); + } + + $s3Repos = $backup->pgbackrestRepos()->where('type', 's3')->where('enabled', true)->get(); + foreach ($s3Repos as $s3Repo) { + $s3 = $s3Repo->s3Storage; + if (!$s3) { + throw new RuntimeException("S3 storage configuration not found for repo {$s3Repo->repo_number}."); + } + + try { + $s3->testConnection(shouldSave: true); + } catch (Throwable $e) { + throw new RuntimeException("S3 connection test failed for repo {$s3Repo->repo_number}: " . $e->getMessage()); + } + } + + $pgdataVolume = $this->database->pgdataVolume(); + if (!$pgdataVolume) { + throw new RuntimeException('PGDATA volume not found.'); + } + + $repoVolume = $this->database->pgbackrestRepoVolume(); + $hasLocalRepo = $backup->hasLocalRepo(); + if (!$repoVolume && $hasLocalRepo) { + throw new RuntimeException('PgBackRest repository volume not found.'); + } + + $this->restore->appendLog('Verifying backup exists in repository via sidecar container.'); + $info = $this->runInfoSidecar($stanza, $backup); + + if (!PgBackrestService::stanzaExists($info)) { + throw new RuntimeException('PgBackRest stanza does not exist or is not healthy.'); + } + + if (!PgBackrestService::hasBackups($info)) { + throw new RuntimeException('No backups found in PgBackRest repository.'); + } + + if ($this->execution && $this->execution->pgbackrest_label) { + $targetBackup = PgBackrestService::findBackupByLabel($info, $this->execution->pgbackrest_label); + if (!$targetBackup) { + throw new RuntimeException("Backup with label '{$this->execution->pgbackrest_label}' not found in repository."); + } + $this->restore->appendLog("Target backup verified: {$this->execution->pgbackrest_label}"); + } else { + $latestBackup = PgBackrestService::getLatestBackup($info); + if ($latestBackup) { + $this->restore->appendLog("Will restore from latest backup: {$latestBackup['label']}"); + } + } + + $this->restore->appendLog('Pre-flight validation passed.'); + } + + private function runInfoSidecar(string $stanza, $backup): array + { + $server = $this->database->destination->server; + $network = $this->database->destination->network; + $containerName = $this->database->uuid; + + // Helper to get info from sidecar. + // Note: Config is generated on fly or mounted. + + $cmd = PgBackrestService::buildSidecarInfoCommand( + $stanza, + $backup, + $network, + $containerName + ); + + $output = instant_remote_process([$cmd], $server, false, false, 120, disableMultiplexing: true); + + if ($output === null || $output === '') { + throw new RuntimeException('Failed to get PgBackRest info - command returned no output. Command: ' . $cmd); + } + + $jsonStart = strpos($output, '['); + if ($jsonStart !== false) { + $output = substr($output, $jsonStart); + } + + $info = PgBackrestService::parseInfoJson($output); + if ($info === null) { + throw new RuntimeException('Failed to parse PgBackRest info output: ' . $output); + } + + return $info; + } + + private function backupCurrentPgData(int $timestamp): ?string + { + $server = $this->database->destination->server; + $pgdataVolume = $this->database->pgdataVolume(); + + if (!$pgdataVolume) { + throw new RuntimeException('PGDATA volume not found.'); + } + + $volumeName = $pgdataVolume->host_path ?: $pgdataVolume->name; + $backupVolumeName = "{$pgdataVolume->name}_backup_{$timestamp}"; + + $sourceMount = escapeshellarg($volumeName . ':/data'); + $checkCmd = 'docker run --rm -v ' . $sourceMount . " alpine sh -c 'test -n \"\$(ls -A /data 2>/dev/null)\" && echo OK || echo EMPTY'"; + $checkResult = instant_remote_process([$checkCmd], $server, false, false, 30, disableMultiplexing: true); + + if (trim($checkResult) === 'EMPTY') { + $this->restore->appendLog('No existing PGDATA to backup (directory is empty).'); + + return null; + } + + instant_remote_process(['docker volume create ' . escapeshellarg($backupVolumeName)], $server, false, false, 30, disableMultiplexing: true); + + $sourceReadOnlyMount = escapeshellarg($volumeName . ':/source:ro'); + $backupMount = escapeshellarg($backupVolumeName . ':/backup'); + $copyCmd = 'docker run --rm -v ' . $sourceReadOnlyMount . ' -v ' . $backupMount . " alpine sh -c 'cp -a /source/. /backup/'"; + instant_remote_process([$copyCmd], $server, true, false, $this->timeout, disableMultiplexing: true); + + $verifyCmd = 'docker run --rm -v ' . $backupMount . " alpine sh -c 'test -n \"\$(ls -A /backup 2>/dev/null)\" && echo OK'"; + $result = instant_remote_process([$verifyCmd], $server, false, false, 30, disableMultiplexing: true); + + if (trim($result) !== 'OK') { + throw new RuntimeException('PGDATA backup verification failed: backup volume is empty or inaccessible.'); + } + + $this->restore->appendLog("PGDATA backed up to volume: {$backupVolumeName}"); + + return $backupVolumeName; + } + + private function recoverFromBackup(string $backupVolumeName): void + { + try { + $server = $this->database->destination->server; + $pgdataVolume = $this->database->pgdataVolume(); + + if (!$pgdataVolume) { + $this->restore->appendLog('Warning: PGDATA volume not found, cannot recover from backup.'); + + return; + } + + $volumeName = $pgdataVolume->host_path ?: $pgdataVolume->name; + + $this->restore->appendLog('Recovering PGDATA from backup...'); + + $dataMount = escapeshellarg($volumeName . ':/data'); + $clearCmd = 'docker run --rm -v ' . $dataMount . " alpine sh -c 'rm -rf /data/* /data/.[!.]* /data/..?* 2>/dev/null || true'"; + instant_remote_process([$clearCmd], $server, false, false, 60, disableMultiplexing: true); + + $sourceMount = escapeshellarg($backupVolumeName . ':/source:ro'); + $copyCmd = 'docker run --rm -v ' . $sourceMount . ' -v ' . $dataMount . " alpine sh -c 'cp -a /source/. /data/'"; + instant_remote_process([$copyCmd], $server, true, false, $this->timeout, disableMultiplexing: true); + + $this->restore->appendLog('PGDATA recovered from backup.'); + } catch (Throwable $e) { + $this->restore->appendLog('Failed to recover from backup: ' . $e->getMessage()); + } + } + + private function removePgDataBackup(string $backupVolumeName): void + { + try { + $server = $this->database->destination->server; + + instant_remote_process(['docker volume rm ' . escapeshellarg($backupVolumeName) . ' 2>/dev/null || true'], $server, false, false, 60, disableMultiplexing: true); + + $this->restore->appendLog('Temporary backup volume removed.'); + } catch (Throwable $e) { + $this->restore->appendLog('Warning: Failed to remove temporary backup volume: ' . $e->getMessage()); + } + } + + private function clearPgData(): void + { + $server = $this->database->destination->server; + $pgdataVolume = $this->database->pgdataVolume(); + + if (!$pgdataVolume) { + throw new RuntimeException('PGDATA volume not found.'); + } + + $mount = $pgdataVolume->host_path ?: $pgdataVolume->name; + + $dataMount = escapeshellarg($mount . ':/data'); + $rmCmd = 'docker run --rm -v ' . $dataMount . " alpine sh -c 'rm -rf /data/* /data/.[!.]* /data/..?* 2>/dev/null || true'"; + instant_remote_process([$rmCmd], $server, false, false, $this->timeout, disableMultiplexing: true); + + $this->restore->appendLog('PGDATA directory cleared.'); + } + + private function verifyRestore(): void + { + try { + $pgdataVolume = $this->database->pgdataVolume(); + if (!$pgdataVolume) { + throw new RuntimeException('PGDATA volume not found.'); + } + + $server = $this->database->destination->server; + $mount = $pgdataVolume->host_path ?: $pgdataVolume->name; + + $dataMount = escapeshellarg($mount . ':/data'); + $checkCmd = 'docker run --rm -v ' . $dataMount . " alpine test -f /data/PG_VERSION && echo 'OK' || echo 'FAIL'"; + $result = instant_remote_process([$checkCmd], $server, false, false, 30, disableMultiplexing: true); + + if (trim($result) !== 'OK') { + throw new RuntimeException('Restored PGDATA does not contain valid PostgreSQL data (PG_VERSION not found).'); + } + + $this->restore->appendLog('Restored database verified successfully.'); + } catch (Throwable $e) { + throw new RuntimeException('Database verification failed: ' . $e->getMessage()); + } + } + + private string $originalOwner = '999:999'; // Default to standard Postgres + + private function getPgDataOwnership(): void + { + $server = $this->database->destination->server; + $pgdataVolume = $this->database->pgdataVolume(); + $pgdataMount = $pgdataVolume->host_path ?: $pgdataVolume->name; + + // Use stat to get uid:gid of the root of pgdata + $cmd = sprintf( + 'docker run --rm -v %s:/var/lib/postgresql/data alpine stat -c "%%u:%%g" /var/lib/postgresql/data', + escapeshellarg($pgdataMount) + ); + + $output = instant_remote_process([$cmd], $server, false, false, 60, disableMultiplexing: true); + + if ($output && preg_match('/^\d+:\d+$/', trim($output))) { + $this->originalOwner = trim($output); + } + } + + private function fixPgDataOwnership(): void + { + $server = $this->database->destination->server; + $pgdataVolume = $this->database->pgdataVolume(); + $pgdataMount = $pgdataVolume->host_path ?: $pgdataVolume->name; + + $cmd = sprintf( + 'docker run --rm -v %s:/var/lib/postgresql/data alpine chown -R %s /var/lib/postgresql/data', + escapeshellarg($pgdataMount), + escapeshellarg($this->originalOwner) + ); + + instant_remote_process([$cmd], $server, true, false, 300, disableMultiplexing: true); + } + + private function runRestoreSidecar(string $stanza): void + { + $server = $this->database->destination->server; + $network = $this->database->destination->network; + + $backup = $this->database->pgbackrestBackups()->where('enabled', true)->first(); + if (!$backup) { + throw new RuntimeException('No enabled PgBackRest backup configuration found.'); + } + + // StopDatabase removes container; resolve mounts explicitly. + $pgdataVolume = $this->database->pgdataVolume(); + $pgdataMount = $pgdataVolume->host_path ?: $pgdataVolume->name; + $mounts = "-v " . escapeshellarg($pgdataMount . ':/var/lib/postgresql/data'); + + $repoVolume = $this->database->pgbackrestRepoVolume(); + if ($repoVolume) { + $repoMount = $repoVolume->host_path ?: $repoVolume->name; + $mounts .= " -v " . escapeshellarg($repoMount . ':/var/lib/pgbackrest'); + } + + $cmd = PgBackrestService::buildSidecarRestoreCommand( + $stanza, + $backup, + $network, + $mounts, + $this->targetTime + ); + + $output = instant_remote_process([$cmd], $server, true, false, $this->timeout, disableMultiplexing: true); + + $this->restore->appendLog('Restore output: ' . substr($output, 0, 2000)); + } + + public function failed(?Throwable $exception): void + { + Log::channel('scheduled-errors')->error('PgBackRest restore permanently failed', [ + 'job' => 'PgBackrestRestoreJob', + 'database' => $this->database->uuid, + 'restore_id' => $this->restore->uuid, + 'error' => $exception?->getMessage(), + 'trace' => $exception?->getTraceAsString(), + ]); + + $this->restore->updateStatus('failed', 'Restore job permanently failed: ' . ($exception?->getMessage() ?? 'Unknown error')); + + $team = $this->database->team(); + if ($team) { + $team->notify(new DatabaseRestoreFailed( + $this->database, + $exception?->getMessage() ?? 'Unknown error', + $this->restore->target_label + )); + } + } +} diff --git a/app/Livewire/Project/Service/Database.php b/app/Livewire/Project/Service/Database.php new file mode 100644 index 0000000000..1e183c6bc7 --- /dev/null +++ b/app/Livewire/Project/Service/Database.php @@ -0,0 +1,228 @@ + 'nullable', + 'description' => 'nullable', + 'image' => 'required', + 'excludeFromStatus' => 'required|boolean', + 'publicPort' => 'nullable|integer', + 'isPublic' => 'required|boolean', + 'isLogDrainEnabled' => 'required|boolean', + ]; + + public function render() + { + return view('livewire.project.service.database'); + } + + public function mount() + { + try { + $this->parameters = get_route_parameters(); + $this->authorize('view', $this->database); + if ($this->database->is_public) { + $this->db_url_public = $this->database->getServiceDatabaseUrl(); + } + $this->refreshFileStorages(); + $this->syncData(false); + } catch (\Throwable $e) { + return handleError($e, $this); + } + } + + private function syncData(bool $toModel = false): void + { + if ($toModel) { + $this->database->human_name = $this->humanName; + $this->database->description = $this->description; + $this->database->image = $this->image; + $this->database->exclude_from_status = $this->excludeFromStatus; + $this->database->public_port = $this->publicPort; + $this->database->is_public = $this->isPublic; + $this->database->is_log_drain_enabled = $this->isLogDrainEnabled; + } else { + $this->humanName = $this->database->human_name; + $this->description = $this->database->description; + $this->image = $this->database->image; + $this->excludeFromStatus = $this->database->exclude_from_status ?? false; + $this->publicPort = $this->database->public_port; + $this->isPublic = $this->database->is_public ?? false; + $this->isLogDrainEnabled = $this->database->is_log_drain_enabled ?? false; + } + } + + public function delete($password) + { + try { + $this->authorize('delete', $this->database); + + if (! verifyPasswordConfirmation($password, $this)) { + return; + } + + $this->database->delete(); + $this->dispatch('success', 'Database deleted.'); + + return redirect()->route('project.service.configuration', $this->parameters); + } catch (\Throwable $e) { + return handleError($e, $this); + } + } + + public function instantSaveExclude() + { + try { + $this->authorize('update', $this->database); + $this->submit(); + } catch (\Throwable $e) { + return handleError($e, $this); + } + } + + public function instantSaveLogDrain() + { + try { + $this->authorize('update', $this->database); + if (! $this->database->service->destination->server->isLogDrainEnabled()) { + $this->isLogDrainEnabled = false; + $this->dispatch('error', 'Log drain is not enabled on the server. Please enable it first.'); + + return; + } + $this->submit(); + $this->dispatch('success', 'You need to restart the service for the changes to take effect.'); + } catch (\Throwable $e) { + return handleError($e, $this); + } + } + + public function convertToApplication() + { + try { + $this->authorize('update', $this->database); + $service = $this->database->service; + $serviceDatabase = $this->database; + + // Check if application with same name already exists + if ($service->applications()->where('name', $serviceDatabase->name)->exists()) { + throw new \Exception('An application with this name already exists.'); + } + + // Create new parameters removing database_uuid + $redirectParams = collect($this->parameters) + ->except('database_uuid') + ->all(); + + DB::transaction(function () use ($service, $serviceDatabase) { + $service->applications()->create([ + 'name' => $serviceDatabase->name, + 'human_name' => $serviceDatabase->human_name, + 'description' => $serviceDatabase->description, + 'exclude_from_status' => $serviceDatabase->exclude_from_status, + 'is_log_drain_enabled' => $serviceDatabase->is_log_drain_enabled, + 'image' => $serviceDatabase->image, + 'service_id' => $service->id, + 'is_migrated' => true, + ]); + $serviceDatabase->delete(); + }); + + return redirect()->route('project.service.configuration', $redirectParams); + } catch (\Throwable $e) { + return handleError($e, $this); + } + } + + public function instantSave() + { + try { + $this->authorize('update', $this->database); + if ($this->isPublic && ! $this->publicPort) { + $this->dispatch('error', 'Public port is required.'); + $this->isPublic = false; + + return; + } + $this->syncData(true); + if ($this->database->is_public) { + if (! str($this->database->status)->startsWith('running')) { + $this->dispatch('error', 'Database must be started to be publicly accessible.'); + $this->isPublic = false; + $this->database->is_public = false; + + return; + } + StartDatabaseProxy::run($this->database); + $this->db_url_public = $this->database->getServiceDatabaseUrl(); + $this->dispatch('success', 'Database is now publicly accessible.'); + } else { + StopDatabaseProxy::run($this->database); + $this->db_url_public = null; + $this->dispatch('success', 'Database is no longer publicly accessible.'); + } + $this->submit(); + } catch (\Throwable $e) { + return handleError($e, $this); + } + } + + public function refreshFileStorages() + { + $this->fileStorages = $this->database->fileStorages()->get(); + } + + public function submit() + { + try { + $this->authorize('update', $this->database); + $this->validate(); + $this->syncData(true); + $this->database->save(); + $this->database->refresh(); + $this->syncData(false); + updateCompose($this->database); + $this->dispatch('success', 'Database saved.'); + } catch (\Throwable $e) { + return handleError($e, $this); + } finally { + $this->dispatch('generateDockerCompose'); + } + } +} diff --git a/app/Livewire/Project/Service/ServiceApplicationView.php b/app/Livewire/Project/Service/ServiceApplicationView.php new file mode 100644 index 0000000000..4302c05fb0 --- /dev/null +++ b/app/Livewire/Project/Service/ServiceApplicationView.php @@ -0,0 +1,345 @@ + 'nullable', + 'description' => 'nullable', + 'fqdn' => 'nullable', + 'image' => 'string|nullable', + 'excludeFromStatus' => 'required|boolean', + 'application.required_fqdn' => 'required|boolean', + 'isLogDrainEnabled' => 'nullable|boolean', + 'isGzipEnabled' => 'nullable|boolean', + 'isStripprefixEnabled' => 'nullable|boolean', + ]; + + public function instantSave() + { + try { + $this->authorize('update', $this->application); + $this->submit(); + } catch (\Throwable $e) { + return handleError($e, $this); + } + } + + public function instantSaveSettings() + { + try { + $this->authorize('update', $this->application); + // Save checkbox states without port validation + $this->application->is_gzip_enabled = $this->isGzipEnabled; + $this->application->is_stripprefix_enabled = $this->isStripprefixEnabled; + $this->application->exclude_from_status = $this->excludeFromStatus; + $this->application->save(); + $this->dispatch('success', 'Settings saved.'); + } catch (\Throwable $e) { + return handleError($e, $this); + } + } + + public function instantSaveAdvanced() + { + try { + $this->authorize('update', $this->application); + if (! $this->application->service->destination->server->isLogDrainEnabled()) { + $this->isLogDrainEnabled = false; + $this->dispatch('error', 'Log drain is not enabled on the server. Please enable it first.'); + + return; + } + // Sync component properties to model + $this->application->human_name = $this->humanName; + $this->application->description = $this->description; + $this->application->fqdn = $this->fqdn; + $this->application->image = $this->image; + $this->application->exclude_from_status = $this->excludeFromStatus; + $this->application->is_log_drain_enabled = $this->isLogDrainEnabled; + $this->application->is_gzip_enabled = $this->isGzipEnabled; + $this->application->is_stripprefix_enabled = $this->isStripprefixEnabled; + $this->application->save(); + $this->dispatch('success', 'You need to restart the service for the changes to take effect.'); + } catch (\Throwable $e) { + return handleError($e, $this); + } + } + + public function delete($password) + { + try { + $this->authorize('delete', $this->application); + + if (! verifyPasswordConfirmation($password, $this)) { + return; + } + + $this->application->delete(); + $this->dispatch('success', 'Application deleted.'); + + return redirect()->route('project.service.configuration', $this->parameters); + } catch (\Throwable $e) { + return handleError($e, $this); + } + } + + public function mount() + { + try { + $this->parameters = get_route_parameters(); + $this->authorize('view', $this->application); + $this->requiredPort = $this->application->getRequiredPort(); + $this->syncData(); + } catch (\Throwable $e) { + return handleError($e, $this); + } + } + + public function confirmRemovePort() + { + $this->forceRemovePort = true; + $this->showPortWarningModal = false; + $this->submit(); + } + + public function cancelRemovePort() + { + $this->showPortWarningModal = false; + $this->syncData(); // Reset to original FQDN + } + + public function syncData(bool $toModel = false): void + { + if ($toModel) { + $this->validate(); + + // Sync to model + $this->application->human_name = $this->humanName; + $this->application->description = $this->description; + $this->application->fqdn = $this->fqdn; + $this->application->image = $this->image; + $this->application->exclude_from_status = $this->excludeFromStatus; + $this->application->is_log_drain_enabled = $this->isLogDrainEnabled; + $this->application->is_gzip_enabled = $this->isGzipEnabled; + $this->application->is_stripprefix_enabled = $this->isStripprefixEnabled; + + $this->application->save(); + } else { + // Sync from model + $this->humanName = $this->application->human_name; + $this->description = $this->application->description; + $this->fqdn = $this->application->fqdn; + $this->image = $this->application->image; + $this->excludeFromStatus = data_get($this->application, 'exclude_from_status', false); + $this->isLogDrainEnabled = data_get($this->application, 'is_log_drain_enabled', false); + $this->isGzipEnabled = data_get($this->application, 'is_gzip_enabled', true); + $this->isStripprefixEnabled = data_get($this->application, 'is_stripprefix_enabled', true); + } + } + + public function convertToDatabase() + { + try { + $this->authorize('update', $this->application); + $service = $this->application->service; + $serviceApplication = $this->application; + + // Check if database with same name already exists + if ($service->databases()->where('name', $serviceApplication->name)->exists()) { + throw new \Exception('A database with this name already exists.'); + } + + $redirectParams = collect($this->parameters) + ->except('database_uuid') + ->all(); + DB::transaction(function () use ($service, $serviceApplication) { + $service->databases()->create([ + 'name' => $serviceApplication->name, + 'human_name' => $serviceApplication->human_name, + 'description' => $serviceApplication->description, + 'exclude_from_status' => $serviceApplication->exclude_from_status, + 'is_log_drain_enabled' => $serviceApplication->is_log_drain_enabled, + 'image' => $serviceApplication->image, + 'service_id' => $service->id, + 'is_migrated' => true, + ]); + $serviceApplication->delete(); + }); + + return redirect()->route('project.service.configuration', $redirectParams); + } catch (\Throwable $e) { + return handleError($e, $this); + } + } + + public function confirmDomainUsage() + { + $this->forceSaveDomains = true; + $this->showDomainConflictModal = false; + $this->submit(); + } + + public function submit() + { + try { + $this->authorize('update', $this->application); + $this->fqdn = str($this->fqdn)->replaceEnd(',', '')->trim()->toString(); + $this->fqdn = str($this->fqdn)->replaceStart(',', '')->trim()->toString(); + $domains = str($this->fqdn)->trim()->explode(',')->map(function ($domain) { + $domain = trim($domain); + Url::fromString($domain, ['http', 'https']); + + return str($domain)->lower(); + }); + $this->fqdn = $domains->unique()->implode(','); + $warning = sslipDomainWarning($this->fqdn); + if ($warning) { + $this->dispatch('warning', __('warning.sslipdomain')); + } + // Sync to model for domain conflict check (without validation) + $this->application->human_name = $this->humanName; + $this->application->description = $this->description; + $this->application->fqdn = $this->fqdn; + $this->application->image = $this->image; + $this->application->exclude_from_status = $this->excludeFromStatus; + $this->application->is_log_drain_enabled = $this->isLogDrainEnabled; + $this->application->is_gzip_enabled = $this->isGzipEnabled; + $this->application->is_stripprefix_enabled = $this->isStripprefixEnabled; + // Check for domain conflicts if not forcing save + if (! $this->forceSaveDomains) { + $result = checkDomainUsage(resource: $this->application); + if ($result['hasConflicts']) { + $this->domainConflicts = $result['conflicts']; + $this->showDomainConflictModal = true; + + return; + } + } else { + // Reset the force flag after using it + $this->forceSaveDomains = false; + } + + // Check for required port + if (! $this->forceRemovePort) { + $requiredPort = $this->application->getRequiredPort(); + + if ($requiredPort !== null) { + // Check if all FQDNs have a port + $fqdns = str($this->fqdn)->trim()->explode(','); + $missingPort = false; + + foreach ($fqdns as $fqdn) { + $fqdn = trim($fqdn); + if (empty($fqdn)) { + continue; + } + + $port = ServiceApplication::extractPortFromUrl($fqdn); + if ($port === null) { + $missingPort = true; + break; + } + } + + if ($missingPort) { + $this->requiredPort = $requiredPort; + $this->showPortWarningModal = true; + + return; + } + } + } else { + // Reset the force flag after using it + $this->forceRemovePort = false; + } + + $this->validate(); + $this->application->save(); + $this->application->refresh(); + $this->syncData(); + updateCompose($this->application); + if (str($this->application->fqdn)->contains(',')) { + $this->dispatch('warning', 'Some services do not support multiple domains, which can lead to problems and is NOT RECOMMENDED.

Only use multiple domains if you know what you are doing.'); + } else { + ! $warning && $this->dispatch('success', 'Service saved.'); + } + $this->dispatch('generateDockerCompose'); + } catch (\Throwable $e) { + $originalFqdn = $this->application->getOriginal('fqdn'); + if ($originalFqdn !== $this->application->fqdn) { + $this->application->fqdn = $originalFqdn; + $this->syncData(); + } + + return handleError($e, $this); + } + } + + public function render() + { + return view('livewire.project.service.service-application-view', [ + 'checkboxes' => [ + ['id' => 'delete_volumes', 'label' => __('resource.delete_volumes')], + ['id' => 'docker_cleanup', 'label' => __('resource.docker_cleanup')], + // ['id' => 'delete_associated_backups_locally', 'label' => 'All backups associated with this Ressource will be permanently deleted from local storage.'], + // ['id' => 'delete_associated_backups_s3', 'label' => 'All backups associated with this Ressource will be permanently deleted from the selected S3 Storage.'], + // ['id' => 'delete_associated_backups_sftp', 'label' => 'All backups associated with this Ressource will be permanently deleted from the selected SFTP Storage.'] + ], + ]); + } +} diff --git a/app/Models/DatabaseRestore.php b/app/Models/DatabaseRestore.php new file mode 100644 index 0000000000..417d008a71 --- /dev/null +++ b/app/Models/DatabaseRestore.php @@ -0,0 +1,80 @@ + 'datetime', + 'finished_at' => 'datetime', + ]; + } + + public function database(): MorphTo + { + return $this->morphTo(); + } + + public function execution(): BelongsTo + { + return $this->belongsTo(ScheduledDatabaseBackupExecution::class, 'scheduled_database_backup_execution_id'); + } + + public function isRunning(): bool + { + return $this->status === 'running'; + } + + public function isPending(): bool + { + return $this->status === 'pending'; + } + + public function isSuccess(): bool + { + return $this->status === 'success'; + } + + public function isFailed(): bool + { + return $this->status === 'failed'; + } + + public function isFinished(): bool + { + return in_array($this->status, ['success', 'failed']); + } + + public function appendLog(string $message, bool $persist = true): void + { + $timestamp = now()->format('Y-m-d H:i:s'); + $this->log = ($this->log ?? '')."[{$timestamp}] {$message}\n"; + + if ($persist) { + $this->save(); + } + } + + public function updateStatus(string $status, ?string $message = null): void + { + $this->status = $status; + + if ($message !== null) { + $this->message = $message; + $this->appendLog($message, persist: false); + } + + if ($this->isFinished()) { + $this->finished_at = now(); + } + + $this->save(); + } +} diff --git a/app/Models/PgbackrestRepo.php b/app/Models/PgbackrestRepo.php new file mode 100644 index 0000000000..0980175e57 --- /dev/null +++ b/app/Models/PgbackrestRepo.php @@ -0,0 +1,72 @@ + 'boolean', + 'repo_number' => 'integer', + 'retention_full' => 'integer', + 'retention_diff' => 'integer', + ]; + } + + protected static function booted(): void + { + static::creating(function ($repo) { + if (empty($repo->uuid)) { + $repo->uuid = (string) new Cuid2; + } + }); + } + + public function scheduledDatabaseBackup(): BelongsTo + { + return $this->belongsTo(ScheduledDatabaseBackup::class); + } + + public function s3Storage(): BelongsTo + { + return $this->belongsTo(S3Storage::class, 's3_storage_id'); + } + + public function isLocal(): bool + { + return $this->type === 'posix'; + } + + public function isS3(): bool + { + return $this->type === 's3'; + } + + public function getRepoKey(): string + { + return "repo{$this->repo_number}"; + } + + public function getDefaultPath(): string + { + if ($this->isS3()) { + $backup = $this->scheduledDatabaseBackup; + $database = $backup?->database; + + return '/'.($database?->uuid ?? 'backup'); + } + + return '/var/lib/pgbackrest'; + } + + public function getEffectivePath(): string + { + return $this->path ?: $this->getDefaultPath(); + } +} diff --git a/app/Models/StandalonePostgresql.php b/app/Models/StandalonePostgresql.php index 5d35f335bf..5de7030f22 100644 --- a/app/Models/StandalonePostgresql.php +++ b/app/Models/StandalonePostgresql.php @@ -43,12 +43,20 @@ protected static function booted() : '/var/lib/postgresql/data'; LocalPersistentVolume::create([ - 'name' => 'postgres-data-'.$database->uuid, + 'name' => 'postgres-data-' . $database->uuid, 'mount_path' => $mountPath, 'host_path' => null, 'resource_id' => $database->id, 'resource_type' => $database->getMorphClass(), ]); + + LocalPersistentVolume::create([ + 'name' => 'postgres-pgbackrest-repo-' . $database->uuid, + 'mount_path' => '/var/lib/pgbackrest', + 'host_path' => null, + 'resource_id' => $database->id, + 'resource_type' => $database->getMorphClass(), + ]); }); static::forceDeleting(function ($database) { $database->persistentStorages()->delete(); @@ -84,7 +92,7 @@ public static function ownedByCurrentTeamCached() public function workdir() { - return database_configuration_dir()."/{$this->uuid}"; + return database_configuration_dir() . "/{$this->uuid}"; } protected function serverStatus(): Attribute @@ -101,7 +109,7 @@ public function deleteConfigurations() $server = data_get($this, 'destination.server'); $workdir = $this->workdir(); if (str($workdir)->endsWith($this->uuid)) { - instant_remote_process(['rm -rf '.$this->workdir()], $server, false); + instant_remote_process(['rm -rf ' . $this->workdir()], $server, false); } } @@ -119,7 +127,7 @@ public function deleteVolumes() public function isConfigurationChanged(bool $save = false) { - $newConfigHash = $this->image.$this->ports_mappings.$this->postgres_initdb_args.$this->postgres_host_auth_method; + $newConfigHash = $this->image . $this->ports_mappings . $this->postgres_initdb_args . $this->postgres_host_auth_method; $newConfigHash .= json_encode($this->environment_variables()->get('value')->sort()); $newConfigHash = md5($newConfigHash); $oldConfigHash = data_get($this, 'config_hash'); @@ -223,16 +231,16 @@ public function isLogDrainEnabled() public function portsMappings(): Attribute { return Attribute::make( - set: fn ($value) => $value === '' ? null : $value, + set: fn($value) => $value === '' ? null : $value, ); } public function portsMappingsArray(): Attribute { return Attribute::make( - get: fn () => is_null($this->ports_mappings) - ? [] - : explode(',', $this->ports_mappings), + get: fn() => is_null($this->ports_mappings) + ? [] + : explode(',', $this->ports_mappings), ); } @@ -245,7 +253,7 @@ public function team() public function databaseType(): Attribute { return new Attribute( - get: fn () => $this->type(), + get: fn() => $this->type(), ); } @@ -344,4 +352,83 @@ public function isBackupSolutionAvailable() { return true; } + + public function hasPgBackrestBackups(): bool + { + return $this->scheduledBackups() + ->where('engine', 'pgbackrest') + ->where('enabled', true) + ->exists(); + } + + public function pgbackrestBackups() + { + return $this->scheduledBackups()->where('engine', 'pgbackrest'); + } + + public function restores() + { + return $this->morphMany(DatabaseRestore::class, 'database'); + } + + public function pgdataVolume(): ?LocalPersistentVolume + { + return $this->persistentStorages() + ->where('mount_path', '/var/lib/postgresql/data') + ->first(); + } + + public function pgbackrestRepoVolume(): ?LocalPersistentVolume + { + return $this->persistentStorages() + ->where('mount_path', '/var/lib/pgbackrest') + ->first(); + } + + public function getCpuMetrics(int $mins = 5) + { + $server = $this->destination->server; + $container_name = $this->uuid; + $from = now()->subMinutes($mins)->toIso8601ZuluString(); + $metrics = instant_remote_process(["docker exec coolify-sentinel sh -c 'curl -H \"Authorization: Bearer {$server->settings->sentinel_token}\" http://localhost:8888/api/container/{$container_name}/cpu/history?from=$from'"], $server, false); + if (str($metrics)->contains('error')) { + $error = json_decode($metrics, true); + $error = data_get($error, 'error', 'Something is not okay, are you okay?'); + if ($error === 'Unauthorized') { + $error = 'Unauthorized, please check your metrics token or restart Sentinel to set a new token.'; + } + throw new \Exception($error); + } + $metrics = json_decode($metrics, true); + $parsedCollection = collect($metrics)->map(function ($metric) { + return [ + (int) $metric['time'], + (float) ($metric['percent'] ?? 0.0), + ]; + }); + + return $parsedCollection->toArray(); + } + + public function getMemoryMetrics(int $mins = 5) + { + $server = $this->destination->server; + $container_name = $this->uuid; + $from = now()->subMinutes($mins)->toIso8601ZuluString(); + $metrics = instant_remote_process(["docker exec coolify-sentinel sh -c 'curl -H \"Authorization: Bearer {$server->settings->sentinel_token}\" http://localhost:8888/api/container/{$container_name}/memory/history?from=$from'"], $server, false); + if (str($metrics)->contains('error')) { + $error = json_decode($metrics, true); + $error = data_get($error, 'error', 'Something is not okay, are you okay?'); + if ($error === 'Unauthorized') { + $error = 'Unauthorized, please check your metrics token or restart Sentinel to set a new token.'; + } + throw new \Exception($error); + } + $metrics = json_decode($metrics, true); + $parsedCollection = collect($metrics)->map(function ($metric) { + return [(int) $metric['time'], (float) $metric['used']]; + }); + + return $parsedCollection->toArray(); + } } diff --git a/app/Notifications/Database/DatabaseRestoreFailed.php b/app/Notifications/Database/DatabaseRestoreFailed.php new file mode 100644 index 0000000000..e6fdca41db --- /dev/null +++ b/app/Notifications/Database/DatabaseRestoreFailed.php @@ -0,0 +1,129 @@ +onQueue('high'); + + $this->name = $database->name; + $this->error = $error; + $this->label = $label; + } + + public function via(object $notifiable): array + { + return $notifiable->getEnabledChannels('backup_failed'); + } + + public function toMail(): MailMessage + { + $mail = new MailMessage; + $mail->subject("Coolify: Database restore failed for {$this->name}"); + $mail->view('emails.database-restore-failed', [ + 'name' => $this->name, + 'error' => $this->error, + 'label' => $this->label, + ]); + + return $mail; + } + + public function toDiscord(): DiscordMessage + { + $description = "Database restore for {$this->name} has failed."; + if ($this->label) { + $description .= " Attempted to restore from backup: {$this->label}"; + } + + $message = new DiscordMessage( + title: ':x: Database restore failed', + description: $description, + color: DiscordMessage::errorColor(), + ); + + $message->addField('Error', $this->error, false); + + return $message; + } + + public function toTelegram(): array + { + $message = "Coolify: Database restore for {$this->name} has failed."; + if ($this->label) { + $message .= " Backup: {$this->label}"; + } + $message .= "\n\nError: {$this->error}"; + + return [ + 'message' => $message, + ]; + } + + public function toPushover(): PushoverMessage + { + $message = "Database restore for {$this->name} has failed."; + if ($this->label) { + $message .= "
Backup: {$this->label}"; + } + $message .= "

Error: {$this->error}"; + + return new PushoverMessage( + title: 'Database restore failed', + level: 'error', + message: $message, + ); + } + + public function toSlack(): SlackMessage + { + $title = 'Database restore failed'; + $description = "Database restore for {$this->name} has failed."; + + if ($this->label) { + $description .= "\n\n*Backup:* {$this->label}"; + } + $description .= "\n\n*Error:* {$this->error}"; + + return new SlackMessage( + title: $title, + description: $description, + color: SlackMessage::errorColor() + ); + } + + public function toWebhook(): array + { + $url = base_url().'/project/'.data_get($this->database, 'environment.project.uuid').'/environment/'.data_get($this->database, 'environment.uuid').'/database/'.$this->database->uuid; + + return [ + 'success' => false, + 'message' => 'Database restore failed', + 'event' => 'restore_failed', + 'database_name' => $this->name, + 'database_uuid' => $this->database->uuid, + 'engine' => 'pgbackrest', + 'backup_label' => $this->label, + 'error' => $this->error, + 'url' => $url, + ]; + } +} diff --git a/app/Notifications/Database/DatabaseRestoreSuccess.php b/app/Notifications/Database/DatabaseRestoreSuccess.php new file mode 100644 index 0000000000..6a001ce917 --- /dev/null +++ b/app/Notifications/Database/DatabaseRestoreSuccess.php @@ -0,0 +1,131 @@ +onQueue('high'); + + $this->name = $database->name; + $this->label = $label; + $this->targetTime = $targetTime; + } + + public function via(object $notifiable): array + { + return $notifiable->getEnabledChannels('backup_success'); + } + + public function toMail(): MailMessage + { + $mail = new MailMessage; + $mail->subject("Coolify: Database restore successful for {$this->name}"); + $mail->view('emails.database-restore-success', [ + 'name' => $this->name, + 'label' => $this->label, + 'target_time' => $this->targetTime, + ]); + + return $mail; + } + + public function toDiscord(): DiscordMessage + { + $description = "Database restore for {$this->name} was successful."; + if ($this->label) { + $description .= " Restored from backup: {$this->label}"; + } + + $message = new DiscordMessage( + title: ':white_check_mark: Database restore successful', + description: $description, + color: DiscordMessage::successColor(), + ); + + if ($this->targetTime) { + $message->addField('Target Time', $this->targetTime, true); + } + + return $message; + } + + public function toTelegram(): array + { + $message = "Coolify: Database restore for {$this->name} was successful."; + if ($this->label) { + $message .= " Restored from backup: {$this->label}"; + } + + return [ + 'message' => $message, + ]; + } + + public function toPushover(): PushoverMessage + { + $message = "Database restore for {$this->name} was successful."; + if ($this->label) { + $message .= "
Backup: {$this->label}"; + } + + return new PushoverMessage( + title: 'Database restore successful', + level: 'success', + message: $message, + ); + } + + public function toSlack(): SlackMessage + { + $title = 'Database restore successful'; + $description = "Database restore for {$this->name} was successful."; + + if ($this->label) { + $description .= "\n\n*Backup:* {$this->label}"; + } + if ($this->targetTime) { + $description .= "\n*Target Time:* {$this->targetTime}"; + } + + return new SlackMessage( + title: $title, + description: $description, + color: SlackMessage::successColor() + ); + } + + public function toWebhook(): array + { + $url = base_url().'/project/'.data_get($this->database, 'environment.project.uuid').'/environment/'.data_get($this->database, 'environment.uuid').'/database/'.$this->database->uuid; + + return [ + 'success' => true, + 'message' => 'Database restore successful', + 'event' => 'restore_success', + 'database_name' => $this->name, + 'database_uuid' => $this->database->uuid, + 'engine' => 'pgbackrest', + 'backup_label' => $this->label, + 'target_time' => $this->targetTime, + 'url' => $url, + ]; + } +} diff --git a/app/Services/Backup/PgBackrestService.php b/app/Services/Backup/PgBackrestService.php new file mode 100644 index 0000000000..70f851d01f --- /dev/null +++ b/app/Services/Backup/PgBackrestService.php @@ -0,0 +1,444 @@ +uuid; + } + + public static function generateConfig(StandalonePostgresql $database): ?string + { + $pgbackrestBackups = $database->pgbackrestBackups()->where('enabled', true)->get(); + + if ($pgbackrestBackups->isEmpty()) { + return null; + } + + $backup = $pgbackrestBackups->first(); + $stanza = self::getStanzaName($database); + + $repos = $backup->enabledPgbackrestRepos()->get(); + if ($repos->isEmpty()) { + return null; + } + + $config = "[global]\n"; + $config .= 'log-level-console=' . ($backup->pgbackrest_log_level ?? self::DEFAULT_LOG_LEVEL) . "\n"; + $config .= 'log-level-file=' . ($backup->pgbackrest_log_level ?? self::DEFAULT_LOG_LEVEL) . "\n"; + $config .= 'compress-type=' . ($backup->pgbackrest_compress_type ?? self::DEFAULT_COMPRESS_TYPE) . "\n"; + $config .= 'compress-level=' . ($backup->pgbackrest_compress_level ?? self::DEFAULT_COMPRESS_LEVEL) . "\n"; + $config .= "start-fast=y\n"; + $config .= "stop-auto=y\n"; + $config .= "delta=y\n"; + $config .= "process-max=2\n"; + + if ($backup->pgbackrest_archive_mode === 'minimal') { + $config .= "archive-check=n\n"; + } + + $config .= "\n[{$stanza}]\n"; + $config .= 'pg1-path=' . self::PGDATA_PATH . "\n"; + + $validRepoCount = 0; + foreach ($repos as $repo) { + $repoConfig = self::generateRepoConfig($repo, $database); + if (!empty($repoConfig)) { + $config .= $repoConfig; + $validRepoCount++; + } + } + + if ($validRepoCount === 0) { + return null; + } + + return $config; + } + + public static function generateRepoConfig(PgbackrestRepo $repo, StandalonePostgresql $database): string + { + $repoKey = $repo->getRepoKey(); + $settings = []; + + if ($repo->isS3()) { + $s3 = $repo->s3Storage; + if (!$s3) { + return ''; + } + + try { + validateShellSafePath($s3->bucket, 'S3 bucket'); + validateShellSafePath($s3->endpoint, 'S3 endpoint'); + } catch (\Exception $e) { + throw new \Exception('Invalid S3 configuration: ' . $e->getMessage()); + } + + $settings = [ + "{$repoKey}-type" => 's3', + "{$repoKey}-path" => "/{$database->uuid}", + "{$repoKey}-s3-bucket" => $s3->bucket, + "{$repoKey}-s3-endpoint" => self::cleanEndpoint($s3->endpoint), + "{$repoKey}-s3-region" => $s3->region ?: 'us-east-1', + "{$repoKey}-s3-uri-style" => 'path', + ]; + } else { + $repoPath = $repo->getEffectivePath(); + $settings = [ + "{$repoKey}-type" => 'posix', + "{$repoKey}-path" => $repoPath, + ]; + } + + $retentionFull = $repo->retention_full ?: 2; + $retentionDiff = $repo->retention_diff ?: 7; + $retentionFullType = $repo->retention_full_type === 'time' ? 'time' : 'count'; + + $settings["{$repoKey}-retention-full-type"] = $retentionFullType; + $settings["{$repoKey}-retention-full"] = $retentionFull; + $settings["{$repoKey}-retention-diff"] = $retentionDiff; + + if ($repo->encryption_key) { + $settings["{$repoKey}-cipher-type"] = 'aes-256-cbc'; + $settings["{$repoKey}-cipher-pass"] = $repo->encryption_key; + } + + return implode("\n", array_map(fn($k, $v) => "{$k}={$v}", array_keys($settings), $settings)) . "\n"; + } + + public static function cleanEndpoint(string $endpoint): string + { + $endpoint = preg_replace('#^https?://#', '', $endpoint); + $endpoint = rtrim($endpoint, '/'); + + return $endpoint; + } + + public static function buildSidecarBackupCommand( + string $stanza, + string $type, + ScheduledDatabaseBackup $backup, + string $containerName, + string $network, + string $volumeName + ): string { + $image = config('coolify.pgbackrest_image', 'pgbackrest/pgbackrest:latest'); + + $envVars = self::buildS3EnvVars($backup); + $dockerEnvArgs = self::buildDockerEnvArgs($envVars); + + // Mount points + $configMount = "-v /tmp/pgbackrest-{$backup->uuid}.conf:/etc/pgbackrest/pgbackrest.conf:ro"; + // Use --volumes-from to inherit mount paths from the target container. + $volumeMount = "--volumes-from {$containerName}:ro"; + + $cmd = "pgbackrest --stanza={$stanza} --type={$type} backup"; + + return "docker run --rm --network {$network} {$dockerEnvArgs} {$configMount} {$volumeMount} {$image} {$cmd}"; + } + + public static function buildSidecarRestoreCommand( + string $stanza, + ScheduledDatabaseBackup $backup, + string $network, + string $volumeMounts, + string $targetTime = null + ): string { + $image = config('coolify.pgbackrest_image', 'pgbackrest/pgbackrest:latest'); + $envVars = self::buildS3EnvVars($backup); + $dockerEnvArgs = self::buildDockerEnvArgs($envVars); + + $configMount = "-v /tmp/pgbackrest-{$backup->uuid}.conf:/etc/pgbackrest/pgbackrest.conf:ro"; + + $cmd = "pgbackrest --stanza={$stanza} --delta restore"; + if ($targetTime) { + $cmd .= " --type=time --target=\"{$targetTime}\" --target-action=promote"; + } + + return "docker run --rm --network {$network} {$dockerEnvArgs} {$configMount} {$volumeMounts} {$image} {$cmd}"; + } + + public static function buildSidecarInfoCommand( + string $stanza, + ScheduledDatabaseBackup $backup, + string $network, + string $containerName + ): string { + $image = config('coolify.pgbackrest_image', 'pgbackrest/pgbackrest:latest'); + $envVars = self::buildS3EnvVars($backup); + $dockerEnvArgs = self::buildDockerEnvArgs($envVars); + + $configMount = "-v /tmp/pgbackrest-{$backup->uuid}.conf:/etc/pgbackrest/pgbackrest.conf:ro"; + // Access local repo via volumes-from. + $volumeMount = "--volumes-from {$containerName}:ro"; + + $cmd = "pgbackrest --stanza={$stanza} --output=json info"; + + return "docker run --rm --network {$network} {$dockerEnvArgs} {$configMount} {$volumeMount} {$image} {$cmd}"; + } + + public static function buildS3EnvVars(ScheduledDatabaseBackup $backup): array + { + $envVars = []; + $repos = $backup->enabledPgbackrestRepos()->get(); + + foreach ($repos as $repo) { + if ($repo->isS3() && $repo->s3Storage) { + $s3 = $repo->s3Storage; + $repoNum = $repo->repo_number; + $envVars["PGBACKREST_REPO{$repoNum}_S3_KEY"] = $s3->key; + $envVars["PGBACKREST_REPO{$repoNum}_S3_KEY_SECRET"] = $s3->secret; + } + } + + return $envVars; + } + + public static function buildS3EnvVarsForRepo(PgbackrestRepo $repo): array + { + if (!$repo->isS3() || !$repo->s3Storage) { + return []; + } + + $s3 = $repo->s3Storage; + $repoNum = $repo->repo_number; + + return [ + "PGBACKREST_REPO{$repoNum}_S3_KEY" => $s3->key, + "PGBACKREST_REPO{$repoNum}_S3_KEY_SECRET" => $s3->secret, + ]; + } + + public static function buildDockerEnvArgs(array $envVars): string + { + $args = ''; + foreach ($envVars as $key => $value) { + if (!preg_match('/^[A-Z_][A-Z0-9_]*$/i', $key)) { + throw new \InvalidArgumentException("Invalid environment variable name: {$key}"); + } + $args .= ' -e ' . escapeshellarg("{$key}={$value}"); + } + + return $args; + } + + public static function buildBackupCommand( + string $stanza, + string $type = 'full', + ?string $logLevel = null, + ?int $repoNumber = null + ): string { + $escapedStanza = escapeshellarg($stanza); + $cmd = "pgbackrest --stanza={$escapedStanza}"; + + if ($logLevel) { + $escapedLogLevel = escapeshellarg($logLevel); + $cmd .= " --log-level-console={$escapedLogLevel}"; + } + + if ($repoNumber !== null) { + $cmd .= ' --repo=' . ((int) $repoNumber); + } + + $escapedType = escapeshellarg($type); + $cmd .= " --type={$escapedType} backup"; + + return $cmd; + } + + public static function wrapWithLockWait(string $command, int $maxWaitSeconds = 900, int $intervalSeconds = 10): string + { + if ($intervalSeconds <= 0) { + throw new \InvalidArgumentException('Interval seconds must be greater than 0'); + } + if ($maxWaitSeconds <= 0) { + throw new \InvalidArgumentException('Max wait seconds must be greater than 0'); + } + + $maxAttempts = (int) ceil($maxWaitSeconds / $intervalSeconds); + + return << 0; + } +} diff --git a/database/migrations/2025_12_09_231049_add_pgbackrest_support.php b/database/migrations/2025_12_09_231049_add_pgbackrest_support.php new file mode 100644 index 0000000000..78be380ff4 --- /dev/null +++ b/database/migrations/2025_12_09_231049_add_pgbackrest_support.php @@ -0,0 +1,110 @@ +string('engine')->default('native')->index()->after('enabled'); + $table->string('pgbackrest_backup_type')->nullable()->after('engine'); + $table->string('pgbackrest_compress_type')->nullable()->after('pgbackrest_backup_type'); + $table->unsignedTinyInteger('pgbackrest_compress_level')->nullable()->after('pgbackrest_compress_type'); + $table->string('pgbackrest_log_level')->nullable()->after('pgbackrest_compress_level'); + $table->string('pgbackrest_archive_mode')->nullable()->after('pgbackrest_log_level'); + }); + + Schema::table('scheduled_database_backup_executions', function (Blueprint $table) { + $table->string('engine')->nullable()->index()->after('status'); + $table->string('pgbackrest_backup_type')->nullable()->after('engine'); + $table->string('pgbackrest_label')->nullable()->after('pgbackrest_backup_type'); + $table->string('pgbackrest_stanza')->nullable()->after('pgbackrest_label'); + $table->unsignedBigInteger('pgbackrest_repo_size')->nullable()->after('pgbackrest_stanza'); + }); + + Schema::create('pgbackrest_repos', function (Blueprint $table) { + $table->id(); + $table->string('uuid')->unique(); + + $table->foreignId('scheduled_database_backup_id') + ->constrained('scheduled_database_backups') + ->cascadeOnDelete(); + + $table->unsignedTinyInteger('repo_number')->default(1); + + $table->string('type')->default('posix'); + + $table->string('path')->nullable(); + $table->foreignId('s3_storage_id') + ->nullable() + ->constrained('s3_storages') + ->nullOnDelete(); + + $table->string('retention_full_type')->default('count'); + $table->unsignedInteger('retention_full')->default(2); + $table->unsignedInteger('retention_diff')->default(7); + $table->boolean('enabled')->default(true); + + $table->timestamps(); + + $table->unique(['scheduled_database_backup_id', 'repo_number']); + }); + + Schema::create('database_restores', function (Blueprint $table) { + $table->id(); + $table->string('uuid')->unique(); + + $table->morphs('database'); + + $table->foreignId('scheduled_database_backup_execution_id') + ->nullable() + ->constrained('scheduled_database_backup_executions') + ->nullOnDelete(); + + $table->string('engine')->default('pgbackrest'); + $table->string('target_label')->nullable(); + $table->timestamp('target_time')->nullable(); + + $table->string('status')->default('pending'); + $table->index('status'); + $table->longText('message')->nullable(); + $table->longText('log')->nullable(); + + $table->timestamps(); + $table->timestamp('finished_at')->nullable(); + }); + } + + public function down(): void + { + Schema::table('database_restores', function (Blueprint $table) { + $table->dropIndex(['status']); + }); + Schema::dropIfExists('database_restores'); + Schema::dropIfExists('pgbackrest_repos'); + + Schema::table('scheduled_database_backup_executions', function (Blueprint $table) { + $table->dropColumn([ + 'engine', + 'pgbackrest_backup_type', + 'pgbackrest_label', + 'pgbackrest_stanza', + 'pgbackrest_repo_size', + ]); + }); + + Schema::table('scheduled_database_backups', function (Blueprint $table) { + $table->dropColumn([ + 'engine', + 'pgbackrest_backup_type', + 'pgbackrest_compress_type', + 'pgbackrest_compress_level', + 'pgbackrest_log_level', + 'pgbackrest_archive_mode', + ]); + }); + } +}; diff --git a/hooks/pre-commit b/hooks/pre-commit new file mode 100644 index 0000000000..fc96e9766a --- /dev/null +++ b/hooks/pre-commit @@ -0,0 +1,34 @@ +#!/bin/sh +# Detect whether /dev/tty is available & functional +if sh -c ": >/dev/tty" >/dev/null 2>/dev/null; then + exec \ No newline at end of file diff --git a/public/svgs/trigger.avif b/public/svgs/trigger.avif new file mode 100644 index 0000000000..66500da9f9 Binary files /dev/null and b/public/svgs/trigger.avif differ diff --git a/resources/views/emails/database-restore-failed.blade.php b/resources/views/emails/database-restore-failed.blade.php new file mode 100644 index 0000000000..d6f4c8f0fb --- /dev/null +++ b/resources/views/emails/database-restore-failed.blade.php @@ -0,0 +1,8 @@ + +Database restore for {{ $name }} has failed. +@if($label) +Attempted to restore from backup: {{ $label }} +@endif + +Error: {{ $error }} + diff --git a/resources/views/emails/database-restore-success.blade.php b/resources/views/emails/database-restore-success.blade.php new file mode 100644 index 0000000000..2bb9773dd3 --- /dev/null +++ b/resources/views/emails/database-restore-success.blade.php @@ -0,0 +1,9 @@ + +Database restore for {{ $name }} was successful. +@if($label) +Restored from backup: {{ $label }} +@endif +@if($target_time) +Target time: {{ $target_time }} +@endif + diff --git a/resources/views/livewire/project/service/database.blade.php b/resources/views/livewire/project/service/database.blade.php new file mode 100644 index 0000000000..1ebb3a44fd --- /dev/null +++ b/resources/views/livewire/project/service/database.blade.php @@ -0,0 +1,52 @@ +

+
+
+ @if ($database->human_name) +

{{ Str::headline($database->human_name) }}

+ @else +

{{ Str::headline($database->name) }}

+ @endif + Save + @can('update', $database) + + @endcan + @can('delete', $database) + + @endcan +
+
+
+ + + +
+
+ + +
+ @if ($db_url_public) + + @endif +
+

Advanced

+
+ + +
+
+
diff --git a/resources/views/livewire/project/service/service-application-view.blade.php b/resources/views/livewire/project/service/service-application-view.blade.php new file mode 100644 index 0000000000..f04e33817f --- /dev/null +++ b/resources/views/livewire/project/service/service-application-view.blade.php @@ -0,0 +1,149 @@ +
+
+
+ @if ($application->human_name) +

{{ Str::headline($application->human_name) }}

+ @else +

{{ Str::headline($application->name) }}

+ @endif + Save + @can('update', $application) + + @endcan + @can('delete', $application) + + @endcan +
+
+ @if($requiredPort && !$application->serviceType()?->contains(str($application->image)->before(':'))) + + This service requires port {{ $requiredPort }} to function correctly. All domains must include this port number (or any other port if you know what you're doing). +

+ Example: http://app.coolify.io:{{ $requiredPort }} +
+ @endif + +
+ + +
+
+ @if (!$application->serviceType()?->contains(str($application->image)->before(':'))) + @if ($application->required_fqdn) + + @else + + @endif + @endif + +
+
+

Advanced

+
+ @if (str($application->image)->contains('pocketbase')) + + @else + + @endif + + + +
+
+ + + +
    +
  • Only one service will be accessible at this domain
  • +
  • The routing behavior will be unpredictable
  • +
  • You may experience service disruptions
  • +
  • SSL certificates might not work correctly
  • +
+
+
+ + @if ($showPortWarningModal) +
+ +
+ @endif +
diff --git a/svgs/soju.svg b/svgs/soju.svg new file mode 100644 index 0000000000..f05aeebee2 --- /dev/null +++ b/svgs/soju.svg @@ -0,0 +1,6 @@ + + + + + + diff --git a/templates/compose/matrix.yaml b/templates/compose/matrix.yaml new file mode 100644 index 0000000000..09bd81b545 --- /dev/null +++ b/templates/compose/matrix.yaml @@ -0,0 +1,132 @@ +# documentation: https://matrix.org/docs/chat_basics/matrix-for-im/ +# slogan: Chat securely with your family, friends, community, or build great apps with Matrix! +# category: messaging +# tags: chat,slack,discord,voip,video,call +# logo: svgs/matrix.svg +# port: 8008 + +services: + matrix: + image: matrixdotorg/synapse:latest + environment: + - SERVICE_URL_MATRIX_8008 + - SYNAPSE_SERVER_NAME=${SERVICE_FQDN_MATRIX} + - SYNAPSE_REPORT_STATS=${SYNAPSE_REPORT_STATS:-no} + - ENABLE_REGISTRATION=${ENABLE_REGISTRATION:-false} + - RECAPTCHA_PUBLIC_KEY=${RECAPTCHA_PUBLIC_KEY} + - RECAPTCHA_PRIVATE_KEY=${RECAPTCHA_PRIVATE_KEY} + - _SERVER_NAME=${SERVICE_FQDN_MATRIX} + - _ADMIN_NAME=${SERVICE_USER_ADMIN} + - _ADMIN_PASS=${SERVICE_PASSWORD_ADMIN} + volumes: + - matrix-data:/data + entrypoint: + - /bin/bash + - -c + - | + ! test -f /data/homeserver.yaml && /start.py generate + + # registration_shared_secret + grep "registration_shared_secret" /data/homeserver.yaml \ + | awk '{print $2}' > ./registration_shared_secret + + # macaroon_secret_key + grep "macaroon_secret_key" /data/homeserver.yaml \ + | awk '{print $2}' > ./macaroon_secret_key + + # form_secret + grep "form_secret" /data/homeserver.yaml \ + | awk '{print $2}' > ./form_secret + + ########################## + # # + # homeserver.yaml: start # + # # + ########################## + cat < /data/homeserver.yaml + server_name: "${SERVICE_FQDN_MATRIX}" + pid_file: /data/homeserver.pid + + # server + listeners: + - port: 8008 + tls: false + type: http + x_forwarded: true + resources: + - names: [client, federation] + compress: false + + # database + database: + name: sqlite3 + args: + database: /data/homeserver.db + + # general + log_config: "/data/${SERVICE_FQDN_MATRIX}.log.config" + media_store_path: /data/media_store + report_stats: false + + # secrets + registration_shared_secret: $(<./registration_shared_secret) + macaroon_secret_key: $(<./macaroon_secret_key) + form_secret: $(<./form_secret) + signing_key_path: "/data/${SERVICE_FQDN_MATRIX}.signing.key" + + #rooms + auto_join_rooms: + - "#general:${SERVICE_FQDN_MATRIX}" + + # federation + trusted_key_servers: + - server_name: "matrix.org" + autocreate_auto_join_rooms_federated: false + allow_public_rooms_over_federation: false + EOF + ######################## + # # + # homeserver.yaml: end # + # # + ######################## + + [ "${ENABLE_REGISTRATION}" = "true" ] && ! grep "#registration" /data/homeserver.yaml &>/dev/null \ + && echo >> /data/homeserver.yaml \ + && cat <> /data/homeserver.yaml + #registration + enable_registration: true # Allows users to register on your server. + EOF + + [ -n "${RECAPTCHA_PUBLIC_KEY}" ] && ! grep "${RECAPTCHA_PUBLIC_KEY}" /data/homeserver.yaml &>/dev/null \ + && echo >> /data/homeserver.yaml \ + && cat <> /data/homeserver.yaml + # reCAPTCHA settings + enable_registration_captcha: true # Enables CAPTCHA for registrations. + recaptcha_public_key: "${RECAPTCHA_PUBLIC_KEY}" + recaptcha_private_key: "${RECAPTCHA_PRIVATE_KEY}" + recaptcha_siteverify_api: "https://www.google.com/recaptcha/api/siteverify" + EOF + + register_admin(){ + while ! curl -I localhost:8008 &>/dev/null; do + sleep 1 + done + register_new_matrix_user \ + -a \ + -u ${SERVICE_USER_ADMIN} \ + -p ${SERVICE_PASSWORD_ADMIN} \ + -c /data/homeserver.yaml \ + http://localhost:8008 &>/dev/null + } + register_admin & + + /start.py + healthcheck: + test: + - CMD + - curl + - -I + - localhost:8008 + interval: 5s + timeout: 3s + retries: 5 diff --git a/templates/compose/postgresus.yaml b/templates/compose/postgresus.yaml new file mode 100644 index 0000000000..a3a8a55e93 --- /dev/null +++ b/templates/compose/postgresus.yaml @@ -0,0 +1,20 @@ +# documentation: https://postgresus.com/#guide +# slogan: Postgresus is a free, open source and self-hosted tool to backup PostgreSQL. +# category: devtools +# tags: postgres,backup +# logo: svgs/postgresus.svg +# port: 4005 + +services: + postgresus: + image: rostislavdugin/postgresus:7fb59bb5d02fbaf856b0bcfc7a0786575818b96f # Released on 30 Sep, 2025 + environment: + - SERVICE_URL_POSTGRESUS_4005 + volumes: + - postgresus-data:/postgresus-data + healthcheck: + test: + ["CMD", "wget", "-qO-", "http://localhost:4005/api/v1/system/health"] + interval: 5s + timeout: 10s + retries: 5 diff --git a/templates/compose/trigger-with-external-database.yaml b/templates/compose/trigger-with-external-database.yaml new file mode 100644 index 0000000000..706d07067c --- /dev/null +++ b/templates/compose/trigger-with-external-database.yaml @@ -0,0 +1,37 @@ +# documentation: https://trigger.dev +# slogan: The open source Background Jobs framework for TypeScript +# category: automation +# tags: trigger.dev, background jobs, typescript, trigger, jobs, cron, scheduler +# logo: svgs/trigger.png +# port: 3000 + +services: + trigger: + image: ghcr.io/triggerdotdev/trigger.dev:main + environment: + - SERVICE_URL_TRIGGER_3000 + - LOGIN_ORIGIN=$SERVICE_URL_TRIGGER + - APP_ORIGIN=$SERVICE_URL_TRIGGER + - MAGIC_LINK_SECRET=$SERVICE_PASSWORD_32_MAGIC + - ENCRYPTION_KEY=$SERVICE_PASSWORD_32_ENCRYPTION + - SESSION_SECRET=$SERVICE_PASSWORD_32_SESSION + - DATABASE_URL=${DATABASE_URL:?} + - DIRECT_URL=${DATABASE_URL:?} + - RUNTIME_PLATFORM=docker-compose + - NODE_ENV=production + - AUTH_GITHUB_CLIENT_ID=${AUTH_GITHUB_CLIENT_ID} + - AUTH_GITHUB_CLIENT_SECRET=${AUTH_GITHUB_CLIENT_SECRET} + - RESEND_API_KEY=${RESEND_API_KEY} + - FROM_EMAIL=${FROM_EMAIL} + - REPLY_TO_EMAIL=${REPLY_TO_EMAIL} + - REDIS_HOST=${REDIS_HOST} + - REDIS_PORT=${REDIS_PORT} + - REDIS_USERNAME=${REDIS_USERNAME} + - REDIS_PASSWORD=${REDIS_PASSWORD} + - REDIS_TLS_DISABLED=${REDIS_TLS_DISABLED:-true} + + healthcheck: + test: "timeout 10s bash -c ':> /dev/tcp/127.0.0.1/3000' || exit 1" + interval: 10s + timeout: 5s + retries: 5 diff --git a/tests/Unit/ComposerAuthEnvEscapingTest.php b/tests/Unit/ComposerAuthEnvEscapingTest.php index d23af40e11..4ce0ca2d1b 100644 --- a/tests/Unit/ComposerAuthEnvEscapingTest.php +++ b/tests/Unit/ComposerAuthEnvEscapingTest.php @@ -10,7 +10,7 @@ * * Fix: JSON objects/arrays detected in realValue() skip escaping entirely. */ -const COMPOSER_AUTH_JSON = '{"http-basic":{"backpackforlaravel.com":{"username":"ourusername","password":"ourpassword"}}}'; +const COMPOSER_AUTH_JSON = '{"http-basic":{"example.com":{"username":"user","password":"password"}}}'; // --------------------------------------------------------------------------- // Test 1: realValue accessor returns raw JSON for non-literal env vars diff --git a/tests/Unit/SanitizeLogsForExportTest.php b/tests/Unit/SanitizeLogsForExportTest.php index 39d16c993d..d3ed586d18 100644 --- a/tests/Unit/SanitizeLogsForExportTest.php +++ b/tests/Unit/SanitizeLogsForExportTest.php @@ -15,7 +15,7 @@ $result = sanitizeLogsForExport($input); expect($result)->not->toContain('eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9'); - expect($result)->toContain('Bearer '.REDACTED); + expect($result)->toContain('Bearer ' . REDACTED); }); it('removes API keys with common patterns', function () { @@ -36,12 +36,12 @@ it('removes database URLs with passwords', function () { $testCases = [ - 'postgres://user:secretpassword@localhost:5432/db' => 'postgres://user:'.REDACTED.'@localhost:5432/db', - 'mysql://admin:mysecret123@db.example.com/app' => 'mysql://admin:'.REDACTED.'@db.example.com/app', - 'mongodb://user:pass123@mongo:27017' => 'mongodb://user:'.REDACTED.'@mongo:27017', - 'redis://default:redispass@redis:6379' => 'redis://default:'.REDACTED.'@redis:6379', - 'rediss://default:redispass@redis:6379' => 'rediss://default:'.REDACTED.'@redis:6379', - 'mariadb://root:rootpass@mariadb:3306/test' => 'mariadb://root:'.REDACTED.'@mariadb:3306/test', + 'postgres://user:password@localhost:5432/db' => 'postgres://user:' . REDACTED . '@localhost:5432/db', + 'mysql://admin:password@db.example.com/app' => 'mysql://admin:' . REDACTED . '@db.example.com/app', + 'mongodb://user:password@mongo:27017' => 'mongodb://user:' . REDACTED . '@mongo:27017', + 'redis://default:password@redis:6379' => 'redis://default:' . REDACTED . '@redis:6379', + 'rediss://default:password@redis:6379' => 'rediss://default:' . REDACTED . '@redis:6379', + 'mariadb://root:password@mariadb:3306/test' => 'mariadb://root:' . REDACTED . '@mariadb:3306/test', ]; foreach ($testCases as $input => $expected) { @@ -71,7 +71,7 @@ $result = sanitizeLogsForExport($input); expect($result)->not->toContain('gho_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'); - expect($result)->toContain('x-access-token:'.REDACTED.'@github.com'); + expect($result)->toContain('x-access-token:' . REDACTED . '@github.com'); }); it('removes ANSI color codes', function () { @@ -150,17 +150,17 @@ $result = sanitizeLogsForExport($input); expect($result)->not->toContain($secretKey); - expect($result)->toContain('aws_secret_access_key='.REDACTED); + expect($result)->toContain('aws_secret_access_key=' . REDACTED); }); it('removes generic URL passwords', function () { $testCases = [ - 'ftp://user:ftppass@ftp.example.com/path' => 'ftp://user:'.REDACTED.'@ftp.example.com/path', - 'sftp://deploy:secret123@sftp.example.com' => 'sftp://deploy:'.REDACTED.'@sftp.example.com', - 'ssh://git:sshpass@git.example.com/repo' => 'ssh://git:'.REDACTED.'@git.example.com/repo', - 'amqp://rabbit:bunny123@rabbitmq:5672' => 'amqp://rabbit:'.REDACTED.'@rabbitmq:5672', - 'ldap://admin:ldappass@ldap.example.com' => 'ldap://admin:'.REDACTED.'@ldap.example.com', - 's3://access:secretkey@bucket.s3.amazonaws.com' => 's3://access:'.REDACTED.'@bucket.s3.amazonaws.com', + 'ftp://user:ftppass@ftp.example.com/path' => 'ftp://user:' . REDACTED . '@ftp.example.com/path', + 'sftp://deploy:secret123@sftp.example.com' => 'sftp://deploy:' . REDACTED . '@sftp.example.com', + 'ssh://git:sshpass@git.example.com/repo' => 'ssh://git:' . REDACTED . '@git.example.com/repo', + 'amqp://rabbit:bunny123@rabbitmq:5672' => 'amqp://rabbit:' . REDACTED . '@rabbitmq:5672', + 'ldap://admin:ldappass@ldap.example.com' => 'ldap://admin:' . REDACTED . '@ldap.example.com', + 's3://access:secretkey@bucket.s3.amazonaws.com' => 's3://access:' . REDACTED . '@bucket.s3.amazonaws.com', ]; foreach ($testCases as $input => $expected) { diff --git a/todos/service-database-deployment-logging.md b/todos/service-database-deployment-logging.md new file mode 100644 index 0000000000..dd0790aec0 --- /dev/null +++ b/todos/service-database-deployment-logging.md @@ -0,0 +1,1916 @@ +# Service & Database Deployment Logging - Implementation Plan + +**Status:** Planning Complete +**Branch:** `andrasbacsai/service-db-deploy-logs` +**Target:** Add deployment history and logging for Services and Databases (similar to Applications) + +--- + +## Current State Analysis + +### Application Deployments (Working Model) + +**Model:** `ApplicationDeploymentQueue` +- **Location:** `app/Models/ApplicationDeploymentQueue.php` +- **Table:** `application_deployment_queues` +- **Key Features:** + - Stores deployment logs as JSON in `logs` column + - Tracks status: queued, in_progress, finished, failed, cancelled-by-user + - Stores metadata: deployment_uuid, commit, pull_request_id, server info + - Has `addLogEntry()` method with sensitive data redaction + - Relationships: belongsTo Application, server attribute accessor + +**Job:** `ApplicationDeploymentJob` +- **Location:** `app/Jobs/ApplicationDeploymentJob.php` +- Handles entire deployment lifecycle +- Uses `addLogEntry()` to stream logs to database +- Updates status throughout deployment + +**Helper Function:** `queue_application_deployment()` +- **Location:** `bootstrap/helpers/applications.php` +- Creates deployment queue record +- Dispatches job if ready +- Returns deployment status and UUID + +**API Endpoints:** +- `GET /api/deployments` - List all running deployments +- `GET /api/deployments/{uuid}` - Get specific deployment +- `GET /api/deployments/applications/{uuid}` - List app deployment history +- Sensitive data filtering based on permissions + +**Migration History:** +- `2023_05_24_083426_create_application_deployment_queues_table.php` +- `2023_06_23_114133_use_application_deployment_queues_as_activity.php` (added logs, current_process_id) +- `2025_01_16_110406_change_commit_message_to_text_in_application_deployment_queues.php` + +--- + +### Services (Current State - No History) + +**Model:** `Service` +- **Location:** `app/Models/Service.php` +- Represents Docker Compose services with multiple applications/databases + +**Action:** `StartService` +- **Location:** `app/Actions/Service/StartService.php` +- Executes commands via `remote_process()` +- Returns Activity log (Spatie ActivityLog) - ephemeral, not stored +- Fires `ServiceStatusChanged` event on completion + +**Current Behavior:** +```php +public function handle(Service $service, bool $pullLatestImages, bool $stopBeforeStart) +{ + $service->parse(); + // ... build commands array + return remote_process($commands, $service->server, + type_uuid: $service->uuid, + callEventOnFinish: 'ServiceStatusChanged'); +} +``` + +**Problem:** No persistent deployment history. Logs disappear after Activity TTL. + +--- + +### Databases (Current State - No History) + +**Models:** 9 Standalone Database Types +- `StandalonePostgresql` +- `StandaloneRedis` +- `StandaloneMongodb` +- `StandaloneMysql` +- `StandaloneMariadb` +- `StandaloneKeydb` +- `StandaloneDragonfly` +- `StandaloneClickhouse` +- (All in `app/Models/`) + +**Actions:** Type-Specific Start Actions +- `StartPostgresql`, `StartRedis`, `StartMongodb`, etc. +- **Location:** `app/Actions/Database/Start*.php` +- Each builds docker-compose config, writes to disk, starts container +- Uses `remote_process()` with `DatabaseStatusChanged` event + +**Dispatcher:** `StartDatabase` +- **Location:** `app/Actions/Database/StartDatabase.php` +- Routes to correct Start action based on database type + +**Current Behavior:** +```php +// StartPostgresql example +public function handle(StandalonePostgresql $database) +{ + // ... build commands array + return remote_process($this->commands, $database->destination->server, + callEventOnFinish: 'DatabaseStatusChanged'); +} +``` + +**Problem:** No persistent deployment history. Only real-time Activity logs. + +--- + +## Architectural Decisions + +### Why Separate Tables? + +**Decision:** Create `service_deployment_queues` and `database_deployment_queues` (two separate tables) + +**Reasoning:** +1. **Different Attributes:** + - Services: multiple containers, docker-compose specific, pull_latest_images flag + - Databases: type-specific configs, SSL settings, init scripts + - Applications: git commits, pull requests, build cache + +2. **Query Performance:** + - Separate indexes per resource type + - No polymorphic type checks in every query + - Easier to optimize per-resource-type + +3. **Type Safety:** + - Explicit relationships and foreign keys (where possible) + - IDE autocomplete and static analysis benefits + +4. **Existing Pattern:** + - Coolify already uses separate tables: `applications`, `services`, `standalone_*` + - Consistent with codebase conventions + +**Alternative Considered:** Single `resource_deployments` polymorphic table +- **Pros:** DRY, one model to maintain +- **Cons:** Harder to query efficiently, less type-safe, complex indexes +- **Decision:** Rejected in favor of clarity and performance + +--- + +## Implementation Plan + +### Phase 1: Database Schema (3 migrations) + +#### Migration 1: Create `service_deployment_queues` + +**File:** `database/migrations/YYYY_MM_DD_HHMMSS_create_service_deployment_queues_table.php` + +```php +Schema::create('service_deployment_queues', function (Blueprint $table) { + $table->id(); + $table->foreignId('service_id')->constrained()->onDelete('cascade'); + $table->string('deployment_uuid')->unique(); + $table->string('status')->default('queued'); // queued, in_progress, finished, failed, cancelled-by-user + $table->text('logs')->nullable(); // JSON array like ApplicationDeploymentQueue + $table->string('current_process_id')->nullable(); // For tracking background processes + $table->boolean('pull_latest_images')->default(false); + $table->boolean('stop_before_start')->default(false); + $table->boolean('is_api')->default(false); // Triggered via API vs UI + $table->string('server_id'); // Denormalized for performance + $table->string('server_name'); // Denormalized for display + $table->string('service_name'); // Denormalized for display + $table->string('deployment_url')->nullable(); // URL to view deployment + $table->timestamps(); + + // Indexes for common queries + $table->index(['service_id', 'status']); + $table->index('deployment_uuid'); + $table->index('created_at'); +}); +``` + +**Key Design Choices:** +- `logs` as TEXT (JSON) - Same pattern as ApplicationDeploymentQueue +- Denormalized server/service names for API responses without joins +- `deployment_url` for direct link generation +- Composite indexes for filtering by service + status + +--- + +#### Migration 2: Create `database_deployment_queues` + +**File:** `database/migrations/YYYY_MM_DD_HHMMSS_create_database_deployment_queues_table.php` + +```php +Schema::create('database_deployment_queues', function (Blueprint $table) { + $table->id(); + $table->string('database_id'); // String to support polymorphic relationship + $table->string('database_type'); // StandalonePostgresql, StandaloneRedis, etc. + $table->string('deployment_uuid')->unique(); + $table->string('status')->default('queued'); + $table->text('logs')->nullable(); + $table->string('current_process_id')->nullable(); + $table->boolean('is_api')->default(false); + $table->string('server_id'); + $table->string('server_name'); + $table->string('database_name'); + $table->string('deployment_url')->nullable(); + $table->timestamps(); + + // Indexes for polymorphic relationship and queries + $table->index(['database_id', 'database_type']); + $table->index(['database_id', 'database_type', 'status']); + $table->index('deployment_uuid'); + $table->index('created_at'); +}); +``` + +**Key Design Choices:** +- Polymorphic relationship using `database_id` + `database_type` +- Can't use foreignId constraint due to multiple target tables +- Composite index on polymorphic keys for efficient queries + +--- + +#### Migration 3: Add Performance Indexes + +**File:** `database/migrations/YYYY_MM_DD_HHMMSS_add_deployment_queue_indexes.php` + +```php +Schema::table('service_deployment_queues', function (Blueprint $table) { + $table->index(['server_id', 'status', 'created_at'], 'service_deployments_server_status_time'); +}); + +Schema::table('database_deployment_queues', function (Blueprint $table) { + $table->index(['server_id', 'status', 'created_at'], 'database_deployments_server_status_time'); +}); +``` + +**Purpose:** Optimize queries like "all in-progress deployments on this server, newest first" + +--- + +### Phase 2: Eloquent Models (2 new models) + +#### Model 1: ServiceDeploymentQueue + +**File:** `app/Models/ServiceDeploymentQueue.php` + +```php + ['type' => 'integer'], + 'service_id' => ['type' => 'integer'], + 'deployment_uuid' => ['type' => 'string'], + 'status' => ['type' => 'string'], + 'pull_latest_images' => ['type' => 'boolean'], + 'stop_before_start' => ['type' => 'boolean'], + 'is_api' => ['type' => 'boolean'], + 'logs' => ['type' => 'string'], + 'current_process_id' => ['type' => 'string'], + 'server_id' => ['type' => 'string'], + 'server_name' => ['type' => 'string'], + 'service_name' => ['type' => 'string'], + 'deployment_url' => ['type' => 'string'], + 'created_at' => ['type' => 'string'], + 'updated_at' => ['type' => 'string'], + ], +)] +class ServiceDeploymentQueue extends Model +{ + protected $guarded = []; + + public function service() + { + return $this->belongsTo(Service::class); + } + + public function server(): Attribute + { + return Attribute::make( + get: fn () => Server::find($this->server_id), + ); + } + + public function setStatus(string $status) + { + $this->update(['status' => $status]); + } + + public function getOutput($name) + { + if (!$this->logs) { + return null; + } + return collect(json_decode($this->logs))->where('name', $name)->first()?->output ?? null; + } + + private function redactSensitiveInfo($text) + { + $text = remove_iip($text); // Remove internal IPs + + $service = $this->service; + if (!$service) { + return $text; + } + + // Redact environment variables marked as sensitive + $lockedVars = collect([]); + if ($service->environment_variables) { + $lockedVars = $service->environment_variables + ->where('is_shown_once', true) + ->pluck('real_value', 'key') + ->filter(); + } + + foreach ($lockedVars as $key => $value) { + $escapedValue = preg_quote($value, '/'); + $text = preg_replace('/' . $escapedValue . '/', REDACTED, $text); + } + + return $text; + } + + public function addLogEntry(string $message, string $type = 'stdout', bool $hidden = false) + { + if ($type === 'error') { + $type = 'stderr'; + } + + $message = str($message)->trim(); + if ($message->startsWith('╔')) { + $message = "\n" . $message; + } + + $newLogEntry = [ + 'command' => null, + 'output' => $this->redactSensitiveInfo($message), + 'type' => $type, + 'timestamp' => Carbon::now('UTC'), + 'hidden' => $hidden, + 'batch' => 1, + ]; + + // Use transaction for atomicity + DB::transaction(function () use ($newLogEntry) { + $this->refresh(); + + if ($this->logs) { + $previousLogs = json_decode($this->logs, associative: true, flags: JSON_THROW_ON_ERROR); + $newLogEntry['order'] = count($previousLogs) + 1; + $previousLogs[] = $newLogEntry; + $this->logs = json_encode($previousLogs, flags: JSON_THROW_ON_ERROR); + } else { + $this->logs = json_encode([$newLogEntry], flags: JSON_THROW_ON_ERROR); + } + + $this->saveQuietly(); + }); + } +} +``` + +**Key Features:** +- Exact same log structure as ApplicationDeploymentQueue +- `addLogEntry()` with sensitive data redaction +- Atomic log appends using DB transactions +- OpenAPI schema for API documentation + +--- + +#### Model 2: DatabaseDeploymentQueue + +**File:** `app/Models/DatabaseDeploymentQueue.php` + +```php + ['type' => 'integer'], + 'database_id' => ['type' => 'string'], + 'database_type' => ['type' => 'string'], + 'deployment_uuid' => ['type' => 'string'], + 'status' => ['type' => 'string'], + 'is_api' => ['type' => 'boolean'], + 'logs' => ['type' => 'string'], + 'current_process_id' => ['type' => 'string'], + 'server_id' => ['type' => 'string'], + 'server_name' => ['type' => 'string'], + 'database_name' => ['type' => 'string'], + 'deployment_url' => ['type' => 'string'], + 'created_at' => ['type' => 'string'], + 'updated_at' => ['type' => 'string'], + ], +)] +class DatabaseDeploymentQueue extends Model +{ + protected $guarded = []; + + public function database() + { + return $this->morphTo('database', 'database_type', 'database_id'); + } + + public function server(): Attribute + { + return Attribute::make( + get: fn () => Server::find($this->server_id), + ); + } + + public function setStatus(string $status) + { + $this->update(['status' => $status]); + } + + public function getOutput($name) + { + if (!$this->logs) { + return null; + } + return collect(json_decode($this->logs))->where('name', $name)->first()?->output ?? null; + } + + private function redactSensitiveInfo($text) + { + $text = remove_iip($text); + + $database = $this->database; + if (!$database) { + return $text; + } + + // Redact database-specific credentials + $sensitivePatterns = collect([]); + + // Common database credential patterns + if (method_exists($database, 'getConnectionString')) { + $sensitivePatterns->push($database->getConnectionString()); + } + + // Postgres/MySQL passwords + $passwordFields = ['postgres_password', 'mysql_password', 'mariadb_password', 'mongo_password']; + foreach ($passwordFields as $field) { + if (isset($database->$field)) { + $sensitivePatterns->push($database->$field); + } + } + + // Redact environment variables + if ($database->environment_variables) { + $lockedVars = $database->environment_variables + ->where('is_shown_once', true) + ->pluck('real_value') + ->filter(); + $sensitivePatterns = $sensitivePatterns->merge($lockedVars); + } + + foreach ($sensitivePatterns as $value) { + if (empty($value)) continue; + $escapedValue = preg_quote($value, '/'); + $text = preg_replace('/' . $escapedValue . '/', REDACTED, $text); + } + + return $text; + } + + public function addLogEntry(string $message, string $type = 'stdout', bool $hidden = false) + { + if ($type === 'error') { + $type = 'stderr'; + } + + $message = str($message)->trim(); + if ($message->startsWith('╔')) { + $message = "\n" . $message; + } + + $newLogEntry = [ + 'command' => null, + 'output' => $this->redactSensitiveInfo($message), + 'type' => $type, + 'timestamp' => Carbon::now('UTC'), + 'hidden' => $hidden, + 'batch' => 1, + ]; + + DB::transaction(function () use ($newLogEntry) { + $this->refresh(); + + if ($this->logs) { + $previousLogs = json_decode($this->logs, associative: true, flags: JSON_THROW_ON_ERROR); + $newLogEntry['order'] = count($previousLogs) + 1; + $previousLogs[] = $newLogEntry; + $this->logs = json_encode($previousLogs, flags: JSON_THROW_ON_ERROR); + } else { + $this->logs = json_encode([$newLogEntry], flags: JSON_THROW_ON_ERROR); + } + + $this->saveQuietly(); + }); + } +} +``` + +**Key Differences from ServiceDeploymentQueue:** +- Polymorphic `database()` relationship +- More extensive sensitive data redaction (database passwords, connection strings) +- Handles all 9 database types + +--- + +### Phase 3: Enums (2 new enums) + +#### Enum 1: ServiceDeploymentStatus + +**File:** `app/Enums/ServiceDeploymentStatus.php` + +```php +id; + $server = $service->destination->server; + $server_id = $server->id; + $server_name = $server->name; + + // Generate deployment URL + $deployment_link = Url::fromString($service->link() . "/deployment/{$deployment_uuid}"); + $deployment_url = $deployment_link->getPath(); + + // Create deployment record + $deployment = ServiceDeploymentQueue::create([ + 'service_id' => $service_id, + 'service_name' => $service->name, + 'server_id' => $server_id, + 'server_name' => $server_name, + 'deployment_uuid' => $deployment_uuid, + 'deployment_url' => $deployment_url, + 'pull_latest_images' => $pullLatestImages, + 'stop_before_start' => $stopBeforeStart, + 'is_api' => $is_api, + 'status' => ServiceDeploymentStatus::IN_PROGRESS->value, + ]); + + return [ + 'status' => 'started', + 'message' => 'Service deployment started.', + 'deployment_uuid' => $deployment_uuid, + 'deployment' => $deployment, + ]; +} +``` + +**Purpose:** Create deployment queue record when service starts. Returns deployment object for passing to actions. + +--- + +#### Helper 2: queue_database_deployment() + +**File:** `bootstrap/helpers/databases.php` (add to existing file) + +```php +use App\Models\DatabaseDeploymentQueue; +use App\Enums\DatabaseDeploymentStatus; +use Spatie\Url\Url; +use Visus\Cuid2\Cuid2; + +function queue_database_deployment( + StandalonePostgresql|StandaloneRedis|StandaloneMongodb|StandaloneMysql|StandaloneMariadb|StandaloneKeydb|StandaloneDragonfly|StandaloneClickhouse $database, + string $deployment_uuid, + bool $is_api = false +): array { + $database_id = $database->id; + $database_type = $database->getMorphClass(); + $server = $database->destination->server; + $server_id = $server->id; + $server_name = $server->name; + + // Generate deployment URL + $deployment_link = Url::fromString($database->link() . "/deployment/{$deployment_uuid}"); + $deployment_url = $deployment_link->getPath(); + + // Create deployment record + $deployment = DatabaseDeploymentQueue::create([ + 'database_id' => $database_id, + 'database_type' => $database_type, + 'database_name' => $database->name, + 'server_id' => $server_id, + 'server_name' => $server_name, + 'deployment_uuid' => $deployment_uuid, + 'deployment_url' => $deployment_url, + 'is_api' => $is_api, + 'status' => DatabaseDeploymentStatus::IN_PROGRESS->value, + ]); + + return [ + 'status' => 'started', + 'message' => 'Database deployment started.', + 'deployment_uuid' => $deployment_uuid, + 'deployment' => $deployment, + ]; +} +``` + +--- + +### Phase 5: Refactor Actions (11 files to update) + +#### Action 1: StartService (CRITICAL) + +**File:** `app/Actions/Service/StartService.php` + +**Before:** +```php +public function handle(Service $service, bool $pullLatestImages = false, bool $stopBeforeStart = false) +{ + $service->parse(); + // ... build commands + return remote_process($commands, $service->server, type_uuid: $service->uuid, callEventOnFinish: 'ServiceStatusChanged'); +} +``` + +**After:** +```php +use App\Models\ServiceDeploymentQueue; +use Visus\Cuid2\Cuid2; + +public function handle(Service $service, bool $pullLatestImages = false, bool $stopBeforeStart = false) +{ + // Create deployment queue record + $deployment_uuid = (string) new Cuid2(); + $result = queue_service_deployment( + service: $service, + deployment_uuid: $deployment_uuid, + pullLatestImages: $pullLatestImages, + stopBeforeStart: $stopBeforeStart, + is_api: false + ); + $deployment = $result['deployment']; + + // Existing logic + $service->parse(); + if ($stopBeforeStart) { + StopService::run(service: $service, dockerCleanup: false); + } + $service->saveComposeConfigs(); + $service->isConfigurationChanged(save: true); + + $commands[] = 'cd ' . $service->workdir(); + $commands[] = "echo 'Saved configuration files to {$service->workdir()}.'"; + // ... rest of command building + + // Pass deployment to remote_process for log streaming + return remote_process( + $commands, + $service->server, + type_uuid: $service->uuid, + model: $deployment, // NEW - link to deployment queue + callEventOnFinish: 'ServiceStatusChanged' + ); +} +``` + +**Key Changes:** +1. Generate deployment UUID at start +2. Call `queue_service_deployment()` helper +3. Pass `$deployment` as `model` parameter to `remote_process()` +4. Return value unchanged (Activity object) + +--- + +#### Actions 2-10: Database Start Actions (9 files) + +**Files to Update:** +- `app/Actions/Database/StartPostgresql.php` +- `app/Actions/Database/StartRedis.php` +- `app/Actions/Database/StartMongodb.php` +- `app/Actions/Database/StartMysql.php` +- `app/Actions/Database/StartMariadb.php` +- `app/Actions/Database/StartKeydb.php` +- `app/Actions/Database/StartDragonfly.php` +- `app/Actions/Database/StartClickhouse.php` + +**Pattern (using StartPostgresql as example):** + +**Before:** +```php +public function handle(StandalonePostgresql $database) +{ + $this->database = $database; + // ... build docker-compose and commands + return remote_process($this->commands, $database->destination->server, callEventOnFinish: 'DatabaseStatusChanged'); +} +``` + +**After:** +```php +use App\Models\DatabaseDeploymentQueue; +use Visus\Cuid2\Cuid2; + +public function handle(StandalonePostgresql $database) +{ + $this->database = $database; + + // Create deployment queue record + $deployment_uuid = (string) new Cuid2(); + $result = queue_database_deployment( + database: $database, + deployment_uuid: $deployment_uuid, + is_api: false + ); + $deployment = $result['deployment']; + + // Existing logic (unchanged) + $container_name = $this->database->uuid; + $this->configuration_dir = database_configuration_dir() . '/' . $container_name; + // ... rest of setup + + // Pass deployment to remote_process + return remote_process( + $this->commands, + $database->destination->server, + model: $deployment, // NEW + callEventOnFinish: 'DatabaseStatusChanged' + ); +} +``` + +**Apply Same Pattern to All 9 Database Start Actions** + +--- + +#### Action 11: StartDatabase (Dispatcher) + +**File:** `app/Actions/Database/StartDatabase.php` + +**Before:** +```php +public function handle(/* all database types */) +{ + switch ($database->getMorphClass()) { + case \App\Models\StandalonePostgresql::class: + $activity = StartPostgresql::run($database); + break; + // ... other cases + } + return $activity; +} +``` + +**After:** No changes needed - already returns Activity from Start* actions + +--- + +### Phase 6: Update Remote Process Handler (CRITICAL) + +**File:** `app/Actions/CoolifyTask/PrepareCoolifyTask.php` + +**Current Behavior:** +- Accepts `$model` parameter (currently only used for ApplicationDeploymentQueue) +- Streams logs to Activity (Spatie ActivityLog) +- Calls event on finish + +**Required Changes:** +1. Check if `$model` is `ServiceDeploymentQueue` or `DatabaseDeploymentQueue` +2. Call `addLogEntry()` on deployment model alongside Activity logs +3. Update deployment status on completion/failure + +**Pseudocode for Changes:** +```php +// In log streaming section +if ($model instanceof ApplicationDeploymentQueue || + $model instanceof ServiceDeploymentQueue || + $model instanceof DatabaseDeploymentQueue) { + $model->addLogEntry($logMessage, $logType); +} + +// On completion +if ($model instanceof ServiceDeploymentQueue || + $model instanceof DatabaseDeploymentQueue) { + if ($exitCode === 0) { + $model->setStatus('finished'); + } else { + $model->setStatus('failed'); + } +} +``` + +**Note:** Exact implementation depends on PrepareCoolifyTask structure. Need to review file in detail during implementation. + +--- + +### Phase 7: API Endpoints (4 new endpoints + 2 updates) + +**File:** `app/Http/Controllers/Api/DeployController.php` + +#### Endpoint 1: List Service Deployments + +```php +#[OA\Get( + summary: 'List service deployments', + description: 'List deployment history for a specific service', + path: '/deployments/services/{uuid}', + operationId: 'list-deployments-by-service-uuid', + security: [['bearerAuth' => []]], + tags: ['Deployments'], + parameters: [ + new OA\Parameter(name: 'uuid', in: 'path', required: true, description: 'Service UUID', schema: new OA\Schema(type: 'string')), + new OA\Parameter(name: 'skip', in: 'query', description: 'Number of records to skip', schema: new OA\Schema(type: 'integer', minimum: 0, default: 0)), + new OA\Parameter(name: 'take', in: 'query', description: 'Number of records to take', schema: new OA\Schema(type: 'integer', minimum: 1, default: 10)), + ], + responses: [ + new OA\Response(response: 200, description: 'List of service deployments'), + new OA\Response(response: 401, ref: '#/components/responses/401'), + new OA\Response(response: 404, ref: '#/components/responses/404'), + ] +)] +public function get_service_deployments(Request $request) +{ + $request->validate([ + 'skip' => ['nullable', 'integer', 'min:0'], + 'take' => ['nullable', 'integer', 'min:1'], + ]); + + $service_uuid = $request->route('uuid', null); + $skip = $request->get('skip', 0); + $take = $request->get('take', 10); + + $teamId = getTeamIdFromToken(); + if (is_null($teamId)) { + return invalidTokenResponse(); + } + + $service = Service::where('uuid', $service_uuid) + ->whereHas('environment.project.team', function($query) use ($teamId) { + $query->where('id', $teamId); + }) + ->first(); + + if (is_null($service)) { + return response()->json(['message' => 'Service not found'], 404); + } + + $this->authorize('view', $service); + + $deployments = $service->deployments($skip, $take); + + return response()->json(serializeApiResponse($deployments)); +} +``` + +#### Endpoint 2: Get Service Deployment by UUID + +```php +#[OA\Get( + summary: 'Get service deployment', + description: 'Get a specific service deployment by deployment UUID', + path: '/deployments/services/deployment/{uuid}', + operationId: 'get-service-deployment-by-uuid', + security: [['bearerAuth' => []]], + tags: ['Deployments'], + parameters: [ + new OA\Parameter(name: 'uuid', in: 'path', required: true, description: 'Deployment UUID', schema: new OA\Schema(type: 'string')), + ], + responses: [ + new OA\Response(response: 200, description: 'Service deployment details'), + new OA\Response(response: 401, ref: '#/components/responses/401'), + new OA\Response(response: 404, ref: '#/components/responses/404'), + ] +)] +public function service_deployment_by_uuid(Request $request) +{ + $teamId = getTeamIdFromToken(); + if (is_null($teamId)) { + return invalidTokenResponse(); + } + + $uuid = $request->route('uuid'); + if (!$uuid) { + return response()->json(['message' => 'UUID is required.'], 400); + } + + $deployment = ServiceDeploymentQueue::where('deployment_uuid', $uuid)->first(); + if (!$deployment) { + return response()->json(['message' => 'Deployment not found.'], 404); + } + + // Authorization check via service + $service = $deployment->service; + if (!$service) { + return response()->json(['message' => 'Service not found.'], 404); + } + + $this->authorize('view', $service); + + return response()->json($this->removeSensitiveData($deployment)); +} +``` + +#### Endpoint 3: List Database Deployments + +```php +#[OA\Get( + summary: 'List database deployments', + description: 'List deployment history for a specific database', + path: '/deployments/databases/{uuid}', + operationId: 'list-deployments-by-database-uuid', + security: [['bearerAuth' => []]], + tags: ['Deployments'], + parameters: [ + new OA\Parameter(name: 'uuid', in: 'path', required: true, description: 'Database UUID', schema: new OA\Schema(type: 'string')), + new OA\Parameter(name: 'skip', in: 'query', schema: new OA\Schema(type: 'integer', minimum: 0, default: 0)), + new OA\Parameter(name: 'take', in: 'query', schema: new OA\Schema(type: 'integer', minimum: 1, default: 10)), + ], + responses: [ + new OA\Response(response: 200, description: 'List of database deployments'), + new OA\Response(response: 401, ref: '#/components/responses/401'), + new OA\Response(response: 404, ref: '#/components/responses/404'), + ] +)] +public function get_database_deployments(Request $request) +{ + $request->validate([ + 'skip' => ['nullable', 'integer', 'min:0'], + 'take' => ['nullable', 'integer', 'min:1'], + ]); + + $database_uuid = $request->route('uuid', null); + $skip = $request->get('skip', 0); + $take = $request->get('take', 10); + + $teamId = getTeamIdFromToken(); + if (is_null($teamId)) { + return invalidTokenResponse(); + } + + // Find database across all types + $database = getResourceByUuid($database_uuid, $teamId); + + if (!$database || !method_exists($database, 'deployments')) { + return response()->json(['message' => 'Database not found'], 404); + } + + $this->authorize('view', $database); + + $deployments = $database->deployments($skip, $take); + + return response()->json(serializeApiResponse($deployments)); +} +``` + +#### Endpoint 4: Get Database Deployment by UUID + +```php +#[OA\Get( + summary: 'Get database deployment', + description: 'Get a specific database deployment by deployment UUID', + path: '/deployments/databases/deployment/{uuid}', + operationId: 'get-database-deployment-by-uuid', + security: [['bearerAuth' => []]], + tags: ['Deployments'], + parameters: [ + new OA\Parameter(name: 'uuid', in: 'path', required: true, description: 'Deployment UUID', schema: new OA\Schema(type: 'string')), + ], + responses: [ + new OA\Response(response: 200, description: 'Database deployment details'), + new OA\Response(response: 401, ref: '#/components/responses/401'), + new OA\Response(response: 404, ref: '#/components/responses/404'), + ] +)] +public function database_deployment_by_uuid(Request $request) +{ + $teamId = getTeamIdFromToken(); + if (is_null($teamId)) { + return invalidTokenResponse(); + } + + $uuid = $request->route('uuid'); + if (!$uuid) { + return response()->json(['message' => 'UUID is required.'], 400); + } + + $deployment = DatabaseDeploymentQueue::where('deployment_uuid', $uuid)->first(); + if (!$deployment) { + return response()->json(['message' => 'Deployment not found.'], 404); + } + + // Authorization check via database + $database = $deployment->database; + if (!$database) { + return response()->json(['message' => 'Database not found.'], 404); + } + + $this->authorize('view', $database); + + return response()->json($this->removeSensitiveData($deployment)); +} +``` + +#### Update: removeSensitiveData() method + +```php +private function removeSensitiveData($deployment) +{ + if (request()->attributes->get('can_read_sensitive', false) === false) { + $deployment->makeHidden(['logs']); + } + return serializeApiResponse($deployment); +} +``` + +**Note:** Already works for ServiceDeploymentQueue and DatabaseDeploymentQueue due to duck typing + +#### Update: deploy_resource() method + +**Before:** +```php +case Service::class: + StartService::run($resource); + $message = "Service {$resource->name} started. It could take a while, be patient."; + break; + +default: // Database + StartDatabase::dispatch($resource); + $message = "Database {$resource->name} started."; + break; +``` + +**After:** +```php +case Service::class: + $this->authorize('deploy', $resource); + $deployment_uuid = new Cuid2; + // StartService now handles deployment queue creation internally + StartService::run($resource); + $message = "Service {$resource->name} deployment started."; + break; + +default: // Database + $this->authorize('manage', $resource); + $deployment_uuid = new Cuid2; + // Start actions now handle deployment queue creation internally + StartDatabase::dispatch($resource); + $message = "Database {$resource->name} deployment started."; + break; +``` + +**Note:** deployment_uuid is now created inside actions, so API just returns message. If we want to return UUID to API, actions need to return deployment object. + +--- + +### Phase 8: Model Relationships (2 model updates) + +#### Update 1: Service Model + +**File:** `app/Models/Service.php` + +**Add Method:** +```php +/** + * Get deployment history for this service + */ +public function deployments(int $skip = 0, int $take = 10) +{ + return ServiceDeploymentQueue::where('service_id', $this->id) + ->orderBy('created_at', 'desc') + ->skip($skip) + ->take($take) + ->get(); +} + +/** + * Get latest deployment + */ +public function latestDeployment() +{ + return ServiceDeploymentQueue::where('service_id', $this->id) + ->orderBy('created_at', 'desc') + ->first(); +} +``` + +--- + +#### Update 2: All Standalone Database Models (9 files) + +**Files:** +- `app/Models/StandalonePostgresql.php` +- `app/Models/StandaloneRedis.php` +- `app/Models/StandaloneMongodb.php` +- `app/Models/StandaloneMysql.php` +- `app/Models/StandaloneMariadb.php` +- `app/Models/StandaloneKeydb.php` +- `app/Models/StandaloneDragonfly.php` +- `app/Models/StandaloneClickhouse.php` + +**Add Methods to Each:** +```php +/** + * Get deployment history for this database + */ +public function deployments(int $skip = 0, int $take = 10) +{ + return DatabaseDeploymentQueue::where('database_id', $this->id) + ->where('database_type', $this->getMorphClass()) + ->orderBy('created_at', 'desc') + ->skip($skip) + ->take($take) + ->get(); +} + +/** + * Get latest deployment + */ +public function latestDeployment() +{ + return DatabaseDeploymentQueue::where('database_id', $this->id) + ->where('database_type', $this->getMorphClass()) + ->orderBy('created_at', 'desc') + ->first(); +} +``` + +--- + +### Phase 9: Routes (4 new routes) + +**File:** `routes/api.php` + +**Add Routes:** +```php +Route::middleware(['auth:sanctum'])->group(function () { + // Existing routes... + + // Service deployment routes + Route::get('/deployments/services/{uuid}', [DeployController::class, 'get_service_deployments']) + ->name('deployments.services.list'); + Route::get('/deployments/services/deployment/{uuid}', [DeployController::class, 'service_deployment_by_uuid']) + ->name('deployments.services.show'); + + // Database deployment routes + Route::get('/deployments/databases/{uuid}', [DeployController::class, 'get_database_deployments']) + ->name('deployments.databases.list'); + Route::get('/deployments/databases/deployment/{uuid}', [DeployController::class, 'database_deployment_by_uuid']) + ->name('deployments.databases.show'); +}); +``` + +--- + +### Phase 10: Policies & Authorization (Optional - If needed) + +**Service Policy:** `app/Policies/ServicePolicy.php` +- May need to add `viewDeployment` and `viewDeployments` methods if they don't exist +- Check existing `view` gate - it should cover deployment viewing + +**Database Policies:** +- Each StandaloneDatabase type may have its own policy +- Verify `view` gate exists and covers deployment history access + +**Action Required:** Review existing policies during implementation. May not need changes if `view` gate is sufficient. + +--- + +## Testing Strategy + +### Unit Tests (Run outside Docker: `./vendor/bin/pest tests/Unit`) + +#### Test 1: ServiceDeploymentQueue Unit Test + +**File:** `tests/Unit/Models/ServiceDeploymentQueueTest.php` + +```php +create([ + 'logs' => null, + ]); + + $deployment->addLogEntry('Test message', 'stdout', false); + + expect($deployment->fresh()->logs)->not->toBeNull(); + + $logs = json_decode($deployment->fresh()->logs, true); + expect($logs)->toHaveCount(1); + expect($logs[0])->toHaveKeys(['command', 'output', 'type', 'timestamp', 'hidden', 'batch', 'order']); + expect($logs[0]['output'])->toBe('Test message'); + expect($logs[0]['type'])->toBe('stdout'); +}); + +it('redacts sensitive environment variables in logs', function () { + $service = Mockery::mock(Service::class); + $envVar = new \StdClass(); + $envVar->is_shown_once = true; + $envVar->key = 'SECRET_KEY'; + $envVar->real_value = 'super-secret-value'; + + $service->shouldReceive('getAttribute') + ->with('environment_variables') + ->andReturn(collect([$envVar])); + + $deployment = ServiceDeploymentQueue::factory()->create(); + $deployment->setRelation('service', $service); + + $deployment->addLogEntry('Deploying with super-secret-value in logs', 'stdout'); + + $logs = json_decode($deployment->fresh()->logs, true); + expect($logs[0]['output'])->toContain(REDACTED); + expect($logs[0]['output'])->not->toContain('super-secret-value'); +}); + +it('sets status correctly', function () { + $deployment = ServiceDeploymentQueue::factory()->create(['status' => 'queued']); + + $deployment->setStatus('in_progress'); + expect($deployment->fresh()->status)->toBe('in_progress'); + + $deployment->setStatus('finished'); + expect($deployment->fresh()->status)->toBe('finished'); +}); +``` + +#### Test 2: DatabaseDeploymentQueue Unit Test + +**File:** `tests/Unit/Models/DatabaseDeploymentQueueTest.php` + +```php +create([ + 'logs' => null, + ]); + + $deployment->addLogEntry('Starting database', 'stdout', false); + + $logs = json_decode($deployment->fresh()->logs, true); + expect($logs)->toHaveCount(1); + expect($logs[0]['output'])->toBe('Starting database'); +}); + +it('redacts database credentials in logs', function () { + $database = Mockery::mock(StandalonePostgresql::class); + $database->shouldReceive('getAttribute') + ->with('postgres_password') + ->andReturn('db-password-123'); + $database->shouldReceive('getAttribute') + ->with('environment_variables') + ->andReturn(collect([])); + $database->shouldReceive('getMorphClass') + ->andReturn(StandalonePostgresql::class); + + $deployment = DatabaseDeploymentQueue::factory()->create([ + 'database_type' => StandalonePostgresql::class, + ]); + $deployment->setRelation('database', $database); + + $deployment->addLogEntry('Connecting with password db-password-123', 'stdout'); + + $logs = json_decode($deployment->fresh()->logs, true); + expect($logs[0]['output'])->toContain(REDACTED); + expect($logs[0]['output'])->not->toContain('db-password-123'); +}); +``` + +--- + +### Feature Tests (Run inside Docker: `docker exec coolify php artisan test`) + +#### Test 3: Service Deployment Integration Test + +**File:** `tests/Feature/ServiceDeploymentTest.php` + +```php +create(); + + // Mock remote_process to prevent actual SSH + // (Implementation depends on existing test patterns) + + StartService::run($service); + + $deployment = ServiceDeploymentQueue::where('service_id', $service->id)->first(); + expect($deployment)->not->toBeNull(); + expect($deployment->service_name)->toBe($service->name); + expect($deployment->status)->toBe('in_progress'); +}); + +it('tracks multiple deployments for same service', function () { + $service = Service::factory()->create(); + + StartService::run($service); + StartService::run($service); + + $deployments = ServiceDeploymentQueue::where('service_id', $service->id)->get(); + expect($deployments)->toHaveCount(2); +}); +``` + +#### Test 4: Database Deployment Integration Test + +**File:** `tests/Feature/DatabaseDeploymentTest.php` + +```php +create(); + + // Mock remote_process + + StartPostgresql::run($database); + + $deployment = DatabaseDeploymentQueue::where('database_id', $database->id) + ->where('database_type', StandalonePostgresql::class) + ->first(); + + expect($deployment)->not->toBeNull(); + expect($deployment->database_name)->toBe($database->name); +}); + +// Repeat for other database types... +``` + +#### Test 5: API Endpoint Tests + +**File:** `tests/Feature/Api/DeploymentApiTest.php` + +```php +create(); + $service = Service::factory()->create([ + 'environment_id' => /* setup team/project/env */ + ]); + + ServiceDeploymentQueue::factory()->count(3)->create([ + 'service_id' => $service->id, + ]); + + $response = $this->actingAs($user) + ->getJson("/api/deployments/services/{$service->uuid}"); + + $response->assertSuccessful(); + $response->assertJsonCount(3); +}); + +it('requires authentication for service deployments', function () { + $service = Service::factory()->create(); + + $response = $this->getJson("/api/deployments/services/{$service->uuid}"); + + $response->assertUnauthorized(); +}); + +// Repeat for database endpoints... +``` + +--- + +## Rollout Plan + +### Phase Order (Safest to Riskiest) + +| Phase | Risk | Can Break Production? | Rollback Strategy | +|-------|------|----------------------|-------------------| +| 1. Schema | Low | No (new tables) | Drop tables | +| 2. Models | Low | No (unused code) | Remove files | +| 3. Enums | Low | No (unused code) | Remove files | +| 4. Helpers | Low | No (unused code) | Remove functions | +| 5. Actions | **HIGH** | **YES** | Revert to old actions | +| 6. Remote Process | **CRITICAL** | **YES** | Revert changes | +| 7. API | Medium | No (new endpoints) | Remove routes | +| 8. Relationships | Low | No (new methods) | Remove methods | +| 9. UI | Low | No (optional) | Remove components | +| 10. Policies | Low | Maybe (if breaking existing) | Revert gates | + +### Recommended Rollout Strategy + +**Week 1: Foundation (No Risk)** +- Complete Phases 1-4 +- Write and run all unit tests +- Verify migrations work in dev/staging + +**Week 2: Critical Changes (High Risk)** +- Complete Phase 5 (Actions) for **Services only** +- Complete Phase 6 (Remote Process handler) for Services +- Test extensively in staging +- Monitor for errors + +**Week 3: Database Support** +- Extend Phase 5 to all 9 database types +- Update Phase 6 for database support +- Test each database type individually + +**Week 4: API & Polish** +- Complete Phases 7-10 +- Feature tests +- API documentation +- User-facing features (if any) + +### Testing Checkpoints + +**After Phase 4:** +- ✅ Migrations apply cleanly +- ✅ Models instantiate without errors +- ✅ Unit tests pass + +**After Phase 5 (Services):** +- ✅ Service start creates deployment queue +- ✅ Service logs stream to deployment queue +- ✅ Service deployments appear in database +- ✅ No disruption to existing service starts + +**After Phase 5 (Databases):** +- ✅ Each database type creates deployment queue +- ✅ Database logs stream correctly +- ✅ No errors on database start + +**After Phase 7:** +- ✅ API endpoints return correct data +- ✅ Authorization works correctly +- ✅ Sensitive data is redacted + +--- + +## Known Risks & Mitigation + +### Risk 1: Breaking Existing Deployments +**Probability:** Medium +**Impact:** Critical + +**Mitigation:** +- Test exhaustively in staging before production +- Deploy during low-traffic window +- Have rollback plan ready (git revert + migration rollback) +- Monitor error logs closely after deploy + +### Risk 2: Database Performance Impact +**Probability:** Low +**Impact:** Medium + +**Details:** Each deployment now writes logs to DB multiple times (via `addLogEntry()`) + +**Mitigation:** +- Use `saveQuietly()` to avoid triggering events +- JSON column is indexed for fast retrieval +- Logs are text (compressed well by Postgres) +- Add monitoring for slow queries + +### Risk 3: Disk Space Growth +**Probability:** Medium (long-term) +**Impact:** Low + +**Details:** Deployment logs accumulate over time + +**Mitigation:** +- Implement log retention policy (delete deployments older than X days/months) +- Add background job to prune old deployment records +- Monitor disk usage trends + +### Risk 4: Polymorphic Relationship Complexity +**Probability:** Low +**Impact:** Low + +**Details:** DatabaseDeploymentQueue uses polymorphic relationship (9 database types) + +**Mitigation:** +- Thorough testing of each database type +- Composite indexes on (database_id, database_type) +- Clear documentation of relationship structure + +### Risk 5: Remote Process Integration +**Probability:** High +**Impact:** Critical + +**Details:** `PrepareCoolifyTask` is core to all deployments. Changes here affect everything. + +**Mitigation:** +- Review `PrepareCoolifyTask` code in detail before changes +- Add type checks (`instanceof`) to avoid breaking existing logic +- Extensive testing of application deployments after changes +- Keep changes minimal and focused + +--- + +## Migration Strategy for Existing Data + +**Q: What about existing services/databases that have been deployed before?** + +**A:** No migration needed. This is a **new feature**, not a data migration. + +- Services/databases deployed before this change won't have history +- New deployments (after feature is live) will be tracked +- This is acceptable - deployment history starts "now" + +**Alternative (if history is critical):** +- Could create fake deployment records for currently running resources +- Not recommended - logs don't exist, would be misleading + +--- + +## Performance Considerations + +### Database Writes During Deployment + +**Current:** ~1 write per deployment (Activity log, TTL-based) + +**New:** ~1 write per deployment + N writes for log entries +- Application deployments: ~50-200 log entries +- Service deployments: ~10-30 log entries +- Database deployments: ~5-15 log entries + +**Impact:** Minimal +- Writes are async (queued) +- Postgres handles small JSON updates efficiently +- `saveQuietly()` skips event dispatching overhead + +### Query Performance + +**Critical Queries:** +- "Get deployment history for service/database" - indexed on (resource_id, status, created_at) +- "Get deployment by UUID" - unique index on deployment_uuid +- "Get all in-progress deployments" - composite index on (server_id, status, created_at) + +**Expected Performance:** +- < 10ms for single deployment lookup +- < 50ms for paginated history (10 records) +- < 100ms for server-wide deployment status + +--- + +## Storage Estimates + +**Per Deployment:** +- Metadata: ~500 bytes +- Logs (avg): ~50KB (application), ~10KB (service), ~5KB (database) + +**1000 deployments/day:** +- Services: ~10MB/day = ~300MB/month +- Databases: ~5MB/day = ~150MB/month +- Total: ~450MB/month (highly compressible) + +**Retention Policy Recommendation:** +- Keep all deployments for 30 days +- Keep successful deployments for 90 days +- Keep failed deployments for 180 days (for debugging) + +--- + +## Alternative Approaches Considered + +### Option 1: Unified Resource Deployments Table + +**Schema:** +```sql +CREATE TABLE resource_deployments ( + id BIGINT PRIMARY KEY, + deployable_id INT, + deployable_type VARCHAR(255), -- App\Models\Service, App\Models\StandalonePostgresql, etc. + deployment_uuid VARCHAR(255) UNIQUE, + -- ... rest of fields + INDEX(deployable_id, deployable_type) +); +``` + +**Pros:** +- Single model to maintain +- DRY (Don't Repeat Yourself) +- Easier to query "all deployments across all resources" + +**Cons:** +- Polymorphic queries are slower +- No foreign key constraints +- Different resources have different deployment attributes +- Harder to optimize indexes per resource type +- More complex to reason about + +**Decision:** Rejected - Separate tables provide better type safety and performance + +--- + +### Option 2: Reuse Activity Log (Spatie) + +**Approach:** Don't create deployment queue tables. Use existing Activity log with longer TTL. + +**Pros:** +- Zero new code +- Activity log already stores logs + +**Cons:** +- Activity log is ephemeral (not designed for permanent history) +- No structured deployment metadata (status, UUIDs, etc.) +- Would need to change Activity TTL globally (affects all activities) +- Mixing concerns (Activity = audit log, Deployment = business logic) + +**Decision:** Rejected - Activity log and deployment history serve different purposes + +--- + +### Option 3: External Logging Service + +**Approach:** Stream logs to external service (S3, CloudWatch, etc.) + +**Pros:** +- Offload storage from main database +- Better for very large log volumes + +**Cons:** +- Additional infrastructure complexity +- Requires external dependencies +- Harder to query deployment history +- Not consistent with application deployment pattern + +**Decision:** Rejected - Keep it simple, follow existing patterns + +--- + +## Future Enhancements (Out of Scope) + +### 1. Deployment Queue System +- Like application deployments, queue service/database starts +- Respect server concurrent limits +- **Complexity:** High +- **Value:** Medium (services/databases deploy fast, queueing less critical) + +### 2. UI for Deployment History +- Livewire components to view past deployments +- Similar to application deployment history page +- **Complexity:** Medium +- **Value:** High (nice-to-have, not critical for first release) + +### 3. Deployment Comparison +- Diff between two deployments (config changes) +- **Complexity:** High +- **Value:** Low + +### 4. Deployment Rollback +- Roll back service/database to previous deployment +- **Complexity:** Very High (databases especially risky) +- **Value:** Medium + +### 5. Deployment Notifications +- Notify on service/database deployment success/failure +- **Complexity:** Low +- **Value:** Medium + +--- + +## Success Criteria + +### Minimum Viable Product (MVP) + +✅ Service deployments create deployment queue records +✅ Database deployments (all 9 types) create deployment queue records +✅ Logs stream to deployment queue during deployment +✅ Deployment status updates (in_progress → finished/failed) +✅ API endpoints to retrieve deployment history +✅ Sensitive data redaction in logs +✅ No disruption to existing application deployments +✅ All unit and feature tests pass + +### Nice-to-Have (Post-MVP) + +⚪ UI components for viewing deployment history +⚪ Deployment notifications +⚪ Log retention policy job +⚪ Deployment statistics/analytics + +--- + +## Questions to Resolve Before Implementation + +1. **Should we queue service/database starts (like applications)?** + - Current: Services/databases start immediately + - With queue: Respect server concurrent limits, better for cloud instance + - **Recommendation:** Start without queue, add later if needed + +2. **Should API deploy endpoints return deployment_uuid for services/databases?** + - Current: Application deploys return deployment_uuid + - Proposed: Services/databases should too + - **Recommendation:** Yes, for consistency. Requires actions to return deployment object. + +3. **What's the log retention policy?** + - **Recommendation:** 90 days for all, with background job to prune + +4. **Do we need UI in first release?** + - **Recommendation:** No, API is sufficient. Add UI iteratively. + +5. **Should we implement deployment cancellation?** + - Applications support cancellation + - **Recommendation:** Not in MVP, add later if requested + +--- + +## Implementation Checklist + +### Pre-Implementation +- [ ] Review this plan with team +- [ ] Get approval on architectural decisions +- [ ] Resolve open questions +- [ ] Set up staging environment for testing + +### Phase 1: Schema +- [ ] Create `create_service_deployment_queues_table` migration +- [ ] Create `create_database_deployment_queues_table` migration +- [ ] Create index optimization migration +- [ ] Test migrations in dev +- [ ] Run migrations in staging + +### Phase 2: Models +- [ ] Create `ServiceDeploymentQueue` model +- [ ] Create `DatabaseDeploymentQueue` model +- [ ] Add `$fillable`, `$guarded` properties +- [ ] Implement `addLogEntry()`, `setStatus()`, `getOutput()` methods +- [ ] Implement `redactSensitiveInfo()` methods +- [ ] Add OpenAPI schemas + +### Phase 3: Enums +- [ ] Create `ServiceDeploymentStatus` enum +- [ ] Create `DatabaseDeploymentStatus` enum + +### Phase 4: Helpers +- [ ] Add `queue_service_deployment()` to `bootstrap/helpers/services.php` +- [ ] Add `queue_database_deployment()` to `bootstrap/helpers/databases.php` +- [ ] Test helpers in Tinker + +### Phase 5: Actions +- [ ] Update `StartService` action +- [ ] Update `StartPostgresql` action +- [ ] Update `StartRedis` action +- [ ] Update `StartMongodb` action +- [ ] Update `StartMysql` action +- [ ] Update `StartMariadb` action +- [ ] Update `StartKeydb` action +- [ ] Update `StartDragonfly` action +- [ ] Update `StartClickhouse` action +- [ ] Test each action in staging + +### Phase 6: Remote Process +- [ ] Review `PrepareCoolifyTask` code +- [ ] Add type checks for ServiceDeploymentQueue +- [ ] Add type checks for DatabaseDeploymentQueue +- [ ] Add `addLogEntry()` calls +- [ ] Add status update logic +- [ ] Test with application deployments (ensure no regression) +- [ ] Test with service deployments +- [ ] Test with database deployments + +### Phase 7: API +- [ ] Add `get_service_deployments()` endpoint +- [ ] Add `service_deployment_by_uuid()` endpoint +- [ ] Add `get_database_deployments()` endpoint +- [ ] Add `database_deployment_by_uuid()` endpoint +- [ ] Update `deploy_resource()` to return deployment_uuid +- [ ] Update `removeSensitiveData()` if needed +- [ ] Add routes to `api.php` +- [ ] Test endpoints with Postman/curl + +### Phase 8: Relationships +- [ ] Add `deployments()` method to `Service` model +- [ ] Add `latestDeployment()` method to `Service` model +- [ ] Add `deployments()` method to all 9 Standalone database models +- [ ] Add `latestDeployment()` method to all 9 Standalone database models + +### Phase 9: Tests +- [ ] Write `ServiceDeploymentQueueTest` (unit) +- [ ] Write `DatabaseDeploymentQueueTest` (unit) +- [ ] Write `ServiceDeploymentTest` (feature) +- [ ] Write `DatabaseDeploymentTest` (feature) +- [ ] Write `DeploymentApiTest` (feature) +- [ ] Run all tests, ensure passing +- [ ] Run full test suite, ensure no regressions + +### Phase 10: Documentation +- [ ] Update API documentation +- [ ] Update CLAUDE.md if needed +- [ ] Add code comments for complex sections + +### Deployment +- [ ] Create PR with all changes +- [ ] Code review +- [ ] Test in staging (full regression suite) +- [ ] Deploy to production during low-traffic window +- [ ] Monitor error logs for 24 hours +- [ ] Verify deployments are being tracked + +### Post-Deployment +- [ ] Monitor disk usage trends +- [ ] Monitor query performance +- [ ] Gather user feedback +- [ ] Plan UI implementation (if needed) +- [ ] Plan log retention job + +--- + +## Contact & Support + +**Implementation Lead:** [Your Name] +**Reviewer:** [Reviewer Name] +**Questions:** Reference this document or ask in #dev channel + +--- + +**Last Updated:** 2025-10-30 +**Status:** Planning Complete, Ready for Implementation +**Next Step:** Review plan with team, get approval, begin Phase 1