[Internal] Self-hosting is difficult due to missing documentation and Docker configuration gaps

[Alex-Alaniz]

OSS Self-Hosting Deep Dive - Investigation Report

Date: October 30, 2025
Engineer: Alex Alaniz
Status: Research Complete - Ready for Implementation Planning

---

Executive Summary

Completed comprehensive investigation of self-hosting difficulties. Mariano's concerns are confirmed and validated. Found critical gaps in documentation, Docker configuration, and environment variable handling that make self-hosting unnecessarily difficult.

Key Finding: Environment variables required for build are not documented in Docker setup, and there are major inconsistencies between documentation sources.

---

Critical Issues Found

1. Environment Variables "Don't Say That in the Docker"

Confirmed Issue: Docker setup is missing critical environment variable documentation.

Specific Problems:

• Dockerfile declares 10 NEXT_PUBLIC_* build args
• docker-compose.yml only passes 1 of those 10 args
• SELF_HOSTING.md doesn't explain build-time vs runtime variables
• turbo.json requires 40+ env vars for build, but Docker docs mention ~23

Impact: Users run docker compose build and get successful builds (due to SKIP_ENV_VALIDATION=true), but apps crash at runtime with undefined values.

2. Documentation is Inconsistent and Hard to Follow

Problems Found:

• SELF_HOSTING.md says use root .env file
• docker-compose.yml actually uses apps/app/.env and apps/portal/.env
• 4 different .env.example files with overlapping variables
• Some variables marked "optional" in docs but "required" in code
• No online self-hosting docs at https://trycomp.ai/docs

Impact: Users following SELF_HOSTING.md create wrong .env file that containers never read.

3. Missing Critical Components

Not in Docker Setup:

• API service (apps/api) completely missing from docker-compose.yml
• PostgreSQL not included (requires external DB)
• No service orchestration (migrations don't auto-run before apps start)

Impact: Incomplete containerization - can't run full stack with Docker alone.

4. Can't Achieve "Single Command" Setup

Current Required Steps: 15+ manual steps taking 2-4 hours
Blockers:

• No setup script/automation
• Must manually create 4 separate .env files
• Must manually generate secrets (AUTH_SECRET, etc.)
• Must manually run database migrations and seeding
• Must manually generate Prisma clients for each app
• External services (Resend, Trigger.dev, AWS S3) have no local mocks

---

Environment Variables - The Complete Picture

Variables in turbo.json but NOT in SELF_HOSTING.md (39 missing):

Critical Missing:

1. AUTH_GOOGLE_ID / AUTH_GOOGLE_SECRET - OAuth
2. AUTH_GITHUB_ID / AUTH_GITHUB_SECRET - OAuth
3. TRIGGER_API_KEY / TRIGGER_API_URL - For self-hosted Trigger.dev
4. APP_AWS_ACCESS_KEY_ID / APP_AWS_SECRET_ACCESS_KEY / APP_AWS_REGION / APP_AWS_BUCKET_NAME - S3 for attachments (marked "optional" but actually required)
5. SECRET_KEY - For encrypting data (not mentioned at all)
6. NEXT_PUBLIC_PORTAL_URL - Marked required in env.mjs, not in Docker args

Also Missing: Vercel vars, Fleet vars, LinkedIn vars, and more.

Build Args Not Wired Through:

Dockerfile declares these build args but docker-compose.yml doesn't pass them:

• NEXT_PUBLIC_PORTAL_URL
• NEXT_PUBLIC_POSTHOG_KEY
• NEXT_PUBLIC_POSTHOG_HOST
• NEXT_PUBLIC_IS_DUB_ENABLED
• NEXT_PUBLIC_GTM_ID
• NEXT_PUBLIC_LINKEDIN_PARTNER_ID
• NEXT_PUBLIC_LINKEDIN_CONVERSION_ID
• NEXT_PUBLIC_GOOGLE_ADS_CONVERSION_LABEL
• NEXT_PUBLIC_API_URL

Impact: These become undefined in client-side code, breaking features.

---

What Prevents Single-Command Setup

Current Manual Process:

1. Clone repo
2. bun install
3. Copy 4 different .env files
4. Manually fill out 40+ variables across files
5. Generate 3 secrets with openssl
6. Start PostgreSQL manually
7. Run migrations manually
8. Seed database manually
9. Generate Prisma clients for each app
10. Finally run dev server

Blockers to Single Command:

1. No setup automation script
2. No unified .env template
3. No secret auto-generation
4. Database initialization not orchestrated in docker-compose
5. No development mode defaults that work out-of-box
6. API not in docker-compose.yml
7. No env validation before startup
8. External service setup not documented clearly

---

Recommended Quick Wins

Priority 1: Fix Docker Setup

1.1 Wire Through All Build Args in docker-compose.yml:

app:
  build:
    args:
      NEXT_PUBLIC_BETTER_AUTH_URL: ${NEXT_PUBLIC_BETTER_AUTH_URL}
      NEXT_PUBLIC_PORTAL_URL: ${NEXT_PUBLIC_PORTAL_URL}
      NEXT_PUBLIC_POSTHOG_KEY: ${NEXT_PUBLIC_POSTHOG_KEY:-}
      NEXT_PUBLIC_POSTHOG_HOST: ${NEXT_PUBLIC_POSTHOG_HOST:-}
      # ... all 10 args

1.2 Add API Service to docker-compose.yml:
Currently missing entirely - needs new stage in Dockerfile + service in compose.

1.3 Add Service Dependencies:

app:
  depends_on:
    postgres:
      condition: service_healthy
    migrator:
      condition: service_completed_successfully

Priority 2: Create Unified .env Template

Create /Users/compai-alex/Documents/GitHub/comp/.env.example with:

• Clear sections: REQUIRED vs OPTIONAL
• Build-time vs runtime distinction
• Comments explaining what each var does
• Example values
• Links to external service setup guides

Fix SELF_HOSTING.md to reference actual file locations:

• Document that docker-compose uses apps/app/.env, not root .env
• Explain build args must be set before docker compose build

Priority 3: Create Setup Script

Create /Users/compai-alex/Documents/GitHub/comp/scripts/setup.sh:

#!/bin/bash
# - Check prerequisites
# - Install dependencies
# - Generate .env from template
# - Auto-generate secrets
# - Start PostgreSQL
# - Run migrations + seed
# - Generate Prisma clients
# - Start apps

Then users can just run: bun run setup

Priority 4: Add Env Validation

Create /Users/compai-alex/Documents/GitHub/comp/scripts/validate-env.sh:

• Check required vars before startup
• Show clear error messages ("Missing DATABASE_URL - set this in apps/app/.env")
• Show which service requires which var

---

Open Questions

Before proceeding with implementation, need clarification on:

1. Scope for Initial Fix

• Option A: Just fix Docker env var documentation
• Option B: Add setup script + Docker fixes
• Option C: Full single-command experience with mocks

2. External Service Dependencies

For local development, should we use mocks?

• Mock email service (maildev) instead of requiring Resend?
• Mock S3 (MinIO) instead of requiring AWS?
• This would let devs run bun run setup with zero external accounts

3. API Service in Docker

Should we add the API to docker-compose.yml?

• Currently API has separate Dockerfile in apps/api/
• Not integrated into root Docker setup
• Is API required for basic self-hosting or optional?

4. PostgreSQL in Docker Compose

Should docker-compose.yml include PostgreSQL for development?

• Currently SELF_HOSTING.md assumes external DB
• But packages/db/ has separate docker-compose for local Postgres
• Should we unify these?

5. Documentation Strategy

Where should self-hosting docs live?

• Keep in SELF_HOSTING.md in repo?
• Move to https://trycomp.ai/docs?
• Both?

6. Priority Order

What hurts OSS adoption most?

• Missing env var docs?
• Complex setup process?
• Unclear external service requirements?

---

Proposed Sprint Work

Sprint Goals

Focus on removing blockers that prevent single-command setup.

Core Tasks

1. Fix docker-compose.yml (wire build args, add API, add orchestration)
2. Create consolidated .env.example with all variables
3. Create setup automation script
4. Add env validation script
5. Rewrite SELF_HOSTING.md with accurate instructions
6. Add QUICK_START.md guide
7. Test on fresh machine

Future Enhancements

• Mock services for local dev (email, S3)
• Interactive setup CLI wizard
• One-click deploy buttons (Railway, Render, etc.)
• Kubernetes Helm charts

---

Files Requiring Changes

Critical Fixes:

1. /docker-compose.yml - Wire build args, add API, add postgres for dev, add orchestration
2. /.env.example - Consolidate all variables with clear docs
3. /SELF_HOSTING.md - Fix file location references, clarify build vs runtime
4. /README.md - Update quick start instructions

New Files to Create:

5. /scripts/setup.sh - Setup automation
6. /scripts/validate-env.sh - Environment validation
7. /scripts/generate-secrets.sh - Secret generation
8. /docs/QUICK_START.md - Quick start guide (optional - could go in README)

---

Success Metrics

Before:

• Setup time: 2-4 hours
• Manual steps: 15+
• Required external accounts: 6+ (Resend, Trigger.dev, AWS, OpenAI, etc.)
• Documentation clarity: Low (inconsistent between files)

After (Target):

• Setup time: 5-15 minutes
• Manual steps: 1-2 (bun run setup or docker compose up)
• Required external accounts: 0 for local dev (with mocks), 3 for production
• Documentation clarity: High (single source of truth)

---

Next Steps

1. Review findings and clarify scope/priorities
2. Create GitHub issue for tracking
3. Begin implementation based on agreed scope
4. Test on fresh machines throughout
5. Update documentation in parallel with code changes

Status: Investigation complete - ready for implementation planning.

---

Technical Deep Dive References

Full detailed findings available in:

• Documentation analysis (40+ pages)
• Docker setup analysis (30+ pages)
• Single-command roadmap (25+ pages)

Available upon request for implementation planning.

[linear]

COMP-277

[Alex-Alaniz]

https://linear.app/compai/issue/ENG-10/internal-oss-offering-have-bad-documentation-hard-to-setup

[matthiscock]

@Alex-Alaniz exceptional post and I hope it gets addressed.

+1 from me for better docs, willing to assist