Self-Hosted Infrastructure

A complete self-hosted data platform running on your Tailscale VPN.

Architecture

┌─────────────────────────────────────────────────────────────────────────────┐
│                         Tailscale VPN (100.97.11.91)                        │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                             │
│  ┌───────────────────────────────────────────────────────────────────────┐  │
│  │  AUTHENTICATION LAYER                                                 │  │
│  │  ┌─────────────────┐  ┌─────────────────┐  ┌─────────────────┐       │  │
│  │  │    Traefik      │  │    Authelia     │  │      Redis      │       │  │
│  │  │  Reverse Proxy  │→→│   Auth + TOTP   │→→│    Sessions     │       │  │
│  │  │      :80        │  │     :9091       │  │                 │       │  │
│  │  └─────────────────┘  └─────────────────┘  └─────────────────┘       │  │
│  └───────────────────────────────────────────────────────────────────────┘  │
│                                                                             │
│  ┌─────────────────┐  ┌─────────────────┐  ┌─────────────────┐             │
│  │     Prefect     │  │     Ollama      │  │      MinIO      │             │
│  │   Orchestration │  │    Local LLM    │  │    Data Lake    │             │
│  │     :4200       │  │     :11434      │  │   :9000/:9001   │             │
│  └─────────────────┘  └─────────────────┘  └─────────────────┘             │
│                                                                             │
│  ┌─────────────────┐  ┌─────────────────┐  ┌─────────────────┐             │
│  │ Iceberg Catalog │  │   DuckDB API    │  │ Prefect Workers │             │
│  │ Table Metadata  │  │   REST Queries  │  │    (3 nodes)    │             │
│  │     :8181       │  │     :8000       │  │                 │             │
│  └─────────────────┘  └─────────────────┘  └─────────────────┘             │
│                                                                             │
└─────────────────────────────────────────────────────────────────────────────┘

Services

Service	Port	URL	Description
Prefect UI	4200	http://100.97.11.91:4200	Workflow orchestration dashboard
Ollama	11434	http://100.97.11.91:11434	Local LLM inference API
MinIO API	9000	http://100.97.11.91:9000	S3-compatible storage API
MinIO Console	9001	http://100.97.11.91:9001	Web-based storage management
Iceberg Catalog	8181	http://100.97.11.91:8181	Table metadata REST API
DuckDB API	8000	http://100.97.11.91:8000	SQL query REST API
Authelia	9091	http://auth.100.97.11.91.nip.io	Authentication portal
Traefik	80	http://100.97.11.91:80	Reverse proxy for protected apps
Traefik Dashboard	8081	http://100.97.11.91:8081	Traefik management UI

Credentials

Service	Username	Password
MinIO	minioadmin	See terraform.tfvars
Authelia	dave	See terraform.tfvars

Generate Authelia password hash:

docker run --rm authelia/authelia:latest authelia crypto hash generate argon2 --password "$MYPASS"

Note: Credentials are stored in terraform/terraform.tfvars (not committed to git).

Quick Start

Option 1: Using Terraform (Recommended)

cd /home/dave/infrastructure/terraform

# Initialize Terraform
terraform init

# Preview changes
terraform plan

# Apply infrastructure
terraform apply

Option 2: Using Docker Compose

cd /home/dave/infrastructure

# Start all services
docker-compose up -d

# View logs
docker-compose logs -f

# Stop all services
docker-compose down

Deploy Script

cd /home/dave/infrastructure
./scripts/deploy.sh

Directory Structure

infrastructure/
├── README.md                 # This file
├── terraform/
│   ├── main.tf              # Docker containers configuration
│   ├── variables.tf         # Configurable variables
│   └── outputs.tf           # Service URLs output
├── docker-compose.yml       # Alternative to Terraform
├── apps/
│   ├── prefect-flows/       # Prefect flow definitions
│   └── lakehouse/           # DuckDB + Iceberg ETL app
│       ├── Dockerfile
│       ├── requirements.txt
│       └── src/
│           ├── config.py         # Configuration helpers
│           ├── minio_client.py   # S3 client
│           ├── iceberg_demo.py   # Iceberg operations
│           ├── etl_example.py    # ETL + DuckDB queries
│           └── prefect_flows.py  # Orchestrated workflows
└── scripts/
    └── deploy.sh            # Deployment script

Components

MinIO (Data Lake)

S3-compatible object storage for your data lake.

# Python example
import boto3

s3 = boto3.client(
    "s3",
    endpoint_url="http://100.97.11.91:9000",
    aws_access_key_id="minioadmin",
    aws_secret_access_key="minioadmin123",
)

# Create bucket
s3.create_bucket(Bucket="data-lake")

# Upload file
s3.upload_file("data.parquet", "data-lake", "raw/data.parquet")

Iceberg REST Catalog

Manages Iceberg table metadata. Data is stored in MinIO.

from pyiceberg.catalog import load_catalog

catalog = load_catalog(
    "rest",
    uri="http://100.97.11.91:8181",
    **{"s3.endpoint": "http://100.97.11.91:9000"}
)

# List tables
tables = catalog.list_tables("lakehouse")

DuckDB API (REST)

Query Parquet files via HTTP REST API.

Endpoints:

• GET / - Health check
• POST /query - Execute SQL query
• GET /tables - List Parquet files
• GET /tables/{path}/stats - Get table statistics
• GET /docs - Interactive API documentation

# Execute a SQL query
curl -X POST http://100.97.11.91:8000/query \
  -H "Content-Type: application/json" \
  -d '{"sql": "SELECT * FROM read_parquet('\''s3://data-lake/sales/*.parquet'\'') LIMIT 10"}'

# List available tables
curl http://100.97.11.91:8000/tables

DuckDB (In-Process)

For direct Python usage, DuckDB is also available in the Lakehouse app.

import duckdb

conn = duckdb.connect()
conn.execute("""
    SET s3_endpoint='100.97.11.91:9000';
    SET s3_access_key_id='minioadmin';
    SET s3_secret_access_key='minioadmin123';
    SET s3_use_ssl=false;
""")

# Query Parquet in MinIO
result = conn.execute("""
    SELECT * FROM read_parquet('s3://data-lake/sales/*.parquet')
    LIMIT 10
""").fetchdf()

Prefect (Orchestration)

Workflow orchestration with a web UI.

from prefect import flow, task

@task
def extract():
    return {"data": [1, 2, 3]}

@flow
def my_pipeline():
    data = extract()
    print(f"Processed: {data}")

# Run flow
my_pipeline()

Ollama (LLM)

Local LLM inference.

# Pull a model
curl http://100.97.11.91:11434/api/pull -d '{"name": "llama2"}'

# Generate text
curl http://100.97.11.91:11434/api/generate -d '{
  "model": "llama2",
  "prompt": "Hello, how are you?"
}'

Configuration

All configuration is in terraform/variables.tf:

Variable	Default	Description
vpn_ip	100.97.11.91	Tailscale VPN IP
prefect_port	4200	Prefect UI port
ollama_port	11434	Ollama API port
minio_api_port	9000	MinIO S3 API port
minio_console_port	9001	MinIO console port
iceberg_port	8181	Iceberg catalog port
duckdb_api_port	8000	DuckDB REST API port
prefect_worker_count	3	Number of Prefect workers
authelia_port	9091	Authelia auth server port
authelia_user	dave	Authelia admin username
traefik_port	80	Traefik HTTP entrypoint
traefik_dashboard_port	8081	Traefik dashboard port

Authelia (Authentication)

Authelia provides SSO with TOTP (2FA) for your apps.

First Time Setup

1. Generate a password hash:

   docker run --rm authelia/authelia:latest authelia crypto hash generate argon2 --password 'your-secure-password'

2. Update terraform/variables.tf:

   • Set authelia_password_hash to the generated hash
   • Change authelia_jwt_secret to a random string
   • Change authelia_session_secret to another random string

3. Apply Terraform:

   cd terraform && terraform apply

4. Set up TOTP:

   • Go to http://auth.100.97.11.91.nip.io
   • Login with username dave and your password
   • Register your TOTP device (Google Authenticator, etc.)

Protecting Your Apps

To protect a future app with Authelia, add a Traefik dynamic config:

# terraform/traefik/dynamic/my-app.yml
http:
  routers:
    my-app:
      rule: "Host(`myapp.100.97.11.91.nip.io`)"
      service: my-app
      middlewares:
        - authelia
      entryPoints:
        - web

  services:
    my-app:
      loadBalancer:
        servers:
          - url: http://my-app-container:8080

Then access via: http://myapp.100.97.11.91.nip.io → Authelia login → Your app

How It Works

User → Traefik (:80) → Authelia (verify auth) → Your App
                              ↓
                         Login Page (if not authenticated)
                              ↓
                         TOTP Verification

Future: DNS Setup

When you're ready to add DNS:

1. Add a DNS service (dnsmasq) to docker-compose
2. Configure entries like:
   • prefect → 100.97.11.91
   • ollama → 100.97.11.91
   • minio → 100.97.11.91
3. Configure Tailscale DNS to use your server

Troubleshooting

Check container status

docker ps

View logs

# All services
docker-compose logs -f

# Specific service
docker logs prefect -f

Restart a service

docker restart prefect

Check Terraform state

cd terraform
terraform state list
terraform show

Access from Other VPN Devices

From any device on your Tailscale VPN:

1. Prefect UI: Open http://100.97.11.91:4200 in browser
2. MinIO Console: Open http://100.97.11.91:9001 in browser
3. Ollama API: curl http://100.97.11.91:11434/api/tags