Terraprem

Terraform Cloud With No DNS

Self-hosted remote Terraform execution with state management, accessible via kubectl port-forward

---

Overview

Terraprem is a self-hosted Terraform Cloud clone that runs entirely on Kubernetes. It provides remote Terraform execution, state management, and workspace organization without requiring external DNS or complex networking setup. Access the system locally through kubectl port-forward, making it perfect for development environments and air-gapped Kubernetes clusters.

📖 Read the full story: Building an open source drop in replacement for Terraform Cloud - A detailed blog post about the motivation, architecture, and implementation of this project.

Demo

Key Features

• 🚀 Terraform Cloud Compatible: Uses standard backend "remote" configuration - no custom plugins needed
• ☸️ Kubernetes Native: Runs entirely in Kubernetes with remote execution in isolated pods
• 🔒 Secure Secret Management: Centralized credential and variable storage with automatic injection
• 📊 State Management: Uses S3 + DynamoDB for state storage and locking (standard Terraform backend)
• 🎯 Workspace Organization: Manage workspaces, variables, and provider configs through UI or API
• 🔄 Remote Execution: Automatically executes terraform plan and terraform apply in Kubernetes pods
• 📝 Real-time Logs: Stream execution logs in real-time via Server-Sent Events
• 🎨 Modern Web UI: Clean, intuitive interface for managing workspaces, runs, and variables
• 🔐 Multi-user Support: Admin and member user roles with token-based authentication

Architecture

System Components

1. Terraform Wrapper Script (terraform-wrapper.sh): Automatically manages kubectl port-forward and hot-swaps backend configuration (recommended - see installation below)
2. Backend Service: Go-based HTTP API implementing Terraform Cloud-compatible endpoints
3. Web UI: Next.js frontend for workspace and run management
4. Remote Execution: Kubernetes Jobs execute Terraform operations in isolated pods
5. State Storage: S3 + DynamoDB for Terraform state files and locking
6. Secret Store: Centralized storage for provider credentials and variables

Data Flow

Terraform CLI → (Wrapper Script) → kubectl port-forward → Backend Service → Kubernetes Jobs
                                                      ↓
                                              S3 + DynamoDB (State)
                                              Secret Store (Credentials)

Note: You can either manually run kubectl port-forward or use the wrapper script for automatic management.

Quick Start

Prerequisites

• Kubernetes cluster with kubectl configured
• AWS account with S3 and DynamoDB access (for state storage)
• Terraform installed locally
• Go 1.21+ (for building from source)

1. Deploy Backend Service

# Build and push Docker image
docker build -t terraprem-backend:latest -f backend/Dockerfile .
kubectl apply -f k8s/

# Set admin credentials (optional, defaults to admin/admin)
kubectl set env deployment/terraprem-backend ADMIN_USERNAME=admin ADMIN_PASSWORD=your-password

2. Configure Port Forward

Option A: Manual Port-Forward (Simplest)

In a separate terminal, run:

kubectl port-forward -n terraprem svc/terraprem-backend 8080:8080

Keep this terminal running while you use Terraform.

Option B: Automatic Port-Forward with Wrapper Script (Recommended)

The wrapper script automatically manages port-forwarding - just configure backend "terraprem" in your Terraform files and run terraform commands normally.

Installation:

Easy way - Run the install script (automatically detects your shell and adds alias):

./scripts/install-alias.sh
source ~/.zshrc  # or source ~/.bashrc

Manual way - Add to your ~/.bashrc or ~/.zshrc (use the full path to your project):

# Replace with your actual project path
alias terraform='/full/path/to/TfCloudCopy/scripts/terraform-wrapper.sh'

For example:

alias terraform='/Users/sidpremkumar/Documents/Projects/TfCloudCopy/scripts/terraform-wrapper.sh'

After adding the alias, reload your shell:

source ~/.bashrc  # or source ~/.zshrc

Usage:

Configure your main.tf:

terraform {
  backend "terraprem" {
    k8s_context = "your-k8s-context"  # Optional - otherwise current context is used
    namespace   = "terraprem"
    service     = "terraprem-backend"
    organization = "terraprem"
    workspaces {
      name = "my-workspace"
    }
  }
}

Then just run terraform normally:

terraform init   # Wrapper automatically starts port-forward!

The wrapper automatically:

• Detects backend "terraprem" configuration
• Starts kubectl port-forward
• Rewrites config to backend "remote" temporarily
• Adds temporary credentials to ~/.terraform.d/credentials.tfrc.json
• Runs terraform commands
• Restores original config & credentials
• Stops port-forward on exit

3. Configure Terraform Backend

If using Option A (Manual Port-Forward):

terraform {
  backend "remote" {
    hostname     = "localhost:8080"
    organization = "terraprem"

    workspaces {
      name = "my-workspace"
    }
  }
}

If using Option B (Wrapper Script):

terraform {
  backend "terraprem" {
    k8s_context = "your-k8s-context"  # Optional - otherwise current context is used
    namespace   = "terraprem"
    service     = "terraprem-backend"
    organization = "terraprem"
    workspaces {
      name = "my-workspace"
    }
  }
}

The wrapper automatically rewrites this to backend "remote" during execution.

4. Set Authentication Token (If not using wrapper script)

Terraform Cloud remote backend requires authentication. Recommended: Use the credentials file:

Create or edit ~/.terraform.d/credentials.tfrc.json:

{
  "credentials": {
    "localhost:8080": {
      "token": "username/token-value"
    }
  }
}

Token Format: {username}/{token_value} (e.g., alice/abc123)

Alternative: Use environment variable:

export TF_TOKEN_localhost_8080="username/token-value"

5. Use Standard Terraform Commands

terraform init    # Auto-creates workspace if needed
terraform plan    # Executes remotely in Kubernetes
terraform apply   # Executes remotely in Kubernetes

What happens:

• terraform init: Connects to backend, creates workspace if needed, uploads configuration
• terraform plan: Creates a run, executes plan remotely in a Kubernetes pod, streams logs
• terraform apply: Creates a run, executes apply remotely in a Kubernetes pod, updates state in S3

Project Structure

TfCloudCopy/
├── backend/          # Go backend service (API, execution, storage)
├── frontend/          # Next.js web UI
├── scripts/          # Build and utility scripts (includes terraform-wrapper.sh)
├── k8s/              # Kubernetes manifests
├── examples/         # Example Terraform configurations
└── assets/           # Architecture diagrams and assets

Documentation

• Architecture & Design: Comprehensive architecture documentation (Note: This is generated and maintained by AI)
• Terraform Wrapper: Wrapper script documentation

How It Works

Remote Execution Flow

1. Setup (one-time): Create workspace and configure provider credentials via UI/API
2. Terraform Init: CLI connects to backend, workspace auto-created if needed
3. Terraform Plan/Apply:
   • Backend creates Kubernetes Job
   • Configuration bundle downloaded from S3 (uploaded during terraform init)
   • Provider configs and variables injected from secret store
   • Terraform executes remotely with S3 + DynamoDB backend
   • Logs streamed back in real-time
   • Plan files stored in S3 for later apply

State Management

• State Storage: Standard Terraform S3 backend (encrypted, versioned)
• State Locking: DynamoDB table prevents concurrent modifications
• No Custom Code: Uses Terraform's native S3 + DynamoDB backend during remote execution

Secret Management

• Provider Credentials: Stored in secret store, injected as environment variables
• Variables: Stored in secret store, injected as terraform.tfvars
• User Tokens: Token-based authentication for API access
• No Local Credentials: Users never handle sensitive credentials directly

Development

Backend Development

cd backend
go run main.go

The backend will start on http://localhost:8080 by default. Set environment variables for configuration:

• ADMIN_USERNAME: Admin username (default: admin)
• ADMIN_PASSWORD: Admin password (default: admin)
• OVERWRITE_ADMIN: Set to true to overwrite existing admin user

Frontend Development

cd frontend
npm install
npm run dev

The frontend will start on http://localhost:3000 by default.

Scripts

Utility scripts are available in the scripts/ directory:

• terraform-wrapper.sh: Wrapper script that automatically manages port-forwarding (recommended)
• terraprem.sh: CLI script to start port-forwards for frontend (UI) and backend (API) services
• install-alias.sh: Helper script to automatically install terraform and terraprem aliases to your shell config (idempotent)
• build-backend.sh: Build and install the backend server (if using backend server approach)
• build-provider.sh: Build and install the Terraform provider (legacy, not recommended)

See scripts/TERRAFORM_WRAPPER.md for detailed wrapper documentation.

Starting the UI and Backend

To start both the frontend UI and backend API port-forwards, use the terraprem command:

# Install the alias first (if not already installed)
./scripts/install-alias.sh
source ~/.zshrc  # or source ~/.bashrc

# Start both port-forwards
terraprem

This will:

• Start port-forward for frontend UI on localhost:3000
• Start port-forward for backend API on localhost:8080
• Display the URLs and wait for Ctrl+C to stop

You can also specify options:

terraprem --context my-cluster          # Use specific Kubernetes context
terraprem --namespace my-namespace      # Use different namespace
terraprem --frontend-port 3001         # Use different frontend port
terraprem --backend-port 8081          # Use different backend port

Press Ctrl+C to gracefully stop both port-forwards.

Installing the Wrapper Script

Recommended approach - Use the install script (automatically detects your shell):

./scripts/install-alias.sh
source ~/.zshrc  # or source ~/.bashrc

The install script is idempotent - it won't add duplicate aliases if run multiple times.

Manual approach - Add alias to your shell config:

# Add to ~/.bashrc or ~/.zshrc
alias terraform='/full/path/to/TfCloudCopy/scripts/terraform-wrapper.sh'

Replace /full/path/to/TfCloudCopy with the actual path to your project.

Alternative - Replace terraform binary:

# Backup existing terraform
mv ~/.local/bin/terraform ~/.local/bin/terraform-real 2>/dev/null || true

# Install wrapper
cp /full/path/to/TfCloudCopy/scripts/terraform-wrapper.sh ~/.local/bin/terraform
chmod +x ~/.local/bin/terraform

After installation, just configure backend "terraprem" in your Terraform files and run terraform commands normally - the wrapper handles everything automatically!

Examples

See the examples/ directory for:

• Basic Terraform configuration: Simple example setup
• Infrastructure setup (infrastructure/): Complete infrastructure setup (S3, DynamoDB, IAM, ECR, etc.)
• Usage example (usage-example/): Example of using Terraform Cloud-compatible remote backend

Each example includes detailed README files with setup instructions.

Security

• Network Isolation: kubectl port-forward provides local-only access
• Token Authentication: Member users authenticate via tokens
• Secret Store: All sensitive data stored in external secret management
• Kubernetes RBAC: Cluster access controlled via Kubernetes RBAC
• Isolated Execution: Each run executes in isolated Kubernetes pods

License

This project is licensed under the GNU Affero General Public License v3.0. See LICENSE for details.

---

Built with ❤️ for the Terraform community