Skip to content
This repository was archived by the owner on Oct 10, 2025. It is now read-only.

hetzner-cloud-setup-guide.md

Latest commit

 

History

History
1432 lines (1029 loc) · 42.7 KB

hetzner-cloud-setup-guide.md

File metadata and controls

1432 lines (1029 loc) · 42.7 KB

Hetzner Cloud Provider Setup Guide

This guide explains how to set up and use the Hetzner Cloud provider with the Torrust Tracker Demo for both staging and production environments.

Overview

This guide covers two deployment environments:

  • Staging Environment: Uses staging-torrust-demo.com domain for testing and validation
  • Production Environment: Uses torrust-demo.com domain for live service

Both environments use floating IPs for stable DNS configuration and leverage Hetzner DNS for complete zone management.

Domain Choice Considerations

.dev Domains (HSTS Preload Required)

  • ⚠️ Browser Behavior: ALL browsers automatically redirect HTTP → HTTPS
  • 🔒 SSL Required: HTTPS certificates mandatory for browser access
  • ✅ Security: Enhanced security with forced encryption
  • 🧪 Testing: Use curl for HTTP API testing during development

.com Domains (Standard HTTP/HTTPS)

  • 🌐 Normal Behavior: Browsers respect server HTTP/HTTPS configuration
  • 🔧 Flexibility: Can start with HTTP and migrate to HTTPS when ready
  • 📈 Production: Standard choice for production services
  • 🛠️ Development: Easier for development and testing workflows

Floating IP Architecture

The deployment uses dedicated floating IPs to maintain stable DNS records across server recreation:

  • IPv4: 78.47.140.132
  • IPv6: 2a01:4f8:1c17:a01d::/64

Benefits:

  • ✅ Stable DNS: Point domain once, recreate servers without DNS changes
  • ✅ Zero Downtime: Switch between servers instantly
  • ✅ Cost Effective: Single floating IP serves both staging and production
  • ✅ Simplified Management: No DNS updates during infrastructure changes

Prerequisites

  1. Hetzner Cloud Account: Create an account at console.hetzner.cloud
  2. Hetzner DNS Account: Enable DNS service in your Hetzner project
  3. API Tokens: Generate both Cloud and DNS API tokens
  4. Domain Registration: Register staging-torrust-demo.com (staging) and/or torrust-demo.com (production)
  5. Floating IPs: Purchase floating IPs for stable addressing
  6. SSH Key: Ensure you have an SSH key pair for server access

Step 1: Create Hetzner Accounts

1.1 Hetzner Cloud Account

  1. Visit console.hetzner.cloud
  2. Sign up for a new account or log in to existing account
  3. Create a new project or use an existing one

1.2 Hetzner DNS Setup

  1. In your Hetzner Cloud project, navigate to DNS
  2. Enable DNS service if not already enabled
  3. Note that you'll configure DNS zones later via API

Step 2: Generate API Tokens

You need two API tokens for complete automation:

2.1 Hetzner Cloud API Token

  1. In the Hetzner Cloud Console, navigate to your project
  2. Go to Security → API Tokens
  3. Click Generate API Token
  4. Give it a descriptive name (e.g., "torrust-tracker-cloud")
  5. Set permissions to Read & Write
  6. Copy the generated token (64 characters)

2.2 Hetzner DNS API Token

  1. In the Hetzner Cloud Console, navigate to DNS
  2. Go to API Tokens (in DNS section)
  3. Click Generate API Token
  4. Give it a descriptive name (e.g., "torrust-tracker-dns")
  5. Set permissions to Zone:Edit
  6. Copy the generated token (32 characters)

Step 3: Secure Token Storage

Store both API tokens securely using the provider configuration system:

Provider Configuration Setup

# Create provider config from template
make infra-config-provider PROVIDER=hetzner

# This creates: infrastructure/config/providers/hetzner.env

Edit the provider configuration file:

# Edit provider configuration
vim infrastructure/config/providers/hetzner.env

Add both API tokens:

# =============================================================================
# Hetzner Cloud Provider Configuration
# =============================================================================

# Cloud Infrastructure Management
HETZNER_TOKEN=your_64_character_cloud_api_token_here

# DNS Zone Management
HETZNER_DNS_API_TOKEN=your_32_character_dns_api_token_here

# Security note: This file is git-ignored. Never commit tokens to version control.

Test Token Configuration

# Source the provider configuration
source infrastructure/config/providers/hetzner.env

# Test Cloud API token
CLOUD_TOKEN="$HETZNER_TOKEN"
echo "Cloud token length: ${#CLOUD_TOKEN} characters"
# Should show: Cloud token length: 64 characters

# Test DNS API token
DNS_TOKEN="$HETZNER_DNS_API_TOKEN"
echo "DNS token length: ${#DNS_TOKEN} characters"
# Should show: DNS token length: 32 characters

# Test Cloud API access (silent mode for clean JSON output)
curl -s -H "Authorization: Bearer $CLOUD_TOKEN" \
     "https://api.hetzner.cloud/v1/servers" | jq
# Expected output: {"servers": []}

# Test DNS API access (silent mode for clean JSON output)
curl -s -H "Auth-API-Token: $DNS_TOKEN" \
     "https://dns.hetzner.com/api/v1/zones" | jq
# Expected output: {"zones": [...]}

Environment Variables (Alternative Method)

If you prefer environment variables over secure files:

# Export both tokens
export HETZNER_TOKEN="your_64_character_cloud_api_token_here"
export HETZNER_DNS_API_TOKEN="your_32_character_dns_api_token_here"

# Or add to your shell profile (~/.bashrc or ~/.zshrc)
echo 'export HETZNER_TOKEN="your_64_character_cloud_api_token_here"' >> ~/.bashrc
echo 'export HETZNER_DNS_API_TOKEN="your_32_character_dns_api_token_here"' >> ~/.bashrc
source ~/.bashrc

Security Warning: Never commit API tokens to git repositories. The provider configuration file (hetzner.env) is automatically git-ignored for security.

Step 4: Purchase and Configure Floating IPs

4.1 Purchase Floating IPs (One-Time Setup)

If you haven't already purchased floating IPs:

  1. IPv4 Floating IP:

    • In Hetzner Cloud Console, go to Floating IPs
    • Click Add Floating IP
    • Select IPv4 and your preferred location
    • Choose Not assigned to any resource (manual assignment)
    • Cost: ~€1.19/month

  2. IPv6 Floating IP (Optional):

    • Add another Floating IP for IPv6
    • Select your preferred location
    • Note: IPv6 floating IPs are currently free

4.2 Note Your Floating IP Addresses

Record your floating IP addresses for environment configuration:

# Example floating IPs (replace with your actual IPs)
IPv4: 78.47.140.132
IPv6: 2a01:4f8:1c17:a01d::/64

4.3 Floating IP Benefits

  • Stable DNS: Domain always points to same IP, even when servers are recreated
  • Zero Downtime: Move IP between servers instantly
  • Backup Ready: Quick failover to backup server
  • Professional: Industry standard for production deployments

Step 5: Configure Environment

This guide supports both staging and production environments. Choose your deployment target:

Option 1: Staging Environment (staging-torrust-demo.com)

For testing and development using the staging domain:

⚠️ Important: .dev domains are on Chrome's HSTS preload list, meaning ALL browsers automatically redirect HTTP to HTTPS. For testing without SSL certificates, use curl commands or consider using a .com domain instead.

# Create staging environment configuration
make infra-config-staging PROVIDER=hetzner

# This creates: infrastructure/config/environments/staging-hetzner.env

Edit the staging configuration:

vim infrastructure/config/environments/staging-hetzner.env

Key staging settings:

# Domain Configuration
TRACKER_DOMAIN=tracker.staging-torrust-demo.com
GRAFANA_DOMAIN=grafana.staging-torrust-demo.com
GRAFANA_DOMAIN=grafana.staging-torrust-demo.com
CERTBOT_EMAIL=admin@staging-torrust-demo.com

# Floating IP Configuration (your actual IPs)
FLOATING_IPV4=78.47.140.132
FLOATING_IPV6=2a01:4f8:1c17:a01d::/64

# Generate secure passwords
MYSQL_ROOT_PASSWORD=$(openssl rand -base64 32)
MYSQL_PASSWORD=$(openssl rand -base64 32)
TRACKER_ADMIN_TOKEN=$(openssl rand -base64 32)
GF_SECURITY_ADMIN_PASSWORD=$(openssl rand -base64 32)

Option 2: Production Environment (torrust-demo.com)

For production deployment using the main domain:

# Create production environment configuration
make infra-config-production PROVIDER=hetzner

# This creates: infrastructure/config/environments/production-hetzner.env

Edit the production configuration:

vim infrastructure/config/environments/production-hetzner.env

Key production settings:

# Domain Configuration
TRACKER_DOMAIN=tracker.torrust-demo.com
GRAFANA_DOMAIN=grafana.torrust-demo.com
GRAFANA_DOMAIN=grafana.torrust-demo.com
CERTBOT_EMAIL=admin@torrust-demo.com

# Floating IP Configuration (your actual IPs)
FLOATING_IPV4=78.47.140.132
FLOATING_IPV6=2a01:4f8:1c17:a01d::/64

# Generate secure passwords
MYSQL_ROOT_PASSWORD=$(openssl rand -base64 32)
MYSQL_PASSWORD=$(openssl rand -base64 32)
TRACKER_ADMIN_TOKEN=$(openssl rand -base64 32)
GF_SECURITY_ADMIN_PASSWORD=$(openssl rand -base64 32)

Step 6: Deploy Infrastructure

The infrastructure scripts automatically load your API tokens from the provider configuration file. Choose your deployment environment:

For Staging Environment

# Initialize infrastructure
make infra-init ENVIRONMENT_TYPE=staging ENVIRONMENT_FILE=staging-hetzner

# Plan the deployment
make infra-plan ENVIRONMENT_TYPE=staging ENVIRONMENT_FILE=staging-hetzner

# Apply the infrastructure (creates server and assigns floating IP)
make infra-apply ENVIRONMENT_TYPE=staging ENVIRONMENT_FILE=staging-hetzner

# Deploy the application
make app-deploy ENVIRONMENT_TYPE=staging ENVIRONMENT_FILE=staging-hetzner

For Production Environment

# Initialize infrastructure
make infra-init ENVIRONMENT_TYPE=production ENVIRONMENT_FILE=production-hetzner

# Plan the deployment
make infra-plan ENVIRONMENT_TYPE=production ENVIRONMENT_FILE=production-hetzner

# Apply the infrastructure (creates server and assigns floating IP)
make infra-apply ENVIRONMENT_TYPE=production ENVIRONMENT_FILE=production-hetzner

# Deploy the application
make app-deploy ENVIRONMENT_TYPE=production ENVIRONMENT_FILE=production-hetzner

Step 6.5: Configure Floating IP Network Interface

After infrastructure deployment, Hetzner assigns the floating IP at the cloud level, but the server requires additional network configuration to use the floating IP for external connectivity. This is a required step for floating IP functionality.

Why This Step is Necessary

Hetzner's floating IP system requires two-phase configuration:

  1. Cloud-level assignment (handled by Terraform during infra-apply)
  2. Server-level network interface configuration (manual step detailed below)

Without the server-side configuration, the floating IP will not be accessible externally, even though it appears assigned in the Hetzner Cloud Console.

6.5.1 Verify Current Network Configuration

First, check the current network setup:

# SSH into your server
make vm-ssh ENVIRONMENT=staging  # or production

# Check current network interfaces
ip addr show eth0

# Check current netplan configuration
sudo cat /etc/netplan/50-cloud-init.yaml

You should see output similar to:

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 96:00:00:12:34:56 brd ff:ff:ff:ff:ff:ff
    inet 188.245.95.154/32 scope global dynamic eth0
       valid_lft 86395sec preferred_lft 86395sec
    inet6 2a01:4f8:c014:333e::1/64 scope global
       valid_lft forever preferred_lft forever

Note that only the DHCP IP (e.g., 188.245.95.154) and IPv6 address are configured.

6.5.2 Configure Floating IP on Network Interface

Create a separate netplan configuration for the floating IP:

# Create floating IP network configuration
sudo tee /etc/netplan/60-floating-ip.yaml > /dev/null << 'EOF'
network:
  version: 2
  renderer: networkd
  ethernets:
    eth0:
      dhcp4: true
      addresses:
        - 78.47.140.132/32
EOF

# Set proper permissions
sudo chmod 600 /etc/netplan/60-floating-ip.yaml

# Validate the configuration
sudo netplan try

# If validation succeeds, apply the configuration
sudo netplan apply

Important: Replace 78.47.140.132 with your actual floating IP address from your environment configuration file.

6.5.3 Verify Floating IP Configuration

After applying the configuration, verify both IP addresses are active:

# Check that both IPs are configured on eth0
ip addr show eth0

# Expected output should show both addresses:
# inet 78.47.140.132/32 scope global eth0        <- Floating IP
# inet 188.245.95.154/32 scope global dynamic eth0  <- DHCP IP

# Test internal connectivity to floating IP
ping -c 3 78.47.140.132

# Check systemd-networkd status
sudo systemctl status systemd-networkd

6.5.4 Test External Connectivity

From your local machine, test connectivity to the floating IP:

# Test SSH connectivity (may take a few minutes for routing to propagate)
timeout 10 ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 \
  torrust@78.47.140.132 "echo 'Floating IP SSH works'"

# Test ping connectivity
ping -c 3 78.47.140.132

# Test HTTP port connectivity
timeout 5 nc -zv 78.47.140.132 22

Note: External connectivity may take 2-5 minutes to become available after configuration due to Hetzner's network routing propagation. If connectivity tests fail initially, wait a few minutes and retry.

6.5.5 Understanding the Network Configuration

The floating IP configuration uses this approach:

network:
  version: 2
  renderer: networkd
  ethernets:
    eth0:
      dhcp4: true # Preserves DHCP for primary IP
      addresses:
        - 78.47.140.132/32 # Adds floating IP as additional address

This configuration:

  • ✅ Preserves DHCP: Maintains automatic IP assignment from Hetzner
  • ✅ Adds Floating IP: Configures floating IP as additional address
  • ✅ Maintains Connectivity: Ensures both IPs work simultaneously
  • ✅ Persistent Setup: Survives server reboots

6.5.6 Troubleshooting

If you encounter issues:

  1. Check netplan syntax:

    sudo netplan try

  2. Verify file permissions:

    ls -la /etc/netplan/60-floating-ip.yaml
# Should show: -rw------- 1 root root

  3. Check systemd-networkd logs:

    sudo journalctl -u systemd-networkd -f

  4. Reset network configuration if needed:

    # Remove floating IP config and restart networking
sudo rm /etc/netplan/60-floating-ip.yaml
sudo netplan apply
# Then recreate the configuration

  5. Verify cloud-level assignment:

    # Check Hetzner Cloud Console or use CLI
HCLOUD_TOKEN="$HETZNER_API_TOKEN" hcloud floating-ip list

6.5.7 Important Notes

  • Two-phase requirement: Both cloud assignment AND server configuration are required
  • Propagation time: External connectivity may take several minutes to become available
  • Persistent configuration: This setup survives server reboots
  • Multiple IPs: The server maintains both DHCP and floating IP addresses
  • Firewall compatibility: UFW rules apply to both IP addresses automatically

After completing this step, your floating IP should be externally accessible and ready for DNS configuration in the next step.

Step 7: Configure DNS

After deployment, you need to configure DNS to point your domain to the floating IP. This section provides complete working examples for Hetzner DNS API configuration.

7.1 Create DNS Zone

First, create a DNS zone for your domain:

# Source your environment configuration with DNS API token
source infrastructure/config/providers/hetzner.env

# Create DNS zone for your domain
curl -s -H "Auth-API-Token: $HETZNER_DNS_API_TOKEN" \
  -H "Content-Type: application/json" \
  -X POST \
  -d '{"name": "staging-torrust-demo.com", "ttl": 86400}' \
  https://dns.hetzner.com/api/v1/zones | jq

Expected Response:

{
  "zone": {
    "id": "Vpew4Pb3YoDjBVHMvV9AHB",
    "name": "staging-torrust-demo.com",
    "ttl": 86400,
    "registrar": "",
    "legacy_dns_host": "",
    "legacy_ns": [],
    "ns": [
      "hydrogen.ns.hetzner.com",
      "oxygen.ns.hetzner.com",
      "helium.ns.hetzner.de"
    ],
    "created": "2025-01-13T19:17:12Z",
    "verified": "0001-01-01T00:00:00Z",
    "modified": "2025-01-13T19:17:12Z",
    "project": "",
    "owner": "",
    "permission": "",
    "zone_type": "PRIMARY",
    "status": "verified",
    "paused": false,
    "is_secondary_dns": false,
    "txt_verification": {
      "name": "",
      "token": ""
    },
    "records_count": 2
  }
}

7.2 Create DNS A Records

Create A records for your tracker and monitoring subdomains:

# Create tracker subdomain A record
curl -s -H "Auth-API-Token: $HETZNER_DNS_API_TOKEN" \
  -H "Content-Type: application/json" \
  -X POST \
  -d '{
    "value": "78.47.140.132",
    "ttl": 86400,
    "type": "A",
    "name": "tracker",
    "zone_id": "Vpew4Pb3YoDjBVHMvV9AHB"
  }' \
  https://dns.hetzner.com/api/v1/records | jq

# Create grafana subdomain A record
curl -s -H "Auth-API-Token: $HETZNER_DNS_API_TOKEN" \
  -H "Content-Type: application/json" \
  -X POST \
  -d '{
    "value": "78.47.140.132",
    "ttl": 86400,
    "type": "A",
    "name": "grafana",
    "zone_id": "Vpew4Pb3YoDjBVHMvV9AHB"
  }' \
  https://dns.hetzner.com/api/v1/records | jq

Expected Response for each record:

{
  "record": {
    "id": "0de308260c254fa933b2c89312d6eb08",
    "type": "A",
    "name": "tracker",
    "value": "78.47.140.132",
    "zone_id": "Vpew4Pb3YoDjBVHMvV9AHB",
    "ttl": 86400,
    "created": "2025-01-13T19:48:51Z",
    "modified": "2025-01-13T19:48:51Z"
  }
}

7.3 Create DNS AAAA Records (IPv6)

Create AAAA records for IPv6 dual-stack connectivity:

# Create tracker subdomain AAAA record
curl -s -H "Auth-API-Token: $HETZNER_DNS_API_TOKEN" \
  -H "Content-Type: application/json" \
  -X POST \
  -d '{
    "type": "AAAA",
    "name": "tracker",
    "value": "2a01:4f8:1c17:a01d::1",
    "ttl": 300,
    "zone_id": "hbpTmpwZJw6xbKqbudCiDb"
  }' \
  "https://dns.hetzner.com/api/v1/records" | jq

# Create grafana subdomain AAAA record
curl -s -H "Auth-API-Token: $HETZNER_DNS_API_TOKEN" \
  -H "Content-Type: application/json" \
  -X POST \
  -d '{
    "type": "AAAA",
    "name": "grafana",
    "value": "2a01:4f8:1c17:a01d::1",
    "ttl": 300,
    "zone_id": "hbpTmpwZJw6xbKqbudCiDb"
  }' \
  "https://dns.hetzner.com/api/v1/records" | jq

Expected Response for each IPv6 record:

{
  "record": {
    "id": "f1a2926dde3b57396b863c66b139fad5",
    "type": "AAAA",
    "name": "tracker",
    "value": "2a01:4f8:1c17:a01d::1",
    "ttl": 300,
    "zone_id": "hbpTmpwZJw6xbKqbudCiDb",
    "created": "2025-08-08T14:33:36.497Z",
    "modified": "2025-08-08T14:33:36.497Z"
  }
}

7.4 Verify DNS Configuration

Verify your DNS records are created correctly:

List All DNS Records

# List all records in your zone
curl -s -H "Auth-API-Token: $HETZNER_DNS_API_TOKEN" \
  "https://dns.hetzner.com/api/v1/records?zone_id=hbpTmpwZJw6xbKqbudCiDb" | jq

Test IPv4 (A Records)

# Test IPv4 DNS resolution
dig A tracker.staging-torrust-demo.com +short
dig A grafana.staging-torrust-demo.com +short

# Expected output for both commands:
# 78.47.140.132

Test IPv6 (AAAA Records)

# Test IPv6 DNS resolution
dig AAAA tracker.staging-torrust-demo.com +short
dig AAAA grafana.staging-torrust-demo.com +short

# Expected output for both commands:
# 2a01:4f8:1c17:a01d::1

Complete DNS Query

# View complete DNS information
dig tracker.staging-torrust-demo.com
dig grafana.staging-torrust-demo.com

Test Dual-Stack Connectivity (Optional)

# Test both IPv4 and IPv6 connectivity (requires deployed application)
curl -4 -s http://tracker.staging-torrust-demo.com/api/health_check || echo "IPv4 not ready"
curl -6 -s http://tracker.staging-torrust-demo.com/api/health_check || echo "IPv6 not ready"

7.5 Configure Nameservers at Domain Registrar

Finally, configure your domain registrar to use Hetzner's nameservers:

hydrogen.ns.hetzner.com
oxygen.ns.hetzner.com
helium.ns.hetzner.de

Important: Replace staging-torrust-demo.com with your actual domain, 78.47.140.132 with your floating IP, and Vpew4Pb3YoDjBVHMvV9AHB with your actual zone ID.

For additional DNS configuration options, see the Deployment Guide - Part 3: DNS Configuration.

Step 5.5: Optional - Configure Persistent Volume for Data Persistence

Important: By default, all data is stored on the main server disk and will be lost when the server is destroyed. For production environments where you need data persistence across server recreation, you must manually set up a persistent volume.

Why Manual Volume Setup?

  • Provider Flexibility: Not all providers create additional volumes automatically
  • Administrative Control: Sysadmins have full control over storage configuration
  • Cost Management: Volumes can be expensive; optional setup allows cost optimization
  • Deployment Simplicity: Basic deployment works without additional storage setup
  • Hetzner Cloud Limitation: As of August 2025, Hetzner has a known issue where servers cannot be created with attached volumes during provisioning (Status Page)

Important: Even if this architectural decision changes in the future, the current Hetzner Cloud service limitation makes manual volume attachment the only reliable approach.

Setting Up Persistent Volume (Optional)

When to do this: After infrastructure provisioning but BEFORE application deployment.

  1. Create and attach volume in Hetzner Cloud Console:

    # Create a 20GB volume for persistent data
HCLOUD_TOKEN="$HETZNER_API_TOKEN" hcloud volume create \
  --name torrust-data \
  --size 20 \
  --location fsn1

# Attach volume to server
HCLOUD_TOKEN="$HETZNER_API_TOKEN" hcloud volume attach \
  torrust-data torrust-tracker-prod

  2. Format and mount the volume (SSH into server):

    # SSH into the server
ssh torrust@YOUR_SERVER_IP

# Format the volume (usually /dev/sdb for first additional volume)
sudo mkfs.ext4 /dev/sdb

# Create mount point
sudo mkdir -p /var/lib/torrust

# Mount the volume
sudo mount /dev/sdb /var/lib/torrust

# Set proper ownership
sudo chown -R torrust:torrust /var/lib/torrust

# Add to fstab for permanent mounting
echo '/dev/sdb /var/lib/torrust ext4 defaults,noatime 0 2' | sudo tee -a /etc/fstab

  3. Verify setup:

    # Check mount
df -h /var/lib/torrust

# Verify ownership
ls -la /var/lib/torrust

Data Persistence Options

Setup Type Data Persistence Cost Complexity Use Case
Main Disk Only (Default) ❌ Lost on server destruction Lower Simple Testing, development
Persistent Volume (Manual) ✅ Survives server recreation Higher Medium Production, staging

What Gets Persisted

With persistent volume setup:

  • ✅ Database data (MySQL)
  • ✅ Configuration files (.env, tracker.toml)
  • ✅ SSL certificates and keys
  • ✅ Application logs and state
  • ✅ Prometheus metrics data

Without persistent volume:

  • ❌ All data lost when server is destroyed
  • ✅ Infrastructure can be recreated identically
  • ✅ Configuration regenerated from templates

Step 6: Verify Deployment

  1. Check infrastructure status:

    make infra-status ENVIRONMENT=production PROVIDER=hetzner

  2. Test SSH access:

    make vm-ssh ENVIRONMENT=production

  3. Verify application health:

    make app-health-check ENVIRONMENT=production

Manual Verification

You can also manually verify the deployment by testing the HTTPS endpoints:

# Get the server IP from infrastructure status
export SERVER_IP=$(make infra-status ENVIRONMENT=production PROVIDER=hetzner | \
  grep vm_ip | cut -d'"' -f2)

# Test HTTPS health check endpoint
curl -k https://$SERVER_IP/health_check

# Expected response:
# {"status":"Ok"}

# Test HTTPS API endpoints (replace with your actual admin token)
curl -k "https://$SERVER_IP/api/v1/stats?token=your_admin_token_here"

# Test tracker announce endpoints
curl -k "https://$SERVER_IP/announce?info_hash=your_info_hash&peer_id=your_peer_id&port=8080"

Note: The -k flag is used to skip SSL certificate verification since we're using self-signed certificates for testing. In production with proper domain names, you would use Let's Encrypt certificates and remove the -k flag.

Deployment Success Indicators

A successful deployment should show:

✅ Infrastructure: Server created and running in Hetzner Cloud Console
✅ SSH Access: Can connect via ssh torrust@SERVER_IP
✅ HTTPS Health Check: https://SERVER_IP/health_check returns {"status":"Ok"}
✅ Docker Services: All containers running via docker compose ps
✅ API Access: Statistics endpoint accessible with admin token
✅ Tracker Functionality: UDP and HTTP tracker endpoints responding

Verified Working (August 2025): HTTPS endpoint https://138.199.166.49/health_check successfully returns the expected JSON response, confirming SSL certificate generation and nginx proxy configuration are working correctly.

Current Implementation Status

✅ Successfully Implemented:

  • Hetzner Cloud Provider: Complete infrastructure provisioning
  • Cloud-init Configuration: Fixed for providers without additional volumes
  • Self-signed SSL Certificates: Automatic generation and nginx configuration
  • Docker Services: All services running with proper orchestration
  • Persistent Volume Architecture: Configuration stored in /var/lib/torrust
  • Twelve-Factor Deployment: Complete Build/Release/Run stages working

📋 Manual Setup Required:

  • Persistent Volumes: Must be created and mounted manually for data persistence
  • Domain Configuration: Point your domain to server IP for Let's Encrypt SSL
  • Production Secrets: Replace default tokens with secure values

🔄 Future Enhancements:

  • Automatic Volume Creation: Providers could optionally create persistent volumes
  • Let's Encrypt Integration: Automatic SSL for real domains
  • Health Check Integration: Automated validation in deployment pipeline

Server Types and Pricing

Choose the appropriate server type based on your needs. Note: Server types are subject to change by Hetzner. Use hcloud server-type list for current availability.

Current Server Types (as of August 2025)

Type vCPU RAM Storage Price/Month* CPU Type Use Case
cx22 2 4GB 40GB SSD ~€5.83 Shared Light staging
cx32 4 8GB 80GB SSD ~€8.21 Shared Recommended
cx42 8 16GB 160GB SSD ~€15.99 Shared High traffic
cx52 16 32GB 320GB SSD ~€31.67 Shared Heavy workloads
cpx11 2 2GB 40GB SSD ~€4.15 AMD Shared Testing only
cpx21 3 4GB 80GB SSD ~€7.05 AMD Shared Light production
cpx31 4 8GB 160GB SSD ~€13.85 AMD Shared Production
ccx13 2 8GB 80GB SSD ~€13.85 Dedicated CPU-intensive

*Prices are approximate and may vary. Check Hetzner Cloud Console for current pricing.

Datacenter Locations

Note: Locations are subject to change. Use hcloud location list for current availability.

Code Location Network Zone Country Description
fsn1 Falkenstein DC Park 1 eu-central DE Default - EU alternative
nbg1 Nuremberg DC Park 1 eu-central DE EU, good latency
hel1 Helsinki DC Park 1 eu-central FI Northern Europe
ash Ashburn, VA us-east US US East Coast
hil Hillsboro, OR us-west US US West Coast
sin Singapore ap-southeast SG Asia Pacific

Security Considerations

  1. API Token Security: Store your token securely, never commit it to version control
  2. SSH Key Management: Use strong SSH keys, rotate regularly
  3. Firewall: The provider automatically configures necessary firewall rules
  4. SSL: Production configuration includes automatic SSL certificates via Let's Encrypt
  5. Updates: Enable automatic security updates in production

Cost Management

  1. Development: Use cx21 or cx31 for cost-effective development

  2. Staging: cx21 is usually sufficient for staging environments

  3. Production: cx31 recommended for most production workloads

  4. Monitoring: Set up billing alerts in Hetzner Cloud Console

  5. Cleanup: Always destroy infrastructure when not needed:

    make infra-destroy ENVIRONMENT=production PROVIDER=hetzner

Troubleshooting

Troubleshooting

Common Issues

1. "server type not found" Error

Problem: Error message server type cx31 not found during deployment.

Cause: Hetzner Cloud server types change over time. Some older types may be deprecated or renamed.

Solution:

  1. Get current server types:

    # Install hcloud CLI if not installed
sudo apt install golang-go
go install github.com/hetznercloud/cli/cmd/hcloud@latest
export PATH=$PATH:$(go env GOPATH)/bin

# List current server types
HCLOUD_TOKEN="$HETZNER_API_TOKEN" hcloud server-type list

  2. Update your configuration with a valid server type:

    vim infrastructure/config/providers/hetzner.env
# Change HETZNER_SERVER_TYPE to a valid type (e.g., cx32)

2. Invalid Token Error

Problem: Token validation fails with "malformed token" or 35-character length.

Cause: Using placeholder token or incorrect token format.

Solution:

  1. Ensure token is exactly 64 characters
  2. Verify token has Read & Write permissions
  3. Check token is correctly set in both:
    • infrastructure/config/providers/hetzner.env
    • Environment variable: export HETZNER_API_TOKEN=your_token_here

3. Browser HTTPS Redirect Issues with .dev Domains

Problem: Web browsers automatically redirect HTTP to HTTPS for .dev domains, even when server only serves HTTP.

Root Cause: .dev domains are on Chrome's HSTS preload list, which ALL browsers respect.

Symptoms:

# These work fine with curl
curl http://tracker.staging-torrust-demo.com/health      # ✅ Works
curl https://tracker.staging-torrust-demo.com/health     # ❌ May fail if no SSL

# But browsers automatically redirect HTTP → HTTPS
http://tracker.staging-torrust-demo.com  →  https://tracker.staging-torrust-demo.com  (automatic)

Solutions:

  1. Use .com domains for testing (no HSTS preload):

    # .com domains work normally with HTTP in browsers
http://tracker.example.com   # Works in browsers if server serves HTTP

  2. Install SSL certificates for .dev domains:

    # Deploy with HTTPS support
make app-deploy ENVIRONMENT_TYPE=staging ENVIRONMENT_FILE=staging-hetzner

# Access via HTTPS
https://tracker.staging-torrust-demo.com

  3. Use curl for HTTP testing with .dev domains:

    # For API testing during development
curl http://tracker.staging-torrust-demo.com/api/health_check
curl "http://tracker.staging-torrust-demo.com/api/v1/stats?token=TOKEN"

Important: This behavior is specific to .dev domains only. Regular .com domains do not have this HSTS preload requirement.

4. Provider Configuration Variable Collision

Problem: Error "Configuration script not found" in provider directory.

Cause: Variable name collision between main provisioning script and provider script.

Solution: This has been fixed in the codebase by using PROVIDER_DIR instead of SCRIPT_DIR in provider scripts.

4. Region/Location Issues

Problem: Some regions may have capacity limits or server type availability issues.

Solution:

  1. Check current locations:

    HCLOUD_TOKEN="$HETZNER_API_TOKEN" hcloud location list

  2. Try different locations:

    # Edit provider configuration
vim infrastructure/config/providers/hetzner.env
# Change HETZNER_LOCATION (e.g., fsn1, nbg1, hel1)

5. SSH Access Issues

Problem: Cannot SSH to deployed server.

Solutions:

  • Verify SSH key is properly configured and accessible
  • Check if server is fully booted (cloud-init can take 5-10 minutes)
  • Verify firewall rules allow SSH (port 22)

6. SSH Connection Refused (Cloud-init Still Running)

Problem: SSH connection is refused with "Connection refused" error.

Cause: Cloud-init is still configuring the system and SSH service hasn't been started yet. This is normal during initial deployment.

Symptoms:

ssh: connect to host X.X.X.X port 22: Connection refused

Diagnosis:

  1. Access server console through Hetzner Cloud Console

  2. Check system status:

    systemctl is-system-running
# Output: "maintenance" means cloud-init is still running

  3. Check cloud-init progress:

    sudo cloud-init status
# Output: "status: running" means configuration is in progress

  4. Check SSH service status:

    systemctl status ssh
# May show "inactive" or "not found" if not yet configured

  5. Monitor what cloud-init is currently doing:

    sudo tail -f /var/log/cloud-init-output.log
# Shows current installation/configuration progress

# Alternative: Check which packages are being installed
ps aux | grep -E "(apt|dpkg|cloud-init)"

Solution: Wait for cloud-init to complete. This process typically takes 5-20 minutes and includes:

  • Package updates and installations (Docker, Git, etc.)
  • User and SSH key configuration
  • SSH service installation and startup
  • Firewall setup
  • Repository cloning
  • System optimization

Expected Timeline:

  • 0-5 minutes: Package updates and system configuration
  • 5-10 minutes: Docker installation and user setup
  • 10-15 minutes: SSH service starts, connection becomes available
  • 15-20 minutes: Final repository cloning and system optimization

The system will automatically transition to "running" state and SSH will become available when complete.

7. Cloud-init Failure During Network Stage

Problem: Cloud-init fails with exit status 1 during network stage.

Symptoms:

cloud-init.service: Main process exited, code=exited, status=1/FAILURE
cloud-init.service: Failed with result 'exit-code'
Failed to start cloud-init.service - Cloud-init: Network Stage.

Cause: Network configuration issues, package repository problems, or cloud-init template errors.

Diagnosis:

  1. Check cloud-init logs for specific errors:

    # Check detailed cloud-init logs
sudo cat /var/log/cloud-init.log
sudo cat /var/log/cloud-init-output.log

# Check for network issues
sudo journalctl -u cloud-init
sudo journalctl -u systemd-networkd

  2. Test basic connectivity:

    # Test network connectivity
ping -c 3 8.8.8.8
ping -c 3 archive.ubuntu.com

# Check DNS resolution
nslookup archive.ubuntu.com

  3. Check package repositories:

    # Test package manager
sudo apt update
sudo apt list --upgradable

Recovery Methods:

Method 1: Manual System Setup (Recommended if cloud-init failed early)

Since cloud-init failed, manually configure the essential components:

# 1. Create torrust user
sudo useradd -m -s /bin/bash torrust
sudo usermod -aG sudo torrust

# 2. Add SSH key for torrust user
sudo mkdir -p /home/torrust/.ssh
sudo chmod 700 /home/torrust/.ssh

# 3. Add the SSH key from cloud-init template
# Replace with your actual public key:
echo "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC..." | sudo tee /home/torrust/.ssh/authorized_keys
sudo chmod 600 /home/torrust/.ssh/authorized_keys
sudo chown -R torrust:torrust /home/torrust/.ssh

# 4. Install and start SSH service
sudo apt update
sudo apt install -y openssh-server
sudo systemctl enable ssh
sudo systemctl start ssh

# 5. Test SSH access
sudo systemctl status ssh

Method 2: Re-run Cloud-init (If network issues are resolved)

# Clean cloud-init state and re-run
sudo cloud-init clean
sudo cloud-init init
sudo cloud-init modules --mode config
sudo cloud-init modules --mode final

Recovery Method (If SSH Still Fails):

If cloud-init completes but SSH access still fails, you can add a backup SSH key:

Note: If using Hetzner web console, you may encounter keyboard layout issues where | becomes /. Use alternative commands without pipes.

  1. Add SSH Key via Hetzner Console:

    • Go to Hetzner Cloud Console → Server → torrust-tracker-prod
    • Click "Rescue" tab
    • Enable rescue system with your personal SSH key
    • Reboot into rescue mode
    • Mount the main filesystem and debug

  2. Alternative - Add Key to Running Server:

    • Access server via Hetzner web console

    • Add your personal public key manually:

      # As root in console
mkdir -p /home/torrust/.ssh
echo "your-personal-ssh-public-key-here" >> /home/torrust/.ssh/authorized_keys
chown -R torrust:torrust /home/torrust/.ssh
chmod 700 /home/torrust/.ssh
chmod 600 /home/torrust/.ssh/authorized_keys

# Test SSH service
systemctl status ssh
systemctl start ssh  # if needed

  3. Then SSH with personal key:

    ssh -i ~/.ssh/your-personal-key torrust@138.199.166.49

8. Billing Issues

Problem: Deployment fails due to insufficient credits.

Solution: Ensure account has sufficient credits/payment method configured in Hetzner Cloud Console.

9. Volume Attachment Issues (Current Hetzner Limitation)

Problem: Attempting to create servers with volumes attached during provisioning fails.

Cause: Hetzner Cloud currently has a service limitation preventing volume attachment during server creation (as of August 2025).

Official Status: Hetzner Cloud Status - Volume Attachment Issue

Solution: This is exactly why our architecture uses manual volume setup:

  1. Create server first without any volumes attached
  2. After server is running, create and attach volumes separately
  3. SSH into server and manually format/mount the volume

This limitation validates our architectural decision to make volume setup manual and optional.

Debug Commands

# Check current server types and availability
HCLOUD_TOKEN="$HETZNER_API_TOKEN" hcloud server-type list

# Check available locations
HCLOUD_TOKEN="$HETZNER_API_TOKEN" hcloud location list

# Validate configuration without applying
make infra-plan ENVIRONMENT=production-hetzner PROVIDER=hetzner

# Check infrastructure status
make infra-status ENVIRONMENT=production-hetzner PROVIDER=hetzner

# Access server console
make vm-ssh ENVIRONMENT=production-hetzner

# Check server details (after deployment)
HCLOUD_TOKEN="$HETZNER_API_TOKEN" hcloud server list
HCLOUD_TOKEN="$HETZNER_API_TOKEN" hcloud server describe torrust-tracker-prod

Real-Time Information Commands

Always verify current Hetzner Cloud offerings before deployment:

# Get current server types with pricing
HCLOUD_TOKEN="$HETZNER_API_TOKEN" hcloud server-type list

# Get current datacenter locations
HCLOUD_TOKEN="$HETZNER_API_TOKEN" hcloud location list

# Check image availability
HCLOUD_TOKEN="$HETZNER_API_TOKEN" hcloud image list --type=system | grep ubuntu

Docker Compose Commands on Deployed Server

Important: The Torrust Tracker Demo uses a persistent volume approach where all configuration files are stored in /var/lib/torrust for backup and snapshot purposes. When running Docker Compose commands on the deployed server, you must specify the correct environment file location.

Correct Docker Compose Usage

All Docker Compose commands must be run from the application directory with the --env-file parameter:

# Connect to server
ssh torrust@YOUR_SERVER_IP

# Navigate to application directory
cd /home/torrust/github/torrust/torrust-tracker-demo/application

# Run Docker Compose commands with explicit env-file path
docker compose --env-file /var/lib/torrust/compose/.env up -d
docker compose --env-file /var/lib/torrust/compose/.env ps
docker compose --env-file /var/lib/torrust/compose/.env logs
docker compose --env-file /var/lib/torrust/compose/.env down

Why Environment Files Are in /var/lib/torrust

  • Persistent Volume: All configuration is stored in /var/lib/torrust for persistence
  • Backup Strategy: You can snapshot only the volume instead of the entire server
  • Configuration Management: All environment variables are centrally managed
  • Infrastructure Separation: Configuration survives server recreation

File Locations

# Environment file for Docker Compose
/var/lib/torrust/compose/.env

# Application configuration files
/var/lib/torrust/tracker/etc/tracker.toml
/var/lib/torrust/proxy/etc/nginx-conf/nginx.conf
/var/lib/torrust/prometheus/etc/prometheus.yml

# Persistent data
/var/lib/torrust/mysql/       # Database data
/var/lib/torrust/proxy/certs/ # SSL certificates

Common Commands

# Check service status
docker compose --env-file /var/lib/torrust/compose/.env ps

# View service logs
docker compose --env-file /var/lib/torrust/compose/.env logs tracker

# Restart specific service
docker compose --env-file /var/lib/torrust/compose/.env restart tracker

# Update and restart all services
docker compose --env-file /var/lib/torrust/compose/.env pull
docker compose --env-file /var/lib/torrust/compose/.env up -d

# Stop all services
docker compose --env-file /var/lib/torrust/compose/.env down

Getting Help

  1. Hetzner Documentation: docs.hetzner.com
  2. Community: community.hetzner.com
  3. Support: Available through Hetzner Cloud Console
  4. Terraform Provider: registry.terraform.io/providers/hetznercloud/hcloud

Next Steps

After successful deployment:

  1. DNS Configuration: Point your domain to the server IP
  2. SSL Verification: Ensure SSL certificates are properly issued
  3. Monitoring Setup: Configure Grafana dashboards and alerts
  4. Backup Strategy: Set up regular database backups
  5. Update Process: Establish update and maintenance procedures