Merge #267: docs: [#266] document Hetzner SSH key dual injection and root access security

josecelano · josecelano · commit 41d826b3186a · 2026-01-10T17:21:04.000Z
c0d7794 docs: [#266] document Hetzner SSH key dual injection and root access security (Jose Celano) Pull request description: ## Summary Documents the SSH key dual injection behavior specific to Hetzner Cloud deployments and provides guidance for users who want stricter security. Closes #266 ## Changes ### New Documentation - **ADR**: `docs/decisions/hetzner-ssh-key-dual-injection.md` - Explains the architectural decision to use both OpenTofu SSH key and cloud-init for SSH access - **Security Guide**: `docs/security/ssh-root-access-hetzner.md` - Comprehensive guide explaining the behavior and how to disable root SSH access ### Updated Documentation - **Hetzner Provider**: Added "SSH Key Behavior" section explaining the dual injection pattern - **LXD Provider**: Added section explaining why LXD doesn't have this behavior (uses `lxc exec` instead) - **ADR Index**: Updated `docs/decisions/README.md` with new decision ### Template Updates - **Hetzner main.tf**: Added detailed comments explaining the purpose of the `hcloud_ssh_key` resource and its role as a debugging fallback ## Context When deploying to Hetzner Cloud, SSH access is configured through two mechanisms: | Mechanism | User | Purpose | |-----------|------|---------| | OpenTofu `hcloud_ssh_key` | `root` | Emergency/debug access | | cloud-init `ssh_authorized_keys` | `torrust` | Normal application access | **Why both?** If cloud-init fails (syntax error, network issue, script error), the server would be completely inaccessible without root SSH access. The OpenTofu SSH key provides a recovery/debugging path. ## Acceptance Criteria - [x] ADR created with proper status, date, and all sections - [x] Security document explains both risk and mitigation - [x] Working commands provided to disable root access - [x] Provider docs updated with cross-references - [x] Template has clear explanatory comment - [x] All documentation links are valid - [x] Pre-commit checks pass ACKs for top commit: josecelano: ACK c0d7794 Tree-SHA512: e75d9911fbe344dcdda5eb7102e83beee2979075bd60ff2f15cde33d71485dc157c0f32915135488d00ef5b0be3f26c66ccc96483de705006c8715d623154763
diff --git a/docs/decisions/README.md b/docs/decisions/README.md
@@ -6,6 +6,7 @@ This directory contains architectural decision records for the Torrust Tracker D
 
 | Status        | Date       | Decision                                                                                                  | Summary                                                                                    |
 | ------------- | ---------- | --------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------ |
+| ✅ Accepted   | 2026-01-10 | [Hetzner SSH Key Dual Injection Pattern](./hetzner-ssh-key-dual-injection.md)                             | Use both OpenTofu SSH key and cloud-init for debugging capability with manual hardening    |
 | ✅ Accepted   | 2026-01-10 | [Configuration and Data Directories as Secrets](./configuration-directories-as-secrets.md)                | Treat envs/, data/, build/ as secrets; no env var injection; users secure via permissions  |
 | ✅ Accepted   | 2026-01-07 | [Configuration DTO Layer Placement](./configuration-dto-layer-placement.md)                               | Keep configuration DTOs in application layer, not domain; defer package extraction         |
 | ✅ Accepted   | 2025-12-23 | [Docker Security Scan Exit Code Zero](./docker-security-scan-exit-code-zero.md)                           | Use exit-code 0 for security scanning - Trivy detects, GitHub Security decides, CI green   |
diff --git a/docs/decisions/hetzner-ssh-key-dual-injection.md b/docs/decisions/hetzner-ssh-key-dual-injection.md
@@ -0,0 +1,110 @@
+# Decision: Hetzner SSH Key Dual Injection Pattern
+
+## Status
+
+Accepted
+
+## Date
+
+2026-01-10
+
+## Context
+
+When deploying to Hetzner Cloud, the deployer needs to configure SSH access for the provisioned server. There are two primary mechanisms available:
+
+1. **Hetzner Provider SSH Key Resource**: The `hcloud_ssh_key` OpenTofu resource registers an SSH public key in Hetzner's account-level key registry. When a server is created with `ssh_keys = [hcloud_ssh_key.id]`, Hetzner automatically injects this key into the root user's `~/.ssh/authorized_keys` during server creation (before the OS boots).
+
+2. **cloud-init SSH Key Injection**: The cloud-init `user-data` configuration can create a non-root user (e.g., `torrust`) with SSH authorized keys via the `ssh_authorized_keys` directive. This runs after the first boot.
+
+The question arose: Do we need both mechanisms, or is cloud-init sufficient?
+
+### The Problem with cloud-init Only
+
+If cloud-init fails (syntax error, network issue, script error, timeout), the server becomes completely inaccessible:
+
+- No SSH access to any user
+- Cannot debug what went wrong
+- Only option is to destroy and recreate the server
+- Root cause may be unclear without access to logs
+
+### Debugging Scenario
+
+During development of this deployer, cloud-init issues were common:
+
+- YAML syntax errors in cloud-init configuration
+- Package installation failures due to network issues
+- User creation failures
+- Script execution errors
+
+Having root SSH access (injected before cloud-init runs) provided a crucial debugging path.
+
+## Decision
+
+**Use both SSH key injection mechanisms for Hetzner deployments:**
+
+1. **OpenTofu `hcloud_ssh_key` resource** with server `ssh_keys` reference → provides root SSH access as a fallback/debugging mechanism
+2. **cloud-init `ssh_authorized_keys`** → provides application user SSH access for normal operations
+
+The same SSH key is used for both mechanisms, so there is no additional key exposure.
+
+### Why Not Make It Configurable?
+
+While a configuration option (e.g., `enable_root_ssh_fallback`) was considered, we opted to always enable it because:
+
+- The security risk is minimal (same key, user can disable post-deployment)
+- The debugging benefit is significant
+- Complexity of additional configuration outweighs the benefit
+- Users who want stricter security can disable root access after deployment
+
+## Consequences
+
+### Positive
+
+- **Debugging capability**: Can SSH as root to diagnose cloud-init failures
+- **Recovery path**: Server is never completely inaccessible after creation
+- **Same key**: No additional key exposure since the same key is used
+- **Visibility**: SSH key appears in Hetzner Console for management
+
+### Negative
+
+- **Root access enabled by default**: Violates principle of least privilege
+- **Provider-specific behavior**: LXD provider doesn't have this (uses `lxc exec` instead)
+- **Manual cleanup needed**: Users wanting strict security must disable root access manually
+- **Key in Hetzner registry**: SSH key visible in Hetzner Console Security section
+
+### Neutral
+
+- **Documentation required**: Users need to understand this behavior
+- **Post-deployment hardening**: Security-conscious users have a clear path to disable
+
+## Alternatives Considered
+
+### Alternative 1: cloud-init Only
+
+Remove the `hcloud_ssh_key` resource and rely solely on cloud-init.
+
+**Rejected because**: Loses debugging capability. Failed cloud-init means inaccessible server.
+
+### Alternative 2: Configurable via Environment Config
+
+Add a configuration option like `enable_root_ssh_fallback: bool`.
+
+**Rejected because**: Added complexity for minimal benefit. Users can disable manually.
+
+### Alternative 3: Disable by Default, Enable for Debug
+
+Only create the Hetzner SSH key when an environment variable or flag is set.
+
+**Rejected because**: Most users would benefit from the fallback, and forgetting to enable it when needed causes frustration.
+
+## Related Decisions
+
+- [Cloud-Init SSH Port Configuration with Reboot](./cloud-init-ssh-port-reboot.md) - Related cloud-init configuration pattern
+- [Configuration Directories as Secrets](./configuration-directories-as-secrets.md) - Security considerations for configuration
+
+## References
+
+- [Hetzner Cloud SSH Keys Documentation](https://docs.hetzner.com/cloud/servers/getting-started/connecting-to-a-server/)
+- [cloud-init User Data Documentation](https://cloudinit.readthedocs.io/en/latest/reference/examples.html#including-users-and-groups)
+- [OpenTofu Hetzner Provider - hcloud_ssh_key](https://registry.terraform.io/providers/hetznercloud/hcloud/latest/docs/resources/ssh_key)
+- [Security Doc: SSH Root Access on Hetzner](../security/ssh-root-access-hetzner.md)
diff --git a/docs/security/ssh-root-access-hetzner.md b/docs/security/ssh-root-access-hetzner.md
@@ -0,0 +1,135 @@
+# SSH Root Access on Hetzner Cloud Deployments
+
+This document explains the SSH key behavior specific to Hetzner Cloud deployments and provides guidance for users who want stricter security.
+
+## Overview
+
+When deploying to Hetzner Cloud, the deployer configures SSH access through **two independent mechanisms**:
+
+| Mechanism                        | User      | When Applied                  | Purpose                   |
+| -------------------------------- | --------- | ----------------------------- | ------------------------- |
+| OpenTofu `hcloud_ssh_key`        | `root`    | Server creation (before boot) | Emergency/debug access    |
+| cloud-init `ssh_authorized_keys` | `torrust` | First boot                    | Normal application access |
+
+**Result**: Both `root` and `torrust` users have SSH access after deployment.
+
+## Why Root Access Is Enabled
+
+The primary reason is **debugging capability**. If cloud-init fails, the server would be completely inaccessible without root SSH access.
+
+### Failure Scenarios Where Root Access Helps
+
+- **YAML syntax errors** in cloud-init configuration
+- **Network issues** during package installation
+- **User creation failures** in cloud-init
+- **Script execution errors** in cloud-init
+- **Timeouts** during cloud-init execution
+
+With root access, you can:
+
+```bash
+# SSH as root to diagnose
+ssh -i ~/.ssh/your_key root@<server-ip>
+
+# Check cloud-init status
+cloud-init status --wait
+
+# View cloud-init logs
+cat /var/log/cloud-init-output.log
+journalctl -u cloud-init
+```
+
+## Security Implications
+
+### Risks
+
+- **Elevated privileges**: Root has unrestricted system access
+- **Larger attack surface**: Compromised SSH key grants full system control
+- **Principle of least privilege**: Violated by default
+
+### Mitigations Already in Place
+
+- **Application runs as non-root**: The `torrust` user runs the tracker
+- **Passwordless sudo**: `torrust` can escalate when needed
+- **Same SSH key**: No additional key exposure (both mechanisms use the same key)
+
+## Disabling Root SSH Access
+
+For production deployments where you want stricter security, you can disable root SSH access after verifying the deployment succeeded.
+
+### Option 1: Remove Root's Authorized Keys
+
+The simplest approach - removes the SSH key from root's configuration:
+
+```bash
+ssh torrust@<server-ip> "sudo rm /root/.ssh/authorized_keys"
+```
+
+**Effect**: Root can no longer SSH in. You can still access via `torrust` user with sudo.
+
+### Option 2: Disable Root Login in SSH Config
+
+Modifies the SSH daemon to reject root logins entirely:
+
+```bash
+ssh torrust@<server-ip> "sudo sed -i 's/#PermitRootLogin yes/PermitRootLogin no/' /etc/ssh/sshd_config && sudo systemctl restart sshd"
+```
+
+**Effect**: SSH daemon rejects all root login attempts, regardless of authentication method.
+
+### Option 3: Remove SSH Key from Hetzner Console
+
+Removes the key from Hetzner's account-level registry:
+
+1. Go to [Hetzner Cloud Console](https://console.hetzner.cloud/)
+2. Navigate to **Security** → **SSH Keys**
+3. Find the key named `torrust-tracker-vm-<environment>-ssh-key`
+4. Click **Delete**
+
+**Effect**: Key is removed from your Hetzner account. Does NOT affect existing servers - only prevents the key from being used for future server creation.
+
+## Verification
+
+After disabling root access, verify the change:
+
+```bash
+# This should fail (connection refused or permission denied)
+ssh -i ~/.ssh/your_key root@<server-ip>
+
+# This should still work
+ssh -i ~/.ssh/your_key torrust@<server-ip>
+```
+
+## Provider Comparison
+
+| Provider    | Root SSH Access       | Reason                                                |
+| ----------- | --------------------- | ----------------------------------------------------- |
+| **Hetzner** | ✅ Enabled by default | Debugging capability for cloud-init failures          |
+| **LXD**     | ❌ Not applicable     | `lxc exec` provides direct console access without SSH |
+
+The LXD provider doesn't need this pattern because:
+
+- LXD runs locally on your machine
+- `lxc exec <instance> -- bash` gives direct console access
+- No SSH required for debugging
+
+## Recommendations
+
+### For Development/Testing
+
+Keep root access enabled. The debugging capability is valuable when iterating on configurations.
+
+### For Production
+
+Consider disabling root access after successful deployment:
+
+1. Verify deployment completed successfully
+2. Test that you can SSH as `torrust` user
+3. Apply one of the disable options above
+4. Verify root access is blocked
+
+## Related Documentation
+
+- [ADR: Hetzner SSH Key Dual Injection Pattern](../decisions/hetzner-ssh-key-dual-injection.md)
+- [Hetzner Provider Documentation](../user-guide/providers/hetzner.md)
+- [SSH Keys Guide](../tech-stack/ssh-keys.md)
diff --git a/docs/user-guide/providers/hetzner.md b/docs/user-guide/providers/hetzner.md
@@ -143,10 +143,27 @@ cat /var/log/cloud-init-output.log
 2. **Restrict SSH access** - Consider using Hetzner Firewall
 3. **Use strong SSH keys** - Ed25519 or RSA 4096-bit minimum
 4. **Regular updates** - Keep server packages updated
+5. **Disable root SSH access** - For production, see [SSH Root Access Guide](../../security/ssh-root-access-hetzner.md)
+
+## SSH Key Behavior
+
+Hetzner deployments configure SSH access through two mechanisms:
+
+| Mechanism                 | User      | Purpose                   |
+| ------------------------- | --------- | ------------------------- |
+| OpenTofu `hcloud_ssh_key` | `root`    | Emergency/debug access    |
+| cloud-init                | `torrust` | Normal application access |
+
+**Why both?** If cloud-init fails, root SSH access provides a debugging path. Without it, a failed cloud-init would leave the server completely inaccessible.
+
+**For stricter security**: You can disable root SSH access after deployment. See [SSH Root Access on Hetzner](../../security/ssh-root-access-hetzner.md) for instructions.
+
+**Note**: The SSH key appears in your Hetzner Console under **Security** → **SSH Keys** with the name `torrust-tracker-vm-<environment>-ssh-key`.
 
 ## Related Documentation
 
 - [Quick Start: Docker](../quick-start/docker.md) - Deploy to Hetzner using Docker
 - [Quick Start: Native](../quick-start/native.md) - Deploy using native installation
 - [SSH Keys Guide](../../tech-stack/ssh-keys.md) - SSH key generation
+- [SSH Root Access Security](../../security/ssh-root-access-hetzner.md) - Disabling root access
 - [LXD Provider](lxd.md) - Local development alternative
diff --git a/docs/user-guide/providers/lxd.md b/docs/user-guide/providers/lxd.md
@@ -95,6 +95,22 @@ lxc network create lxdbr0
 | Storage  | 20 GB   | 50+ GB        |
 | OS       | Linux   | Ubuntu 22.04+ |
 
+## SSH Key Behavior
+
+Unlike the [Hetzner provider](hetzner.md), LXD does **not** create a provider-level SSH key resource. This is because:
+
+1. **Direct console access**: `lxc exec` provides shell access without SSH
+2. **No account-level keys**: LXD doesn't have an SSH key registry concept
+3. **Local environment**: No need for remote debugging fallback
+
+SSH access is configured solely through cloud-init for the `torrust` user.
+
+**Debugging without SSH**: If cloud-init fails, use direct console access:
+
+```bash
+lxc exec torrust-tracker-vm-<environment> -- bash
+```
+
 ## Related Documentation
 
 - [LXD Tech Guide](../../tech-stack/lxd.md) - Installation and detailed LXD operations
diff --git a/templates/tofu/hetzner/main.tf b/templates/tofu/hetzner/main.tf
@@ -78,6 +78,22 @@ variable "server_labels" {
 # ============================================================================
 
 # Create or import the SSH key for server access
+#
+# PURPOSE: This resource registers the SSH public key in Hetzner's account-level
+# registry and enables ROOT SSH access as a fallback/debugging mechanism.
+#
+# WHY BOTH THIS AND CLOUD-INIT?
+# - This SSH key (via ssh_keys on server) → root user access
+# - cloud-init ssh_authorized_keys → torrust user access
+#
+# The root access is intentional: if cloud-init fails (syntax error, network
+# issue, script error), the server would be completely inaccessible without it.
+# Root SSH provides a recovery/debugging path.
+#
+# SECURITY NOTE: For production deployments, consider disabling root SSH access
+# after verifying deployment succeeded. See docs/security/ssh-root-access-hetzner.md
+#
+# This key will appear in Hetzner Console → Security → SSH Keys.
 resource "hcloud_ssh_key" "torrust" {
   name       = var.ssh_key_name
   public_key = var.ssh_public_key
