Skip to content

Commit c0d7794

Browse files
josecelanojosecelano
committed
docs: [#266] document Hetzner SSH key dual injection and root access security
- Add ADR explaining the dual SSH key injection pattern - Create security documentation for SSH root access on Hetzner - Update Hetzner provider docs with SSH key behavior section - Update LXD provider docs explaining why it differs - Add explanatory comments to Hetzner OpenTofu template
1 parent 15c0681 commit c0d7794

File tree

6 files changed

+295
-0
lines changed

6 files changed

+295
-0
lines changed

‎docs/decisions/README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@ This directory contains architectural decision records for the Torrust Tracker D
66

77
| Status | Date | Decision | Summary |
88
| ------------- | ---------- | --------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------ |
9+
| ✅ Accepted | 2026-01-10 | [Hetzner SSH Key Dual Injection Pattern](./hetzner-ssh-key-dual-injection.md) | Use both OpenTofu SSH key and cloud-init for debugging capability with manual hardening |
910
| ✅ Accepted | 2026-01-10 | [Configuration and Data Directories as Secrets](./configuration-directories-as-secrets.md) | Treat envs/, data/, build/ as secrets; no env var injection; users secure via permissions |
1011
| ✅ Accepted | 2026-01-07 | [Configuration DTO Layer Placement](./configuration-dto-layer-placement.md) | Keep configuration DTOs in application layer, not domain; defer package extraction |
1112
| ✅ Accepted | 2025-12-23 | [Docker Security Scan Exit Code Zero](./docker-security-scan-exit-code-zero.md) | Use exit-code 0 for security scanning - Trivy detects, GitHub Security decides, CI green |

‎docs/decisions/hetzner-ssh-key-dual-injection.md‎

Lines changed: 110 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,110 @@
1+
# Decision: Hetzner SSH Key Dual Injection Pattern
2+
3+
## Status
4+
5+
Accepted
6+
7+
## Date
8+
9+
2026-01-10
10+
11+
## Context
12+
13+
When deploying to Hetzner Cloud, the deployer needs to configure SSH access for the provisioned server. There are two primary mechanisms available:
14+
15+
1. **Hetzner Provider SSH Key Resource**: The `hcloud_ssh_key` OpenTofu resource registers an SSH public key in Hetzner's account-level key registry. When a server is created with `ssh_keys = [hcloud_ssh_key.id]`, Hetzner automatically injects this key into the root user's `~/.ssh/authorized_keys` during server creation (before the OS boots).
16+
17+
2. **cloud-init SSH Key Injection**: The cloud-init `user-data` configuration can create a non-root user (e.g., `torrust`) with SSH authorized keys via the `ssh_authorized_keys` directive. This runs after the first boot.
18+
19+
The question arose: Do we need both mechanisms, or is cloud-init sufficient?
20+
21+
### The Problem with cloud-init Only
22+
23+
If cloud-init fails (syntax error, network issue, script error, timeout), the server becomes completely inaccessible:
24+
25+
- No SSH access to any user
26+
- Cannot debug what went wrong
27+
- Only option is to destroy and recreate the server
28+
- Root cause may be unclear without access to logs
29+
30+
### Debugging Scenario
31+
32+
During development of this deployer, cloud-init issues were common:
33+
34+
- YAML syntax errors in cloud-init configuration
35+
- Package installation failures due to network issues
36+
- User creation failures
37+
- Script execution errors
38+
39+
Having root SSH access (injected before cloud-init runs) provided a crucial debugging path.
40+
41+
## Decision
42+
43+
**Use both SSH key injection mechanisms for Hetzner deployments:**
44+
45+
1. **OpenTofu `hcloud_ssh_key` resource** with server `ssh_keys` reference → provides root SSH access as a fallback/debugging mechanism
46+
2. **cloud-init `ssh_authorized_keys`** → provides application user SSH access for normal operations
47+
48+
The same SSH key is used for both mechanisms, so there is no additional key exposure.
49+
50+
### Why Not Make It Configurable?
51+
52+
While a configuration option (e.g., `enable_root_ssh_fallback`) was considered, we opted to always enable it because:
53+
54+
- The security risk is minimal (same key, user can disable post-deployment)
55+
- The debugging benefit is significant
56+
- Complexity of additional configuration outweighs the benefit
57+
- Users who want stricter security can disable root access after deployment
58+
59+
## Consequences
60+
61+
### Positive
62+
63+
- **Debugging capability**: Can SSH as root to diagnose cloud-init failures
64+
- **Recovery path**: Server is never completely inaccessible after creation
65+
- **Same key**: No additional key exposure since the same key is used
66+
- **Visibility**: SSH key appears in Hetzner Console for management
67+
68+
### Negative
69+
70+
- **Root access enabled by default**: Violates principle of least privilege
71+
- **Provider-specific behavior**: LXD provider doesn't have this (uses `lxc exec` instead)
72+
- **Manual cleanup needed**: Users wanting strict security must disable root access manually
73+
- **Key in Hetzner registry**: SSH key visible in Hetzner Console Security section
74+
75+
### Neutral
76+
77+
- **Documentation required**: Users need to understand this behavior
78+
- **Post-deployment hardening**: Security-conscious users have a clear path to disable
79+
80+
## Alternatives Considered
81+
82+
### Alternative 1: cloud-init Only
83+
84+
Remove the `hcloud_ssh_key` resource and rely solely on cloud-init.
85+
86+
**Rejected because**: Loses debugging capability. Failed cloud-init means inaccessible server.
87+
88+
### Alternative 2: Configurable via Environment Config
89+
90+
Add a configuration option like `enable_root_ssh_fallback: bool`.
91+
92+
**Rejected because**: Added complexity for minimal benefit. Users can disable manually.
93+
94+
### Alternative 3: Disable by Default, Enable for Debug
95+
96+
Only create the Hetzner SSH key when an environment variable or flag is set.
97+
98+
**Rejected because**: Most users would benefit from the fallback, and forgetting to enable it when needed causes frustration.
99+
100+
## Related Decisions
101+
102+
- [Cloud-Init SSH Port Configuration with Reboot](./cloud-init-ssh-port-reboot.md) - Related cloud-init configuration pattern
103+
- [Configuration Directories as Secrets](./configuration-directories-as-secrets.md) - Security considerations for configuration
104+
105+
## References
106+
107+
- [Hetzner Cloud SSH Keys Documentation](https://docs.hetzner.com/cloud/servers/getting-started/connecting-to-a-server/)
108+
- [cloud-init User Data Documentation](https://cloudinit.readthedocs.io/en/latest/reference/examples.html#including-users-and-groups)
109+
- [OpenTofu Hetzner Provider - hcloud_ssh_key](https://registry.terraform.io/providers/hetznercloud/hcloud/latest/docs/resources/ssh_key)
110+
- [Security Doc: SSH Root Access on Hetzner](../security/ssh-root-access-hetzner.md)

‎docs/security/ssh-root-access-hetzner.md‎

Lines changed: 135 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,135 @@
1+
# SSH Root Access on Hetzner Cloud Deployments
2+
3+
This document explains the SSH key behavior specific to Hetzner Cloud deployments and provides guidance for users who want stricter security.
4+
5+
## Overview
6+
7+
When deploying to Hetzner Cloud, the deployer configures SSH access through **two independent mechanisms**:
8+
9+
| Mechanism | User | When Applied | Purpose |
10+
| -------------------------------- | --------- | ----------------------------- | ------------------------- |
11+
| OpenTofu `hcloud_ssh_key` | `root` | Server creation (before boot) | Emergency/debug access |
12+
| cloud-init `ssh_authorized_keys` | `torrust` | First boot | Normal application access |
13+
14+
**Result**: Both `root` and `torrust` users have SSH access after deployment.
15+
16+
## Why Root Access Is Enabled
17+
18+
The primary reason is **debugging capability**. If cloud-init fails, the server would be completely inaccessible without root SSH access.
19+
20+
### Failure Scenarios Where Root Access Helps
21+
22+
- **YAML syntax errors** in cloud-init configuration
23+
- **Network issues** during package installation
24+
- **User creation failures** in cloud-init
25+
- **Script execution errors** in cloud-init
26+
- **Timeouts** during cloud-init execution
27+
28+
With root access, you can:
29+
30+
```bash
31+
# SSH as root to diagnose
32+
ssh -i ~/.ssh/your_key root@<server-ip>
33+
34+
# Check cloud-init status
35+
cloud-init status --wait
36+
37+
# View cloud-init logs
38+
cat /var/log/cloud-init-output.log
39+
journalctl -u cloud-init
40+
```
41+
42+
## Security Implications
43+
44+
### Risks
45+
46+
- **Elevated privileges**: Root has unrestricted system access
47+
- **Larger attack surface**: Compromised SSH key grants full system control
48+
- **Principle of least privilege**: Violated by default
49+
50+
### Mitigations Already in Place
51+
52+
- **Application runs as non-root**: The `torrust` user runs the tracker
53+
- **Passwordless sudo**: `torrust` can escalate when needed
54+
- **Same SSH key**: No additional key exposure (both mechanisms use the same key)
55+
56+
## Disabling Root SSH Access
57+
58+
For production deployments where you want stricter security, you can disable root SSH access after verifying the deployment succeeded.
59+
60+
### Option 1: Remove Root's Authorized Keys
61+
62+
The simplest approach - removes the SSH key from root's configuration:
63+
64+
```bash
65+
ssh torrust@<server-ip> "sudo rm /root/.ssh/authorized_keys"
66+
```
67+
68+
**Effect**: Root can no longer SSH in. You can still access via `torrust` user with sudo.
69+
70+
### Option 2: Disable Root Login in SSH Config
71+
72+
Modifies the SSH daemon to reject root logins entirely:
73+
74+
```bash
75+
ssh torrust@<server-ip> "sudo sed -i 's/#PermitRootLogin yes/PermitRootLogin no/' /etc/ssh/sshd_config && sudo systemctl restart sshd"
76+
```
77+
78+
**Effect**: SSH daemon rejects all root login attempts, regardless of authentication method.
79+
80+
### Option 3: Remove SSH Key from Hetzner Console
81+
82+
Removes the key from Hetzner's account-level registry:
83+
84+
1. Go to [Hetzner Cloud Console](https://console.hetzner.cloud/)
85+
2. Navigate to **Security** → **SSH Keys**
86+
3. Find the key named `torrust-tracker-vm-<environment>-ssh-key`
87+
4. Click **Delete**
88+
89+
**Effect**: Key is removed from your Hetzner account. Does NOT affect existing servers - only prevents the key from being used for future server creation.
90+
91+
## Verification
92+
93+
After disabling root access, verify the change:
94+
95+
```bash
96+
# This should fail (connection refused or permission denied)
97+
ssh -i ~/.ssh/your_key root@<server-ip>
98+
99+
# This should still work
100+
ssh -i ~/.ssh/your_key torrust@<server-ip>
101+
```
102+
103+
## Provider Comparison
104+
105+
| Provider | Root SSH Access | Reason |
106+
| ----------- | --------------------- | ----------------------------------------------------- |
107+
| **Hetzner** | ✅ Enabled by default | Debugging capability for cloud-init failures |
108+
| **LXD** | ❌ Not applicable | `lxc exec` provides direct console access without SSH |
109+
110+
The LXD provider doesn't need this pattern because:
111+
112+
- LXD runs locally on your machine
113+
- `lxc exec <instance> -- bash` gives direct console access
114+
- No SSH required for debugging
115+
116+
## Recommendations
117+
118+
### For Development/Testing
119+
120+
Keep root access enabled. The debugging capability is valuable when iterating on configurations.
121+
122+
### For Production
123+
124+
Consider disabling root access after successful deployment:
125+
126+
1. Verify deployment completed successfully
127+
2. Test that you can SSH as `torrust` user
128+
3. Apply one of the disable options above
129+
4. Verify root access is blocked
130+
131+
## Related Documentation
132+
133+
- [ADR: Hetzner SSH Key Dual Injection Pattern](../decisions/hetzner-ssh-key-dual-injection.md)
134+
- [Hetzner Provider Documentation](../user-guide/providers/hetzner.md)
135+
- [SSH Keys Guide](../tech-stack/ssh-keys.md)

‎docs/user-guide/providers/hetzner.md‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -143,10 +143,27 @@ cat /var/log/cloud-init-output.log
143143
2. **Restrict SSH access** - Consider using Hetzner Firewall
144144
3. **Use strong SSH keys** - Ed25519 or RSA 4096-bit minimum
145145
4. **Regular updates** - Keep server packages updated
146+
5. **Disable root SSH access** - For production, see [SSH Root Access Guide](../../security/ssh-root-access-hetzner.md)
147+
148+
## SSH Key Behavior
149+
150+
Hetzner deployments configure SSH access through two mechanisms:
151+
152+
| Mechanism | User | Purpose |
153+
| ------------------------- | --------- | ------------------------- |
154+
| OpenTofu `hcloud_ssh_key` | `root` | Emergency/debug access |
155+
| cloud-init | `torrust` | Normal application access |
156+
157+
**Why both?** If cloud-init fails, root SSH access provides a debugging path. Without it, a failed cloud-init would leave the server completely inaccessible.
158+
159+
**For stricter security**: You can disable root SSH access after deployment. See [SSH Root Access on Hetzner](../../security/ssh-root-access-hetzner.md) for instructions.
160+
161+
**Note**: The SSH key appears in your Hetzner Console under **Security** → **SSH Keys** with the name `torrust-tracker-vm-<environment>-ssh-key`.
146162

147163
## Related Documentation
148164

149165
- [Quick Start: Docker](../quick-start/docker.md) - Deploy to Hetzner using Docker
150166
- [Quick Start: Native](../quick-start/native.md) - Deploy using native installation
151167
- [SSH Keys Guide](../../tech-stack/ssh-keys.md) - SSH key generation
168+
- [SSH Root Access Security](../../security/ssh-root-access-hetzner.md) - Disabling root access
152169
- [LXD Provider](lxd.md) - Local development alternative

‎docs/user-guide/providers/lxd.md‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -95,6 +95,22 @@ lxc network create lxdbr0
9595
| Storage | 20 GB | 50+ GB |
9696
| OS | Linux | Ubuntu 22.04+ |
9797

98+
## SSH Key Behavior
99+
100+
Unlike the [Hetzner provider](hetzner.md), LXD does **not** create a provider-level SSH key resource. This is because:
101+
102+
1. **Direct console access**: `lxc exec` provides shell access without SSH
103+
2. **No account-level keys**: LXD doesn't have an SSH key registry concept
104+
3. **Local environment**: No need for remote debugging fallback
105+
106+
SSH access is configured solely through cloud-init for the `torrust` user.
107+
108+
**Debugging without SSH**: If cloud-init fails, use direct console access:
109+
110+
```bash
111+
lxc exec torrust-tracker-vm-<environment> -- bash
112+
```
113+
98114
## Related Documentation
99115

100116
- [LXD Tech Guide](../../tech-stack/lxd.md) - Installation and detailed LXD operations

‎templates/tofu/hetzner/main.tf‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -78,6 +78,22 @@ variable "server_labels" {
7878
# ============================================================================
7979

8080
# Create or import the SSH key for server access
81+
#
82+
# PURPOSE: This resource registers the SSH public key in Hetzner's account-level
83+
# registry and enables ROOT SSH access as a fallback/debugging mechanism.
84+
#
85+
# WHY BOTH THIS AND CLOUD-INIT?
86+
# - This SSH key (via ssh_keys on server) → root user access
87+
# - cloud-init ssh_authorized_keys → torrust user access
88+
#
89+
# The root access is intentional: if cloud-init fails (syntax error, network
90+
# issue, script error), the server would be completely inaccessible without it.
91+
# Root SSH provides a recovery/debugging path.
92+
#
93+
# SECURITY NOTE: For production deployments, consider disabling root SSH access
94+
# after verifying deployment succeeded. See docs/security/ssh-root-access-hetzner.md
95+
#
96+
# This key will appear in Hetzner Console → Security → SSH Keys.
8197
resource "hcloud_ssh_key" "torrust" {
8298
name = var.ssh_key_name
8399
public_key = var.ssh_public_key

0 commit comments

Comments
 (0)