Managing Proxmox VE via Terraform and GitOps

[joevizcara]

i am not a devops engineer. i appreciate any critique or correction.

code: gitlab github

Managing Proxmox VE via Terraform and GitOps

This program enables a declarative, IaC method of provisioning multiple resources in a Proxmox Virtual Environment.

Deployment

1. Clone this GitLab/Hub repository.

2. Go to the GitLab Project/Repository > Settings > CI/CD > Runner > Create project runner, mark Run untagged jobs and click Create runner.

3. On Step 1, copy the runner authentication token, store it somewhere and click View runners.

4. On the PVE Web UI, right-click on the target Proxmox node and click Shell.

5. Execute this command in the PVE shell.

bash <(curl -s https://gitlab.com/joevizcara/terraform-proxmox/-/raw/master/prep.sh)

Caution

The content of this shell script can be examined before executing it. It can be executed on a virtualized Proxmox VE to observe what it does. It will create a privileged PAM user to authenticate via an API token. It creates a small LXC environment for GitLab Runner to manage the Proxmox resources. Because of the API limitations between the Terraform provider and PVE, it will necessitate to add the SSH public key from the LXC to the authorized keys of the PVE node to write the cloud-init configuration YAML files to the local Snippets datastore. It will also add a few more data types that can be accepeted in the local datastore (e.g. Snippets, Import). Consider enabling two-factor authentication on GitLab if this is to be applied on a real environment.

6. Go to GitLab Project/Repository > Settings > CI/CD > Variables > Add variable:

Key: PM_API_TOKEN_SECRET
Value: the token secret value from credentials.txt

7. If this repository is cloned locally, adjust the values of the .tf files to conform with the PVE onto which this will be deployed.

Note

The Terraform provider resgistry is bpg/proxmox for reference.
git push signals will trigger the GitLab Runner and will apply the infrastructure changes.

8. If the first job stage succeeded, go to GitLab Project/Repository > Build > Jobs and click Run ▶️ button of the apply infra job.

9. If the second job stage succeeded, go to the PVE WUI to start the new VMs to test or configure.

Note

To configure the VMs, go to PVE WUI and right-click the gitlab-runner LXC and click Console.
The GitLab Runner LXC credentials are in the credentials.txt.
Inside the console, do ssh k3s@<ip-address-of-the-VM>.
They can be converted into Templates, converted into an HA cluster, etc.
The IP addresses are declared in variables.tf.

Diagramme

[bpg]

This is a no-go for me:

You should never execute scripts from untrusted sources on your PVE host under root@pam account, period.

[joevizcara]

there's already caution banner there in red telling the user to test on a virtualized pve

[bpg]

Yeah, but who actually reads them :)) Anyway, you can't guarantee that the script content hasn't changed on the remote since it was previously downloaded and reviewed.
There's also the possibility of serving different scripts depending on the user agent, or using other tricks, especially if the content is served from a custom domain (not your case, just an example).

And for context, we're talking about a hypervisor host that runs other workloads, VMs, containers, and so on. Root access there is really sensitive and shouldn't be used lightly.

[bpg]

Also, I don't really see how "enabling two-factor authentication on GitLab" could mitigate these concerns. 2FA should be used everywhere regardless, it's just basic security hygiene.

[joevizcara]

the 2fa doesn't mitigate your concerns of the user not reading cautionary texts and being served an altered/different malicious script.
it's for a scenario when an attacker gains control over your repository and starts remotely managing your pve by altering those yml and tf codes.
let's also admit and be honest to ourselves that there are countless shell scripts online that publicly tells users to run them, and almost all of them don't even have a warning text.

[svengreb]

To be fair, yes, *nix shell scripts are often used, but as soon as I see one I close the browser tab.
If you're seeking feedback I'd suggest that you resolve the shell script into documentation which means that it would be better for users to know exactly what needs to be done to set up your solution. This way every step is documented and everyone knows what's happening (and why!) while manually executed command helps others to learn the techstack.

[bpg]

Also, I'm not 100% sure about the purpose of this post here (and all over the Reddit to be fair). Promoting your repo?

[joevizcara]

i'm seeking critique, like what you did about the "untrusted script"