Error: creating custom disk: unable to authenticate user over SSH

[testdcpractice]

Hello! When authenticating using the API token, I get the error:

│ Error: creating custom disk: unable to authenticate user "opentofu" over SSH to "10.10.50.32:22". Please verify that ssh-agent is correctly loaded with an authorized key via 'ssh-add -L' (NOTE: configurations in ~/.ssh/config are not considered by the provider): failed to dial 10.10.50.32:22: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none password], no supported methods remain
│ 
│   with proxmox_virtual_environment_vm.rocky_vm[0],
│   on main.tf line 47, in resource "proxmox_virtual_environment_vm" "rocky_vm":
│   47: resource "proxmox_virtual_environment_vm" "rocky_vm" {
│ 

When I authenticate with the same API token in the telmate/proxmox provider, the virtual machine is created without errors. The API token has been granted all necessary access permissions (administrator privileges). SSH keys for the user 'opentofu' have been generated and added to the file /etc/pve/priv/authorized_keys. The user 'opentofu' has been granted administrator privileges. If I authenticate using the username and password of the 'opentofu' user, the virtual machine is created without errors. What's wrong?

terraform {
  required_providers {
    proxmox = {
      source  = "bpg/proxmox"
      version = "0.82.0"
    }
  }
}

provider "proxmox" {
  endpoint = var.api_url
  #username = var.pm_user
  #password = var.pm_password
  #api_token = "${var.pm_user}!${var.token_id}=${var.token_secret}"
  api_token = "opentofu@pam!provider=my-secret-api-token"
  insecure = true // necessary because of self-signed certificates

  ssh {
    agent = true
    username = "opentofu" 
  }
}

resource "proxmox_virtual_environment_download_file" "rocky_cloud_image" {
  content_type = "iso"
  overwrite_unmanaged = true
  datastore_id = var.datastore_id
  node_name    = var.node_name
  url          = "https://dl.rockylinux.org/pub/rocky/9/images/x86_64/Rocky-9-GenericCloud.latest.x86_64.qcow2"
  file_name    = "Rocky-9-GenericCloud.latest.x86_64.qcow2.img"
}

resource "proxmox_virtual_environment_vm" "rocky_vm" {
  count     = var.vm_count
  name      = var.vm_name
  #vm_id     = var.vm_id
  node_name = var.node_name

  operating_system {
    type = "l26"
  }

  # should be true if qemu agent is not installed / enabled on the VM
  stop_on_destroy = true

  initialization {

    datastore_id = var.datastore_id

    ip_config {
      ipv4 {
        address = "dhcp"
    }
  }
    user_account {
      username = "admin2"
      password = var.vm_password
      keys = var.ssh_public_keys
    }

  }

  scsi_hardware = "virtio-scsi-single" 

  disk {
    datastore_id = var.datastore_id
    file_id      = proxmox_virtual_environment_download_file.rocky_cloud_image.id
    interface    = "scsi0"
    iothread     = true
    replicate    = false
    backup       = false
    discard      = "on"
    ssd          = true
    size         = 40
  }

  cpu {
    cores = 16
    type  = "x86-64-v2-AES"
    numa = true
    sockets = 2
    units = 10240
  }

  memory {
    dedicated = 98304
  }

  cdrom {
    file_id = "none"
    interface = "ide3"
  }

  network_device {
    bridge      = "vmbr0"
    model       = "virtio"
    firewall    = true
    vlan_id     = 51
  }

   serial_device {
    device = "socket"
  }

  # qemu-guest-agent enabled
  agent {
    enabled = true 
  }

}

[bpg]

Hey @testdcpractice! 👋🏼

Since you don't have password specified in the provider config (which is good!) and have SSH agent enabled (which is correct!), your machine running tofu must have SSH Agent properly configured so the provider can establish an SSH connection to the PVE host.
See https://registry.terraform.io/providers/bpg/proxmox/latest/docs#ssh-agent for more details.

However, if you're running PVE 8.4+, you can change your config to use import content type in proxmox_virtual_environment_download_file when downloading the base image. This allows the provider to use imported disks directly via PVE API, without SSH hacks used to convert the source image to the format required by the VM.
You can find a working example here: https://github.com/bpg/terraform-provider-proxmox/tree/main/examples/guides/cloud-image/centos-qcow2

[testdcpractice]

I solved the issue. The SSH block should look like this.

provider "proxmox" {
endpoint = var.api_url
#username = var.pm_user
#password = var.pm_password
#api_token = "${var.pm_user}!${var.token_id}=${var.token_secret}"
#api_token = "opentofu@pam!provider=secret-opentofu-token"
api_token = "root@pam!root=secret-root-token"
insecure = true // necessary because of self-signed certificates

ssh {
agent = true
username = "root"
#username = "opentofu"

private_key = file("~/.ssh/root_pve/id_rsa")
#private_key = file("~/.ssh/opentofu/id_rsa")

}
}

I still couldn't create a virtual machine with the "opentofu" API token — only with the "root" API token(I didn’t update Proxmox to version 8.4+ — I’m using 8.3.3.). In fact, there's no point in creating a second "opentofu" user. For the block resource "proxmox_virtual_environment_download_file" "rocky_cloud_image" to work for the "opentofu" user, they must have the same privileges as "root" (and why create a second root when one already exists?)

[bpg]

It looks like your SSH agent is not working or not configured as expected if you need to provide a private key via the argument.

I still couldn't create a virtual machine with the "opentofu" API token — only with the "root" API token
...
For the block resource "proxmox_virtual_environment_download_file" "rocky_cloud_image" to work for the "opentofu" user, they must have the same privileges as "root" (and why create a second root when one already exists?)

There are two layers of authentication for the provider:

1. API token (or username/password) for API access
2. SSH user for SSH access

It is completely possible to use a non-root API token with limited permissions, combined with a non-root SSH user, to manage (almost) any resource in PVE.

However, it is important that the non-root user is configured properly. If you follow the steps in the docs to configure the SSH agent and grant the necessary privileges via sudo, the provider will be able to connect to the PVE host securely without exposing passwords, private keys, or root credentials. This is the setup I use in both production and test labs, and it works reliably.

Admittedly, this setup is not the most straightforward, but stronger security always comes with some extra complexity.

[testdcpractice]

I solved the issue. You need to create the authorized_keys file in the user's directory /home/opentofu/.ssh on the Proxmox server and copy the user's public SSH key into it. This should be done on each node. My mistake was that I copied the public SSH keys of the opentofu user into the /etc/pve/priv/authorized_keys file – I thought this file was shared, but it is only for the root user.

provider "proxmox" {
endpoint = var.api_url
#username = var.pm_user
#password = var.pm_password
api_token = "opentofu@pam!provider=secret-opentofu-token"
insecure = true // necessary because of self-signed certificates

ssh {
agent = true
username = "opentofu"

private_key = file("~/.ssh/opentofu/id_rsa")

}
}