Tech Notes

Automating the Proxmox ecosystem: the first VMs from OpenTofu

How I read three hand-built VMs from one Proxmox host over the API and built them again on another host with OpenTofu, and what broke on the way.

#Linux#Homelab#Proxmox

In the previous post I built the controller: one VM with Postgres for the OpenTofu state and Semaphore for the playbooks. The post ended with one line, that the actual VMs are still to be done. This post is about them.

All the code is in the proxmox-iac repository. The long files are linked below, and QUICKSTART.md has every step in order.

What existed before

Two Proxmox hosts.

The VMs on the old one are all Debian 13, all installed by hand long before this project:

vmid  name            cores  memory  disk
 101  external-proxy  2      2 GiB   63.9 GiB
 102  claude          2      8 GiB   59 GiB
 103  jenkins         2      4 GiB   59 GiB
 104  applications    2      8 GiB   160 GiB

Every VM there has its own directory storage with one qcow2 file inside. None of them has a tag, and none of them has the guest agent.

The new one is pve-test at 192.168.0.155. On it there was only ctrl, vmid 200, the controller from the previous post.

The repository had the module pve-vm, which builds one VM from my autoinstall image, and one environment intra. In that environment the fleet was empty, fleet = {}, and a file imported.tf was empty too. The plan was to import the old machines into OpenTofu state as they are.

The task for today: take three of the old machines, external-proxy, jenkins and applications, and create them on pve-test with the same name, the same vmid and the same hardware. Fresh install from the image, not a copy of the disk.

Reading the old machines

I did not want to write the numbers from the web interface by hand, so I read them from the API. For this a read-only token is enough. token add prints the secret once, so copy it right away:

pveum user token add root@pam read-machines --privsep 1
pveum acl modify / --tokens 'root@pam!read-machines' --roles PVEAuditor

The second command is not optional. Without it the token logs in fine, and every answer is empty:

curl -sk -H "Authorization: PVEAPIToken=root@pam!read-machines=$SECRET" \
  https://$OLD_PVE:8006/api2/json/access/permissions
{"data":{}}

With the role in place, one request per VM gives the whole configuration:

SRC_TOKEN='root@pam!read-machines=<secret>'
api() { curl -sk -H "Authorization: PVEAPIToken=$SRC_TOKEN" "https://$OLD_PVE:8006/api2/json$1"; }
api /cluster/resources?type=vm
api /nodes/$OLD_NODE/qemu/101/config

For external-proxy it returned this, trimmed:

cores = 2
cpu = x86-64-v2-AES
memory = 2048
net0 = virtio=BC:24:11:34:48:F5,bridge=vmbr0,firewall=1
scsi0 = external-proxy:101/vm-101-disk-0.qcow2,discard=on,iothread=1,size=68612102553,ssd=1
scsihw = virtio-scsi-single
sockets = 1

What the module could not say

Four things from that output did not exist in the module.

The CPU type. The module had host written inside. The old machines use x86-64-v2-AES, which is what the Proxmox interface gives to a new VM. It is slower than host, but a VM with it can move to a node with a different CPU.

The balloon device.

iothread on the disk and firewall on the network card.

So the module got four new variables, and the resource now uses them. This is only the changed part, the whole resource is in pve-vm/main.tf:

cpu {
  cores = var.cores
  type  = var.cpu_type
}

memory {
  dedicated = var.memory
  floating  = var.balloon ? var.memory : 0
}

disk {
  datastore_id = var.storage_disk
  interface    = "scsi0"
  size         = var.disk
  discard      = "on"
  ssd          = var.ssd
  iothread     = var.iothread
}

network_device {
  bridge   = var.bridge
  model    = "virtio"
  firewall = var.firewall
}

floating = 0 means Proxmox balloon=0, no balloon device at all. floating equal to dedicated is what a VM from the web interface gets: the device is there, but with minimum equal to maximum it never takes memory back.

The fleet file

The environment passes the new fields through, and the defaults are the values of the old machines. The module call is in main.tf, and the full type in variables.tf:

type = map(object({
  vmid   = number
  node   = string
  ip     = string
  cores  = optional(number, 2)
  memory = optional(number, 4096)
  disk   = optional(number, 32)
  tags   = optional(list(string), [])

  ssd = optional(bool, false)

  cpu_type = optional(string, "x86-64-v2-AES")
  balloon  = optional(bool, true)
  firewall = optional(bool, true)
}))

So the file which I actually edit, fleet.auto.tfvars, stays short:

fleet = {
  external-proxy = { vmid = 101, node = "pve-test", ip = "192.168.0.158/24", cores = 2, memory = 2048, disk = 64 }
  jenkins        = { vmid = 103, node = "pve-test", ip = "192.168.0.159/24", cores = 2, memory = 4096, disk = 59 }
  applications   = { vmid = 104, node = "pve-test", ip = "192.168.0.160/24", cores = 2, memory = 8192, disk = 160 }
}

tags is empty on purpose. The tags decide which playbooks run on a machine, and I do not know yet what runs on the old ones.

The disks are on local-lvm, raw, and not in directories like on the old host. I thought about directories for some time. pve-test has one disk, the thin pool takes almost all of it, and a directory there means a thin volume, mkfs and a mount on the node. The Proxmox API cannot do any of this. Backups work the same on both, so I stayed with the thin pool.

discard stays on, so when the guest runs fstrim, the freed blocks go back to the thin pool.

The key for the state

The backend block reads the encryption from the environment variable TF_ENCRYPTION. The passphrase is kept in the same vault file as the controller passwords:

openssl rand -hex 24
# into ansible/inventory/group_vars/tag_controller.yml as tofu_state_passphrase
export TF_ENCRYPTION='key_provider "pbkdf2" "k" { passphrase = "<pass>" }
                      method "aes_gcm" "m" { keys = key_provider.pbkdf2.k }
                      state { method = method.aes_gcm.m }
                      plan { method = method.aes_gcm.m }'

The plan line encrypts saved plan files too. They contain the same secrets as the state.

The first apply failed on the network card

The write token on pve-test got the roles which I thought are enough:

pveum user token add root@pam write --privsep 1
pveum acl modify / --tokens 'root@pam!write' --roles PVEVMAdmin,PVEDatastoreUser,PVEAuditor

The plan was fine. The apply was not:

error creating VM: received an HTTP 403 response - Reason: Permission
check failed (/sdn/zones/localnetwork/vmbr0, SDN.Use)

A plain bridge like vmbr0 belongs to a zone called localnetwork, and connecting a network card to it needs SDN.Use. None of the three roles has it:

pveum acl modify /sdn/zones/localnetwork --tokens 'root@pam!write' --roles PVESDNUser

The apply waits for the install

The provider waits for the guest agent, up to fifteen minutes, and the agent starts only when the installed system boots. Three installs at once on one HDD took about 25 minutes. apply printed a timeout warning for every machine and still finished:

timeout while waiting for the QEMU agent on VM "103" to publish the network interfaces
Apply complete! Resources: 3 added, 0 changed, 0 destroyed.

SSH answered a few minutes later.

Once more from zero

A setup which worked one time does not prove much. The machines were built step by step, with fixes on the way. So I removed everything from pve-test and did it again, only with the commands from the quickstart.

OpenTofu removed the three VMs, while the state still knew about them:

tofu plan -destroy -out=destroy.tfplan
tofu apply destroy.tfplan

The controller itself is not in OpenTofu, so I removed it over the API:

curl -sk -H "Authorization: PVEAPIToken=$TOKEN" -X POST \
  "https://192.168.0.155:8006/api2/json/nodes/pve-test/qemu/200/status/stop"
curl -sk -H "Authorization: PVEAPIToken=$TOKEN" -X DELETE \
  "https://192.168.0.155:8006/api2/json/nodes/pve-test/qemu/200?purge=1&destroy-unreferenced-disks=1"

Then I created it again with pve-provision.sh, the same way as in the previous post:

./scripts/pve-provision.sh --insecure \
    --node pve-test --vmid 200 --name ctrl \
    --installer-url http://files.mirror.intra/downloads/os/autoinstall/debian-13.6.0-amd64-netinst-autoinstall.iso \
    --ciuser ansible --ssh-key ~/.ssh/id_ed25519.pub \
    --ip 192.168.0.156/24 --gw 192.168.0.1 --nameserver 192.168.0.1

Without a web server for the ISO, --installer images/debian/dist/debian-13.6.0-amd64-netinst-autoinstall.iso uploads the local build instead.

The script waits for the guest agent and returns when the install is finished. After that I set the tag on the machine and ran the playbook:

curl -sk -H "Authorization: PVEAPIToken=$TOKEN" -X PUT \
    "https://192.168.0.155:8006/api2/json/nodes/pve-test/qemu/200/config" \
    --data-urlencode "tags=controller"

make controller

The playbook took two minutes, most of it Docker and the first start of the stack. Semaphore answered on port 3000, and the tofu role could log in with the password from the vault. The database was new and empty. The state from the first round was deleted together with the old controller, and it did not matter, because the three machines were already destroyed.

For the fleet, the provider and the backend take everything from the environment:

export PROXMOX_VE_ENDPOINT=https://192.168.0.155:8006
export PROXMOX_VE_API_TOKEN='root@pam!write=<secret>'
export PROXMOX_VE_INSECURE=true
export PG_CONN_STR='postgres://tofu:<password>@192.168.0.156:5432/tofu_state?sslmode=disable'
export TF_ENCRYPTION='...'   # the block from above

cp tofu/environments/intra/local.auto.tfvars.example tofu/environments/intra/local.auto.tfvars
$EDITOR tofu/environments/intra/local.auto.tfvars   # your SSH public key

make init
make plan
git commit -am 'Add the fleet'   # apply refuses a dirty checkout
make apply

The plan showed the same three machines as the first time, only now with ssd = false:

Plan: 3 to add, 0 to change, 0 to destroy.

All three installs finished successfully. The check is one login per machine:

ssh [email protected] hostname

How it looks now

pve-test runs four VMs. ctrl is created by a script, and the other three by one file of three lines. Each of the three has the hardware of its original, a new address from 192.168.0.158 to .160, Debian 13 from my image and nothing else.

The copies have the same hardware as the originals, but they do not run anything yet, and this is the larger part of the work. In the next post I will look inside the old machines over SSH: packages, services, containers, config files, cron. What I find becomes playbooks and tags, and the new machines get the same software from git. The data, like the Jenkins jobs, is a separate question, and I will decide it for every machine separately.