Tech Notes

Automating the Proxmox ecosystem: the first VMs from OpenTofu

How I read three hand-built VMs from one Proxmox host over the API and built them again on another host with OpenTofu, and what broke on the way.

#Linux#Homelab#Proxmox

In the previous post I built the controller: one VM with Postgres for the OpenTofu state and Semaphore for the playbooks. The post ended with one line, that the actual VMs are still to be done. This post is about them.

What existed before

Two Proxmox hosts.

The old one is 192.168.11.8, its node is called host. Five VMs run there, all Debian 13, all installed by hand long before this project:

vmid  name            cores  memory  disk
 100  pw              2      2 GiB   59.2 GiB
 101  external-proxy  2      2 GiB   63.9 GiB
 102  claude          2      8 GiB   59 GiB
 103  jenkins         2      4 GiB   59 GiB
 104  applications    2      8 GiB   160 GiB

Every VM there has its own directory storage with one qcow2 file inside. None of them has a tag, and none of them has the guest agent.

The new one is pve-test at 192.168.0.155. On it there was only ctrl, vmid 200, the controller from the previous post.

The repository had the module pve-vm, which builds one VM from my autoinstall image, and one environment intra. In that environment the fleet was empty, fleet = {}, and a file imported.tf was empty too. The plan was to import the five old machines into OpenTofu state as they are. The vault file with the controller passwords existed only on my workstation and was not in git. There was no key for state encryption anywhere.

The task for today: take three of the old machines, external-proxy, jenkins and applications, and create them on pve-test with the same name, the same vmid and the same hardware. Fresh install from the image, not a copy of the disk. claude is my workstation, and pw I do not want there, so both stay.

Reading the old machines

I did not want to write the numbers from the web interface by hand, so I read them from the API. For this a read-only token is enough:

pveum user token add root@pam read-machines --privsep 1
pveum acl modify / --tokens 'root@pam!read-machines' --roles PVEAuditor

The second command is not optional. Without it the token logs in fine, and every answer is empty:

curl -sk -H "Authorization: PVEAPIToken=root@pam!read-machines=$SECRET" \
  https://192.168.11.8:8006/api2/json/access/permissions
{"data":{}}

With the role in place, one request per VM gives the whole configuration:

api() { curl -sk -H "Authorization: PVEAPIToken=$SRC_TOKEN" "https://192.168.11.8:8006/api2/json$1"; }
api /cluster/resources?type=vm
api /nodes/host/qemu/101/config

For external-proxy it returned this, trimmed:

cores = 2
cpu = x86-64-v2-AES
memory = 2048
net0 = virtio=BC:24:11:34:48:F5,bridge=vmbr0,firewall=1
scsi0 = external-proxy:101/vm-101-disk-0.qcow2,discard=on,iothread=1,size=68612102553,ssd=1
scsihw = virtio-scsi-single
sockets = 1

The disk size is in bytes here. 68612102553 bytes is 63.9 GiB, and my module takes whole gigabytes, so it became 64. Rounding up is safe. Rounding down could give a disk which is too small for what will go on it later.

What the module could not say

Four things from that output did not exist in the module.

The CPU type. The module had host written inside. The old machines use x86-64-v2-AES, which is what the Proxmox interface gives to a new VM. It is slower than host, but a VM with it can move to a node with a different CPU.

The balloon device. pw had it off, the others had it on.

iothread on the disk and firewall on the network card. Both were set on all five.

So the module got four new variables, and the resource now uses them:

cpu {
  cores = var.cores
  type  = var.cpu_type
}

memory {
  dedicated = var.memory
  floating  = var.balloon ? var.memory : 0
}

disk {
  datastore_id = var.storage_disk
  interface    = "scsi0"
  size         = var.disk
  discard      = "on"
  ssd          = var.ssd
  iothread     = var.iothread
}

network_device {
  bridge   = var.bridge
  model    = "virtio"
  firewall = var.firewall
}

floating = 0 means Proxmox balloon=0, no balloon device at all. floating equal to dedicated is what a VM from the web interface gets: the device is there, but with minimum equal to maximum it never takes memory back.

The fleet file

The environment passes the new fields through, and the defaults are the values of the old machines:

type = map(object({
  vmid   = number
  node   = string
  ip     = string
  cores  = optional(number, 2)
  memory = optional(number, 4096)
  disk   = optional(number, 32)
  tags   = optional(list(string), [])

  ssd = optional(bool, false)

  cpu_type = optional(string, "x86-64-v2-AES")
  balloon  = optional(bool, true)
  firewall = optional(bool, true)
}))

So the file which I actually edit stays short:

fleet = {
  external-proxy = { vmid = 101, node = "pve-test", ip = "192.168.0.158/24", cores = 2, memory = 2048, disk = 64 }
  jenkins        = { vmid = 103, node = "pve-test", ip = "192.168.0.159/24", cores = 2, memory = 4096, disk = 59 }
  applications   = { vmid = 104, node = "pve-test", ip = "192.168.0.160/24", cores = 2, memory = 8192, disk = 160 }
}

tags is empty on purpose. The tags decide which playbooks run on a machine, and I do not know yet what runs on the old ones.

The disks are on local-lvm, raw, and not in directories like on the old host. I thought about directories for some time. pve-test has one disk, the thin pool takes almost all of it, and a directory there means a thin volume, mkfs and a mount on the node. The Proxmox API cannot do any of this, so OpenTofu cannot either. Backups work the same on both, so I stayed with the thin pool.

ssd is false, because the one disk in pve-test is a spinning Samsung HDD. discard stays on, so when the guest runs fstrim, the freed blocks go back to the thin pool.

The key for the state

The backend block reads the encryption from the environment variable TF_ENCRYPTION. Until today nobody had set it, because nothing had ever been written into the real state. Now it had to exist, and it had to be kept somewhere better than my shell history. It went into the same vault file as the controller passwords:

openssl rand -hex 24
# into ansible/inventory/group_vars/tag_controller.yml as tofu_state_passphrase
export TF_ENCRYPTION='key_provider "pbkdf2" "k" { passphrase = "<pass>" }
                      method "aes_gcm" "m" { keys = key_provider.pbkdf2.k }
                      state { method = method.aes_gcm.m }
                      plan { method = method.aes_gcm.m }'

The plan line encrypts saved plan files too. They contain the same secrets as the state.

After this the vault file was committed. It is encrypted, and if it lives on one laptop only then the state lives on one laptop only too. The vault password itself stays outside of git.

One small thing about reading from the vault. The first time I took the Postgres password out of it with awk, and the login failed with a password which looked correct. It had one character more than the real one. Now it is parsed as YAML:

ansible-vault view --vault-password-file .vault-pass ansible/inventory/group_vars/tag_controller.yml |
  python3 -c "import sys, yaml, urllib.parse
print(urllib.parse.quote(yaml.safe_load(sys.stdin)['controller_pg_tofu_password'], safe=''), end='')"

quote is there because the password goes into a URL, in PG_CONN_STR.

The first apply failed on the network card

The write token on pve-test got the roles which I thought are enough:

pveum acl modify / --tokens 'root@pam!write' --roles PVEVMAdmin,PVEDatastoreUser,PVEAuditor

The plan was fine. The apply was not:

error creating VM: received an HTTP 403 response - Reason: Permission
check failed (/sdn/zones/localnetwork/vmbr0, SDN.Use)

I have no SDN configured on this node. It does not matter. In Proxmox 9.2, which runs here, a plain bridge like vmbr0 belongs to a zone called localnetwork, and connecting a network card to it needs SDN.Use. None of the three roles has it:

pveum acl modify /sdn/zones/localnetwork --tokens 'root@pam!write' --roles PVESDNUser

Nothing was created by the failed run, so the second run started clean.

A value which is never equal

In the plan I saw this line:

+ cdrom {
    + enabled   = false
    + file_id   = "local:iso/debian-13.6.0-amd64-netinst-autoinstall.iso"

A VM without the installer boots from an empty disk and stays there. So I wrote enabled = true into the module, and the provider printed a warning that the attribute is deprecated.

The VMs installed without problems. Then I ran the plan again, and it wanted to change all three machines, enabled = false -> true. The provider ignores that attribute: file_id alone attaches the ISO, and enabled is read back as false every time. With true in the module the plan never ends with zero changes. I removed the line.

After that:

No changes. Your infrastructure matches the configuration.

The apply waits for the install

In the README I had written that apply returns about ten minutes before Debian is installed. It is not true. The provider waits for the guest agent, up to fifteen minutes, and the agent starts only when the installed system boots. Three installs at once on one HDD took about 25 minutes. apply printed a timeout warning for every machine and still finished:

timeout while waiting for the QEMU agent on VM "103" to publish the network interfaces
Apply complete! Resources: 3 added, 0 changed, 0 destroyed.

SSH answered a few minutes later.

One environment, not two

Until today there were two environments: intra for the old host with its import plan, and pve-test for the new machines. When the copies were running, the question was simple. Why do I need to manage the old host at all? It is only the place where I read the numbers. Nothing will be imported from it.

So intra with imported.tf was removed, and pve-test took its name. The state had to move with it. The backend keeps every environment in its own Postgres schema, and there was already an empty intra schema, created by an early init and never written:

pg_dump -U semaphore -d tofu_state -n pve_test > pve_test-schema.sql

psql -U semaphore -d tofu_state -v ON_ERROR_STOP=1 \
  -c "BEGIN" \
  -c "DROP SCHEMA intra RESTRICT" \
  -c "ALTER SCHEMA pve_test RENAME TO intra" \
  -c "COMMIT"

RESTRICT makes the drop fail if something is inside. Then schema_name = "intra" in the backend block, tofu init again, and the plan showed zero changes with three resources in the state.

Everything which still pointed to the old host moved with it. The Ansible inventory now takes 192.168.0.155 and the subnet 192.168.0. by default, and fleet_network became 192.168.0.0/24.

Once more from zero

A setup which worked one time does not prove much. The machines were built step by step, with fixes on the way. So I removed everything from pve-test and did it again, only with the commands from the quickstart.

The three VMs went away through OpenTofu, while the state still knew them:

tofu plan -destroy -out=destroy.tfplan
tofu apply destroy.tfplan

Then the controller, over the API:

curl -sk -H "Authorization: PVEAPIToken=$TOKEN" -X POST \
  "https://192.168.0.155:8006/api2/json/nodes/pve-test/qemu/200/status/stop"
curl -sk -H "Authorization: PVEAPIToken=$TOKEN" -X DELETE \
  "https://192.168.0.155:8006/api2/json/nodes/pve-test/qemu/200?purge=1&destroy-unreferenced-disks=1"

I also wanted to delete the ISO, so the node would download it from the mirror again. The token could not do this, it needs Datastore.Allocate. So that path was not tested this time, the script found the file and used it.

The controller, like in the previous post:

./scripts/pve-provision.sh --insecure \
    --node pve-test --vmid 200 --name ctrl \
    --installer-url http://files.mirror.intra/downloads/os/autoinstall/debian-13.6.0-amd64-netinst-autoinstall.iso \
    --ciuser ansible --ssh-key ~/.ssh/id_ed25519.pub \
    --ip 192.168.0.156/24 --gw 192.168.0.1 --nameserver 192.168.0.1

It was ready after 503 seconds. Then the tag, then the playbook:

curl -sk -H "Authorization: PVEAPIToken=$TOKEN" -X PUT \
    "https://192.168.0.155:8006/api2/json/nodes/pve-test/qemu/200/config" \
    --data-urlencode "tags=controller"

make controller

The playbook took two minutes, most of it Docker and the first start of the stack. Semaphore answered on port 3000, and the tofu role could log in with the password from the vault. The database was new and empty. The state from the first round died with the old controller, and it did not matter, because the three machines were already destroyed.

And the fleet:

make init
make plan
make apply

The plan showed the same three machines as the first time, only now with ssd = false:

Plan: 3 to add, 0 to change, 0 to destroy.

While I write this, the three installs are still running. In the first round they took about 25 minutes together, and this round goes at the same speed on the same disk.

How it looks now

pve-test runs four VMs. ctrl is created by a script, and the other three by one file of three lines. Each of the three has the hardware of its original, a new address from 192.168.0.158 to .160, Debian 13 from my image and nothing else.

The copies have the same hardware as the originals, but they do not run anything yet, and this is the bigger half of the work. In the next post I will look inside the old machines over SSH: packages, services, containers, config files, cron. What I find becomes playbooks and tags, and the new machines get the same software from git. The data, like the Jenkins jobs, is a separate question, and I will decide it for every machine separately.