Automating the Proxmox ecosystem: a controller VM for OpenTofu state

In the previous post I built one Debian 13 image which installs itself, and every machine after that became one API call to Proxmox. At the end of it I wrote that the next step is Ansible, Semaphore and playbooks. This is that step.

The API call works, but it is a script. It cannot tell me what exists, it cannot remove a machine which I do not need any more, and if I run it twice I get two machines. I wanted the list of my machines to be a file which I edit, and the tool to make reality match that file.

So now there are three layers. The image holds what never changes. OpenTofu owns the machine lifecycle. Ansible owns what runs inside the machine.

Why one machine should be provisioned manually

OpenTofu keeps a state file. It is the memory of what it created. On four workstations one state file in git does not work, because I will forget to push it, and then the next machine plans against a fleet which it does not know. An encrypted state file is worse, because two versions of it cannot be merged.

So the state goes into Postgres, and Postgres needs a machine. That machine is the controller. Semaphore runs on it too, on the same Postgres, in a different database.

The controller is the one machine which OpenTofu does not manage. It holds the state, so a plan which decides to replace it would destroy the database it is writing into, in the middle of the run. It is created by the script from the image repository and configured by one playbook.

Local state is the default, so OpenTofu runs before the backend exists. The playbook does not read OpenTofu state, so Ansible runs before OpenTofu ever ran. Each layer needs only the layer below it.

Preconditions

I did the whole bootstrap on a test node first, pve-test at 192.168.0.155. Five things must be true before anything works.

The API token must have privilege separation turned off. A token created in the web interface gets it on by default, and then it has no rights at all, also when its user is root@pam. Every list comes back empty and every write returns 403.

pveum user token modify root@pam Test --privsep 0

The check is one request. If it returns {} then the token can do nothing:

curl -sk -H "Authorization: PVEAPIToken=root@pam!Test=$SECRET" \
  https://192.168.0.155:8006/api2/json/access/permissions

There must be an ssh key, and an agent which holds it. The image creates one account and installs a key into it. When the account has a key, sshd turns password login off, so the key is the only way in. The tools run in a container which gets the forwarded agent socket and not my key files, so the agent has to run:

ssh-keygen -t ed25519 -C "$(whoami)@$(hostname)"
eval "$(ssh-agent)" && ssh-add ~/.ssh/id_ed25519

The node name, the storage ids and the bridge come from the node itself:

api() { curl -sk -H "Authorization: $PVE_TOKEN" "https://$PVE_HOST:8006/api2/json$1"; }
api /nodes                    # the node name
api /nodes/pve-test/storage   # local for the ISO, local-lvm for the disk
api /nodes/pve-test/network   # vmbr0

The image has to be published somewhere the node can reach. In the previous post I uploaded the ISO from my workstation. Now it sits on my mirror and the node downloads it itself.

Docker must be on the workstation.

The toolbox image

Both tools live in one image. It is a client and not a service: bind mount the repository, run one command, exit.

FROM debian:trixie-slim

ARG TOFU_VERSION=1.12.6
ARG TOFU_SHA256=5dc43da4f750f33873dc25e94587128709e819e544b7be9016b255316153c3a8
ARG ANSIBLE_CORE_VERSION=2.21.3

RUN curl -fsSLo /tmp/tofu.zip \
        "https://github.com/opentofu/opentofu/releases/download/v${TOFU_VERSION}/tofu_${TOFU_VERSION}_linux_amd64.zip" \
    && echo "${TOFU_SHA256}  /tmp/tofu.zip" | sha256sum -c - \
    && unzip -j /tmp/tofu.zip tofu -d /usr/local/bin

RUN python3 -m venv /opt/ansible \
    && /opt/ansible/bin/pip install --no-cache-dir \
        "ansible-core==${ANSIBLE_CORE_VERSION}" proxmoxer requests
ENV PATH="/opt/ansible/bin:${PATH}"

COPY requirements.yml /tmp/requirements.yml
RUN ansible-galaxy collection install -r /tmp/requirements.yml \
        -p /usr/share/ansible/collections
ENV ANSIBLE_COLLECTIONS_PATH=/usr/share/ansible/collections

ENV HOME=/work
WORKDIR /work

RUN chmod 0666 /etc/passwd
COPY entrypoint.sh /usr/local/bin/entrypoint.sh
ENTRYPOINT ["/usr/local/bin/entrypoint.sh"]
CMD ["bash"]

The container runs as my uid, so the files which it writes on the bind mount belong to me and not to root. The price of that is the last four lines, and I explain them below.

#!/bin/sh
set -e
if ! getent passwd "$(id -u)" >/dev/null 2>&1; then
	printf 'toolbox:x:%s:%s:toolbox:/work:/bin/bash\n' "$(id -u)" "$(id -g)" >>/etc/passwd
fi
exec "$@"

Credentials never go into the image and never into the repository. They arrive from the environment:

DOCKER = docker run --rm -it \
	-u "$$(id -u):$$(id -g)" \
	-v "$(CURDIR):/work" \
	$(if $(SSH_AUTH_SOCK),-v "$(SSH_AUTH_SOCK):/ssh-agent" -e SSH_AUTH_SOCK=/ssh-agent,) \
	-e PROXMOX_VE_HOST -e PROXMOX_VE_USER \
	-e PROXMOX_VE_TOKEN_ID -e PROXMOX_VE_TOKEN_SECRET \
	-e PROXMOX_VE_ENDPOINT -e PROXMOX_VE_API_TOKEN -e PROXMOX_VE_INSECURE \
	-e PG_CONN_STR -e TF_ENCRYPTION \
	-e ANSIBLE_VAULT_PASSWORD_FILE \
	-e FLEET_SUBNET_RE \
	$(IMAGE)

Creating the controller machine

The provisioning script is the one from the image repository. It got one new flag. Before, it uploaded the ISO through my workstation, which is 790 megabytes over the network for every new node. Now the node fetches the file itself:

export PVE_HOST=192.168.0.155
export PVE_TOKEN='PVEAPIToken=root@pam!Test=<secret>'

../debian-images/examples/proxmox-provision.sh --insecure \
    --node pve-test --vmid 200 --name ctrl \
    --installer-url http://files.mirror.intra/downloads/os/autoinstall/debian-13.6.0-amd64-netinst-autoinstall.iso \
    --ciuser ansible --ssh-key ~/.ssh/id_ed25519.pub \
    --ip 192.168.0.156/24 --gw 192.168.0.1 --nameserver 192.168.0.1

The user is ansible and it gets no password, only the key. The image sees an account without a password and writes /etc/sudoers.d/90-ansible with NOPASSWD:ALL.

The inventory reads Proxmox, not the state file

OpenTofu says what should exist. Proxmox says what exists. Ansible reads the second one. Nothing generated is passed between OpenTofu and Ansible, and a machine which somebody created by hand still appears in the inventory.

plugin: community.proxmox.proxmox

url: "https://{{ lookup('ansible.builtin.env', 'PROXMOX_VE_HOST') }}:8006"
user: "{{ lookup('ansible.builtin.env', 'PROXMOX_VE_USER') }}"
token_id: "{{ lookup('ansible.builtin.env', 'PROXMOX_VE_TOKEN_ID') }}"
token_secret: "{{ lookup('ansible.builtin.env', 'PROXMOX_VE_TOKEN_SECRET') }}"
validate_certs: false

want_facts: true
want_proxmox_nodes_ansible_host: false
exclude_nodes: true

filters:
  - "proxmox_status == 'running'"

keyed_groups:
  - key: proxmox_tags_parsed
    prefix: tag
    separator: "_"

compose:
  ansible_host: >-
    ((proxmox_agent_interfaces | default([]))
      | rejectattr('name', 'equalto', 'lo')
      | map(attribute='ip-addresses') | flatten
      | map('regex_replace', '/.*$', '')
      | select('match', lookup('ansible.builtin.env', 'FLEET_SUBNET_RE')
                        | default('^192\.168\.11\.', true))
      | list | first)
    | default(proxmox_name, true)

A VM with the Proxmox tag web lands in the group tag_web. The tag is the only thing which crosses from the first layer to the second one.

The address comes from the guest agent, which the image installs.

The playbook

The secrets are in a vault file which belongs to the group tag_controller:

make vault                       # ansible-vault create, four passwords
echo '<vault-password>' > .vault-pass
export ANSIBLE_VAULT_PASSWORD_FILE=/work/.vault-pass

export PROXMOX_VE_HOST=192.168.0.155 PROXMOX_VE_USER=root@pam
export PROXMOX_VE_TOKEN_ID=Test PROXMOX_VE_TOKEN_SECRET=<secret>
export FLEET_SUBNET_RE='^192\.168\.0\.'

make controller

The role writes one compose file and starts two services:

services:
  db:
    image: {{ controller_postgres_image }}
    restart: unless-stopped
    environment:
      POSTGRES_USER: {{ controller_semaphore_db_user }}
      POSTGRES_PASSWORD: {{ controller_pg_semaphore_password }}
      POSTGRES_DB: {{ controller_semaphore_db }}
    volumes:
      - pgdata:/var/lib/postgresql/data
    ports:
      - "{{ ansible_default_ipv4.address }}:{{ controller_pg_port }}:5432"
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U {{ controller_semaphore_db_user }}"]
      interval: 10s
      timeout: 5s
      retries: 10

  semaphore:
    image: {{ controller_semaphore_image }}
    restart: unless-stopped
    depends_on:
      db:
        condition: service_healthy
    environment:
      SEMAPHORE_DB_DIALECT: postgres
      SEMAPHORE_DB_HOST: db
      SEMAPHORE_DB_PORT: "5432"
      SEMAPHORE_DB_USER: {{ controller_semaphore_db_user }}
      SEMAPHORE_DB_PASS: {{ controller_pg_semaphore_password }}
      SEMAPHORE_DB: {{ controller_semaphore_db }}
      SEMAPHORE_ADMIN: {{ controller_semaphore_admin }}
      SEMAPHORE_ADMIN_EMAIL: {{ controller_semaphore_admin_email }}
      SEMAPHORE_ADMIN_PASSWORD: {{ controller_semaphore_admin_password }}
      SEMAPHORE_ACCESS_KEY_ENCRYPTION: {{ controller_semaphore_access_key_encryption }}
    volumes:
      - semaphore_data:/var/lib/semaphore
      - semaphore_config:/etc/semaphore
    ports:
      - "{{ ansible_default_ipv4.address }}:{{ controller_semaphore_port }}:3000"

volumes:
  pgdata:
  semaphore_data:
  semaphore_config:

Postgres is published on the LAN address and not on 0.0.0.0, because the workstations connect to it for the state. Semaphore does not use that port, it reaches the database over the compose network.

OpenTofu gets a separate role and a separate database from Semaphore, so one set of credentials is not both.

Five things which did not work at first

ssh does not start without a passwd entry. The container runs as my uid, and that uid is not in /etc/passwd inside the image. ssh calls getpwuid() before it does anything else and stops:

Failed to connect to the host via ssh: No user exists for uid 1000

Every playbook failed on gathering facts. The uid is different on every workstation, so the entry cannot be written into the image at build time, and this is why there is an entrypoint script.

The collection changed the shape of its data. The Proxmox inventory plugin moved out of community.general into community.proxmox. In the new one the guest agent interfaces look like this:

{"name": "ens18", "ip-addresses": ["192.168.0.156/24", "fe80::be24:11ff:fe1b:9a5b/64"]}

The old plugin returned a list of objects with ip-address and ip-address-type inside. My filter still selected on ip-address-type, matched nothing, and every host fell back to its own name. It looked like it worked, because a name is a valid value there. It stopped working only when the name did not resolve in DNS.

psql does not substitute variables in -c. The role creates the OpenTofu role with the password as a psql variable, so the value is quoted by psql and does not appear in the process list:

CREATE ROLE tofu LOGIN PASSWORD :'pw'

With -c this goes to the server as it is written, and the server answers syntax error at or near ":". psql substitutes variables only when it reads from a file or from standard input. Moving the statement to stdin fixed it, and ON_ERROR_STOP=1 had to be added with it, because psql reading from stdin prints the SQL error and still exits with zero.

Group variables follow the group, not the target. The playbook takes the controller by name on the first run, because the machine has no tags yet. The secrets are in group_vars/tag_controller.yml. Group variables are applied by group membership and not by what the playbook targets, so the host was reached and no secrets were loaded, and the run stopped on the assert which checks them. The machine needs the Proxmox tag before the playbook, not after it:

curl -sk -H "Authorization: $PVE_TOKEN" -X PUT \
    "https://192.168.0.155:8006/api2/json/nodes/pve-test/qemu/200/config" \
    --data-urlencode "tags=controller"

The token id has two different forms. The Ansible plugin builds the header itself from {user}!{token_id}={secret}, so PROXMOX_VE_TOKEN_ID must be the bare name Test. The OpenTofu provider wants the whole thing, root@pam!Test=<secret>, in PROXMOX_VE_API_TOKEN. The same token, two shapes, and the wrong one gives a 401 with no hint about which half is wrong.

Checking that the backend really works

tofu init says the backend is configured. That is not the same as OpenTofu being able to log in, create its table and write a state. When make plan fails later, these two look identical.

So I check them separately, in a schema which is not the real one:

terraform {
  backend "pg" {
    schema_name = "tofu_probe"
  }
}

resource "terraform_data" "probe" {
  input = "round-trip"
}

docker run --rm -u "$(id -u):$(id -g)" -v /tmp/pgprobe:/work \
    -e PG_CONN_STR intra-toolbox \
    sh -c 'tofu init && tofu apply -auto-approve && tofu state list'

One resource is added, terraform_data.probe is listed back, and in the database there is a states table with one row of 506 bytes. Then I destroy it and drop the schema. The probe needs no passphrase, so it can run before the real backend is initialised.

It does not check everything. It sets no encryption key, and one operation alone never competes for the advisory lock, so those two get their first real test on the first apply.

How it looks now

The controller is a VM with 4 gigabytes of memory and a 32 gigabyte disk. On it there is Postgres 17.11 with two databases and Semaphore v2.19.9. The backup timer dumps both databases every night.

The playbook is idempotent. The second run reports ok=23 changed=0, which is the number I wanted to see, because it means the file describes the machine and not a list of commands which happened to work one time.

The real result is smaller than the amount of work. One machine now exists which no script creates, and everything after it is a line in a file. My four workstations plan against the same state, and it does not matter which one I am sitting at.

The work is not finished. The machines which exist today were installed by hand, so they have to be imported into the state before OpenTofu can be trusted with them, and that is slow work with many iterations. After that the apply moves into Semaphore and the write tokens on the workstations can be removed.