Skip to main content

Proxmox Cloud Init VM Setup

Overview

Basically what we're trying to setup is a Proxmox Host that can run Ansible playbooks against to create cloud-init enabled virtual machines from a pre-made template we source from official cloud sources.

Systems / Architecture Diagram

What you can see here is the following:

  • Ansible desktop client running ansible-playbook with proxmox user api token credentials
  • a console session as [email protected] user which sources an official cloud image and create a cloud-init enabled template from it
  • the playbook successfully creates clone vm 1 and clone vm 2 from the template.

proxmox architecture

Proxmox Host Setup

Install the following for the Proxmox host to handle requests from the ansible modules proxmox and proxmox_kvm

apt install -y python-pip3 python3-dev build-essential libguestfs-tools
pip3 install virtualenv
pip3 install proxmoxer requests
Caution

tbh I don't know if proxmoxer or requests actually do anything with my setup, so ymmv.

User API Token

I've opted to use API tokens instead of my users account credentials for security reasons where you can set an expiry date and also limit permissions on the tokens. Just easier to manage and keep separate from your user.

On your proxmox host, go to Datacenter-> API Tokens:

api tokens

and create a token under your chosen user:

api tokens

here are your token id and secret, make a note as you'll need them later on in our ansible playbooks:

api tokens

tokenid

note the format of the tokenid [email protected]!ansible_pve_token:

Cloud image + Customizations

You will download and customize a cloud image on the proxmox host, that's the easiest most straight-forward way until this step gets automated.

As per ianddays blog, you need to have the following files available:

  • /root/keys/id_mykeys.pub with the public key you allow to ssh to this image
  • /root/ansible is just a file with the ansible users sudo permissions i.e. just a file with this text: ansible ALL=(ALL) NOPASSWD: ALL
machine-id

Important note on the machine-id inside the cloud image: the file /etc/machine-id has an ID in it, if you don't delete it and leave the file blank before you convert the image to template, the clones you create from the template will all have the same machine-id and be assigned the same network details.

I added in the >/etc/machine-id command making the machine-id file blank.

Run these commands on your Proxmox Host:

# download cloud-init enabled cloud image -- Ubuntu
wget https://cloud-images.ubuntu.com/impish/current/impish-server-cloudimg-amd64.img

We use virt-customize to modify the disk image in place and add our customizations without needing the image to be running.

note

Does not have to be run as root, preferrably not according to the authors of the tool.

# customize the image using `virt-customize` provided by libguestfs-tools package.
virt-customize -a impish-server-cloudimg-amd64.img --update
virt-customize -a impish-server-cloudimg-amd64.img --install qemu-guest-agent
virt-customize -a impish-server-cloudimg-amd64.img --run-command 'useradd --shell /bin/bash ansible'
virt-customize -a impish-server-cloudimg-amd64.img --run-command 'mkdir -p /home/ansible/.ssh'
virt-customize -a impish-server-cloudimg-amd64.img --ssh-inject ansible:file:/root/keys/id_mykeys.pub
virt-customize -a impish-server-cloudimg-amd64.img --run-command 'chown -R ansible:ansible /home/ansible'
virt-customize -a impish-server-cloudimg-amd64.img --upload /root/ansible:/etc/sudoers.d/ansible
virt-customize -a impish-server-cloudimg-amd64.img --run-command 'chmod 0440 /etc/sudoers.d/ansible'
virt-customize -a impish-server-cloudimg-amd64.img --run-command 'chown root:root /etc/sudoers.d/ansible'
virt-customize -a impish-server-cloudimg-amd64.img --run-command '>/etc/machine-id' # important step so your clones get unique mac address / network details.

Now we create a VM in proxmox and use our customized, cloud-init enabled image as the disk image for the VM:

#create vm + customize
qm create 9000 --name "ubuntu-impish-cloudinit-template" --memory 2048 --cores 2 --net0 virtio,bridge=vmbr0
qm importdisk 9000 impish-server-cloudimg-amd64.img ZFS01
qm set 9000 --scsihw virtio-scsi-pci --scsi0 ZFS01:vm-9000-disk-0
qm set 9000 --boot c --bootdisk scsi0
qm set 9000 --ide2 ZFS01:cloudinit
qm set 9000 --serial0 socket --vga serial0
qm set 9000 --agent enabled=1

Finally, convert to template:

# convert vm to template
qm template 9000

You should now have a template in your GUI that looks like this:

api tokens

Create VMs from cloud-init template

Ansible Playbooks

I have ansible installed on my desktop and will run the playbooks on my local machine which will call my Proxmox Host/Server, and the proxmoxer module will execute the request and create our vms.

First, simple hosts file setup, in a folder called playbook/, a file named hosts:

[proxmox]
your.proxmox.host ansible_user=root

And then, a very simple playbook, let's call it playbook/create-cloud-vm.yml

---
# create-cloud-vm.yml

- name: 'Deploy our Cloud-init VMs'

hosts: proxmox

tasks:
- name: Clone cloud-init template
community.general.proxmox_kvm:
node: proxmox
vmid: 9000
clone: gemini
name: cloud-1
api_user: [email protected]
api_token_id: ansible_pve_token
api_token_secret: 1daf3b05-5f94-4f10-b924-888ba30b038b
api_host: your.proxmox.host
storage: ZFS01
timeout: 90

- name: Start VM
community.general.proxmox_kvm:
node: proxmox
name: cloud-1
api_user: [email protected]
api_token_id: ansible_pve_token
api_token_secret: 1daf3b05-5f94-4f10-b924-888ba30b038b
api_host: your.proxmox.host
state: started
SECURITY.SECURITY.SECURITY

Obviously!!

DON'T hard code your creds like this and DO look into an ansible-vault setup for fetching/injecting creds at runtime so you don't have them lying around or accidentally commit to source. Version 2 of this write-up will have a security section at a later date. These are just general notes for now.

Run Playbooks

Right, there's nothing left to it but to do it, from your local desktop ansible setup, run: ansible-playbook -i hosts playbooks/create-cloud-vm.yml

Verify

If everything was setup correctly and more importantly, that cloud init actually did its magic, you will see

  • each cloned VM has a hostname of the vm name you specified in the playbook e.g. cloud-1
  • each cloned VM has a unique IP address and a unique /etc/machine-id number.

Troubleshooting

Just some random troubleshooting notes:

Check the Proxmox API

Checking Proxmox API is working: curl -k -d "[email protected]&password=yourpassword" https://10.0.0.1:8006/api2/json/access/ticket

Proxmox: VM wont stop

  1. console into the proxmox server
  2. try qemu stop <VMID>
  3. if it says "can't lock file '/var/lock/..." do rm -f /var/lock/.... whatever that file is
  4. do qm stop <VIMID> again.

One other place to look for the lock file is /run/lock/qemu-server/*

References

Credit for various parts of the process goes to these folks, some of the docs were not easy to follow, but credit where due: