How to actually test a Proxmox backup restore

A backup job that reports success tells you a file was written. It does not tell you that you can get your VM back. The only thing that tells you that is a restore you performed.

This is the drill I run in my own lab. It takes under an hour the first time, less after that, and it does not touch the running VM.

Step 0: decide what "restored" means

Before you start, write down the pass condition. For most people it is three things: the guest boots, its main service answers, and the data inside it is from the expected point in time. If you skip this, you'll end up declaring victory the moment a VM shows a login prompt — which proves the disk image is readable and nothing more.

Write the condition per VM, not once for the whole lab. "Nextcloud boots and the last file I uploaded before the backup is present" is a test. "The restore worked" is a feeling.

Step 1: let Proxmox Backup Server verify the chunks

If you back up to Proxmox Backup Server, you already have half of an integrity check available and possibly turned off. PBS stores backups as deduplicated chunks and can re-read them, checking each against its recorded digest. That is a verify job, and it catches bit rot and a datastore that has quietly gone bad.

In the PBS web UI, under the datastore's Verify Jobs, schedule one. Two settings matter: how often it runs, and whether it re-verifies snapshots that already passed. Re-verifying everything every night is expensive on spinning disks; a common shape is to verify new snapshots promptly and re-verify older ones on a longer interval. Pick something your storage can finish, because a verify job that never completes is worse than none — it looks green in a list nobody reads.

What a verify job does not prove: that the guest inside those chunks boots, that the filesystem inside it is consistent, or that the application in it starts. It proves the bytes came back the same. That's necessary, not sufficient — which is why the next step exists.

Step 2: restore to a scratch VMID, never over the original

This is the step people skip because the obvious path in the UI is "restore," and the obvious target is the VM you're looking at. Don't. Restore into a new, unused VMID so the production guest is never at risk and both copies exist at once.

In the Proxmox VE UI: pick the storage or PBS datastore, choose the backup, hit Restore, and change the VM ID to something outside your normal range — I keep 9000–9099 free purely for this. From the shell it's the same idea:

qmrestore <storage>:backup/<archive> 9001 --storage <target-pool>

Two things to set before you power it on. First, disconnect the network — untick the NIC's "Connected" box or move it to an isolated bridge. A second machine claiming the same static IP, or a restored copy of a DHCP server, mail server or Active Directory member, does real damage on a live network. Second, expect the restored guest to keep the original's MAC address; that's another reason to keep it off the wire.

For containers the equivalent is pct restore 9001 <archive>, and the same isolation rule applies.

Step 3: boot it and check the pass condition

Start the restored guest and open its console. Watch the boot rather than tabbing away: filesystem errors, a failed mount, or a service that fails to start all show up here and all mean your backup restores a machine you'd still have to repair during an outage.

Then check the condition you wrote in step 0. Log in. Start the service if it isn't set to start on boot. Look at the actual data — the newest row in the database, the most recent file, yesterday's log line. If it's a database VM, this is where you find out whether the backup caught a consistent state or a half-written one, which is the argument for having the guest quiesce or dump its database before the backup window rather than relying on a snapshot alone.

Step 4: audit retention, not just the last backup

Restoring last night's backup proves last night works. The restore you'll actually need is often older — you discover a corruption or a bad change days later. So check what you still have.

Look at the prune settings on the datastore or job (keep-daily, keep-weekly, keep-monthly) and compare them against a simple question: how far back can I go, and does that cover how long it would realistically take me to notice a problem? If your retention is seven daily copies and it took you nine days to notice a corrupted share last time, your retention is set wrong. Then confirm the snapshot list actually matches the policy — pruning that silently stopped is a common and quiet failure.

While you're there, check the other half of 3-2-1: is any copy of this off the machine and off the site? A verified restore from the same box that dies in the fire is not a disaster recovery plan.

Step 5: destroy the scratch VM and write down what happened

Delete VMID 9001 — leaving restores lying around fills your pool and eventually confuses you at 2am. Then write two lines somewhere durable: the date, which VM you restored, how long it took, and what broke. The elapsed time is the most useful number you own. It converts "we have backups" into "we can be back in about forty minutes," which is the sentence that lets you decide what else needs work.

Put the drill on a schedule — quarterly per critical VM is a reasonable floor, and rotate which VM you pick so you're not proving the same easy one every time.

This drill is one chapter of The Homelab Ops Handbook ($29), which covers backup, DR, monitoring and patching as fill-in worksheets. The free 96-item Homelab Ops Checklist has the short version.