Automating Proxmox ZFS backups with pve-zsync

2
pve-zsync visualization
pve-zsync visualization

Today we have a quick how-to guide on Proxmox VE’s new feature: pve-zsync.  The pve-zsync features allows one to take automated backups of ZFS / virtual machine volumes and copy those backups to another node (more info here.) After having to recover from a major crash years ago, we now make more backups and also do VM level backups of our web hosting nodes. The reason for this is simple, recovering and moving applications when you have snapshots of virtual machines is a trivial task. Since we do use Proxmox in our infrastructure, we have been doing a few articles on the nice free hyper-converged platform. This is a guide we would suggest for anyone using two or more Proxmox hosts and ZFS on Linux for storage. Proxmox integrates ZFS on Linux fairly well so it is available to users on the newest versions out of the box and pve-zsync will allow you to replicate data across two hosts.

Getting ready for pve-zsync

In this guide we are going to be backing-up ZFS backed VMs from fmt-pve-07 to fmt-pve-01. We have some extra capacity in our Fremont colocation facility so this guide is essentially moving data from a mirrored set of NVMe drives to another set of mirrored NVMe drives in a second server. We have this guide on setting up ZFS mirrored disks in Proxmox. Here is a quick diagram of what we are going to setup:

pve-zsync visualization
pve-zsync visualization

The first step is to create a target/ destination for the ZFS send operations to land. If you have this setup already then great. If not, then you will need to do a bit of prep work on your destination server. We are going to use “zfs create -s -V 400g” to create a sparse (thin provisioned) 400GB volume for a backup target.

pve-zsync create sparse target volume
pve-zsync create sparse target volume

One can see the “fmt-01-nvme1/thinbackup2” entry in our zfs list as having only 64K used. We use sparse/ thin provisioning because we are not exactly sure how big our volume will need to be.

Automating Proxmox ZFS backups with pve-zsync

The next thing we need to do is login to our source server and create a pve-zsync job. We use the following command to install pve-zsync on the server:

apt-get install pve-zsync

We use the command:

pve-zsync create --source 100 --dest 10.0.104.201:fmt-01-nvme1/thinbackup2 --verbose --maxsnap 7

pve-zsync create recurring job
pve-zsync create recurring job

Bereaking down that command:

  • pve-zsync – the command we are using.
  • create – we are making a new recurring job.
  • –source 100 – we are going to use the volume associated with Proxmox VM number 100 and use that to snapshot and send to the other server
  • –dest 10.0.104.201:fmt-01-nvme1/thinbackup2 – tells pve-zsync to use the private (non routable) IP of our 10Gb network to send backups to the volume we created on our destination server. We could have used the server name but we instead use the static 10Gb storage network IP address for fmt-pve-01.
  • –verbose – we want to see what is going on
  • –maxsnap 7 – this tells pve-zsync the number of snapshots we want to keep. We have 7 in this example. Keep in mind the default is to snapshot every 15 minutes (we will see this later) so if you wanted a full day’s worth of 15min increment snapshots, you would want this to be 96.
  • –name STHwp1bak just a name given to the job so you can see it easily later on

One can see the command run and the initial snapshot taking place.

pve-zsync initiate creation of a recurring job
pve-zsync initiate creation of a recurring job

We have a 10Gb dedicated storage network that s also being used for Ceph and other traffic on both nodes. Still with NVMe storage we see 300MB/s speeds. If you were using 1GbE and had a larger dataset, this initial sync can take a long time. If you did not have a dedicated storage network, this would also put a lot of stress on your NICs handling VM traffic.

When we use ZFS list on the target machine after compeltion, we can see fmt-01-nvme1/thinbackup2 has increased by approximately 40.5GB in size due to the sync occuring.

pve-zsync see initial backups completed
pve-zsync see initial backups completed

If you want to change settings, you can edit /etc/cron.d/pve-zsync and edit the cron job that is taking snapshots.

pve-zsync cron job
pve-zsync cron job

One can see the pve-zsync list and pve-zsync status has both of these working. It also shows the last time that the last sync took place.

pve-zsync list time elapsed
pve-zsync list time elapsed

 

Conclusion

Overall, this is a simple guide. There are a few features we would like to see added:

  • The ability set and retain different time frames of snapshots (e.g. one monthly snapshot for 12 months, one daily snapshot for 30 days and etc.)
  • The ability to use servers without root login access. Proxmox to Proxmox works well. We can also use Proxmox to rsync.net as a backup method. Some other NAS units such as FreeNAS will be a bit harder to use with existing setups where one does not have a root password login.
  • The ability to specify multiple backup targets using the same set of snapshots of a VM.

It is a great feature and if you have spare capacity in your Proxmox network, or a backup service like rsync.net with ZFS and root access in a jail, this works very well.

2 COMMENTS

  1. Nice bit of automation.
    Have you executed a few disaster recovery scenarios?
    What is the process of reintegrating the old snapshots into the existing zfs sets, is it all or nothing?

    There is an extremely entertaining article on youpoop (teksyndicate) on an instance of recovering zfs files after a non-recoverable disaster in a small unbacked up pool that provides some interesting information on tools to use in a worst case situation.

  2. Just discovered your site and love all the Proxmox articles. I started using it many years ago in my homelab and soon afterwards I consolidated all our physical production servers onto Proxmox. Now have 5 licenses and a couple are multi-CPU.

    So far it is rock solid and I’ve had almost no issues (1 forced reboot), but backing up VMs is starting to get messy because of the size. I’m super happy to see your pve-zsync because it may be exactly what I’m looking for.

    Currently reinstalling two of my test boxes with 4.3 and ZFS so I can start testing.

LEAVE A REPLY

Please enter your comment!
Please enter your name here