At STH, we have been using Proxmox VE as an open source hyper-converged solution for years. Underlying the project is a standard Debian installation with KVM virtualization and LXC containers. About a year ago we provided a guide to creating the Ultimate Virtualization and Container Setup (KVM, LXC, Docker) with Management GUIs. This setup allowed one to use bare metal Docker containers on Proxmox VE for maximum performance. It also allows one to utilize Portainer.io as a graphical management interface. Due to some software ecosystem changes, there is a new Docker and ZFS interaction that can cause the setup to fail upon reboot if the root pool is set as a ZFS pool. Today we have a method to fix this and get everything working as normal.
Setup Docker on Proxmox VE Using ZFS Storage
If you install Docker CE as normal, and Proxmox VE was setup with the installation on a ZFS rpool, then the system will fail to boot if you make a container using default settings. After installing Docker, we will see that it automatically selects the ZFS storage driver:
# docker info | grep Storage
Storage Driver: zfs
The issue is that this will interfere with the Debian Linux ZFS boot. Another concern is that the if you are installing on a boot device such as a SATA DOM you may want to change the Docker container storage directory to a zfs pool using the same procedure.
What we need to do now is stop Docker and change the configuration to mount to a different location. We are using “rpool/docker” in this example, however, you will likely want to move this to a different ZFS pool. If you have Proxmox VE installed on a single ZFS pool in the system, then you can use our location as a template.
# service docker stop
# rm -rf /var/lib/docker
# zfs create -o mountpoint=/var/lib/docker rpool/docker
# mkdir /etc/systemd/system/docker.service.d
The next step is that we want to create a storage-driver.conf file:
# nano /etc/systemd/system/docker.service.d/storage-driver.conf
We then want to define storage driver usage in the service:
[Service]
ExecStart=
ExecStart=/usr/bin/dockerd --storage-driver=zfs -H fd://
Once this is complete, we can restart daemons and Docker:
# systemctl daemon-reload
# service docker start
There may be more elegant ways, and there are certainly recommended changes such as moving the Docker storage directory to another ZFS pool, but this configuration will let you at least reboot the system.
The Setup Docker on Proxmox VE Using ZFS Storage Video
If you want to see the above in action, we have a video you can find here:
That should show you a bit more of the process, but the basics are all in the text here.
Final Words
Although we do not recommend this in production, if you just want a simple development system, this will work. We recommend that if you follow our original guide and use a ZFS root, that you apply this fix. Even if you are not using a ZFS rpool, we strongly suggest using this guide and changing the storage location to something other than the rpool for your Docker container storage.
The proxmox guys are too stubborn. They should also support the docker and all this mess would be over. It’s unfortunate that to setup a few docker containers on proxmox I have to crate a kvm machine and then setup the docker inside that if I don’t want a complete mess.
Nice withthe sharing of the knowledge. Would it not be easier to have a LXC dedicated to containing docker apps? Would this offer pretty much the same performance?
But still, shows yet again the power of using a solution that is, at the end of the day, just a Debian platform.
I had to remove -H fd:// to make it work for me. It might have something to do with the latest version of docker.
The following worked for me:
“`
[Service]
ExecStart=bash -c “/usr/bin/dockerd –storage-driver=zfs”
“`
Hello Everyone,
I agree with Miha, Joyents SmartOS supports Bhyve and Docker through api’s exposed to the system. We use them at work it would be as simple as exposing the api to Proxmox infrastructure. I honestly wish this would happen.
Have a great day everyone