For FreeBSD administrators, ZFS and jails combine to make virtualization easy, fast, and secure. A FreeBSD jail is a virtual machine which can only access the resources assigned to it when it was created, so its processes have no access to the rest of the machine. ZFS is an advanced filesystem that makes it very easy to create and destroy filesystems whenever they are needed. Together, they make it a matter of moments to create a new virtual system for testing, walling off network services, or other projects.
This article, Part 1, will walk you through setting up a host FreeBSD system to be ready for jails. Part 2 will cover creating a jail to run a network service. In the terminology I’ll use here, the “host” system is the main OS, which can control and look inside its jails. The “jailed” or “guest” system can only see what resources the host has assigned to it, and cannot see outside itself.
Create a ZFS pool (if necessary)
If you already have a ZFS pool on your system and want to put your jails in it, you can skip this step. This sets up a ZFS pool named zf on one or more hard drives, in which you will then create your ZFS filesystems. Depending on how many free drives you have available, use one of these commands, substituting in the device names for your drives:
zpool create zf /dev/ada2 # one drive, no redundancy
zpool create zf mirror /dev/ada2 /dev/ada3 # two drives, mirrored
zpool create zf raidz /dev/ada2 /dev/ada3 /dev/ada4 # 3+ drives, striped
Create and mount a filesystem for your jails
We will create one filesystem called zf/jail, mount it on /usr/jail, and give it the options we want all our jails to have. Then those options will be inherited by all filesystems created beneath it:
zfs create zf/jail
zfs set mountpoint=/usr/jail zf/jail
zfs set compression=on zf/jail
You probably want to turn on compression, unless you know you’re going to be storing mostly already-compressed files in the jail. You can also turn that on and off per-jail later, so use whatever you want as the default here.
If your pool has a single drive, you may also want to use what I call “poor man’s RAID,” by telling ZFS to store two copies of every file. If the drive fails entirely, you will still lose everything, so it’s not as good as multiple drives or a replacement for regular backups. But if individual sectors fail or there are occasional bit errors, ZFS will be able to repair a file by making a new copy based on the other good sector, so you might be able to get by until you’re ready to replace it. To turn on two copies:
zfs set copies=2 zf/jail
Now create a filesystem in which to build a fresh FreeBSD install. Give it a dotfile name, because you won’t actually be using this one as a live system, so that’s an easy way to keep it separate from them in scripts:
zfs create zf/jail/.freebsd-10x64
Unpack FreeBSD into the new jail
Go to your favorite FreeBSD mirror site and fetch the distribution files matching your architecture and the release you want to use. You can get your architecture with uname -p, and see your release with uname -r (dropping any -pX patchlevel from the end). In my case, my architecture is amd64 and my release is 10.2-RELEASE, so I fetched from ftp://ftp5.us.freebsd.org/pub/FreeBSD/releases/amd64/10.2-RELEASE.
You don’t need the kernel, ports, or doc archives, so grab the other four. (You probably don’t need games either, but it’s small.) Download them into somewhere handy. I put them in /root.
Unpack them into your new jail:
cd /usr/jail/.freebsd-10x64
tar -xJvf /root/base.txz
tar -xJvf /root/lib32.txz
tar -xJvf /root/src.txz
tar -xJvf /root/games.txz
Setup the fresh install
You need to copy a few things into your new FreeBSD install and setup a few things to make it a bootable OS of its own:
cp /etc/resolv.conf /usr/jail/.freebsd-10x64/etc/ # so the jail can do DNS
Edit /root/.profile and add this line, if you aren’t already defining and exporting ENV. The reason for this will appear later:
ENV=$HOME/.shrc ; export ENV
Now we’ll chroot into the filesystem, so that the following commands will treat the jailed filesystem as if it is the root filesystem. These are setup details that would normally be handled by the installer. The last line updates the guest OS with any available updates.
chroot /usr/jail/.freebsd-10x64
passwd # (set the password for root in the jail)
mkdir /usr/ports
mkdir /usr/home
ln -s /usr/home /home
cd /etc/mail
make aliases
freebsd-update fetch install
Now edit /root/.shrc (still chrooted into the jailed filesystem) and add the following line, plus any other environment variables or aliases that you want to set when you run a shell within the jail. This will put JAIL:{hostname} in your command prompt later whenever you enter a jail as root, so you won’t get confused about whether you’re in the host or the guest. You don’t want to do a rm -rf * at some point, thinking you’re in the jail, and then realize you already exited and are wiping something out on the host.
PS1='JAIL:{\h} \$ '
Edit /etc/rc.conf and add a few lines to keep the jail from running things it doesn’t need to:
sendmail_enable="NONE"
syslogd_flags="-ss"
rpcbind_enable="NO"
Edit /etc/make.conf and add these lines. The important thing here is that we’re going to have each jail mount /usr/ports read-only from the host system, so all your jails don’t have to download and maintain their own copies of the ports tree. But since they won’t be able to write in /usr/ports, they need to download distfiles, build ports, and store packages somewhere local. If you don’t want them in /var, choose somewhere else, just not under /usr/ports.
WITH_PKGNG=yes
WRKDIRPREFIX=/var/ports
DISTDIR=/var/ports/distfiles
PACKAGES=/var/ports/packages
INDEXDIR=/usr/ports
Now run pkg once to setup the pkg directories, and ignore the error it spits out.
If there’s anything else you can think of that you want all your jails to have, go ahead and put it in place now. For instance, if you want a particular user account in every jail, create it now. When you’re ready, exit to get out of chroot and back to the full host.
Create a snapshot of this prepared FreeBSD image
Now that you have this fresh install of FreeBSD configured to your satisfaction and have exited back to the host, take a ZFS snapshot of its filesystem. You will clone this snapshot later to create individual jails. I name it “ready” to show that it is ready for cloning:
zfs snapshot zf/jail/.freebsd-10x64@ready
Setup the host to support jails
First enable jails:
echo jail_enable="YES" >>/etc/rc.conf
Now create /etc/jail.conf and add the following lines.
# file: /etc/jail.conf
# Defaults
exec.prestart = "/sbin/mount -t nullfs -o ro /usr/ports/ /usr/jail/$name/usr/ports";
exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.poststop = "/sbin/umount -f /usr/jail/$name/usr/ports";
exec.clean;
mount.devfs;
mount.fstab = "/etc/fstab.blank";
host.hostname = "$name.domain.com"; # replace 'domain.com' with your own
allow.nomount;
path = "/usr/jail/$name";
You’ll add a few lines here later when you create your first jail, but this sets up defaults for all your jails. To explain some of these lines: the jail commands replace $name in these settings with the name of a jail. The exec.prestart lines runs before the jail starts and mounts /usr/ports read-only so the jail can see it. The exec.poststop line likewise unmounts it when you stop the jail. It gives a blank fstab so the jailed OS won’t complain on boot. Set the hostname domain to whatever you like; the $name will match the jail name, which makes things easy.
Now create that empty fstab:
touch /etc/fstab.blank
Make jails!
Now your host system is ready to create all the jails you like! The first time you do all this, it may take a few hours, as you get things just the way you want them. With some experience, it can all be done in 30 minutes or so.
Coming up in Part 2: creating a jail to support a single network service.
(Hat-tip to Savagedlight, whose article on FreeBSD jails and ZFS clones was a major source of the procedure I’ve outlined here.)