Introduction

The examples continue from the previous post: a jail named fbsd15 on the dataset rpool/vm/jls/fbsd15, with a snapshot called base-minimal. Adapt names and IP addresses to your setup.

1. What a clone really is

A snapshot is a read-only, point-in-time view of a dataset. A clone, created with zfs-clone(8), is a new writable dataset that starts out sharing every block with that snapshot.

ZFS is copy-on-write, so from then on:

• when the clone changes a file, only the new blocks are written to the clone;
• the original dataset and the snapshot are untouched;
• blocks that nobody has changed stay stored once and are shared.

That is why a clone is created instantly and costs almost nothing at first. It also means two clones of the same snapshot are logically independent: what happens in one jail never shows up in the other.

You can see the link with the origin property:

zfs list -o name,origin -r rpool/vm/jls

2. Prerequisite: a snapshot of a clean jail

Clones are made from a snapshot, not from the live dataset. Anything you changed in fbsd15 after taking the snapshot will not be in them. If needed, stop the jail and take a fresh snapshot with zfs-snapshot(8):

zfs snapshot rpool/vm/jls/fbsd15@base-minimal

3. What must be different in every clone

A clone is a byte-for-byte copy, so it also copies everything that should be unique. Besides the zfs clone itself, five things need attention:

Two optional items: change the root password in each clone (it is the one you set before the snapshot), and check /etc/hosts inside the clone in case you added the hostname there.

4. Creating four jails in one go

The script below works in both sh and zsh. Edit the names and IP addresses in the list at the bottom, and use addresses that are free on your network.

In short, we must clone the dataset of fbsd15 and define the associated jail.conf file.

The following script do those 2 actions for the 4 jails: web1 web2 db1 db2

SNAP=rpool/vm/jls/fbsd15@base-minimal

while read n ip; do
  zfs clone $SNAP rpool/vm/jls/$n
  J=/vm/jls/$n

  sysrc -R $J hostname="$n"
  rm -f $J/etc/ssh/ssh_host_*

  cat > /etc/jail.conf.d/$n.conf <<CONF
$n {
  exec.start = "/bin/sh /etc/rc";
  exec.stop = "/bin/sh /etc/rc.shutdown";
  exec.consolelog = "/var/log/jail_console_$n.log";
  exec.timeout = 60;
  stop.timeout = 30;
  persist;

  allow.raw_sockets;
  exec.clean;
  mount.devfs;
  devfs_ruleset = 4;

  host.hostname = "$n";
  path = "$J";

  ip4.addr = $ip;
  interface = igc1;
}
CONF

  sysrc jail_list+="$n"
done <<'LIST'
web1 192.168.2.11
web2 192.168.2.12
db1 192.168.2.13
db2 192.168.2.14
LIST

Start them and check:

service jail start web1 web2 db1 db2
jls

service(8) starts the jails through their rc script, and jls(8) lists the running ones. Then set a root password in each with jexec(8):

jexec web1 passwd root

5. What happens when you upgrade

Because each clone writes its own blocks, upgrades are completely independent:

• Upgrading the original fbsd15 does not change the clones. They keep the old files, still referenced through the snapshot.
• Upgrading a clone does not change the original or the other clones.
• Each jail has its own package database (/var/db/pkg is inside its dataset), so the upgrade has to be run separately for every jail.

zfs snapshot rpool/vm/jls/web1@pre-upgrade
pkg -r /vm/jls/web1 -o IGNORE_OSVERSION=yes upgrade -r FreeBSD-base
service jail restart web1

This also means a clone is not a shared template: updating fbsd15 will not update its clones. If that is what you want, you need a different design (thin jails with a shared, read-only base), which is outside the scope of this post.

As blocks diverge during upgrades, each jail uses more space of its own. That is normal.

6. The link that remains: the origin snapshot

Here is the part that surprises people. Independent behaviour does not mean independent storage. Every clone depends on its origin snapshot as long as it shares blocks with it. You can see it from both sides:

zfs list -o name,origin -r rpool/vm/jls
zfs get clones rpool/vm/jls/fbsd15@base-minimal

Each clone shows rpool/vm/jls/fbsd15@base-minimal as its origin, and the second command lists all four jails as clones of that snapshot.

The consequence, with zfs-destroy(8):

• I cannot destroy the snapshot fbsd15@base-minimal while a clone depends on it;
• I cannot destroy fbsd15 itself either, because the snapshot lives inside it;
• zfs destroy -r on fbsd15 fails for the same reason.

ZFS offers -R, which also destroys the dependent clones. Do not use it here: your four jails would be deleted with the original.

Shared blocks are only freed when the snapshot and every clone referencing them are gone, so deleting files inside a clone does not necessarily give space back.

7. Three ways to deal with the dependency

Option 1: promote one clone

zfs-promote(8) reverses the relationship between a clone and its origin. It is instant and uses no extra space.

zfs promote rpool/vm/jls/web1

The origin snapshot now belongs to web1, and fbsd15 becomes a clone of it. With nothing depending on fbsd15 any more, you can destroy it:

zfs destroy rpool/vm/jls/fbsd15

The limit of this approach: web2, web3 and web4 also depend on the snapshot, which now belongs to web1. So web1 becomes the dataset you cannot destroy. The dependency has moved, it has not disappeared.

Option 2: make real, independent copies

If you want a jail you can destroy at any time without touching the others, copy it with zfs-send(8) and zfs receive instead of cloning:

zfs send rpool/vm/jls/fbsd15@base-minimal | zfs receive rpool/vm/jls/web1

The result has no link to the original. The price is disk space: every copy is a full jail (roughly 150 to 250 MB for this minimal system), with no block sharing. For small jails that is a fair trade, but it adds up quickly with large ones. The other changes from section 3 still apply to the received copy.

Option 3: keep the original as a permanent template

This is the approach I prefer. Stop using fbsd15 as a real jail, keep it with its snapshot, and clone from it whenever you need a new jail. It costs almost nothing, because the shared blocks are stored once, and the dependency stops being a problem: the template is never meant to be destroyed. When you want a fresher template, update fbsd15, take a new snapshot (base-minimal-2) and clone from that one for new jails. Existing jails keep their older origin.

8. Quick reference

# create a clone
zfs clone rpool/vm/jls/fbsd15@base-minimal rpool/vm/jls/web1

# see dependencies
zfs list -o name,origin -r rpool/vm/jls
zfs get clones rpool/vm/jls/fbsd15@base-minimal

# reverse the dependency
zfs promote rpool/vm/jls/web1

# make an independent copy
zfs send rpool/vm/jls/fbsd15@base-minimal | zfs receive rpool/vm/jls/web1

# destroy a jail (stop it first, and unmount its devfs if it hung)
service jail stop web1
zfs destroy rpool/vm/jls/web1

Conclusion

• A clone is created instantly from a snapshot and shares its blocks, so it costs almost no space at first.
• Upgrades are independent: each jail has its own files and package database, and they must be upgraded one by one.
• Change the jail config, IP address, hostname and SSH host keys in every clone, and add it to jail_list.
• A clone stays linked to its origin snapshot. You cannot destroy the snapshot or its dataset while clones exist, and you should never reach for zfs destroy -R to get around that.
• Use zfs promote to move the dependency, zfs send | zfs receive for truly independent copies, or keep the original as a permanent template.