Introduction

To build Jail, I've decided to use iocage.

I must first note that development of iocage is not stopped as we could guess by looking at original website: iocoage.io
Indeed, we can see on their github page that there are no more commits since November 2023. More over we can see this discussion in the issue list.

So, the new repo for iosage is: https://github.com/freebsd.iocage
Many thanks to dgeo and others to continue the development of it.

How to setup a Jail with FreeBSD

We have first to install the correct and last version of iocage

fbsd:~ # pkg install py311-iocage
fbsd:~ # iocage --version
Version 1.10

Which is the last release made by the new developers.

Then we have to activate it and create our associated zfs dataset

fbsd:~ # iocage activate zroot
fbsd:~ # zfs set mountpoint=/mnt/iocage zroot/iocage

(on my machine my zpool is called "zroot")

Then we can fetch available releases:

fbsd:~ # iocage fetch
[0] 13.3-RELEASE (EOL)
[1] 13.4-RELEASE
[2] 14.1-RELEASE
[3] 14.2-RELEASE

Type de number of the desired RELEASE
Press [Enter] to fetch the default selection: (14.1-RELEASE)
Type EXIT to quite:

Extraction: base.tgz...
Extraction: lib32.tgz...
Extraction: src.tgz...

* Updateting 14.1-RELEASE to the latest patch level...
Looking up update.FreeBSD.org mirrors... 3 mirrors found
...
Installing updates... 
Scanning /vm/iocage/releases/14.1-RELEASE/root/usr/share/certs/trusted for certificates...
Done.

Since I'm running GhostBSD 24.10, I'm running FreeBSD 14.1. This is why he proposes me this version of FreeBSD

Now we can create our Jail called "iodemo":

fbsd:~ # iocage create -n iodemo -r 14.1-RELEASE
iodemo successfully created

Before starting it, we will assign the correct network interface:

fbsd:~ # iocage set ip4_addr="wlan0|192.168.3.241" iodemo
fbsd:~ # iocage start iodemo

I'm on a laptop connected to internet via wlan0 (IP 192.168.3.24), so this command has created a kind of alias with a new IP in the same Vlan, so I do not need to play with NAT, ...

We can go into this new Jail machine thanks to the console command:

fbsd:~ # iocage console iodemo

Some remarks about Jails

Having played with the Solaris Jails in my young time, I find back some old habits ;).

I would just mentions that Jails are not VM. So, Jails are a kind of special process running on the host.
So, we can reach the machine's files via the host by just walking to the correct directory.
But the opposite is not possible !!! (This is a jail, so the name)

fbsd:~ # ls -al /vm/iocage/jails/iodemo/root/root
total 36
drwxr-x---   2 root wheel    8 May 31  2024 .
drwxr-xr-x  18 root wheel   22 Feb  9 10:01 ..
-rw-r--r--   2 root wheel 1011 May 31  2024 .cshrc
-rw-r--r--   1 root wheel   66 May 31  2024 .k5login
-rw-r--r--   1 root wheel  316 May 31  2024 .login
-rw-r--r--   2 root wheel  495 May 31  2024 .profile
-rw-r--r--   1 root wheel 1174 May 31  2024 .shrc

This is the home directory of the user root on the Jail's VM.
We can add or modify files directly from the host.

As consequences, the Jail's VM has not to deal with a VirtualIO process, the Jail VM is directly doing disk IO on the disk as any other processes. In term of performance this should be a game changer. We will see this just bellow.

In a Jail normally, there is no network virtualisation. I will not cover it, but we could use vnet and play with Vlan on Jails.

Memory allocation is made like any other application running on the host.

So, in term of performance a Jail should out perform a bhyve VM.

The main goal of Jail is to isolate this VM in a part of the disk.
We could say that a virtual machine is in a Jail, but the VM breath the same air as any neighbor of the jail.

As a down side, a Jail's VM could eat all CPU or all Memory of the host, and thus impacting other processes running on the host.
This is the benefit / risk to evaluate between a Jail's VM and bhyve VM

Let's see big is the gain of a Jail's VM in term of performance versus a bhyve VM

As said in a previous blog post, I'll mainly use bhyve VM with the disk_type="nvme".

# insert
Test Statistic: 0.0
P-Value: 1.9073486328125e-06
Reject the null hypothesis: There is a significant difference between the systems.
Nvme  has globally bigger values.
Sample size:  20
average nvme: 9.887179255485535
stdev nvme: 0.08201595318848469
average jail: 9.580141353607178
stdev jail: 0.08633756579768952
# select
Test Statistic: 0.0
P-Value: 1.9073486328125e-06
Reject the null hypothesis: There is a significant difference between the systems.
Nvme has globally bigger values.
Sample size:  20
average nvme: 2.6247466087341307
stdev nvme: 0.12543457354863238
average jail: 2.3807085871696474
stdev jail: 0.024377845319168694
# update
Test Statistic: 0.0
P-Value: 1.9073486328125e-06
Reject the null hypothesis: There is a significant difference between the systems.
Nvme has globally bigger values.
Sample size:  20
average nvme: 27.86021978855133
stdev nvme: 0.37119499183363347
average jail: 21.178079068660736
stdev jail: 0.18693284161860768
# delete
Test Statistic: 0.0
P-Value: 1.9073486328125e-06
Reject the null hypothesis: There is a significant difference between the systems.
Nvme has globally bigger values.
Sample size:  20
average nvme: 21.99110244512558
stdev nvme: 0.23639644985291663
average jail: 18.007801282405854
stdev jail: 0.22756337304402485

As expected we can see that a bhyve VM with disk_type="nvme" is slower than a Jail.
What attract my attention is the Standard Deviation when doing "updates". Indeed, "update" are the majority of actions we will perform on a real production database. And in this case, the nvme has a 2x the standard deviation of a Jail (host filesystem which is zfs in my case).
I think this is something to take into consideration.

How does a Jail could be compared to the host on which it runs ?

This is weird test, but since I have the data, let's compare this.

# insert
Test Statistic: 4.0
P-Value: 1.33514404296875e-05
Reject the null hypothesis: There is a significant difference between the systems.
GhostBSD has globally bigger values.
Sample size:  20
average GhostBSD: 9.713736152648925
stdev GhostBSD: 0.09923602826382205
average VM with nvme: 9.580141353607178
stdev VM with nvme: 0.08633756579768952
# select
Test Statistic: 45.0
P-Value: 0.023950576782226562
Reject the null hypothesis: There is a significant difference between the systems.
GhostBSD has globally bigger values.
Sample size:  20
average GhostBSD: 2.396692383289337
stdev GhostBSD: 0.01860527693325904
average VM with nvme: 2.3807085871696474
stdev VM with nvme: 0.024377845319168694
# update
Test Statistic: 57.0
P-Value: 0.0758514404296875
Fail to reject the null hypothesis: There is no significant difference between the systems.
Sample size:  20
average GhostBSD: 21.084751880168916
stdev GhostBSD: 0.15135644039599663
average VM with nvme: 21.178079068660736
stdev VM with nvme: 0.18693284161860768
# delete
Test Statistic: 2.0
P-Value: 5.7220458984375e-06
Reject the null hypothesis: There is a significant difference between the systems.
VM with nvme has globally bigger values.
Sample size:  20
average GhostBSD: 17.545048415660858
stdev GhostBSD: 0.18618484257386217
average VM with nvme: 18.007801282405854
stdev VM with nvme: 0.22756337304402485

I would expect that the test of Wilcoxon would conclude that set of values are comparable. But it does this only for the "updates".
What I observe is the very close value of Standard Deviation. This shows that, indeed, in both cases we are using the same components; there is no virtualisation components.

I have no explanations for the average differences. Maybe this a variation due to other context on the host it self.
But at least they are exceeding the 5% threshold put in the Wilcoxon test.

Conclusion

As expected, on a performance point of view Jails are better than bhyve VM. But we can see that the "cost of bhyve VM" is about 10% to 30%.
So, to my eyes this is not that big.

Thus, the choice between bhyve VM or Jail VM would mainly focus on isolation aspects.
Are we confident the VM share resources with other VMs ?, then a Jail could be a good choice.

We also see that, thanks to iocage, the setup of Jail is really simple. Many thanks to developers.

In the next tests, I plan to check what could be the results in case we have Linux in the VM instead of FreeBSD.
I'm also interested to see if we can move VM with impacts for the end-users. But, as a beginner in FreeBSD VMs world, this blog post could take long time.