Thank you for visiting!
My little window on internet allowing me to share several of my passions
Categories:
- Firewall
- OpenBSD
- VoidLinux
- vdcron
- Log2Table
- zig
- fapws
- FreeBSD
- VM
- ZFS
- Tunnel
- Nvim
- got
- PEKwm
- Zsh
- High Availability
- My Sysupgrade
- Nas
- VPN
- DragonflyBSD
- Alpine Linux
- Openbox
- Desktop
- Security
- yabitrot
- nmctl
- Tint2
- Project Management
- Hifi
- Alarm
Most Popular Articles:
Last Articles:
Keeping two OpenBSD CARP firewalls in sync with fwcompare
Posted on 2026-11-19 20:42:00 from Vincent in Firewall OpenBSD
Running two OpenBSD firewalls in a CARP pair gives you high availability: when the master fails, the backup takes over the shared addresses within seconds. But this protection is only real if both machines are configured the same way. A rule added to pf.conf on one box, a DHCP reservation forgotten on the other, a daemon enabled on only one side: nothing shows until the day of the failover, when the backup suddenly behaves differently. To catch these drifts early, I wrote fwcompare, a small shell script that compares a list of configuration files between both firewalls and shows a readable diff. This post explains how it works and how to configure it.
The context: two firewalls, one set of addresses
My setup has two OpenBSD machines, fw1 and fw2. Both act as pf firewall and DHCP server for several VLANs. Each VLAN has a carp(4) interface carrying the gateway address the clients use, for example 192.168.1.1. The advskew value decides who is master: 0 on fw2, 100 on fw1. DHCP leases are synchronized between both dhcpd instances with the -Y/-y options.
In such a pair, most files must be strictly identical: pf.conf, dhcpd.conf, the tables loaded by pf, the crontab. Some files are almost identical: the VLAN interface carries .11 on fw1 and .12 on fw2, the CARP advskew differs, and the physical network card is an em1 on one box and an ix1 on the other. A naive diff reports all of these expected differences, and the real problems get lost in the noise. fwcompare is built to handle both cases.
What fwcompare does
fwcompare.sh is a plain POSIX shell script. It needs only what is in the OpenBSD base system: sh, ssh, diff, sed and grep. There is nothing to install from packages.
For every file listed in its configuration file, it fetches the file from both firewalls, applies optional normalization rules, and compares the result. Each file gets one status line, IDENTICAL, DIFFERENT, MISSING or ERROR, followed by a unified diff when the file differs. A summary ends the report, and the exit code tells a script what happened: 0 when everything matches, 1 when something differs or is missing, 2 on an error.
The script works from an admin workstation as well as from one of the firewalls. When one of the configured hosts is the machine running the script (matched by its hostname, or by the keyword local), its files are read locally and only the other firewall is reached over ssh. This way, the same script and the same configuration file can be copied to both firewalls without changes.
Getting the script
The script is released under the ISC license, the same permissive license as most of OpenBSD. It and a commented sample configuration can be downloaded here:
- fwcompare.sh, the script
- fwcompare.conf.sample, a sample configuration whose header is a complete README
On OpenBSD, ftp(1) downloads them and sha256(1) checks them:
mkdir -p ~/fwcompare && cd ~/fwcompare
ftp https://example.org/files/fwcompare/fwcompare.sh \
https://example.org/files/fwcompare/fwcompare.conf.sample \
https://example.org/files/fwcompare/SHA256
sha256 -C SHA256 fwcompare.sh fwcompare.conf.sample
chmod 755 fwcompare.sh
cp fwcompare.conf.sample fwcompare.conf
The script looks for fwcompare.conf in its own directory. Use -c to point it to another file.
Preparing access
The script connects with ssh in batch mode, so authentication must work with a key and without any prompt. I use a dedicated, unprivileged account called fwcompare on both firewalls. Its authorized_keys restricts the key to the source address of the peer and disables terminals and forwarding:
from="192.168.1.11",restrict ssh-ed25519 AAAA... fwcompare
Some files, pf.conf first of all, can only be read by root. The script therefore reads files with a configurable command, which I set to doas cat. On both firewalls, /etc/doas.conf gets a matching rule. It must be nopass, since nobody is there to type a password:
permit nopass fwcompare as root cmd cat
If you prefer to limit the account to the compared files only, write one rule per file with args /etc/pf.conf and so on.
Filling in fwcompare.conf
The configuration file has two parts: global settings first, then the list of files. Lines starting with # are comments.
Global settings
The global settings tell the script which machines to compare and how to reach them:
host1 = fw1
host2 = fw2
name1 = fw1
name2 = fw2
ssh_user = fwcompare
ssh_opts = -i ~/.ssh/id_ed25519_fwcompare
read_cmd = doas cat
diff_opts = -u
strip_comments = yes
host1 and host2 are ssh host names. name1 and name2 are only labels for the report. With strip_comments = yes, comments and blank lines are removed before comparing. A comment present on only one box (or a "generated at" timestamp) is not something I want to be warned about.
A host can also be a local directory, starting with /. Files are then read from that directory. This is handy for testing: copy /etc from both firewalls into /tmp/fw1 and /tmp/fw2, point the configuration to them, and tune your rules without touching the real machines.
The list of files
Each file is a line with an absolute path. The simplest entries need nothing more:
/etc/pf.conf
/etc/dhcpd.conf
/etc/sysctl.conf
Don't forget the files that pf.conf loads itself, such as tables declared with table <name> persist file "...". If pf.conf is identical but its tables differ, the filtering still changes at failover.
Shell wildcards are allowed and are expanded on both firewalls. /etc/hostname.* catches every interface file, including one that exists on only one side, which then shows as MISSING. An explicit path is never picked up again by a wildcard. Among wildcards, the first one that matches a file wins, so list specific patterns such as /etc/hostname.carp* before the generic /etc/hostname.*.
Expected differences: sed, sed1 and sed2
Option lines placed under a path apply to that file. The most useful ones turn values that are supposed to differ into a placeholder, so the rest of the line is still compared.
sed applies a sed -E script to the file on both hosts. For the CARP interfaces, only advskew may differ:
/etc/hostname.carp*
sed = s/advskew [0-9]+/advskew <SKEW>/
sed1 applies only to the copy from the first host, and sed2 only to the copy from the second. This is how the node's own address and network card are handled on the VLAN interfaces:
/etc/hostname.vlan*
sed1 = s/^(inet [0-9]+\.[0-9]+\.[0-9]+)\.11 /\1.<NODE> /; s/vlandev em1$/vlandev <NIC>/
sed2 = s/^(inet [0-9]+\.[0-9]+\.[0-9]+)\.12 /\1.<NODE> /; s/vlandev ix1$/vlandev <NIC>/
The same technique covers lines where each firewall points to its peer. That is the case for the dhcpd synchronization target in rc.conf.local, and for an other_fw alias in /etc/hosts:
/etc/rc.conf.local
sed1 = s/^(dhcpd_flags=.*-Y )192\.168\.1\.12 /\1<PEER> /
sed2 = s/^(dhcpd_flags=.*-Y )192\.168\.1\.11 /\1<PEER> /
/etc/hosts
sed1 = s/^192\.168\.1\.12[[:space:]]+fw2 other_fw$/<PEER> <PEER_NAME> other_fw/
sed2 = s/^192\.168\.1\.11[[:space:]]+fw1 other_fw$/<PEER> <PEER_NAME> other_fw/
Keep these expressions as narrow as possible. The CARP rule above hides a different advskew, but a wrong vhid or a wrong address on the same line is still reported. If an expression stops matching after you edit the file, the difference simply appears again. The script never hides more than you asked.
Files with different names: file2
The physical interface is hostname.em1 on fw1 and hostname.ix1 on fw2. The file2 option tells the script which path to use on the second host:
/etc/hostname.em1
file2 = /etc/hostname.ix1
Lines that do not matter: ignore
Sometimes a whole line is irrelevant. My crontab has a job that only one firewall runs. ignore takes an extended regular expression, and matching lines are dropped on both sides:
/var/cron/tabs/root
ignore = /root/bin/local_task\.sh
Remember to escape the dot. To skip several lines, combine them with | in a single expression. Use ignore sparingly: unlike sed, it removes the whole line from the comparison.
The options are applied in a fixed order: comment stripping, then sed, then sed1/sed2, then ignore, and finally the diff.
Reading the report
A typical run, here with a mistake on the vhid of one CARP interface, looks like this:
Comparing fw1 (fw1, local) <-> fw2 (fw2)
IDENTICAL /etc/pf.conf
IDENTICAL /etc/dhcpd.conf
DIFFERENT /etc/hostname.carp20 (normalized: sed)
--- fw1:/etc/hostname.carp20
+++ fw2:/etc/hostname.carp20
@@ -1 +1 @@
-inet 192.168.1.1 255.255.255.0 NONE vhid 20 pass passcarp carpdev vlan20 advskew <SKEW>
+inet 192.168.1.1 255.255.255.0 NONE vhid 21 pass passcarp carpdev vlan20 advskew <SKEW>
IDENTICAL /etc/hostname.vlan20 (normalized: sed1,sed2)
IDENTICAL /etc/hostname.em1 <-> /etc/hostname.ix1
Summary: 4 identical, 1 different, 0 missing, 0 errors
The normalized: note reminds you that the diff shows the text after normalization, with its placeholders, and not the raw file. An ERROR line usually points to an ssh key or a doas rule that needs fixing. The option -q keeps only the status lines and the summary. Without diff lines, no file content appears in the output, which can matter for a file such as hostname.wg0 that holds a private key.
Running it every day from daily.local
On OpenBSD, daily(8) runs /etc/daily.local every night and mails its output to root. That makes it the natural place for this check, and I added these lines on fw1:
echo ""
echo "Configuration differences with the CARP peer:"
/root/fwcompare/fwcompare.sh -n
-n disables colours, so the mail stays clean. Since daily.local runs as root, the local files are read as root. The ssh connection still uses the dedicated fwcompare account through the ssh_user and ssh_opts settings, so root's key is never trusted by the peer. If you use doas cat and doas refuses to run for root, add a permit nopass root rule in doas.conf. Every morning, the daily mail now ends with a short report. As long as it says only IDENTICAL, both firewalls are ready to take over from each other.
Conclusion
A CARP pair protects you against hardware failure, but not against configuration drift. That drift happens slowly, one quick fix at a time, and stays invisible until the failover. fwcompare makes it visible with very little: a shell script built only on base tools, and a configuration file that tells it which files must match and which differences are expected. The time spent writing precise sed1/sed2 rules pays off quickly. Once the expected differences are normalized, any line that still shows up in the daily mail is a real problem worth looking at. Running the check from daily.local turns it into a habit, and the next failover should hold no surprises.