I wanted to setup a Raspberry Pi 4B as an lightweight internal server to handle DHCP and DNS.  The existing server was already FreeBSD so when I found support for the Pi, it seemed a natural choice.  I love ZFS and use it wherever I can, so if I can also get it running on the Pi, that will make moving the jails and running backups just that much easier.  

Yes, it won't be the most performative, but we're just looking at DHCP and DNS, so shouldn't require much.

I had a spare 250GB SATA SSD drive laying around and picked up a SATA->USB3 adaptor.  Testing showed this was at least 5 times faster than using a microSD card on the Pi.

All the FreeBSD installation instructions for the Pi just show writing (dd) the image file to the boot drive, but that will give you a ufs partition and my goal is to run zfs as the root on the SSD drive (and not need a SD card at all).

I also found a number of other people on the internet doing the same thing, but none of those insturctions were fully complete or exactly what I wanted, so I'm documenting everything I did here for my own future reference.

Most of these instructions are based off of the following reddit post, which was the best example I found:
https://www.reddit.com/r/freebsd/comments/v4b5oj/freebsd_on_zfs_on_raspberry_pi/

FreeBSD Images for the Pi found here: https://download.freebsd.org/releases/arm64/aarch64/ISO-IMAGES/14.0/ (the one that ends in -RPI.img.xz)

Step 1: Install FreeBSD on an SD card

This was just following the default instructions found here: https://wiki.freebsd.org/arm/Raspberry%20Pi

I used the Raspberry Pi Imager to write the image to the SD card from another computer.

Step 2: Boot the Pi from the SD card, then attach the USB SSD drive

The rest of the instructions are done after SSHing into the Pi and running as root.

Step 3: Prepare the SSD drive

Download the Installer Image to the Pi (sdcard):

curl -L -O https://download.freebsd.org/releases/arm64/aarch64/ISO-IMAGES/14.0/FreeBSD-14.0-RELEASE-arm64-aarch64-RPI.img.xz
unxz FreeBSD-14.0-RELEASE-arm64-aarch64-RPI.img.xz

Write the image to the target disk (in my case, da0).  This will create the partitions, a fat32 boot partition(1) and a ufs partition (2) with freebsd.  We'll blow away this 2nd partition and replace it with our zpool, but this step gets things bootable.

dd if=FreeBSD-14.0-RELEASE-arm64-aarch64-RPI.img  of=/dev/da0 bs=4M iflag=direct oflag=direct status=progress
gpart resize -i 2 da0
gpart resize -i 1 da0s2

Next, replace the ufs partition just created (da0s2) with a zpool.

zpool create -O compress=lz4 -O atime=off -o altroot=/zpialt -m none zroot da0s2

zfs create -o mountpoint=none zroot/ROOT
zfs create -o mountpoint=/ zroot/ROOT/default
zfs create -o mountpoint=/tmp -o exec=on -o setuid=off zroot/tmp
zfs create -o mountpoint=/usr -o canmount=off zroot/usr
zfs create zroot/usr/home
zfs create -o setuid=off zroot/usr/ports
zfs create zroot/usr/src
zfs create -o mountpoint=/var -o canmount=off zroot/var
zfs create -o exec=off -o setuid=off zroot/var/audit
zfs create -o exec=off -o setuid=off zroot/var/crash
zfs create -o exec=off -o setuid=off zroot/var/log
zfs create -o atime=on zroot/var/mail
zfs create -o setuid=off zroot/var/tmp

zfs set mountpoint=/zroot zroot

zpool set bootfs=zroot/ROOT/default zroot

zfs set canmount=noauto zroot/ROOT/default

Create a loopback device (md0) and mount that installer .img previously downloaded.

mdconfig -a -t vnode -f FreeBSD-14.0-RELEASE-arm64-aarch64-RPI.img -u 0
mount /dev/md0s2a /mnt

And copy the installation files into the new zfs filesystem (that's been mounted under /zpialt)

cd /mnt
tar cf - . | ( cd /zpialt && tar xvf - )

Step 4: Cleanup

Before rebooting into the new zfs root install, I made sure to make a few little changes to the install.

echo 'zfs_load="YES"' >> /zpialt/boot/loader.conf
echo 'kern.geom.label.disk_ident.enable=0' >> /zpialt/boot/loader.conf
echo 'kern.geom.label.gptid.enable=0' >> /zpialt/boot/loader.conf
echo 'net.fibs=5' >> /zpialt/boot/loader.conf

sed -i '' '/boot_serial/ s/^/#/' /zpialt/boot/loader.conf

sed -i '' /ufs/d /zpialt/etc/fstab
 

Edit the /zpialt/etc/rc.conf.  Since this Pi will be running DHCP, I need to set a static IP, but YMMV.

hostname="digitama.int.axcella.net"
ifconfig_genet0="inet 10.0.80.13 netmask 255.255.255.0"
defaultrouter="10.0.80.1"

vlans_genet0="10 11 12 13"
ifconfig_genet0_10="inet 10.0.10.4/24"
ifconfig_genet0_11="inet 10.0.11.4/24"
ifconfig_genet0_12="inet 10.0.12.4/24"
ifconfig_genet0_13="inet 10.0.13.4/24"

sshd_enable="YES"
sendmail_enable="NONE"
sendmail_submit_enable="NO"
sendmail_outbound_enable="NO"
sendmail_msp_queue_enable="NO"

ntpdate_enable="YES"
ntpdate_hosts="10.0.80.1"

# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
dumpdev="NO"
nfs_reserved_port_only="NO"
powerd_enable="YES"
zfs_enable="YES"

Add a /etc/resolv.conf:

search int.axcella.net
#nameserver 127.0.0.1
nameserver 10.0.80.1

And finally

zpool export zroot

Step 5: Reboot

Shutdown, remove the sdcard and reboot into newly install disk.