Kernel Extension Won't Load on macOS Sequoia

[dmzimmerman]

System information

Type | Version/Name
macOS | Sequoia Developer Beta 1 (24A5264n)
ZFS Version | 2.2.3 rc1 (Apple Silicon)

Describe the problem you're observing

The kernel extension does not load on macOS Sequoia, because it can't link to an implementation of _sprintf. Here's what it says when trying to manually kextload it on the command line:

Error Domain=KMErrorDomain Code=31 "Error occurred while building a collection:
    1: One or more binaries has an error which prevented linking.  See other errors.
    2: Could not use 'org.openzfsonosx.zfs' because: Failed to bind '_sprintf' in 'org.openzfsonosx.zfs' (at offset 0x558 in __DATA_CONST, __auth_got) as could not find a kext which exports this symbol
org.openzfsonosx.zfs specific:
    1: Failed to bind '_sprintf' in 'org.openzfsonosx.zfs' (at offset 0x558 in __DATA_CONST, __auth_got) as could not find a kext which exports this symbol
" UserInfo={NSLocalizedDescription=Error occurred while building a collection:
    1: One or more binaries has an error which prevented linking.  See other errors.
    2: Could not use 'org.openzfsonosx.zfs' because: Failed to bind '_sprintf' in 'org.openzfsonosx.zfs' (at offset 0x558 in __DATA_CONST, __auth_got) as could not find a kext which exports this symbol
org.openzfsonosx.zfs specific:
    1: Failed to bind '_sprintf' in 'org.openzfsonosx.zfs' (at offset 0x558 in __DATA_CONST, __auth_got) as could not find a kext which exports this symbol
}

I can't imagine that it's going to be too hard to make the jump to Sequoia in general - they didn't announce any major changes to kernel extensions as far as I've been able to find - but clearly there are some minor issues, at least.

Describe how to reproduce the problem

Install OpenZFS on Sequoia and try to load the kernel extension.

[lundman]

Awesome, we shouldn't be calling sprintf anyway, I've been frustrated with Linux devs adding more of them.
(We should be calling snprintf)

I'll grab the beta, and roll out a build with a fix and see if other things break.

[lundman]

While I wait for VM to install, I produced:
https://openzfsonosx.org/forum/viewtopic.php?f=20&t=3896

I have not yet tested it, but all I did was change sprintf() calls into snprintf() should you feel interested in testing it. If you get a boot-loop, just run the --trigger-panic-medic

[dmzimmerman]

Excellent. Sadly, I won't get back to a machine where I can test it for a few hours, but I'll try it when I do (unless your VM reveals other problems first).

[lundman]

Hmm remotely I can only do i386 VMs (thanks apple!) but I can at least check sprintf is fixed.

[dmzimmerman]

Well, I actually got away for a couple of minutes to try it, and the kext seems to load fine. Of course I have no ZFS pools on this machine yet so I don't know what'll happen when I try to actually use one, but at least no boot loop.

[lundman]

create an empty file: mkfile -n 1G /var/tmp/pool.img and create a pool on it? I mean, if you wanted to :)

[lundman]

Assuming /var/tmp is still writable, not sure if macOS made them all readonly

[dmzimmerman]

That seems to work fine. I copied some disk images to it, mounted them, they seemed to work. That bodes well. Don't have time for any more detailed testing right now, though, I'm afraid.

[lundman]

Awesome, thanks for your report

[fskaeh]

@lundman Just for information, do you have an ETA in mind for a full release of 2.2.3? Using an rc isn't a dealbreaker by any stretch, but it is convenient to be able to count on Homebrew to pull the latest release for me.
Thanks for the good work!

[lundman]

Ah that is a fair point, I'll roll out a version, don't think there are any major issues with it.

[rottegift]

Fortunately macOS kexts have been highly forward-compatible, i.e., they work reasonably across major system upgrades. (Reasonably in that you can still use existing pools, filesystems, and zvols reliably).

Any recent macOS will tend to alert you if there are any kernel extensions disabled at boot / across an upgrade.

A panic (including, annoyingly, a watchdog panic tripped by slow I/O) at shutdown in some circumstances will also caues macOS to require you to visit System Settings > Privacy & Security to reauthorize one or more extensions.

Even for unprivileged users, "zpool version" will report the userland and kernel extension ("kmod" in that case) versions; they should match. "sysctl zfs" will also report the kernel extension version. If those work, then the zfs kernel extension is active.

When upgrading across macOS versions and keeping the same kernel extension (& userland), I tend to do:

(in one window, to watch what the kext is doing)

$ log stream --source --predicate '(sender == "zfs" OR sender == "spl" OR subsystem == "com.apple.kext" OR (sender == "logger" and message contains "eid")) AND NOT (message contains "metaslab.c:3898")' --style compact | tee -a zfs.log.out | egrep --line-buffered -v 'metaslab.c:3953:metaslab_flush|metaslab.c:2479:metaslab_load_impl|metaslab.c:2586:metaslab_unload'

And in another, things like this (most recently looked-at options preserved in cut-and-paste from shell history):

$ sudo -s
# mkfile -n 20g /tmp/pool
# zpool create -f -o ashift=12 -O compression=zstd -O checksum=skein -O normalization=formD -O casesensitivity=insensitive -O xattr=sa tstpool /tmp/pool
# zfs create -o xattr=sa -o dnodesize=auto tstpool/b
# zfs create -s -V 18g tstpool/zv
# ls -Rl /var/run/zfs/zvol ### to find the diskNN for tstpool/zv
# diskutil partitiondisk diskNN 1 gpt apfs tstapfs r
# mount|grep tstapfs
# rsync --exclude '.fseventsd/' --exclude '.Spotlight-V100/' -c -i -avh --progress --inplace /opt/local/ /Volumes/tstpool/b/
# run that again, see if they match
# repeat previous two rsyncs, changing /Volumes/tstpool/b/ to /Volumes/tstapfs/
# diskutil unmountdisk diskNN
# diskutil unmountdisk force diskNN ### if necessary: spotlight/mds can grab hold hard
# zpool export tstpool
# zpool export -f tstpool ### if necessary
# zpool list  ## should show no pools imported
# kextunload /Library/Extensions/zfs.kext # optionally, drops significant memory

More complicated testing involves fio, for example:

$ sudo fio --name=fulltest --randrepeat=0 --ioengine=posixaio --directory=/Volumes/tstpool/b  --bs=128k --iodepth=64 --readwrite=randrw --fadvise_hint=sequential --verify=crc32c --verify_backlog=100000 --verify_async=8 --group_reporting --numjobs=8 --size=2g --thread --gtod_reduce=1 --nrfiles=8 --fallocate=none --runtime=30m --time_based

varying options.

Typically I do this on an M1 mini that doesn't normally do zfs, before deploying a new kernel extension onto anything that relies on zfs working. (Likewise, if an Apple software update breaks that M1 mini, it won't hurt too much).

And of course, one can also run the full zfs test suite in the source tree (although probably not the best idea to do it on a system where one is normally running zfs).