Big Sur Input/output error when unmounting dataset whose mountpoint is gone

[ilovezfs]

This behavior seems specific to Big Sur as it does not occur on Catalina.

joe@joes-Mac ~ % ./bigsur-unmount-fail.sh 
+ sw_vers
ProductName:	macOS
ProductVersion:	11.3
BuildVersion:	20E232
+ zpool version
zfs-2.0.1-0
zfs-kmod-macos-2.0.0-rc6-2-g29afe03e29-dirty
+ NUMSECTORS=1280000
++ hdiutil attach -nomount ram://1280000
+ mydev='/dev/disk2          	                               	'
+ sudo zpool create foo /dev/disk2
+ sudo zfs snapshot foo@s1
+ sudo zfs create foo/bar
+ sudo zfs rollback foo@s1
+ sudo zfs unmount foo/bar
Unmount failed for /Volumes/foo/bar
Fallback umount called
umount: unmount(/Volumes/foo/bar): Input/output error
cannot unmount '/Volumes/foo/bar': unmount failed
+ sudo zfs unmount -f foo/bar
Unmount failed for /Volumes/foo/bar
Fallback umount called
umount: unmount(/Volumes/foo/bar): Input/output error
cannot unmount '/Volumes/foo/bar': unmount failed
+ sudo zfs unmount foo
Unmount failed for /Volumes/foo/bar
Fallback umount called
umount: unmount(/Volumes/foo/bar): Input/output error
cannot unmount '/Volumes/foo/bar': unmount failed
+ sudo zfs unmount -f foo
Unmount failed for /Volumes/foo/bar
Fallback umount called
umount: unmount(/Volumes/foo/bar): Input/output error
cannot unmount '/Volumes/foo/bar': unmount failed
+ sudo umount -f /Volumes/foo/bar
umount: unmount(/Volumes/foo/bar): Input/output error
+ sudo umount -f /Volumes/foo
+ sudo zfs mount -a
+ sudo zpool export foo
Unmount successful for /Volumes/foo/bar
Volume foo on disk3s1 unmounted
joe@joes-Mac ~ % 

The script is here: https://gist.github.com/ilovezfs/92978595c91e28dce0090a5484988cb5

Note that the only way to recover is to umount -f the mountpoint of the parent dataset. zfs unmount -f is ineffective (I suspect the umount fallback is not passing the -f option even zfs unmount -f was called.

On FreeBSD and OmniOS, you can zfs unmount foo/bar && zfs mount foo/bar and do not need to force unmount the parent dataset to recover. On Linux, the operating system automatically removes the mount after a few seconds if its mountpoint has gone missing, and so you just need to run zfs mount foo/bar.

[captain-haddock17]

JFYI,

I did same test to check on my MB / Catalina.
ProductName: Mac OS X
ProductVersion: 10.15.7
BuildVersion: 19H1217
zfs-macOS-2.0.1-1
zfs-kmod-2.0.1-1

Result is OK
"Unmount successful for /Volumes/foo/bar"

(no such issue as you stated for Catalina)

Kind regards
William

[captain-haddock17]

Did same check on another MB-Air 2012 with BigSur

ProductName: macOS
ProductVersion: 11.4
BuildVersion: 20F71
zfs-macOS-2.0.1-1
zfs-kmod-2.0.1-1

Indeed, same issue
$ sudo zfs unmount foo/bar

Unmount failed for /Volumes/foo/bar
umount: unmount(/Volumes/foo/bar): Input/output error
cannot unmount '/Volumes/foo/bar': unmount failed

[captain-haddock17]

On BigSur

$ diskutil list disk2 disk3

/dev/disk2 (disk image):
#: TYPE NAME SIZE IDENTIFIER
0: GUID_partition_scheme +655.4 MB disk2
1: ZFS ⁨foo⁩ 644.9 MB disk2s1
2: 6A945A3B-1DD2-11B2-99A6-080020736631 ⁨⁩ 8.4 MB disk2s9

/dev/disk3 (internal, virtual):
#: TYPE NAME SIZE IDENTIFIER
0: GUID_partition_scheme +604.0 MB disk3
1: ZFS Dataset ⁨foo⁩ 604.0 MB disk3s1

after

$ sudo diskutil unmount force  disk3s1

(should be same effect as 'umount -f /Volumes/foo', as stated by ilovezfs)

it was possible to do successfully

$ sudo zpool export  foo

Sorry , no added value in all this, just check/confirm.
William

[ilovezfs]

Thanks for confirming @captain-haddock17

One unfortunate thing is that zfs unmount -f foo will first insist on unmounting foo/bar when in this particular situation skipping unmounting foo/bar is a way out of the mess. We could probably detect that situation so at least the zfs unmount -f would work (or always try umount -f even if the child datasets failed to unmount?), but maybe it would be better to block the rollback on macOS if there are mounted file systems inside the dataset (or perhaps just the top level directory since that covers most cases) so that we avoid ending up in this territory in the first place.

I wonder what Apple is trying to achieve with the verification of the mountpoint directory's presence on Big Sur, given that the node is clearly still mounted on and in need of some way to get unmounted directly, and given that it unmounts fine when the parent is unmounted with force. It might be interesting to hunt down the change in xnu to see if there's a comment that might explain what the motivation was.

[ilovezfs]

It would also probably be helpful if at minimum sudo zfs unmount foo/bar suggested (in the error text displayed to the user) sudo umount -f /Volumes/foo (or sudo diskutil unmount force /Volumes/foo) when it hits an Input/output error. Or if we do get sudo zfs unmount -f foo to work in this situation, then the failed sudo zfs unmount foo/bar could suggest sudo zfs unmount -f foo instead of sudo umount -f /Volumes/foo.