lxd recover fails to import external zpool

[simondeziel]

I just lxd recover'ed images: which used a single storage pool (ZFS) that survived the loss of the root disk (LXD's config/DB). The zpool was attached to /dev/vdb but lxd recover would simply not pick it up:

root@prod-lxd-images:~# lsblk
NAME    MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
loop0     7:0    0  10.7M  1 loop /snap/canonical-livepatch/286
loop1     7:1    0  73.9M  1 loop /snap/core22/1722
loop2     7:2    0 104.8M  1 loop /snap/lxd/31214
loop3     7:3    0  44.3M  1 loop /snap/snapd/23258
vda     253:0    0    50G  0 disk 
├─vda1  253:1    0  49.9G  0 part /
├─vda14 253:14   0     4M  0 part 
└─vda15 253:15   0   106M  0 part /boot/efi
vdb     253:16   0     1T  0 disk 
├─vdb1  253:17   0  1024G  0 part 
└─vdb9  253:25   0     8M  0 part 

root@prod-lxd-images:~# lxd recover
This LXD server currently has the following storage pools:
Would you like to recover another storage pool? (yes/no) [default=no]: yes
Name of the storage pool: default
Name of the storage backend (zfs, btrfs, ceph, cephfs, cephobject, dir, lvm, powerflex): zfs
Source of the storage pool (block device, volume group, dataset, path, ... as applicable): /dev/vdb
Additional storage pool configuration property (KEY=VALUE, empty when done): 
Would you like to recover another storage pool? (yes/no) [default=no]: 
The recovery process will be scanning the following storage pools:
 - NEW: "default" (backend="zfs", source="/dev/vdb")
Would you like to continue with scanning for lost volumes? (yes/no) [default=yes]: 
Scanning for unknown volumes...
Error: Failed validation request: Failed mounting pool "default": Failed to run: zpool import -f -d /var/snap/lxd/common/lxd/disks default: exit status 1 (cannot import 'default': no such pool available)

Entering LXD's ns shows the zpool to be importable though:

root@prod-lxd-images:~# LD_LIBRARY_PATH=/snap/lxd/current/lib/:/snap/lxd/current/lib/x86_64-linux-gnu/:/snap/lxd/current/zfs-2.2/lib PATH=/snap/lxd/current/zfs-2.2/bin:/snap/lxd/current/bin:$PATH nsenter --mount=/run/snapd/ns/lxd.mnt -- bash
root@prod-lxd-images:/# zpool list
no pools available
root@prod-lxd-images:/# zpool import
   pool: default
     id: 15939611234363387516
  state: ONLINE
 action: The pool can be imported using its name or numeric identifier.
 config:

	default     ONLINE
	  vdb       ONLINE
root@prod-lxd-images:/# zpool import default
root@prod-lxd-images:/# zpool export default
root@prod-lxd-images:/# 
exit

Still unable to lxd recover it if it was exported:

root@prod-lxd-images:~# lxd recover
This LXD server currently has the following storage pools:
Would you like to recover another storage pool? (yes/no) [default=no]:   
The recovery process will be scanning the following storage pools:
Would you like to continue with scanning for lost volumes? (yes/no) [default=yes]: 
Scanning for unknown volumes...
No unknown storage pools or volumes found. Nothing to do.
root@prod-lxd-images:~# lxd recover
This LXD server currently has the following storage pools:
Would you like to recover another storage pool? (yes/no) [default=no]: yes
Name of the storage pool: default
Name of the storage backend (zfs, btrfs, ceph, cephfs, cephobject, dir, lvm, powerflex): zfs
Source of the storage pool (block device, volume group, dataset, path, ... as applicable): /dev/vdb
Additional storage pool configuration property (KEY=VALUE, empty when done): 
Would you like to recover another storage pool? (yes/no) [default=no]: 
The recovery process will be scanning the following storage pools:
 - NEW: "default" (backend="zfs", source="/dev/vdb")
Would you like to continue with scanning for lost volumes? (yes/no) [default=yes]: 
Scanning for unknown volumes...
Error: Failed validation request: Failed mounting pool "default": Failed to run: zpool import -f -d /var/snap/lxd/common/lxd/disks default: exit status 1 (cannot import 'default': no such pool available)

Manually importing it from LXD's ns allows lxd recover to work:

root@prod-lxd-images:~# LD_LIBRARY_PATH=/snap/lxd/current/lib/:/snap/lxd/current/lib/x86_64-linux-gnu/:/snap/lxd/current/zfs-2.2/lib PATH=/snap/lxd/current/zfs-2.2/bin:/snap/lxd/current/bin:$PATH nsenter --mount=/run/snapd/ns/lxd.mnt -- bash
root@prod-lxd-images:/# zpool import default
root@prod-lxd-images:/# 
exit

root@prod-lxd-images:~# lxd recover
This LXD server currently has the following storage pools:
Would you like to recover another storage pool? (yes/no) [default=no]: yes
Name of the storage pool: default
Name of the storage backend (zfs, btrfs, ceph, cephfs, cephobject, dir, lvm, powerflex): zfs
Source of the storage pool (block device, volume group, dataset, path, ... as applicable): /dev/vdb
Additional storage pool configuration property (KEY=VALUE, empty when done): 
Would you like to recover another storage pool? (yes/no) [default=no]: 
The recovery process will be scanning the following storage pools:
 - NEW: "default" (backend="zfs", source="/dev/vdb")
Would you like to continue with scanning for lost volumes? (yes/no) [default=yes]: 
Scanning for unknown volumes...
The following unknown storage pools have been found:
 - Storage pool "default" of type "zfs"
The following unknown volumes have been found:
 - Container "images" on pool "default" in project "default" (includes 3 snapshots)
 - Volume "simplestreams" on pool "default" in project "default" (includes 1 snapshots)
You are currently missing the following:
 - Network "lxdbr0" in project "default"
Please create those missing entries and then hit ENTER: 
The following unknown storage pools have been found:
 - Storage pool "default" of type "zfs"
The following unknown volumes have been found:
 - Container "images" on pool "default" in project "default" (includes 3 snapshots)
 - Volume "simplestreams" on pool "default" in project "default" (includes 1 snapshots)
Would you like those to be recovered? (yes/no) [default=no]: yes
Starting recovery...