How does import -d work

[fliiiix]

I don't really understand the -d option. And i never used it. But in NixOS it's used.

The docs state:

Uses device or searches for devices or files in dir. The -d option can be specified multiple times.

I would just assume it uses the disks in this dir to find pools on it to limit the search space.
If i use -d /dev/disk/by-id with is the default on NixOS the dir looks something like this:

[l33tname@telesto:~]$ ls -la /dev/disk/by-id/
total 0
drwxr-xr-x 2 root root 340 Aug 26 19:15 .
drwxr-xr-x 9 root root 180 Aug 26 19:14 ..
lrwxrwxrwx 1 root root  13 Aug 26 19:14 nvme-eui.e8238fa6bf530001001b448b4e246dab -> ../../nvme0n1
lrwxrwxrwx 1 root root  15 Aug 26 19:15 nvme-eui.e8238fa6bf530001001b448b4e246dab-part1 -> ../../nvme0n1p1
lrwxrwxrwx 1 root root  15 Aug 26 19:14 nvme-eui.e8238fa6bf530001001b448b4e246dab-part9 -> ../../nvme0n1p9
lrwxrwxrwx 1 root root  13 Aug 26 19:14 nvme-WD_Blue_SN570_2TB_23073J800884 -> ../../nvme0n1
lrwxrwxrwx 1 root root  13 Aug 26 19:14 nvme-WD_Blue_SN570_2TB_23073J800884_1 -> ../../nvme0n1
lrwxrwxrwx 1 root root  15 Aug 26 19:15 nvme-WD_Blue_SN570_2TB_23073J800884_1-part1 -> ../../nvme0n1p1
lrwxrwxrwx 1 root root  15 Aug 26 19:14 nvme-WD_Blue_SN570_2TB_23073J800884_1-part9 -> ../../nvme0n1p9
lrwxrwxrwx 1 root root  15 Aug 26 19:15 nvme-WD_Blue_SN570_2TB_23073J800884-part1 -> ../../nvme0n1p1
lrwxrwxrwx 1 root root  15 Aug 26 19:14 nvme-WD_Blue_SN570_2TB_23073J800884-part9 -> ../../nvme0n1p9

And if I check zpool import it shows my pool correctly:

(And only one pool as expected)

If I try to import this pool now with

zpool" import -d "/dev/disk/by-id" -N tank

It fails to import the pool because it is not unique.

cannot import 'tank': more than one matching pool
import by numeric ID instead

Note: import works fine with -d by-uuid where the disk shows up only once or without the -d param.

• Why does this happen? (My theory is because the same disk is showing up with different IDs in the by-id dir)
• Is this a bug?
• When should -d be used and why? (I don't really understand the usecase for this feature or what it is suppose to do)

[rincebrain]

If it says more than one matching, your problem isn't that you have multiple devices with distinct names and the same pool, it's that you have multiple pools named that, almost certainly, because by-id almost always has multiple names for the same device, and that's not a problem for most people.

No, it's not a bug, unless you show more evidence.

-d is used for explicitly naming what name to use for a device because, for example, on Linux, there are no promises made about device names being stable across hotplugging or rebooting, and if you don't use a name that's stable and your disk vanishes under you, you might be sad.

It's not just hardcoded because, among other reasons, some systems, by-id or by-partuuid or by-uuid don't work in some or all cases. One of the wiki articles goes into detail about the pros and cons of each.

[fliiiix]

zpool import -d /dev/disk/by-id/
   pool: tank
     id: 15525408456443063927
  state: ONLINE
 action: The pool can be imported using its name or numeric identifier.
 config:

	tank                                         ONLINE
	  nvme-eui.e8238fa6bf530001001b448b4e246dab  ONLINE

   pool: tank
     id: 16638860066397443734
  state: ONLINE
 action: The pool can be imported using its name or numeric identifier.
 config:

	tank                                         ONLINE
	  nvme-eui.e8238fa6bf530001001b448b4e246dab  ONLINE

Seems like you are right that there are 2 lable for the same? pool

As i can import only one

[root@telesto:/home/l33tname]# zpool status 
no pools available

[root@telesto:/home/l33tname]# zpool import 15525408456443063927 foo

[root@telesto:/home/l33tname]# zpool import 16638860066397443734 bar
cannot import '16638860066397443734': no such pool available

[root@telesto:/home/l33tname]# zpool list
NAME   SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
foo   1.81T  4.57G  1.81T        -         -     0%     0%  1.00x    ONLINE  -

[rincebrain]

You might get useful information out of asking zdb -l on the whole device and the first partition to see what it means, it's possible that what it means is something like, at the exact offset if it was using the whole disk for a pool, it found labels for one, and at the common partition layout locations, it found the other.

wipefs without the "actually delete it" flags would be similarly informative, though I don't think it'll tell you which is which without more pointers.

[fliiiix]

Seems like you are correct

$ sudo zdb -l /dev/nvme0n1
failed to unpack label 0
------------------------------------
LABEL 1
------------------------------------
    version: 5000
    name: 'tank'
    state: 1
    txg: 47
    pool_guid: 16638860066397443734
    errata: 0
    hostid: 2138265770
    hostname: 'telesto'
    top_guid: 4799150557898763025
    guid: 4799150557898763025
    vdev_children: 1
    vdev_tree:
        type: 'disk'
        id: 0
        guid: 4799150557898763025
        path: '/dev/disk/by-id/nvme-eui.e8238fa6bf530001001b448b4e246dab'
        whole_disk: 0
        metaslab_array: 64
        metaslab_shift: 34
        ashift: 9
        asize: 2000394125312
        is_log: 0
        create_txg: 4
    features_for_read:
        com.delphix:hole_birth
        com.delphix:embedded_data
    labels = 1 2 3 

and

$ sudo zdb -l /dev/nvme0n1p1
------------------------------------
LABEL 0
------------------------------------
    version: 5000
    name: 'tank'
    state: 0
    txg: 19217
    pool_guid: 15525408456443063927
    errata: 0
    hostid: 2138265770
    hostname: 'telesto'
    top_guid: 9100411320409522586
    guid: 9100411320409522586
    vdev_children: 1
    vdev_tree:
        type: 'disk'
        id: 0
        guid: 9100411320409522586
        path: '/dev/disk/by-uuid/15525408456443063927'
        whole_disk: 1
        metaslab_array: 256
        metaslab_shift: 34
        ashift: 9
        asize: 2000384688128
        is_log: 0
        DTL: 2208
        create_txg: 4
    features_for_read:
        com.delphix:hole_birth
        com.delphix:embedded_data
    labels = 0 1 2 3 

And i think the main difference it the path used.
So it seems like i fucked up somewhere in my trying things to cleanly wipe the disk and I used different paths in between tries.

Can i safely remove a pool? I assume no because i can not import both at the same time and if i would delete one the data on the 'other' pool is gone as well.

[rincebrain]

The one time I had this happen to me, I had wipefs tell me the offsets of the labels for the partition (which I didn't want to remove) and the ones for the whole disk (which I did, for the ones that weren't the whole partition pool), figured out which one was the one I did want to remove, and carefully just scribbled over that one.

wipefs will tell you the ID of the two different pools, so you should be able to distinguish which labels are which, I think.

A less hairy workaround if you don't want to mess with that, would be renaming the pool so it's unambiguous which you mean at import, and just waiting until ZFS inevitably overwrites the free space that has the old label.

A less hairy workaround that doesn't involve just waiting would be just running zpool initialize which writes a constant value to all free space on the pool.

[fliiiix]

Ok i might even just backup the data and reset the disk. Since it is just a single 2TB ssd.

Thanks for all the help and explanations. ✨

I still think zfs could be a bit more helpful in cases like this but idk in the end it looks like a weird edge case which is my own fault.