zpool import failing but can see all drives with zdb -l

[toddpfaff]

Sorry, I realize this initial question is going to seem vague and probably lacking crucial information but I'm not sure what information to provide yet and I don't want to just start by dumping everything from zdb output.

I'm having a problem trying to import a pool after some changes to drive device paths. I'm suprised that the zpool import is not able to find the devices since I can clearly see them all with a zdb -l run against each of the individual drives.

This is a pool comprised of 35 drives, 17 mirror vdevs, 2 drives per mirror, and 1 spare drive.

All 35 drives exist. I can, for example, run zdb -l /dev/disk/by-id/wwn*-part1 and the pool_guid is identical for each drive.

After some system changes, the path and phys_path for these drives has changed from what is shown by zdb -l. I thought that should not be a problem since I would have expected the zpool import to just scan every available drive, find the existing drives, and update the drive device paths to reflect the changes. But that's clearly not happening as I expected. Is there a way to force that to happen?

Can anyone point me to next steps? What zdb -l output would help in diagnosing this problem? What other information should I be looking at or can provide here to help you to help me?

[tonyhutter]

I would suggest looking at the zpool status -d <dir> option to say "please look for my vdev devices in this directory" if you're not using /dev (you can pass multiple -d flags to look in multiple dirs). You could also look in /proc/spl/kstat/zfs/dbgmsg after the failed import for clues to why it's not working.

[toddpfaff]

I expect you meant zpool import -d <dir> rather than status.

I've used the -d option to import many times over the years, and I've tried that in this case as well, but it's not helping, which is one thing that is puzzling me.

I've also been looking at /proc/spl/kstat/zfs/dbgmsg based on some other searching I've done. It's showing evidence of vdev.c trying to access the disks in question by their old device names and failing with open error=16.

[tonyhutter]

I expect you meant zpool import -d dir rather than status.

Oops, yes you're right

It's showing evidence of vdev.c trying to access the disks in question by their old device names and failing with open error=16.

If you could post the relevant lines from dbgmsg we could take a look.

[toddpfaff]

Ok, I'll try to get more useful information posted here. In the meantime, I'm wondering about the following...

Looking at zdb -l output for one of the disks in question, there are these three items for each disk: path, devid, and phys_path.

If I understand correctly, the path item is what can be changed via the import -d option, and that is only used for display in output such as zpool status.

I'm not sure how devid and phys_path figure in but I suspect my problem has to do with the fact that the devid has changed for my devices and there's no way to change this in the zdb.

Does this ring any bells?

The reason the devid has changed is that the drives in question were previously iscsi targets on another backend host, but they are now local drives in the host that is trying to import the pool. As I said, I can clearly see those drives on the new host with zdb, but zpool import refuses to import them.

[toddpfaff]

This is what I'm seeing in dbgmsg for each of the drives that exist but which zpool import is not able to import:

1728583771 vdev.c:162:vdev_dbgmsg(): disk vdev '/dev/disk/by-path/ip-10.10.10.101:3260-iscsi-iqn.0000-00.bar.foo:target-ps1-wwn-0x5000c500dbea09ea-lun-0-part1': open error=16 timeout=7561/1000000000

That's the old path to the drive from when it was an iscsi target on another host. I don't understand why it's trying to open that old path rather than find the drive as it exists now on this new host.

[amotin]

I don't know why ZFS does not switch to the new paths, it should if there is a newer option available, but just for a note earlier this year @ixhamza fixed a number of issue with paths not being updated in the labels after successful import, like #15737, #15816, #15817.