Zpool spare in faulted state, but also online? Schrödinger's vdev.

[mattico]

This started about two weeks ago, on one of my pools, the shared hot spare disappeared and now one disk shows up as both ONLINE and as a FAULTED spare. Here's zpool status -P:

  pool: tank
 state: ONLINE
  scan: scrub repaired 0B in 13:02:11 with 0 errors on Mon Sep  9 13:40:48 2024
config:

        NAME                                                                   STATE     READ WRITE CKSUM
        tank                                                                   ONLINE       0     0     0
          raidz2-0                                                             ONLINE       0     0     0
            /dev/disk/by-id/ata-WDC_WUH721414ALE6L4_Y6G4X2YC-part1  (sda)      ONLINE       0     0     0
            /dev/disk/by-id/ata-WDC_WUH721414ALE6L4_Y6G3N88C-part1  (sdb)      ONLINE       0     0     0
            /dev/disk/by-id/ata-WDC_WUH721414ALE6L4_Y6G56R5C-part1  (sdd)      ONLINE       0     0     0
            /dev/disk/by-id/ata-WDC_WUH721414ALE6L4_Y6G2UNGC-part1  (sdc)      ONLINE       0     0     0
            /dev/disk/by-id/ata-WDC_WUH721414ALE6L4_Y6G3KUZC-part1  (sde)      ONLINE       0     0     0
            /dev/disk/by-id/ata-WDC_WUH721414ALE6L4_Y6G4ZJNC-part1  (sdf)      ONLINE       0     0     0
        special
          mirror-2                                                             ONLINE       0     0     0
            /dev/disk/by-id/nvme-INTEL_SSDPE21D280GA_PHM27472009E280AGN-part2  ONLINE       0     0     0
            /dev/disk/by-id/nvme-INTEL_SSDPE21D280GA_PHM2747200BU280AGN-part2  ONLINE       0     0     0
        logs
          mirror-1                                                             ONLINE       0     0     0
            /dev/disk/by-id/nvme-INTEL_SSDPE21D280GA_PHM27472009E280AGN-part1  ONLINE       0     0     0
            /dev/disk/by-id/nvme-INTEL_SSDPE21D280GA_PHM2747200BU280AGN-part1  ONLINE       0     0     0
        spares
          /dev/sde1                                                            FAULTED   corrupted data

As I annotated, sde is listed as both a FAULTED spare and as an ONLINE member of raidz2-0. It seems similar to this issue (https://utcc.utoronto.ca/~cks/space/blog/solaris/ZFSFaultedSpares) something to do with GUIDs? I looked through dmesg and syslog at the time this started and there was nothing ZFS related. My zpool events don't go back far enough to see anything relevant, it seems. I tried re-adding the missing spare, and that appeared to fix the issue briefly from the zpool status results but it went back almost immediately. Rebooting also did not help.

Looking at /proc/spl/kstat/zfs/dbgmsg, this seems relevant:

1725979605   spa.c:6109:spa_import(): spa_import: importing tank
1725979605   spa_misc.c:418:spa_load_note(): spa_load(tank, config trusted): LOADING
1725979605   vdev.c:160:vdev_dbgmsg(): disk vdev '/dev/disk/by-id/ata-WDC_WUH721414ALE6L4_Y6G4X2YC-part1': best uberblock found for spa tank. txg 2615301
1725979605   spa_misc.c:418:spa_load_note(): spa_load(tank, config untrusted): using uberblock with txg=2615301
1725979605   spa_misc.c:418:spa_load_note(): spa_load(tank, config trusted): read 20 log space maps (20 total blocks - blksz = 131072 bytes) in 2 ms
1725979605   mmp.c:240:mmp_thread_start(): MMP thread started pool 'tank' gethrtime 8919011459
1725979605   spa.c:8371:spa_async_request(): spa=tank async request task=1
1725979605   spa.c:8371:spa_async_request(): spa=tank async request task=2048
1725979605   spa_misc.c:418:spa_load_note(): spa_load(tank, config trusted): LOADED
...
1725979605   vdev.c:160:vdev_dbgmsg(): disk vdev '/dev/sde1': vdev_disk_t is NULL for VDEV:/dev/sde1

What should I do? I'm tempted to try to detach either the missing hot spare or the faulted-and-online disk.

[mattico]

It's strange that the spare shows up as sde when I've never used sdX devices in my zpools, always used the /dev/disk/by-id/ names.

I'm not able to offline or detach sde: cannot detach /dev/sde: device is reserved as a hot spare.

I replaced "sde" from raidz2-0, waited for reslivering, powered the drive off, physically removed it, and rebooted. Now zfs thinks the sde spare is in use:

  pool: tank
 state: ONLINE
  scan: scrub canceled on Wed Sep 18 10:52:52 2024
config:

        NAME                                                   STATE     READ WRITE CKSUM
        tank                                                   ONLINE       0     0     0
          raidz2-0                                             ONLINE       0     0     0
            ata-WDC_WUH721414ALE6L4_Y6G4X2YC                   ONLINE       0     0     0
            ata-WDC_WUH721414ALE6L4_Y6G3N88C                   ONLINE       0     0     0
            ata-WDC_WUH721414ALE6L4_Y5HA97NC                   ONLINE       0     0     0
            ata-WDC_WUH721414ALE6L4_Y6G2UNGC                   ONLINE       0     0     0
            ata-WDC_WUH721414ALE6L4_Y6G3KUZC                   ONLINE       0     0     0
            ata-WDC_WUH721414ALE6L4_Y6G4ZJNC                   ONLINE       0     0     0
        special
          mirror-2                                             ONLINE       0     0     0
            nvme-INTEL_SSDPE21D280GA_PHM27472009E280AGN-part2  ONLINE       0     0     0
            nvme-INTEL_SSDPE21D280GA_PHM2747200BU280AGN-part2  ONLINE       0     0     0
        logs
          mirror-1                                             ONLINE       0     0     0
            nvme-INTEL_SSDPE21D280GA_PHM27472009E280AGN-part1  ONLINE       0     0     0
            nvme-INTEL_SSDPE21D280GA_PHM2747200BU280AGN-part1  ONLINE       0     0     0
        spares
          sde                                                  INUSE     currently in use

Due to removing the disk, the sdX got moved around and now sde is ata-WDC_WUH721414ALE6L4_Y5HA97NC which is indeed in use.

[mattico]

I just did sudo zpool remove tank sde and that worked (once I reverted back to the original situation where the spare was not INUSE). Not sure why I was so afraid of doing that.