EL7 (CentOS 7.6) After Native ZFS Installation, Second Array causes grub to fail: No Device found

[Makr91]

So after following the procedures and learning the hard way to read the instructions fully, a few times. I was able to get a bootable installation using a ZFS stripe with two SSDs using the instructions. I am fairly new to ZFS forgive my ignorance. I wanted to have two Arrays, rpool for the base OS with the two aforementioned striped SSDs, and then add another array to the machine as Array-0. I would prefer this second array to be striped like the first, however mirroring it is fine as well. This is on a physical machine and not a VM. I create Array-0 with the same command line instructions used to create the rpool.

The exact error is after post: error : no such device XXXXXXXXXXXXXXXX

When I take the two drives out in the second array, the machine will boot correctly. However placing them back in, and recreating Array-0, then rebooting recreates the issue.

For the secondary array, do I need to modify some file like /etc/rc.conf as I have seen in a few searches? ZFS_enabled=true .

I have tried to create these via GPT Labels (ie /dev/sda-d etc) and by /dev/disk/by-id/

Should I try to create the arrays via the WWNXXXXXXXXXXXX?

Here is what my current /dev/disk/by-id/ looks like after booting the without the two drives:

ata-KINGSTON_SA400S37240G_50026B7682B69107
ata-KINGSTON_SA400S37240G_50026B7682B69107-part1
ata-KINGSTON_SA400S37240G_50026B7682B69107-part9
ata-KINGSTON_SA400S37240G_50026B7682B69AAD
ata-KINGSTON_SA400S37240G_50026B7682B69AAD-part1
ata-KINGSTON_SA400S37240G_50026B7682B69AAD-part9
wwn-0x50026b7682b69107
wwn-0x50026b7682b69107-part1
wwn-0x50026b7682b69107-part9
wwn-0x50026b7682b69aad
wwn-0x50026b7682b69aad-part1
wwn-0x50026b7682b69aad-part9

and /dev
sda
sda1
sda9
sdb
sdb1
sdb9

and After I add the two drives:

ata-KINGSTON_SA400S37240G_50026B7682B69107
ata-KINGSTON_SA400S37240G_50026B7682B69107-part1
ata-KINGSTON_SA400S37240G_50026B7682B69107-part9
ata-KINGSTON_SA400S37240G_50026B7682B69AAD
ata-KINGSTON_SA400S37240G_50026B7682B69AAD-part1
ata-KINGSTON_SA400S37240G_50026B7682B69AAD-part9
ata-Hitachi_HUA723030ALA640_MK0371YHJ2RLSA
ata-Hitachi_HUA723030ALA640_MK0371YHJ2RLSA-part1
ata-Hitachi_HUA723030ALA640_MK0371YHJ2RLSA-part9
ata-Hitachi_HUA723030ALA640_MK0371YHJ2AW1
ata-Hitachi_HUA723030ALA640_MK0371YHJ2AW1-part1
ata-Hitachi_HUA723030ALA640_MK0371YHJ2AW1-part9
wwn-0x50026b7682b69107
wwn-0x50026b7682b69107-part1
wwn-0x50026b7682b69107-part9
wwn-0x50026b7682b69aad
wwn-0x50026b7682b69aad-part1
wwn-0x50026b7682b69aad-part9
wwn-0x5000cca225dd4090
wwn-0x5000cca225dd4090-part1
wwn-0x5000cca225dd4090-part9
wwn-0x5000cca225dd6c9b
wwn-0x5000cca225dd6c9b-part1
wwn-0x5000cca225dd6c9b-part9
`

lsblk

NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 223.6G 0 disk
├─sda1 8:1 0 223.1G 0 part
└─sda9 8:9 0 512.6M 0 part
sdb 8:16 0 223.6G 0 disk
├─sdb1 8:17 0 223.1G 0 part
└─sdb9 8:25 0 512.6M 0 part

After:

lsblk

NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 223.6G 0 disk
├─sda1 8:1 0 223.1G 0 part
└─sda9 8:9 0 512.6M 0 part
sdb 8:16 0 223.6G 0 disk
├─sdb1 8:17 0 223.1G 0 part
└─sdb9 8:25 0 512.6M 0 part
sdc 8:32 0 2.7T 0 disk
├─sdc1 8:33 0 2.7T 0 part
└─sdc9 8:41 0 512.5M 0 part
sdd 8:48 0 2.7T 0 disk
├─sdd1 8:49 0 2.7T 0 part
└─sdd9 8:57 0 512.5M 0 part

Here is the output of zpool status and list forafter Array-0 is created:

zfs list

NAME USED AVAIL REFER MOUNTPOINT
Array-0 640K 5.27T 136K /Array-0
rpool 2.13G 428G 156K /rpool
rpool/ROOT 2.13G 428G 2.13G /rpool/ROOT

Commands used to create the pools:

rpool

zpool create -d -o feature@async_destroy=enabled -o feature@empty_bpobj=enabled -o feature@lz4_compress=enabled -o ashift=12 -O compression=lz4 rpool ata-KINGSTON_SA400S37240G_50026B7682B69107-part1 ata-KINGSTON_SA400S37240G_50026B7682B69AAD-part1

Array-0

zpool create -d -o feature@async_destroy=enabled -o feature@empty_bpobj=enabled -o feature@lz4_compress=enabled -o ashift=12 -O compression=lz4 Array-0 /dev/disk/by-id/ata-Hitachi_HUA723030ALA640_MK0371YHJ2RLSA-part1 /dev/disk/by-id/ata-Hitachi_HUA723030ALA640_MK0371YHJ2AW1D-part1

I also run these same command with the GPT labels in lieu of the id.

I install grub to all disks:
grub2-install --boot-directory=/boot /dev/disk/by-id/ata-Hitachi_HUA723030ALA640_MK0371YHJ2AW1D
grub2-install --boot-directory=/boot /dev/disk/by-id/ata-Hitachi_HUA723030ALA640_MK0371YHJ2RLSA
grub2-install --boot-directory=/boot /dev/disk/by-id/ata-KINGSTON_SA400S37240G_50026B7682B69107
grub2-install --boot-directory=/boot /dev/disk/by-id/ata-KINGSTON_SA400S37240G_50026B7682B69AAD

[Makr91]

Solved: Looks like my grub2 device map file was incorrect. I adjusted it to:

/boot/grub2/device.map
(hd0) /dev/sda
(hd1) /dev/sdb

from:
(hd0) /dev/sdc
(hd1) /dev/sdc

[Makr91]

This issue still occurs. I found that if I remove the second array before boot. Then add it after Grub has loaded it is able to boot full and the ZFS volume is mounted.