parallel resilver on FreeBSD 13.1 is sequential

[abaumgaertner]

Hi, with FreeBSD12 resilvering in parallel was possible (at least from user perspective). In FreeBSD13 with zfs 2.1.4, resilvering seems to be sequential. Enlarging the vdev with zpool replace to enlarge the pool: raidz2 -> 2 disks in parallel per pool

The NAS (offline, mainteinance) with no disk activity other then the resilver.

It seems from progress further down, first replace / resilver blocks the additional. Since all the data is already read for resilvering a device, it would make sense to reuse the data to resilver additoinal device at the same time instead of resilvering the pool multiple times for each single disk.

zpool replace only accepts one device at a time.

Is it possible for resilver to run in parallel?

Did not want to start the commands in background with & but would not make much of a differrence most likely.

Thanks

• command sequence

scripts# zpool version
zfs-2.1.4-FreeBSD_g52bad4f23
zfs-kmod-2.1.4-FreeBSD_g52bad4f23
scripts# uname -a
FreeBSD xxxx.starfleet 13.1-RELEASE-p7 FreeBSD 13.1-RELEASE-p7 #XigmaNAS: Wed Mar 29 00:12:52 CEST 2023     root@[email protected]:/usr/obj/xigmanas/usr/src/amd64.amd64/sys/XIGMANAS-amd64 amd64
scripts# date;
zpool replace Pool_A /dev/gptid/4c7b7166-6c57-11e9-86e0-001517321a31 ada0;
date;
zpool replace Pool_A /dev/gptid/527876b7-6c57-11e9-86e0-001517321a31 ada2;
date;
zpool replace Pool_B /dev/gptid/854ba09e-6c57-11e9-86e0-001517321a31 ada1;
date;
zpool replace Pool_B /dev/gptid/8b370eb4-6c57-11e9-86e0-001517321a31 ada3;
date;
zpool status Pool_A Pool_B;
date;   Fri Apr  7 23:32:05 CEST 2023
scripts# zpool replace Pool_A /dev/gptid/4c7b7166-6c57-11e9-86e0-001517321a31 ada0;
scripts# date;
Fri Apr  7 23:32:15 CEST 2023
scripts# zpool replace Pool_A /dev/gptid/527876b7-6c57-11e9-86e0-001517321a31 ada2;
scripts# date;
Fri Apr  7 23:32:37 CEST 2023
scripts# zpool replace Pool_B /dev/gptid/854ba09e-6c57-11e9-86e0-001517321a31 ada1;
scripts# date;
Fri Apr  7 23:32:47 CEST 2023
scripts# zpool replace Pool_B /dev/gptid/8b370eb4-6c57-11e9-86e0-001517321a31 ada3;
scripts# date;
Fri Apr  7 23:33:11 CEST 2023
scripts# zpool status Pool_A Pool_B;
  pool: Pool_A
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Fri Apr  7 23:32:06 2023
        152G scanned at 2.33G/s, 10.2G issued at 160M/s, 21.0T total
        3.39G resilvered, 0.05% done, 1 days 14:10:09 to go
config:

        NAME                                            STATE     READ WRITE CKSUM
        Pool_A                                          DEGRADED     0     0     0
          raidz2-0                                      DEGRADED     0     0     0
            gptid/3cc42dae-6c57-11e9-86e0-001517321a31  ONLINE       0     0     0
            gptid/450fc446-6c57-11e9-86e0-001517321a31  ONLINE       0     0     0
            replacing-2                                 DEGRADED     0     0     0
              8552346261489016874                       UNAVAIL      0     0     0  was /dev/gptid/4c7b7166-6c57-11e9-86e0-001517321a31
              ada0                                      ONLINE       0     0     0  (resilvering)
            replacing-3                                 DEGRADED     0     0     0
              15661576951159810779                      UNAVAIL      0     0     0  was /dev/gptid/527876b7-6c57-11e9-86e0-001517321a31
              ada2                                      ONLINE       0     0     0  (resilvering)
            ada8                                        ONLINE       0     0     0
            ada10                                       ONLINE       0     0     0

errors: No known data errors

  pool: Pool_B
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Fri Apr  7 23:32:40 2023
        299G scanned at 9.66G/s, 2.12G issued at 69.9M/s, 19.5T total
        716M resilvered, 0.01% done, 3 days 09:03:48 to go
config:

        NAME                                            STATE     READ WRITE CKSUM
        Pool_B                                          DEGRADED     0     0     0
          raidz2-0                                      DEGRADED     0     0     0
            gptid/75678bbe-6c57-11e9-86e0-001517321a31  ONLINE       0     0     0
            gptid/7db38670-6c57-11e9-86e0-001517321a31  ONLINE       0     0     0
            replacing-2                                 DEGRADED     0     0     0
              16112458548400503143                      UNAVAIL      0     0     0  was /dev/gptid/854ba09e-6c57-11e9-86e0-001517321a31
              ada1                                      ONLINE       0     0     0  (resilvering)
            replacing-3                                 DEGRADED     0     0     0
              14272397202807754731                      UNAVAIL      0     0     0  was /dev/gptid/8b370eb4-6c57-11e9-86e0-001517321a31
              ada3                                      ONLINE       0     0     0  (resilvering)
            ada9                                        ONLINE       0     0     0
            ada11                                       ONLINE       0     0     0

errors: No known data errors
scripts# date;
Fri Apr  7 23:33:11 CEST 2023

• resilvering progress ...

scripts# date ; zpool status Pool_A Pool_B ;
Sun Apr  9 09:25:45 CEST 2023
  pool: Pool_A
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Fri Apr  7 23:32:06 2023
        16.8T scanned at 144M/s, 16.6T issued at 143M/s, 21.0T total
        5.54T resilvered, 79.12% done, 08:56:34 to go
config:

        NAME                                            STATE     READ WRITE CKSUM
        Pool_A                                          DEGRADED     0     0     0
          raidz2-0                                      DEGRADED     0     0     0
            gptid/3cc42dae-6c57-11e9-86e0-001517321a31  ONLINE       0     0     0
            gptid/450fc446-6c57-11e9-86e0-001517321a31  ONLINE       0     0     0
            replacing-2                                 DEGRADED     0     0     0
              8552346261489016874                       UNAVAIL      0     0     0  was /dev/gptid/4c7b7166-6c57-11e9-86e0-001517321a31
              ada0                                      ONLINE       0     0     0  (resilvering)
            replacing-3                                 DEGRADED     0     0     0
              15661576951159810779                      UNAVAIL      0     0     0  was /dev/gptid/527876b7-6c57-11e9-86e0-001517321a31
              ada2                                      ONLINE       0     0     0  (resilvering)
            ada8                                        ONLINE       0     0     0
            ada10                                       ONLINE       0     0     0

errors: No known data errors

  pool: Pool_B
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Fri Apr  7 23:32:40 2023
        17.1T scanned at 147M/s, 16.8T issued at 144M/s, 19.5T total
        5.61T resilvered, 86.35% done, 05:21:17 to go
config:

        NAME                                            STATE     READ WRITE CKSUM
        Pool_B                                          DEGRADED     0     0     0
          raidz2-0                                      DEGRADED     0     0     0
            gptid/75678bbe-6c57-11e9-86e0-001517321a31  ONLINE       0     0     0
            gptid/7db38670-6c57-11e9-86e0-001517321a31  ONLINE       0     0     0
            replacing-2                                 DEGRADED     0     0     0
              16112458548400503143                      UNAVAIL      0     0     0  was /dev/gptid/854ba09e-6c57-11e9-86e0-001517321a31
              ada1                                      ONLINE       0     0     0  (resilvering)
            replacing-3                                 DEGRADED     0     0     0
              14272397202807754731                      UNAVAIL      0     0     0  was /dev/gptid/8b370eb4-6c57-11e9-86e0-001517321a31
              ada3                                      ONLINE       0     0     0  (resilvering)
            ada9                                        ONLINE       0     0     0
            ada11                                       ONLINE       0     0     0

errors: No known data errors
scripts# date ; zpool status Pool_A Pool_B ;
Sun Apr  9 18:11:17 CEST 2023
  pool: Pool_A
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Fri Apr  7 23:32:06 2023
        21.0T scanned at 143M/s, 20.9T issued at 143M/s, 21.0T total
        6.98T resilvered, 99.68% done, 00:08:09 to go
config:

        NAME                                            STATE     READ WRITE CKSUM
        Pool_A                                          DEGRADED     0     0     0
          raidz2-0                                      DEGRADED     0     0     0
            gptid/3cc42dae-6c57-11e9-86e0-001517321a31  ONLINE       0     0     0
            gptid/450fc446-6c57-11e9-86e0-001517321a31  ONLINE       0     0     0
            replacing-2                                 DEGRADED     0     0     0
              8552346261489016874                       UNAVAIL      0     0     0  was /dev/gptid/4c7b7166-6c57-11e9-86e0-001517321a31
              ada0                                      ONLINE       0     0     0  (resilvering)
            replacing-3                                 DEGRADED     0     0     0
              15661576951159810779                      UNAVAIL      0     0     0  was /dev/gptid/527876b7-6c57-11e9-86e0-001517321a31
              ada2                                      ONLINE       0     0     0  (resilvering)
            ada8                                        ONLINE       0     0     0
            ada10                                       ONLINE       0     0     0

errors: No known data errors

  pool: Pool_B
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Sun Apr  9 14:59:34 2023
        1.82T scanned at 166M/s, 1.41T issued at 129M/s, 19.5T total
        241G resilvered, 7.25% done, 1 days 16:52:06 to go
config:

        NAME                                            STATE     READ WRITE CKSUM
        Pool_B                                          DEGRADED     0     0     0
          raidz2-0                                      DEGRADED     0     0     0
            gptid/75678bbe-6c57-11e9-86e0-001517321a31  ONLINE       0     0     0
            gptid/7db38670-6c57-11e9-86e0-001517321a31  ONLINE       0     0     0
            ada1                                        ONLINE       0     0     0
            replacing-3                                 DEGRADED     0     0     0
              14272397202807754731                      UNAVAIL      0     0     0  was /dev/gptid/8b370eb4-6c57-11e9-86e0-001517321a31
              ada3                                      ONLINE       0     0     0  (resilvering)
            ada9                                        ONLINE       0     0     0
            ada11                                       ONLINE       0     0     0

errors: No known data errors
scripts# date ; zpool status Pool_A Pool_B ;
Mon Apr 10 07:19:11 CEST 2023
  pool: Pool_A
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Sun Apr  9 18:21:58 2023
        5.83T scanned at 131M/s, 5.36T issued at 121M/s, 21.0T total
        916G resilvered, 25.54% done, 1 days 13:46:25 to go
config:

        NAME                                            STATE     READ WRITE CKSUM
        Pool_A                                          DEGRADED     0     0     0
          raidz2-0                                      DEGRADED     0     0     0
            gptid/3cc42dae-6c57-11e9-86e0-001517321a31  ONLINE       0     0     0
            gptid/450fc446-6c57-11e9-86e0-001517321a31  ONLINE       0     0     0
            ada0                                        ONLINE       0     0     0
            replacing-3                                 DEGRADED     0     0     0
              15661576951159810779                      UNAVAIL      0     0     0  was /dev/gptid/527876b7-6c57-11e9-86e0-001517321a31
              ada2                                      ONLINE       0     0     0  (resilvering)
            ada8                                        ONLINE       0     0     0
            ada10                                       ONLINE       0     0     0

errors: No known data errors

  pool: Pool_B
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Sun Apr  9 14:59:34 2023
        7.17T scanned at 128M/s, 6.63T issued at 118M/s, 19.5T total
        1.11T resilvered, 34.07% done, 1 days 07:35:25 to go
config:

        NAME                                            STATE     READ WRITE CKSUM
        Pool_B                                          DEGRADED     0     0     0
          raidz2-0                                      DEGRADED     0     0     0
            gptid/75678bbe-6c57-11e9-86e0-001517321a31  ONLINE       0     0     0
            gptid/7db38670-6c57-11e9-86e0-001517321a31  ONLINE       0     0     0
            ada1                                        ONLINE       0     0     0
            replacing-3                                 DEGRADED     0     0     0
              14272397202807754731                      UNAVAIL      0     0     0  was /dev/gptid/8b370eb4-6c57-11e9-86e0-001517321a31
              ada3                                      ONLINE       0     0     0  (resilvering)
            ada9                                        ONLINE       0     0     0
            ada11                                       ONLINE       0     0     0

errors: No known data errors
scripts#

[rincebrain]

You're hitting the resilver_defer feature, which attempts to prevent restarting if another disk faults in the middle of a resilver, but mostly surprises people who try to resilver multiple things at once. See #14505 for a discussion.

You can force it to do them all immediately with zpool resilver.