Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Encrypted ZFS performance issues #103

Open
W9CR opened this issue Oct 11, 2021 · 5 comments
Open

Encrypted ZFS performance issues #103

W9CR opened this issue Oct 11, 2021 · 5 comments

Comments

@W9CR
Copy link

W9CR commented Oct 11, 2021

System information

Type Version/Name
Distribution Name OSX
Distribution Version 10.13.4
Kernel root:xnu-4570.51.1~1/RELEASE_X86_64 x86_64
Architecture Intel
ZFS Version zfs-macOS-2.1.0-1
SPL Version zfs-kmod-2.1.0-1
RAM 64gb DDR4 2666 MHz
CPU Intel Core i7-8700

Summary of Problem

Poor read performance on encrypted ZFS dataset.

I have a 12 disk zraid2 zvol in a zpool. The disks I'm using are ST4000NM0033, Dell branded with Firmware GA6E.

I've created this using the following commands 
sudo zpool create -f -o ashift=12 \
-O casesensitivity=insensitive \
-O normalization=formD \
-O compression=lz4 \
-O atime=off \
-O recordsize=256k \
ZfsMediaPool raidz2 \
/var/run/disk/by-path/PCI0@0-SAT0@17-PRT5@5-PMP@0-@0:0 \
/var/run/disk/by-path/PCI0@0-SAT0@17-PRT4@4-PMP@0-@0:0  \
/var/run/disk/by-path/PCI0@0-RP21@1B,4-PXSX@0-PRT31@1f-PMP@0-@0:0  \
/var/run/disk/by-path/PCI0@0-SAT0@17-PRT3@3-PMP@0-@0:0  \
/var/run/disk/by-path/PCI0@0-SAT0@17-PRT2@2-PMP@0-@0:0  \
/var/run/disk/by-path/PCI0@0-SAT0@17-PRT1@1-PMP@0-@0:0  \
/var/run/disk/by-path/PCI0@0-SAT0@17-PRT0@0-PMP@0-@0:0  \
/var/run/disk/by-path/PCI0@0-RP21@1B,4-PXSX@0-PRT2@2-PMP@0-@0:0  \
/var/run/disk/by-path/PCI0@0-RP21@1B,4-PXSX@0-PRT3@3-PMP@0-@0:0  \
/var/run/disk/by-path/PCI0@0-RP21@1B,4-PXSX@0-PRT28@1c-PMP@0-@0:0  \
/var/run/disk/by-path/PCI0@0-RP21@1B,4-PXSX@0-PRT4@4-PMP@0-@0:0  \
/var/run/disk/by-path/PCI0@0-RP21@1B,4-PXSX@0-PRT29@1d-PMP@0-@0:0

zpool add ZfsMediaPool log /dev/disk5s3
zpool add ZfsMediaPool cache /dev/disk5s4

zpool set feature@encryption=enabled ZfsMediaPool
zfs create -o encryption=on -o keylocation=prompt -o keyformat=passphrase ZfsMediaPool/bryan

zfs set com.apple.mimic_hfs=hfs ZfsMediaPool/bryan

Reading and writing from the non-encrypted dataset works as expected with speeds exceeding 500 mbytes/s.

When reading from the encrypted dataset the performance goes way down, and any random I/O will bring the system to a crawl. Writing to the dataset causes the CPU load to go to 600%. A program such as thunderbird is almost unusable on the encrypted ZFS dataset.

I've tested this against a ZFS dataset with an encrypted APFS container and it has performed much better than native ZFS encryption.

Describe how to reproduce the problem

I created a random 10gb file by concatenating 10 copies of 1g from /dev/urandom

# zfs list
NAME                           USED  AVAIL     REFER  MOUNTPOINT
ZfsMediaPool                  8.18T  24.9T     7.07T  /Volumes/ZfsMediaPool
ZfsMediaPool/bryan            1.08T  24.9T     1.08T  /Users/bryan
ZfsMediaPool/mailcorestorage  31.6G  24.9T     31.6G  -

My encrypted pool is mounted on /Users/bryan/
I did the following while logged in as root and no processes were accessing the dataset

I have the ZfsMediaPool/bryan dataset encrypted and mounted at /Users/bryan
/Volumes/ZfsMediaPool is the same zpool, but not encrypted

Below is a 10g random file being copied with no other processes using the pool
# this is writing it from the root, which is a NVME disk 
dd if=/random-10g of=/Users/bryan/random-10g bs=1m
10737418240 bytes transferred in 71.818303 secs (149508103 bytes/sec)

# encrypted read
dd if=/Users/bryan/random-10g bs=1m of=/dev/null
10737418240 bytes transferred in 179.890006 secs (59688798 bytes/sec)

# non encrypted dataset
dd if=/Volumes/ZfsMediaPool/random-10g bs=1m of=/dev/null
10737418240 bytes transferred in 18.207343 secs (589730098 bytes/sec)


# First two are from the crypto disk, 1 is first, 2 is second but cache
$time du -hc /Users/bryan/Library/Thunderbird
real	0m3.202s
user	0m0.006s
sys	0m0.153s

$ time du -hc /Users/bryan/Library/Thunderbird
real	0m0.024s
user	0m0.003s
sys	0m0.020s

# This is from the non-crypto disk and not cached:
$ time du -hc /Users/bryan/Library/Thunderbird
real	0m0.552s
user	0m0.005s
sys	0m0.071s

I created an encrypted apfs disk on a zfs dataset
# zfs create -s -V 50g ZfsMediaPool/mailcorestorage
# ls /var/run/zfs/zvol/dsk/ZfsMediaPool/mailcorestorage -al
lrwxr-xr-x 1 root daemon 11 Oct  2 02:27 /var/run/zfs/zvol/dsk/ZfsMediaPool/mailcorestorage -> /dev/disk16
# diskutil eraseDisk JHFS+ dummy GPT /dev/disk16
(I was lazy and used diskutility to make an APFS encrypted disk)

copying from the same pool to the APFS dataset:
# dd if=/Volumes/ZfsMediaPool/random-10g of=/Volumes/encryptedMail/random-10g bs=1m
10240+0 records in
10240+0 records out
10737418240 bytes transferred in 83.464137 secs (128647089 bytes/sec)

Reading that file from the APFS dataset after remounting it:
dd if=/Users/bryan/Library/Thunderbird/random-10g of=/dev/null bs=1m
10240+0 records in
10240+0 records out
10737418240 bytes transferred in 34.071565 secs (315143090 bytes/sec)

Include any warning/errors/backtraces from the system logs

I've attached three spindumps during these operation and one while unison was running.
zfs-read-from-crypto-dataset-Spindump.txt
zfs-while-unison-running-Spindump.txt
zfs-write-to-crypto-big-dataset-Spindump.txt

zpool layout:

ComicBookGuy:~ root# zpool list -v
NAME                                                                       SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
ZfsMediaPool                                                              43.7T  10.7T  32.9T        -         -     0%    24%  1.00x    ONLINE  -
  raidz2                                                                  43.7T  10.7T  32.9T        -         -     0%  24.6%      -    ONLINE
    PCI0@0-SAT0@17-PRT5@5-PMP@0-@0:0                                          -      -      -        -         -      -      -      -    ONLINE
    PCI0@0-SAT0@17-PRT4@4-PMP@0-@0:0                                          -      -      -        -         -      -      -      -    ONLINE
    PCI0@0-RP21@1B,4-PXSX@0-PRT31@1f-PMP@0-@0:0                               -      -      -        -         -      -      -      -    ONLINE
    PCI0@0-SAT0@17-PRT3@3-PMP@0-@0:0                                          -      -      -        -         -      -      -      -    ONLINE
    PCI0@0-SAT0@17-PRT2@2-PMP@0-@0:0                                          -      -      -        -         -      -      -      -    ONLINE
    PCI0@0-SAT0@17-PRT1@1-PMP@0-@0:0                                          -      -      -        -         -      -      -      -    ONLINE
    PCI0@0-SAT0@17-PRT0@0-PMP@0-@0:0                                          -      -      -        -         -      -      -      -    ONLINE
    PCI0@0-RP21@1B,4-PXSX@0-PRT2@2-PMP@0-@0:0                                 -      -      -        -         -      -      -      -    ONLINE
    PCI0@0-RP21@1B,4-PXSX@0-PRT3@3-PMP@0-@0:0                                 -      -      -        -         -      -      -      -    ONLINE
    PCI0@0-RP21@1B,4-PXSX@0-PRT28@1c-PMP@0-@0:0                               -      -      -        -         -      -      -      -    ONLINE
    PCI0@0-RP21@1B,4-PXSX@0-PRT4@4-PMP@0-@0:0                                 -      -      -        -         -      -      -      -    ONLINE
    PCI0@0-RP21@1B,4-PXSX@0-PRT29@1d-PMP@0-@0:0                               -      -      -        -         -      -      -      -    ONLINE
logs                                                                          -      -      -        -         -      -      -      -  -
  PCI0@0-RP09@1D-PXSX@0-IONVMeController-IONVMeBlockStorageDevice@1-@1:3  15.5G  1.19M  15.5G        -         -     0%  0.00%      -    ONLINE
cache                                                                         -      -      -        -         -      -      -      -  -
  PCI0@0-RP09@1D-PXSX@0-IONVMeController-IONVMeBlockStorageDevice@1-@1:4   128G   108G  20.1G        -         -     0%  84.3%      -    ONLINE
@GRBaset
Copy link

GRBaset commented Dec 8, 2021 •
edited
Loading

I have the same problem on the two Macs I tested, with the same version of ZFS as yours. One is an iMac with an i5-5675R on macOS 10.15.7 (kernel root:xnu-6153.141.43~1/RELEASE_X86_64), the other is a MacBook Air with an i5-5350U on macOS 10.14.6 (kernel root:xnu-4903.278.70~1/RELEASE_X86_64).

I tested on a RAM disk (see my script) with dd from /dev/zero, and performance on an unencrypted dataset is over 700 MB/s on the iMac, 208 MB/s on the MBA, while on AES-256-GCM (the default) it falls to about 150 MB/s on the iMac and 20 MB/s on the MBA with full CPU usage in both cases.

As a comparison, openssl speed -evp aes-256-gcm gives 400-1500 MB/s on the iMac and 310-1200 MB/s on the MBA with only a single core saturated in both cases, and I tested on a RAM disk on a Linux VM on the iMac and it gives over 500 MB/s for AES-256-GCM, so it's clearly not a problem of OpenZFS. Maybe this release of OpenZFS on OSX doesn't use AES-NI?

Here are the results for the dd benchmark:

iMac

Running benchmark for noencryption...
10+0 records in
10+0 records out
1048576000 bytes transferred in 1.427054 secs (734783724 bytes/sec)
Running benchmark for aes128ccm...
10+0 records in
10+0 records out
1048576000 bytes transferred in 2.966284 secs (353498177 bytes/sec)
Running benchmark for aes256ccm...
10+0 records in
10+0 records out
1048576000 bytes transferred in 3.687919 secs (284327275 bytes/sec)
Running benchmark for aes128gcm...
10+0 records in
10+0 records out
1048576000 bytes transferred in 6.448514 secs (162607386 bytes/sec)
Running benchmark for aes256gcm...
10+0 records in
10+0 records out
1048576000 bytes transferred in 6.668878 secs (157234244 bytes/sec)

MacBook Air

Running benchmark for noencryption...
10+0 records in
10+0 records out
1048576000 bytes transferred in 5.040728 secs (208020742 bytes/sec)
Running benchmark for aes128ccm...
10+0 records in
10+0 records out
1048576000 bytes transferred in 9.648026 secs (108682958 bytes/sec)
Running benchmark for aes256ccm...
10+0 records in
10+0 records out
1048576000 bytes transferred in 9.792953 secs (107074546 bytes/sec)
Running benchmark for aes128gcm...
10+0 records in
10+0 records out
1048576000 bytes transferred in 50.366285 secs (20819006 bytes/sec)
Running benchmark for aes256gcm...
10+0 records in
10+0 records out
1048576000 bytes transferred in 53.425810 secs (19626768 bytes/sec)

And here are the results of openssl speed -evp aes-256-gcm:

iMac

LibreSSL 2.8.3
built on: date not available
options:bn(64,64) rc4(16x,int) des(idx,cisc,16,int) aes(partial) blowfish(idx) 
compiler: information not available
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-256-gcm     409696.18k  1008532.69k  1437244.00k  1539713.79k  1580495.42k

MacBook Air

LibreSSL 2.6.5
built on: date not available
options:bn(64,64) rc4(16x,int) des(idx,cisc,16,int) aes(partial) blowfish(idx) 
compiler: information not available
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-256-gcm     311740.97k   739539.17k  1127034.30k  1191862.55k  1227247.06k

@lundman
Copy link
Contributor

lundman commented Jan 12, 2022

1d90a79

We should now be detecting the correct CPUID features (yikes) and use AESNI when available. This could also be why AVX fails - need to test that separately.

There is a test PKG on the forum for those we can test it:

https://openzfsonosx.org/forum/viewtopic.php?f=26&t=3651&p=11593#p11588

@digitalgunslinger
Copy link

Has this performance issue been fixed in version OpenZFSonOsX-2.1.0-Catalina-10.15?

I have not been able to get on to the openzfsonosx.org forum... site appears to be down.

@andrewc12
Copy link
Contributor

The website is broken at the moment due to a problem with the server that runs the vm's.
If you have any urgent enquires you might want to jump on #openzfs-macos on irc.libera.chat:6697

@digitalgunslinger
Copy link

It's nothing I'd consider urgent... was going to try the test PKG linked above. Is there a way to get that PKG from github?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@lundman @andrewc12 @GRBaset @W9CR @digitalgunslinger