feat: added additional zfs services to support encrypted volumes

[runningman84]

This should work fine but I am not sure if the dependencies work that way. Please review...

With this config zfs filesystems with encrypted volumes and local keys (for example stored in /var) will be auto mounted.

This is an example from my test system:

root@worker2:~# df
Filesystem                      1K-blocks     Used Available Use% Mounted on
overlay                         975037276 13135376 961901900   2% /
tmpfs                               65536        0     65536   0% /dev
overlay                               256      256         0 100% /host
rootfs                           16242936   104588  16138348   1% /host/.extra
devtmpfs                         16242936        0  16242936   0% /host/dev
tmpfs                            16295300        0  16295300   0% /dev/shm
tmpfs                            16295300        0  16295300   0% /host/proc/acpi
tmpfs                            16295300        0  16295300   0% /host/proc/scsi
efivarfs                              192       92        96  50% /host/sys/firmware/efi/efivars
tmpfs                            16295300     2628  16292672   1% /host/run
overlay                         975037276 13135376 961901900   2% /run/containerd/io.containerd.runtime.v2.task/system/kubelet/rootfs
shm                                 65536        0     65536   0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/c31fad6a21cd91e62facb7b20a61a020ac83600aeb67a51259623e4c93c5157c/shm
shm                                 65536        0     65536   0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/c6851ae888bc2afad1c246e089c3902cad9a9362a87169a84bb7b476774414c4/shm
shm                                 65536        0     65536   0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/a382de4b321b6324fe8cba0052ec3a372e3209963568f3b6cf9e9866f4bea094/shm
shm                                 65536        0     65536   0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/9e6166cc734d1cdecc0763eafe2ec6ed8abc435e18af581c387ad3c0bf28c824/shm
overlay                         975037276 13135376 961901900   2% /run/containerd/io.containerd.runtime.v2.task/k8s.io/c31fad6a21cd91e62facb7b20a61a020ac83600aeb67a51259623e4c93c5157c/rootfs
overlay                         975037276 13135376 961901900   2% /run/containerd/io.containerd.runtime.v2.task/k8s.io/c6851ae888bc2afad1c246e089c3902cad9a9362a87169a84bb7b476774414c4/rootfs
overlay                         975037276 13135376 961901900   2% /run/containerd/io.containerd.runtime.v2.task/k8s.io/6b249ed4e115676c6ac82de699264f07701de5d26398e78d1e4b2b7a184e099f/rootfs
overlay                         975037276 13135376 961901900   2% /run/containerd/io.containerd.runtime.v2.task/k8s.io/a3afd6d213e959957cf5edf2b9db2ae42ddf660121c32399bcbc5ff9bc8886db/rootfs
overlay                         975037276 13135376 961901900   2% /run/containerd/io.containerd.runtime.v2.task/k8s.io/9e6166cc734d1cdecc0763eafe2ec6ed8abc435e18af581c387ad3c0bf28c824/rootfs
overlay                         975037276 13135376 961901900   2% /run/containerd/io.containerd.runtime.v2.task/k8s.io/a382de4b321b6324fe8cba0052ec3a372e3209963568f3b6cf9e9866f4bea094/rootfs
overlay                         975037276 13135376 961901900   2% /run/containerd/io.containerd.runtime.v2.task/k8s.io/84ee344d0c6af8b4d9a4be0f0cecb3667229043c61e98003f8f2edf7bcb73271/rootfs
overlay                         975037276 13135376 961901900   2% /run/containerd/io.containerd.runtime.v2.task/k8s.io/dce879378a9651f24fd5e9c72ae50bed34946d67ed57f0226ec4e8f832275488/rootfs
tmpfs                            16295300      240  16295060   1% /host/system
overlay                               256      256         0 100% /host/system/libexec/apid/apid
/dev/mapper/nvme0n1p3-encrypted     80544     5140     75404   7% /host/system/state
tmpfs                               65536        0     65536   0% /host/tmp
overlay                          16295300      240  16295060   1% /host/usr/etc/udev
/dev/mapper/nvme0n1p4-encrypted 975037276 13135376 961901900   2% /host/var
tmpfs                            32291592        4  32291588   1% /host/var/lib/kubelet/pods/ecd04ced-2d4e-4e1a-a2e0-64b300abc082/volumes/kubernetes.io~secret/memberlist
tmpfs                            32291592        0  32291592   0% /host/var/lib/kubelet/pods/a0b74911-c60d-4d3a-8187-863e39930ea9/volumes/kubernetes.io~secret/longhorn-grpc-tls
tmpfs                            32291592       12  32291580   1% /host/var/lib/kubelet/pods/0d0689d2-e263-42b1-82a9-22ad3ade5ccc/volumes/kubernetes.io~projected/kube-api-access-qmg8n
tmpfs                            32291592       12  32291580   1% /host/var/lib/kubelet/pods/eb279ec9-bf74-4102-ab89-3fe44cac45e8/volumes/kubernetes.io~projected/kube-api-access-m2v2n
tmpfs                            32291592       12  32291580   1% /host/var/lib/kubelet/pods/b21d24d3-f21b-4dfe-ae43-4108b9241327/volumes/kubernetes.io~projected/kube-api-access-kwtmz
tmpfs                            32291592       12  32291580   1% /host/var/lib/kubelet/pods/a0b74911-c60d-4d3a-8187-863e39930ea9/volumes/kubernetes.io~projected/kube-api-access-h5nvx
tmpfs                            32291592       12  32291580   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs                            32291592       12  32291580   1% /host/var/lib/kubelet/pods/784bf64e-0f81-4e15-ae98-d98f055a5748/volumes/kubernetes.io~projected/kube-api-access-lhwhw
tmpfs                            32291592       12  32291580   1% /host/var/lib/kubelet/pods/ecd04ced-2d4e-4e1a-a2e0-64b300abc082/volumes/kubernetes.io~projected/kube-api-access-s6lfj
tmpfs                            32291592       12  32291580   1% /host/var/lib/kubelet/pods/fbbe174c-8888-4302-a8be-3ffaa7f8fc9b/volumes/kubernetes.io~projected/kube-api-access-jpsv5
tmpfs                            32291592       12  32291580   1% /host/var/lib/kubelet/pods/55c9398f-7fa1-4a01-99fa-7ad13a04e506/volumes/kubernetes.io~projected/kube-api-access-hflnz
tmpfs                            32291592       12  32291580   1% /host/var/lib/kubelet/pods/11bc1b90-1509-4084-bb11-85acd8276f0f/volumes/kubernetes.io~projected/kube-api-access-8t4zh
overlay                         975037276 13135376 961901900   2% /host/etc/cni
overlay                         975037276 13135376 961901900   2% /host/etc/kubernetes
overlay                         975037276 13135376 961901900   2% /host/usr/libexec/kubernetes
overlay                         975037276 13135376 961901900   2% /host/opt
overlay                          16295300      240  16295060   1% /host/usr/local/lib/containers/tgtd
overlay                          16295300      240  16295060   1% /host/usr/local/lib/containers/zpool-importer
overlay                          16295300      240  16295060   1% /host/usr/local/lib/containers/iscsid
root@worker2:~# zfs list
NAME           USED  AVAIL  REFER  MOUNTPOINT
tank          1.57T  9.21T    96K  /var/hddstorage
tank/private   195G  9.21T   195G  /var/hddstorage/private
tank/public   1.38T  9.21T  1.38T  /var/hddstorage/public
root@worker2:~# zfs load-key -a         
2 / 2 key(s) successfully loaded
root@worker2:~# zfs mount -a
root@worker2:~# df
Filesystem                        1K-blocks       Used  Available Use% Mounted on
overlay                           975037276   13135432  961901844   2% /
tmpfs                                 65536          0      65536   0% /dev
overlay                                 256        256          0 100% /host
rootfs                             16242936     104588   16138348   1% /host/.extra
devtmpfs                           16242936          0   16242936   0% /host/dev
tmpfs                              16295300          0   16295300   0% /dev/shm
tmpfs                              16295300          0   16295300   0% /host/proc/acpi
tmpfs                              16295300          0   16295300   0% /host/proc/scsi
efivarfs                                192         92         96  50% /host/sys/firmware/efi/efivars
tmpfs                              16295300       2628   16292672   1% /host/run
overlay                           975037276   13135432  961901844   2% /run/containerd/io.containerd.runtime.v2.task/system/kubelet/rootfs
shm                                   65536          0      65536   0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/c31fad6a21cd91e62facb7b20a61a020ac83600aeb67a51259623e4c93c5157c/shm
shm                                   65536          0      65536   0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/c6851ae888bc2afad1c246e089c3902cad9a9362a87169a84bb7b476774414c4/shm
shm                                   65536          0      65536   0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/a382de4b321b6324fe8cba0052ec3a372e3209963568f3b6cf9e9866f4bea094/shm
shm                                   65536          0      65536   0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/9e6166cc734d1cdecc0763eafe2ec6ed8abc435e18af581c387ad3c0bf28c824/shm
overlay                           975037276   13135432  961901844   2% /run/containerd/io.containerd.runtime.v2.task/k8s.io/c31fad6a21cd91e62facb7b20a61a020ac83600aeb67a51259623e4c93c5157c/rootfs
overlay                           975037276   13135432  961901844   2% /run/containerd/io.containerd.runtime.v2.task/k8s.io/c6851ae888bc2afad1c246e089c3902cad9a9362a87169a84bb7b476774414c4/rootfs
overlay                           975037276   13135432  961901844   2% /run/containerd/io.containerd.runtime.v2.task/k8s.io/6b249ed4e115676c6ac82de699264f07701de5d26398e78d1e4b2b7a184e099f/rootfs
overlay                           975037276   13135432  961901844   2% /run/containerd/io.containerd.runtime.v2.task/k8s.io/a3afd6d213e959957cf5edf2b9db2ae42ddf660121c32399bcbc5ff9bc8886db/rootfs
overlay                           975037276   13135432  961901844   2% /run/containerd/io.containerd.runtime.v2.task/k8s.io/9e6166cc734d1cdecc0763eafe2ec6ed8abc435e18af581c387ad3c0bf28c824/rootfs
overlay                           975037276   13135432  961901844   2% /run/containerd/io.containerd.runtime.v2.task/k8s.io/a382de4b321b6324fe8cba0052ec3a372e3209963568f3b6cf9e9866f4bea094/rootfs
overlay                           975037276   13135432  961901844   2% /run/containerd/io.containerd.runtime.v2.task/k8s.io/84ee344d0c6af8b4d9a4be0f0cecb3667229043c61e98003f8f2edf7bcb73271/rootfs
overlay                           975037276   13135432  961901844   2% /run/containerd/io.containerd.runtime.v2.task/k8s.io/dce879378a9651f24fd5e9c72ae50bed34946d67ed57f0226ec4e8f832275488/rootfs
tmpfs                              16295300        240   16295060   1% /host/system
overlay                                 256        256          0 100% /host/system/libexec/apid/apid
/dev/mapper/nvme0n1p3-encrypted       80544       5140      75404   7% /host/system/state
tmpfs                                 65536          8      65528   1% /host/tmp
overlay                            16295300        240   16295060   1% /host/usr/etc/udev
/dev/mapper/nvme0n1p4-encrypted   975037276   13135432  961901844   2% /host/var
tmpfs                              32291592          4   32291588   1% /host/var/lib/kubelet/pods/ecd04ced-2d4e-4e1a-a2e0-64b300abc082/volumes/kubernetes.io~secret/memberlist
tmpfs                              32291592          0   32291592   0% /host/var/lib/kubelet/pods/a0b74911-c60d-4d3a-8187-863e39930ea9/volumes/kubernetes.io~secret/longhorn-grpc-tls
tmpfs                              32291592         12   32291580   1% /host/var/lib/kubelet/pods/0d0689d2-e263-42b1-82a9-22ad3ade5ccc/volumes/kubernetes.io~projected/kube-api-access-qmg8n
tmpfs                              32291592         12   32291580   1% /host/var/lib/kubelet/pods/eb279ec9-bf74-4102-ab89-3fe44cac45e8/volumes/kubernetes.io~projected/kube-api-access-m2v2n
tmpfs                              32291592         12   32291580   1% /host/var/lib/kubelet/pods/b21d24d3-f21b-4dfe-ae43-4108b9241327/volumes/kubernetes.io~projected/kube-api-access-kwtmz
tmpfs                              32291592         12   32291580   1% /host/var/lib/kubelet/pods/a0b74911-c60d-4d3a-8187-863e39930ea9/volumes/kubernetes.io~projected/kube-api-access-h5nvx
tmpfs                              32291592         12   32291580   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs                              32291592         12   32291580   1% /host/var/lib/kubelet/pods/784bf64e-0f81-4e15-ae98-d98f055a5748/volumes/kubernetes.io~projected/kube-api-access-lhwhw
tmpfs                              32291592         12   32291580   1% /host/var/lib/kubelet/pods/ecd04ced-2d4e-4e1a-a2e0-64b300abc082/volumes/kubernetes.io~projected/kube-api-access-s6lfj
tmpfs                              32291592         12   32291580   1% /host/var/lib/kubelet/pods/fbbe174c-8888-4302-a8be-3ffaa7f8fc9b/volumes/kubernetes.io~projected/kube-api-access-jpsv5
tmpfs                              32291592         12   32291580   1% /host/var/lib/kubelet/pods/55c9398f-7fa1-4a01-99fa-7ad13a04e506/volumes/kubernetes.io~projected/kube-api-access-hflnz
tmpfs                              32291592         12   32291580   1% /host/var/lib/kubelet/pods/11bc1b90-1509-4084-bb11-85acd8276f0f/volumes/kubernetes.io~projected/kube-api-access-8t4zh
overlay                           975037276   13135432  961901844   2% /host/etc/cni
overlay                           975037276   13135432  961901844   2% /host/etc/kubernetes
overlay                           975037276   13135432  961901844   2% /host/usr/libexec/kubernetes
overlay                           975037276   13135432  961901844   2% /host/opt
overlay                            16295300        240   16295060   1% /host/usr/local/lib/containers/tgtd
overlay                            16295300        240   16295060   1% /host/usr/local/lib/containers/zpool-importer
overlay                            16295300        240   16295060   1% /host/usr/local/lib/containers/iscsid
tank                             9889162240        128 9889162112   1% /host/var/hddstorage
tank/private                    10093786496  204624384 9889162112   3% /host/var/hddstorage/private
tank/public                     11371600512 1482438400 9889162112  14% /host/var/hddstorage/public

[smira]

I would really love to land some support for mounting/volumes in Talos 1.7 to avoid such workaround if possible.

[runningman84]

Okay if you bring native support for that in 1.7 I am totally fine… if not please consider merging this workaround.

[runningman84]

@smira I played arround with zfs a bit. Zfs encryption in talos is tricky at the moment:

• if you mount an encrypted volume (zfs load-key -a; zfs mount -a) using a debug pod the files are not accessible in another pod
• having a mix of encrypted and unencrypted zfs datasets results in none of them being automounted anymore
• normally all datasets are automated if you use zfs but sometimes this does not happen, so you have to call zfs mount --a , this is also problematic because pods might write data to some local fs instead of the real zfs fs. This can result in data loos.

I think there should be some tests and documentation for zfs usage to make this usable without these issues,