"Too many levels of symbolic links" when accessing snapshots

[odoucet]

This is the same error as #816 but this old ticket was closed as fixed because it happens on old ZFS versions with an old kernel. I'm opening a new issue with additional informations to get it fixed.

This happens on two different systems (with same data, replicated with zfs send/recv).
System 1 is Kernel 4.4.4 + SPL/ZFS 0.6.5.5
System 2 is Kernel 4.5.0 + SPL/ZFS 0.6.5.6

This worked on ZFS 0.6.3 with kernel 3.10 ...

$ ls -lah /backup/xxxxobfuscatedxxx/.zfs/snapshot/2016-04-11.000000Z/
ls: cannot access /backup/xxxxobfuscatedxxx/.zfs/snapshot/2016-04-11.000000Z/: Too many levels of symbolic links

But this almost work :

$ cd /backup/xxxxobfuscatedxxx/.zfs/snapshot/2016-04-11.000000Z/
$ ls
home/
# Great, I have it ! 
$ pwd
/backup/xxxxobfuscatedxxx/.zfs/snapshot/2016-04-11.000000Z/
$ cd home && pwd
(unreachable)/home

But as the path seems completely f**ed up, tools like rsync do not work to restore data...

Fortunately, I can still have access to my data with mount :

$ mount -t zfs backupxx/[email protected] /mnt/test
# works \o/

stracing ls does not result to segfault like old ticket :)

stat("/backup/xxxxobfuscatedxxx/.zfs/snapshot/2016-04-11.000000Z/", 0x1ac50d0) = -1 ELOOP

This volume currently have 57 snapshots.

This can be reproduced easily if needed. Just tell me how can I help ...

[tuxoko]

Is the snapshot already mounted somewhere else?

[odoucet]

None of the snapshot in this filesystem is mounted elsewhere.
Initial filesystem is mounted though.

[tuxoko]

When you saw the ELOOP error, can you manual mount the snapshot at the .zfs/snapshot/xxx location and would every thing work afterward?

[odoucet]

I'm not sure what you mean ... Yes, I can mount the same snapshot I have the ELOOP error with.

Or do you mean

mount -t zfs [email protected] /backup/xxxx/.zfs/snapshot/2016-04-10.000000Z/ ?

This does not have any sense nope ? mount does no error, but ls still failed with ELOOP.

And 3rd try :

$ mount -t zfs [email protected] /mnt/test
$ ls /backup/xxx/.zfs/snapshot/2016-04-10.000000Z/
Too many levels of symbolic links

[tuxoko]

Do this mount -t zfs [email protected] /backup/xxxx/.zfs/snapshot/2016-04-10.000000Z/
It might not make sense to you, but I want to have an idea of which part in the code went wrong.

[odoucet]

Yes, that's what I tested in my previous post ;) mount command does no error, but ELOOP is still there, and /proc/mounts does not show the mount.

[odoucet]

This bug also prevents "zfs diff" to work (failed with message "Unable to obtain diffs: No such file or directory"), if it can help.
Folder .zfs/shares is empty

[tuxoko]

@odoucet
Please try strace mount.zfs [email protected] /backup/xxxx/.zfs/snapshot/2016-04-10.000000Z/

[odoucet]

[...]
stat("xxxxxxx/[email protected]", 0x7ffe33b45500) = -1 ENOENT (No such file or directory)
getcwd("/root", 4096)                   = 6
lstat("/xx", {st_mode=S_IFDIR|0755, st_size=10, ...}) = 0
lstat("/xx/xx", {st_mode=S_IFDIR|0755, st_size=50, ...}) = 0
lstat("/xx/xx/xx", {st_mode=S_IFDIR|0755, st_size=25, ...}) = 0
lstat("/xx/xx/xx/xx", {st_mode=S_IFDIR|0755, st_size=3, ...}) = 0
lstat("/xx/xx/xx/xx/.zfs", {st_mode=S_IFDIR|0555, st_size=0, ...}) = 0
lstat("/xx/xx/xx/xx/.zfs/snapshot", {st_mode=S_IFDIR|0555, st_size=2, ...}) = 0
lstat("/xx/xx/xx/xx/.zfs/snapshot/2016-04-10.000000Z", {st_mode=S_IFDIR|0555, st_size=0, ...}) = 0
access("/sys/module/zfs", F_OK)         = 0
access("/sys/module/zfs", F_OK)         = 0
open("/dev/zfs", O_RDWR)                = 3
close(3)                                = 0
open("/dev/zfs", O_RDWR)                = 3
open("/etc/mtab", O_RDONLY)             = 4
open("/etc/dfs/sharetab", O_RDONLY)     = 5
open("/dev/zfs", O_RDWR)                = 6
open("/usr/share/locale/locale.alias", O_RDONLY) = 7
fstat(7, {st_mode=S_IFREG|0644, st_size=2512, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f2c28673000
read(7, "# Locale name alias data base.\n#"..., 4096) = 2512
read(7, "", 4096)                       = 0
close(7)                                = 0
munmap(0x7f2c28673000, 4096)            = 0
open("/usr/share/locale/en_US.UTF-8/LC_MESSAGES/zfs-linux-user.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/en_US.utf8/LC_MESSAGES/zfs-linux-user.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/en_US/LC_MESSAGES/zfs-linux-user.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/en.UTF-8/LC_MESSAGES/zfs-linux-user.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/en.utf8/LC_MESSAGES/zfs-linux-user.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/en/LC_MESSAGES/zfs-linux-user.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
ioctl(3, 0x5a12, 0x7ffe33b41a70)        = 0
ioctl(3, 0x5a05, 0x7ffe33b3e420)        = 0
ioctl(3, 0x5a13, 0x7ffe33b41e80)        = 0
close(3)                                = 0
close(4)                                = 0
close(5)                                = 0
close(6)                                = 0
mount("xxxxxxx/[email protected]", "/xxxxxxx/.zfs/snapshot/2016-04-10.000000Z", "zfs", 0, ",mntpoint=/xxxxxxxxx"...) = 0
lstat("/etc/mtab", {st_mode=S_IFREG|0644, st_size=17740, ...}) = 0
open("/etc/mtab", O_RDWR|O_CREAT, 0644) = 3
close(3)                                = 0
open("/etc/mtab", O_RDWR|O_CREAT|O_APPEND, 0666) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=17740, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f2c28673000
fstat(3, {st_mode=S_IFREG|0644, st_size=17740, ...}) = 0
lseek(3, 16384, SEEK_SET)               = 16384
read(3, "rw,noatime 0 0\nxxxxxxx"..., 1356) = 1356
write(3, "xxxxxx"..., 147) = 147
close(3)                                = 0
munmap(0x7f2c28673000, 4096)            = 0
exit_group(0)                           = ?
+++ exited with 0 +++

[odoucet]

any update ? What can I do to help ?

[tuxoko]

The mount command returns success, I don't know why it wouldn't work for you.

[odoucet]

Just tested different kernel with same SPL/ZFS version (v0.6.5.6-1).
On kernel 4.4.4 : not working
Rebooted same system on kernel 3.10.101 : working

on kernel 4.4.4, behaviour is really really strange :

$ ls /backup/xxxobfuscatedxxx/.zfs/snapshot/2016-04-26.000000Z/
ls: cannot access /backup/xxxobfuscatedxxx/.zfs/snapshot/2016-04-26.000000Z/: Too many levels of symbolic links
$ cd xxobfuscatedxxx && ls 2016-04-26.000000Z/
home/

As stated above, mount on snapshot is working ...

[m-r-r]

Hello @odoucet,

I have the same problem with a ZFS filesystem which is bind-mounted in a LXC container.
Is your ZFS filesystem also accessible from a LXC container ? If so, that could be related…

[odoucet]

Hi @m-r-r , sorry, no LXC involved in my setup ...

[JuliaVixen]

I'm having, was appears to be, the same issue. Short summary: 'ls' on a snapshot when the CWD is under .zfs works as expected. When the CWD is elsewhere: "Too many symbolic links".

localhost ~ # uname -a
Linux localhost 4.4.6-gentoo #1 SMP Mon Apr 25 02:58:59 Local time zone must be set--see zic  x86_64 Intel(R) Xeon(R) CPU E3-1220 v3 @ 3.10GHz GenuineIntel GNU/Linux

localhost ~ # modinfo zfs
filename:       /lib/modules/4.4.6-gentoo/extra/zfs/zfs.ko
version:        0.6.5.4-r1-gentoo
license:        CDDL
author:         OpenZFS on Linux
description:    ZFS
srcversion:     4251E810337436FD7B850DA
depends:        spl,znvpair,zunicode,zcommon,zavl
vermagic:       4.4.6-gentoo SMP mod_unload modversions 
[... And so on.]

localhost ~ # ls /l/photos/.zfs/snapshot/send/
ls: cannot open directory /l/photos/.zfs/snapshot/send/: Too many levels of symbolic links
[Wait a minute]
localhost ~ # ls /l/photos/.zfs/snapshot/send/
ls: cannot open directory /l/photos/.zfs/snapshot/send/: Too many levels of symbolic links

localhost ~ # pwd
/root

localhost ~ # ls /l/photos/.zfs/snapshot/send/
ls: cannot open directory /l/photos/.zfs/snapshot/send/: Too many levels of symbolic links
localhost ~ # ls /l/photos/.zfs/snapshot/send
ls: cannot access /l/photos/.zfs/snapshot/send/stuff: Too many levels of symbolic links
ls: cannot access /l/photos/.zfs/snapshot/send/thing: Too many levels of symbolic links
ls: cannot access /l/photos/.zfs/snapshot/send/whatever: Too many levels of symbolic links
[...]
localhost ~ # ls /l/photos/.zfs/snapshot/
[Expected output, no error]
localhost ~ # ls /l/photos/.zfs/
[Expected output, no error]
localhost ~ # ls /l/photos/
[Expected output, no error]
localhost ~ # ls /l/
[Expected output, no error]

localhost photos # cd /l/photos
localhost photos # pwd
/l/photos

localhost photos # ls .zfs/snapshot/send/
ls: cannot open directory .zfs/snapshot/send/: Too many levels of symbolic links

localhost photos # cd .zfs
localhost .zfs # pwd
/l/photos/.zfs
localhost .zfs # ls snapshot/send/
thing
stuff
whatever
[...And the correctly expected results with no errors.]

Using an absolute path, even with CWD being .zfs

localhost .zfs # ls /l/photos/.zfs/snapshot/send/
ls: cannot open directory /l/photos/.zfs/snapshot/send/: Too many levels of symbolic links

Only relative paths to .zfs and below work without error.

There's nothing in dmesg, and I've been scrubbing the pool, but there are no errors detected so far.

Oh, also...

localhost .zfs # zpool get all l
NAME  PROPERTY                    VALUE                       SOURCE
l     size                        36.2T                       -
l     capacity                    67%                         -
l     altroot                     -                           default
l     health                      ONLINE                      -
l     guid                        4946876290228094116         default
l     version                     -                           default
l     bootfs                      -                           default
l     delegation                  on                          default
l     autoreplace                 off                         default
l     cachefile                   -                           default
l     failmode                    wait                        default
l     listsnapshots               off                         default
l     autoexpand                  off                         default
l     dedupditto                  0                           default
l     dedupratio                  1.00x                       -
l     free                        11.9T                       -
l     allocated                   24.3T                       -
l     readonly                    off                         -
l     ashift                      12                          local
l     comment                     -                           default
l     expandsize                  -                           -
l     freeing                     0                           default
l     fragmentation               37%                         -
l     leaked                      0                           default
l     feature@async_destroy       enabled                     local
l     feature@empty_bpobj         active                      local
l     feature@lz4_compress        active                      local
l     feature@spacemap_histogram  active                      local
l     feature@enabled_txg         active                      local
l     feature@hole_birth          active                      local
l     feature@extensible_dataset  active                      local
l     feature@embedded_data       active                      local
l     feature@bookmarks           enabled                     local
l     feature@filesystem_limits   enabled                     local
l     feature@large_blocks        active                      local

localhost .zfs # zfs get all l/photos
NAME      PROPERTY              VALUE                                                       SOURCE
l/photos  type                  filesystem                                                  -
l/photos  creation              Tue Apr 26 19:43 2016                                       -
l/photos  used                  13.8T                                                       -
l/photos  available             8.64T                                                       -
l/photos  referenced            13.2T                                                       -
l/photos  compressratio         1.01x                                                       -
l/photos  mounted               yes                                                         -
l/photos  quota                 none                                                        default
l/photos  reservation           none                                                        default
l/photos  recordsize            128K                                                        default
l/photos  mountpoint            /l/photos                                                   default
l/photos  sharenfs              fsid=25,rw=172.16.111.0/24,sec=sys,insecure,insecure_locks  received
l/photos  checksum              on                                                          default
l/photos  compression           on                                                          inherited from l
l/photos  atime                 off                                                         inherited from l
l/photos  devices               off                                                         inherited from l
l/photos  exec                  on                                                          default
l/photos  setuid                off                                                         inherited from l
l/photos  readonly              off                                                         default
l/photos  zoned                 off                                                         default
l/photos  snapdir               hidden                                                      default
l/photos  aclinherit            restricted                                                  default
l/photos  canmount              on                                                          default
l/photos  xattr                 on                                                          default
l/photos  copies                1                                                           default
l/photos  version               1                                                           -
l/photos  utf8only              off                                                         default
l/photos  normalization         none                                                        default
l/photos  casesensitivity       sensitive                                                   default
l/photos  vscan                 off                                                         default
l/photos  nbmand                off                                                         default
l/photos  sharesmb              off                                                         default
l/photos  refquota              none                                                        default
l/photos  refreservation        none                                                        default
l/photos  primarycache          all                                                         default
l/photos  secondarycache        all                                                         default
l/photos  usedbysnapshots       680G                                                        -
l/photos  usedbydataset         13.2T                                                       -
l/photos  usedbychildren        0                                                           -
l/photos  usedbyrefreservation  0                                                           -
l/photos  logbias               latency                                                     default
l/photos  dedup                 off                                                         default
l/photos  mlslabel              none                                                        default
l/photos  sync                  standard                                                    default
l/photos  refcompressratio      1.01x                                                       -
l/photos  written               0                                                           -
l/photos  logicalused           14.0T                                                       -
l/photos  logicalreferenced     13.3T                                                       -
l/photos  filesystem_limit      none                                                        default
l/photos  snapshot_limit        none                                                        default
l/photos  filesystem_count      none                                                        default
l/photos  snapshot_count        none                                                        default
l/photos  snapdev               hidden                                                      default
l/photos  acltype               off                                                         default
l/photos  context               none                                                        default
l/photos  fscontext             none                                                        default
l/photos  defcontext            none                                                        default
l/photos  rootcontext           none                                                        default
l/photos  relatime              off                                                         default
l/photos  redundant_metadata    all                                                         default
l/photos  overlay               off                                                         default

There are 43 snapshots under /l/photos/.zfs/snapshots. The filesystem has been "zfs send" and "zfs received" a few times over the years, from older versions of ZFS on Solaris, FreeBSD, and Linux. It's been running under ZFSonLinux for two years now. I just did a...

zfs send -eLRv -I May_20_2015 h/photos@send | zfs recv -evF l

...a few days ago. (The h pool was an older version without many features turned on. And I don't remember this snapshot error on that pool, but I'd have to plug the drives back in to check...)

[JuliaVixen]

I plugged the drives back in; there is no error when using the old pool.

localhost ~ # ls /l/photos/.zfs/snapshot/send/
ls: cannot open directory /l/photos/.zfs/snapshot/send/: Too many levels of symbolic links
localhost ~ # ls /h/photos/.zfs/snapshot/send/
things
stuff
[...expected results, no errors]

localhost ~ # zpool get all h

[Ignore the "DEGRADED" state, I only plugged in just enough drives to import this to test.]

h     size                        21.8T                       -
h     capacity                    96%                         -
h     altroot                     -                           default
h     health                      DEGRADED                    -
h     guid                        5105680881284105628         default
h     version                     -                           default
h     bootfs                      -                           default
h     delegation                  on                          default
h     autoreplace                 off                         default
h     cachefile                   -                           default
h     failmode                    wait                        default
h     listsnapshots               off                         default
h     autoexpand                  off                         default
h     dedupditto                  0                           default
h     dedupratio                  1.00x                       -
h     free                        825G                        -
h     allocated                   20.9T                       -
h     readonly                    on                          -
h     ashift                      12                          local
h     comment                     -                           default
h     expandsize                  -                           -
h     freeing                     0                           default
h     fragmentation               0%                          -
h     leaked                      0                           default
h     feature@async_destroy       enabled                     local
h     feature@empty_bpobj         active                      local
h     feature@lz4_compress        active                      local
h     feature@spacemap_histogram  disabled                    local
h     feature@enabled_txg         disabled                    local
h     feature@hole_birth          disabled                    local
h     feature@extensible_dataset  disabled                    local
h     feature@embedded_data       active                      local
h     feature@bookmarks           disabled                    local
h     feature@filesystem_limits   disabled                    local
h     feature@large_blocks        disabled                    local

localhost ~ # zfs get all h/photos
NAME      PROPERTY              VALUE                                                       SOURCE
h/photos  type                  filesystem                                                  -
h/photos  creation              Sat Mar 21 20:06 2015                                       -
h/photos  used                  13.9T                                                       -
h/photos  available             85.7G                                                       -
h/photos  referenced            13.0T                                                       -
h/photos  compressratio         1.00x                                                       -
h/photos  mounted               yes                                                         -
h/photos  quota                 none                                                        default
h/photos  reservation           none                                                        default
h/photos  recordsize            128K                                                        default
h/photos  mountpoint            /h/photos                                                   default
h/photos  sharenfs              fsid=25,rw=172.16.111.0/24,sec=sys,insecure,insecure_locks  received
h/photos  checksum              on                                                          default
h/photos  compression           lz4                                                         local
h/photos  atime                 off                                                         local
h/photos  devices               off                                                         local
h/photos  exec                  off                                                         local
h/photos  setuid                off                                                         local
h/photos  readonly              on                                                          temporary
h/photos  zoned                 off                                                         default
h/photos  snapdir               hidden                                                      default
h/photos  aclinherit            restricted                                                  default
h/photos  canmount              on                                                          default
h/photos  xattr                 on                                                          default
h/photos  copies                1                                                           default
h/photos  version               1                                                           -
h/photos  utf8only              off                                                         default
h/photos  normalization         none                                                        default
h/photos  casesensitivity       sensitive                                                   default
h/photos  vscan                 off                                                         default
h/photos  nbmand                off                                                         default
h/photos  sharesmb              off                                                         default
h/photos  refquota              none                                                        default
h/photos  refreservation        none                                                        default
h/photos  primarycache          all                                                         default
h/photos  secondarycache        all                                                         default
h/photos  usedbysnapshots       988G                                                        -
h/photos  usedbydataset         13.0T                                                       -
h/photos  usedbychildren        0                                                           -
h/photos  usedbyrefreservation  0                                                           -
h/photos  logbias               latency                                                     default
h/photos  dedup                 off                                                         default
h/photos  mlslabel              none                                                        default
h/photos  sync                  standard                                                    default
h/photos  refcompressratio      1.00x                                                       -
h/photos  written               234K                                                        -
h/photos  logicalused           13.9T                                                       -
h/photos  logicalreferenced     13.0T                                                       -
h/photos  filesystem_limit      none                                                        default
h/photos  snapshot_limit        none                                                        default
h/photos  filesystem_count      none                                                        default
h/photos  snapshot_count        none                                                        default
h/photos  snapdev               hidden                                                      default
h/photos  acltype               off                                                         default
h/photos  context               none                                                        default
h/photos  fscontext             none                                                        default
h/photos  defcontext            none                                                        default
h/photos  rootcontext           none                                                        default
h/photos  relatime              off                                                         default
h/photos  redundant_metadata    all                                                         default
h/photos  overlay               off                                                         default

I have this in my zpool history:
2016-04-26.06:53:29 zpool set feature@embedded_data=enabled h
But I haven't modified any data in the pool at all, so it has probably had no tangible effect.
I also set these on h/photos, just before I make the h/photos@send snapshot:
h/photos compression lz4 local
h/photos atime off local
h/photos devices off local
h/photos exec off local
h/photos setuid off local
These were all the default values (off), for all snapshots prior to that last snapshot.

All 43 snapshots on the pool "h" work without error, all 43 snapshots on the pool "l" don't return the "too many symbolic links" error message.

`localhost ~ # zdb -C l

MOS Configuration:
version: 5000
name: 'l'
state: 0
txg: 623065
pool_guid: 4946876290228094116
errata: 0
hostname: 'localhost'
vdev_children: 1
vdev_tree:
type: 'root'
id: 0
guid: 4946876290228094116
create_txg: 4
children[0]:
type: 'raidz'
id: 0
guid: 3883405302801716307
nparity: 1
metaslab_array: 35
metaslab_shift: 38
ashift: 12
asize: 40007743569920
is_log: 0
create_txg: 4
children[0]:
type: 'disk'
id: 0
guid: 560335158339124174
path: '/dev/disk/by-id/ata-WDC_WD80EFZX-68UW8N0_VKGU6V2X-part1'
whole_disk: 1
create_txg: 4
children[1]:
type: 'disk'
id: 1
guid: 17515656895715273874
path: '/dev/disk/by-id/ata-WDC_WD80EFZX-68UW8N0_VKH408MX-part1'
whole_disk: 1
DTL: 425
create_txg: 4
children[2]:
type: 'disk'
id: 2
guid: 8921389355845190819
path: '/dev/disk/by-id/ata-WDC_WD80EFZX-68UW8N0_VKH52T6X-part1'
whole_disk: 1
create_txg: 4
children[3]:
type: 'disk'
id: 3
guid: 13116042119914339975
path: '/dev/disk/by-id/ata-WDC_WD80EFZX-68UW8N0_VKHLNHZX-part1'
whole_disk: 1
create_txg: 4
children[4]:
type: 'disk'
id: 4
guid: 6003650805274103645
path: '/dev/disk/by-id/ata-WDC_WD80EFZX-68UW8N0_VKHNJ93X-part1'
whole_disk: 1
create_txg: 4
features_for_read:
com.delphix:embedded_data
com.delphix:hole_birth
space map refcount mismatch: expected 150 != actual 146

localhost ~ # zdb -C h
zdb: can't open 'h': No such file or directory
[That's unexpected...]

localhost ~ # zpool list
NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
h 21.8T 20.9T 825G - 0% 96% 1.00x DEGRADED -
l 36.2T 24.3T 11.9T - 37% 67% 1.00x ONLINE -

localhost ~ # zdb -d l
Dataset mos [META], ID 0, cr_txg 4, 356M, 672 objects
Dataset l/photos@May_20_2015 [ZPL], ID 632, cr_txg 550939, 8.96T, 1946909 objects
[etc...]
Dataset l/photos@Oct_19_2015 [ZPL], ID 861, cr_txg 601790, 12.0T, 2214649 objects
Dataset l/photos [ZPL], ID 218, cr_txg 69828, 13.2T, 2290873 objects
Dataset l/test [ZPL], ID 356, cr_txg 131705, 153K, 6 objects
Dataset l/new [ZPL], ID 42, cr_txg 75, 4.40T, 35274 objects
Dataset l/CDs [ZPL], ID 49, cr_txg 1607, 78.8G, 144 objects
Dataset l [ZPL], ID 21, cr_txg 1, 1.11T, 2351 objects
Verified large_blocks feature refcount is correct (1)
space map refcount mismatch: expected 150 != actual 146

localhost ~ # zdb -d h
zdb: can't open 'h': No such file or directory

`

That's strange, zdb isn't seeing the other pool. Is this another bug? "zdb -C" only lists the "l" pool and not the "h" pool. Also the "h" pool doesn't appear in "/etc/zfs/zpool.cache". Could this be because I imported "h" with:

zpool import -o readonly=on h

[JuliaVixen]

So, I just checked, and yes, when a pool is imported "readonly=on", it doesn't appear in zpool.cache and "zdb" can't see it. I'm going to create an issue for this if there isn't one already...

localhost ~ # zpool import -o readonly=on h
localhost ~ # zpool list
NAME   SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
h     21.8T  20.9T   825G         -     0%    96%  1.00x  ONLINE  -
l     36.2T  24.3T  11.9T         -    37%    67%  1.00x  ONLINE  -

localhost ~ # strings /etc/zfs/zpool.cache 
[There's just one pool there]

localhost ~ # zdb -C  
l:
    version: 5000
    name: 'l'
    state: 0
[etc etc, just this one pool]

[But, re-import read-write...]

localhost ~ # zpool export h
localhost ~ # zpool import h
localhost ~ # zdb -C
h:
    version: 5000
    name: 'h'
    state: 0
    txg: 5195339
    pool_guid: 5105680881284105628
    errata: 0
    hostname: 'localhost'
    vdev_children: 1
    vdev_tree:
        type: 'root'
        id: 0
        guid: 5105680881284105628
        children[0]:
            type: 'raidz'
            id: 0
            guid: 16056632966898540210
            nparity: 1
            metaslab_array: 34
            metaslab_shift: 37
            ashift: 12
            asize: 24004646141952
            is_log: 0
            create_txg: 4
            children[0]:
                type: 'disk'
                id: 0
                guid: 2091825546480782105
                path: '/dev/sdn1'
                whole_disk: 1
                DTL: 381
                create_txg: 4
            children[1]:
                type: 'disk'
                id: 1
                guid: 10983704478065269639
                path: '/dev/sdo1'
                whole_disk: 1
                DTL: 261
                create_txg: 4
            children[2]:
                type: 'disk'
                id: 2
                guid: 16892970404048100612
                path: '/dev/sdm1'
                whole_disk: 1
                DTL: 380
                create_txg: 4
    features_for_read:
        com.delphix:embedded_data

l:
    version: 5000
    name: 'l'
    state: 0
    txg: 623065
    pool_guid: 4946876290228094116
    errata: 0
[etc. etc.]
36.2T  24.3T  11.9T         -    37%    67%  1.00x  ONLINE  -
localhost ~ # zdb -d h
Dataset mos [META], ID 0, cr_txg 4, 354M, 680 objects
Dataset h/photos@2015_Jul_8 [ZPL], ID 330, cr_txg 1261118, 10.1T, 1910968 objects
[etc. etc].

[JuliaVixen]

I have updated to the Gentoo zfs-9999 package (as of May 17, 2016) and with this version I can no longer reproduce the "Too many symbolic links" error when I attempt to "ls" the ".zfs/snapshots" directory with an absolute path. As far as I know, nothing has changed about the configuration of my filesystem, except I made a new snapshot yesterday, but I haven't even written any data to this filesystem since I wrote this bug. The only things which have changed about the configuration of the zpool which this filesystem sits upon, is the creation of one or two new zfs filesystems, and I think I destroyed a filesystem, and I wrote a few TiB of data into the other zfs filesystems, and did some "zfs send"'s of all the filesystems.

The exact version of zfs-9999 I'm using is this one:
#4582 (comment)

[pivot69]

Im using zfs on ubuntu server 16.04.1 and I had the same issue with the symlink error when accessing the snapshots. I got the error after sending incremental snapshots from another ubuntu server (running ubuntu server 14.04).

After updating the affected server and trying everything in my mind (atime off, compression off, mountpoints etc) it still did not work. I did a reboot and suddenly everything worked again - until I transferred new incremental snapshots.

This led me to try unmounting and remounting the filesystem after each time I transferred snapshots, and that seemed to do the trick! Now I just put the remount-commands into my script, and I am no longer bothered by this bug.

This is not a fix, it is a only workaround. But in case someone cannot get it working, even with the newest versions of everything, then try this! :)

Just some additional info:
The ubuntu server sending snapshots (14.04) has the ubuntu-zfs package installed.
[ 1.570547] ZFS: Loaded module v0.6.5.7-1~trusty, ZFS pool version 5000, ZFS filesystem version 5
The ubuntu server receiving snapshots (16.04.1) has zfs native
[ 17.440504] ZFS: Loaded module v0.6.5.6-0ubuntu10, ZFS pool version 5000, ZFS filesystem version 5

[zffocussss]

I have same error messages when using autofs + docker volume mount

[jgoerzen]

I encountered this when attempting to access a snapshot under /usr/.zfs/snapshots on my system. On this sytem, I have filesystems like this:

tank/hephaestus-1                    20.2G   135G       96K  /tank/hephaestus-1
tank/hephaestus-1/ROOT               15.9G  8.78G     1.22G  /
tank/hephaestus-1/ROOT/opt           1.51G   135G      823M  /opt
tank/hephaestus-1/ROOT/usr           12.4G  9.47G     10.5G  /usr
tank/hephaestus-1/var                4.35G  6.96G     3.04G  legacy

There are no bind mounts involved.

The relevant snapshots would have been created with zfs snapshot -r tank@foo. I observed df showing directories under /.zfs/snapshot -- showing snapshots of tank/hephaestus-1/ROOT. It did not make the /usr snapshot available in any way.

[helamonster]

I have just encountered this for the first time myself.
My environment:

root@myserver ~ # lsb_release  -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.2 LTS
Release:        20.04
Codename:       focal

root@myserver ~ # uname -a
Linux myserver.mydomain.com 5.4.0-66-generic #74-Ubuntu SMP Wed Jan 27 22:54:38 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

root@myserver ~ # zfs version
zfs-2.0.3-0york0~20.04
zfs-kmod-2.0.3-0york0~20.04

Observe:

root@myserver ~ # pwd
/root

root@myserver ~ # ls -l  /var/.zfs/snapshot/zrepl_20211116_231549_000/
ls: cannot access '/var/.zfs/snapshot/zrepl_20211116_231549_000/': Too many levels of symbolic links

root@myserver ~ # cd   /var/.zfs/snapshot/zrepl_20211116_231549_000/
-bash: cd: /var/.zfs/snapshot/zrepl_20211116_231549_000/: Too many levels of symbolic links

root@myserver ~ # cd /var/.zfs

root@myserver .zfs # cd snapshot/

root@myserver snapshot # cd zrepl_20211116_231549_000
-bash: cd: zrepl_20211116_231549_000: Too many levels of symbolic links

root@myserver snapshot # mount.zfs rpool/ROOT/ubuntu/var@zrepl_20211116_231549_000 /var/.zfs/snapshot/zrepl_20211116_231549_000/  
filesystem 'rpool/ROOT/ubuntu/var@zrepl_20211116_231549_000' is already mounted

Interestingly, I found that the snapshot is indeed mounted but at a different level in the filesystem (/.zfs/snapshot instead of /var/.zfs/snapshot). Strange...

root@myserver snapshot # mount.zfs rpool/ROOT/ubuntu/var@zrepl_20211116_231549_000 /mnt/var
filesystem 'rpool/ROOT/ubuntu/var@zrepl_20211116_231549_000' is already mounted

root@myserver snapshot # mount | grep zrepl
rpool/ROOT/ubuntu/var@zrepl_20211116_231549_000 on /.zfs/snapshot/zrepl_20211116_231549_000 type zfs (ro,relatime,xattr,posixacl)

root@myserver snapshot # ls /.zfs/snapshot/zrepl_20211116_231549_000
backups  cache  empty  lib  local  lock  log  mail  opt  run  snap  spool  tmp

root@myserver snapshot # zfs list -r rpool -o name,used,avail,refer,canmount,mounted,mountpoint
NAME                    USED  AVAIL     REFER  CANMOUNT  MOUNTED  MOUNTPOINT
rpool                  30.9G   182G       96K  off       no       none
rpool/ROOT             30.8G   182G       96K  off       no       none
rpool/ROOT/ubuntu      30.8G   182G     2.88G  on        yes      /
rpool/ROOT/ubuntu/tmp  48.3M   182G      440K  on        yes      /tmp
rpool/ROOT/ubuntu/var  27.9G   182G     11.7G  on        yes      /var
rpool/temp               96K   182G       96K  on        yes      /temp

I can confirm that trying to access the correct .zfs/snapshot directory causes the snapshot to be mounted at the parent directory's .zfs/snapshot directory instead. Odd.

root@myserver ~ # mount | grep 'rpool.*zfs'
rpool/ROOT/ubuntu on / type zfs (rw,relatime,xattr,posixacl)
rpool/ROOT/ubuntu/tmp on /tmp type zfs (rw,relatime,xattr,posixacl)
rpool/ROOT/ubuntu/var on /var type zfs (rw,relatime,xattr,posixacl)
rpool/temp on /temp type zfs (rw,noatime,xattr,posixacl)
rpool/ROOT/ubuntu/var@zrepl_20211116_231549_000 on /.zfs/snapshot/zrepl_20211116_231549_000 type zfs (ro,relatime,xattr,posixacl)

root@myserver ~ # umount /.zfs/snapshot/zrepl_20211116_231549_000

root@myserver ~ # mount | grep 'rpool.*zfs'
rpool/ROOT/ubuntu on / type zfs (rw,relatime,xattr,posixacl)
rpool/ROOT/ubuntu/tmp on /tmp type zfs (rw,relatime,xattr,posixacl)
rpool/ROOT/ubuntu/var on /var type zfs (rw,relatime,xattr,posixacl)
rpool/temp on /temp type zfs (rw,noatime,xattr,posixacl)

root@myserver ~ # ls /var/.zfs/snapshot/zrepl_20211116_231549_000
ls: cannot open directory '/var/.zfs/snapshot/zrepl_20211116_231549_000': Too many levels of symbolic links

root@myserver ~ # mount | grep 'rpool.*zfs'
rpool/ROOT/ubuntu on / type zfs (rw,relatime,xattr,posixacl)
rpool/ROOT/ubuntu/tmp on /tmp type zfs (rw,relatime,xattr,posixacl)
rpool/ROOT/ubuntu/var on /var type zfs (rw,relatime,xattr,posixacl)
rpool/temp on /temp type zfs (rw,noatime,xattr,posixacl)
rpool/ROOT/ubuntu/var@zrepl_20211116_231549_000 on /.zfs/snapshot/zrepl_20211116_231549_000 type zfs (ro,relatime,xattr,posixacl)

Nothing is written to dmesg, syslog, or /proc/spl/kstat/zfs/dbgmsg as a result of executing these failing commands.

I can access files under the snapshot where it was mounted (/.zfs/snapshot/zrepl_20211116_231549_000) just fine.

[parke]

Fyi, it appears this issue may be related to #9958.

[rptb1]

Just another "me too", since this issue is marked closed but is still happening.

$ uname -a
Linux plover 5.15.0-56-generic #62-Ubuntu SMP Tue Nov 22 19:54:14 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 22.04.1 LTS
Release:	22.04
Codename:	jammy
$ dpkg-query -l 'zfs*' | grep ii
ii  zfs-initramfs  2.1.4-0ubuntu0.1 amd64        OpenZFS root filesystem capabilities for Linux - initramfs
ii  zfs-zed        2.1.4-0ubuntu0.1 amd64        OpenZFS Event Daemon
ii  zfsutils-linux 2.1.4-0ubuntu0.1 amd64        command-line tools to manage OpenZFS filesystems
$ 
$ ls -ld /var/lib/.zfs/snapshot/backup-pelican-2022-11-13-09-53-19
ls: cannot access '/var/lib/.zfs/snapshot/backup-pelican-2022-11-13-09-53-19': Too many levels of symbolic links

The system is set up according to https://openzfs.github.io/openzfs-docs/Getting%20Started/Ubuntu/Ubuntu%2022.04%20Root%20on%20ZFS.html

It's not consistent for all filesystems:

$ zfs list -H -o mountpoint | grep '^/' | sort -u | while read m; do sudo ls -ld "$m/.zfs/snapshot/backup-pelican-2022-11-13-09-53-19"; done
drwx------ 2 root root 30 Nov 13 09:48 //.zfs/snapshot/backup-pelican-2022-11-13-09-53-19
ls: cannot access '/boot/.zfs/snapshot/backup-pelican-2022-11-13-09-53-19': No such file or directory
drwxr-xr-x 28 rb rb 46 Nov 12 19:55 /home/rb/.zfs/snapshot/backup-pelican-2022-11-13-09-53-19
ls: cannot access '/home/rb/.cache/.zfs/snapshot/backup-pelican-2022-11-13-09-53-19': No such file or directory
ls: cannot access '/home/rb/private/.zfs/snapshot/backup-pelican-2022-11-13-09-53-19': No such file or directory
ls: cannot access '/home/rb/snap/firefox/common/.cache/.zfs/snapshot/backup-pelican-2022-11-13-09-53-19': No such file or directory
ls: cannot access '/home/rb/tmp/.zfs/snapshot/backup-pelican-2022-11-13-09-53-19': No such file or directory
drwx------ 9 root root 16 Nov  8 14:43 /root/.zfs/snapshot/backup-pelican-2022-11-13-09-53-19
ls: cannot access '/tmp/.zfs/snapshot/backup-pelican-2022-11-13-09-53-19': No such file or directory
ls: cannot access '/usr/.zfs/snapshot/backup-pelican-2022-11-13-09-53-19': No such file or directory
ls: cannot access '/var/.zfs/snapshot/backup-pelican-2022-11-13-09-53-19': No such file or directory
ls: cannot access '/var/cache/.zfs/snapshot/backup-pelican-2022-11-13-09-53-19': No such file or directory
ls: cannot access '/var/lib/.zfs/snapshot/backup-pelican-2022-11-13-09-53-19': Too many levels of symbolic links
ls: cannot access '/var/lib/AccountsService/.zfs/snapshot/backup-pelican-2022-11-13-09-53-19': Too many levels of symbolic links
ls: cannot access '/var/lib/docker/.zfs/snapshot/backup-pelican-2022-11-13-09-53-19': Too many levels of symbolic links
ls: cannot access '/var/lib/NetworkManager/.zfs/snapshot/backup-pelican-2022-11-13-09-53-19': Too many levels of symbolic links
ls: cannot access '/var/log/.zfs/snapshot/backup-pelican-2022-11-13-09-53-19': Too many levels of symbolic links
ls: cannot access '/var/snap/.zfs/snapshot/backup-pelican-2022-11-13-09-53-19': Too many levels of symbolic links
ls: cannot access '/var/spool/.zfs/snapshot/backup-pelican-2022-11-13-09-53-19': Too many levels of symbolic links
ls: cannot access '/var/tmp/.zfs/snapshot/backup-pelican-2022-11-13-09-53-19': No such file or directory

I noticed a large number of mounts of snapshots:

$ mount | grep 'type zfs' | grep '@' | wc -l
76

Perhaps there some resource limit that's being exceeded?

I frequently browse lists of snapshots in Emacs dired, which will be doing at least ls -l on .zfs/snapshot. I wonder if that triggers a lot of mounts at once?

EDIT: Later on, I'm getting messages from zfs-auto-snapshot like this:

cannot destroy snapshot kiwi-rpool/ROOT/ubuntu_8lgjho/var/lib@zfs-auto-snap_frequent-2023-01-04-1100: dataset is busy
cannot destroy snapshot kiwi-rpool/ROOT/ubuntu_8lgjho/var/lib@zfs-auto-snap_frequent-2023-01-04-1045: dataset is busy

Notice that these are in the same filesystem as the original problem. Does this also suggest a problem with automatic mount/unmount?

Can someone point me at whatever code/daemon/agent is responsible for this mounting and unmounting of snapshots? Perhaps I can figure it out.

[rptb1]

Can someone point me at whatever code/daemon/agent is responsible for this mounting and unmounting of snapshots? Perhaps I can figure it out.

Digging deeper, I have bogus entries like this in /etc/mtab:

kiwi-rpool/ROOT/ubuntu_8lgjho/var/lib@backup-pelican-2022-11-13-09-53-19 /.zfs/snapshot/backup-pelican-2022-11-13-09-53-19 zfs ro,relatime,xattr,posixacl 0 0

Notice that this is a snapshot for filesystem kiwi-rpool/ROOT/ubuntu_8lgjho/var/lib but look at where it's mounted -- it's mounted at /.zfs in the wrong place. (Something like this was mentioned by #4514 (comment) .)

And if I unmount it from there, then the problem is fixed:

rb@plover:/var/lib/.zfs/snapshot$ sudo umount /.zfs/snapshot/backup-pelican-2022-12-15-07-36-11
rb@plover:/var/lib/.zfs/snapshot$ ls -ld /.zfs/snapshot/backup-pelican-2023-01-03-08-08-09
drwxr-xr-x 79 root root 81 Dec 11 08:28 /.zfs/snapshot/backup-pelican-2023-01-03-08-08-09
rb@plover:/var/lib/.zfs/snapshot$ 

But even more weirdness, I can get the wrong directory contents to appear at /.zfs/snapshot:

$ ls -l /.zfs/snapshot/backup-pelican-2022-12-15-07-36-11
[shows correctly the things I expect to see in / ]
$ mount | grep backup-pelican-2023-01-03-08-08-09
kiwi-rpool/ROOT/ubuntu_8lgjho/var/lib@backup-pelican-2023-01-03-08-08-09 on /.zfs/snapshot/backup-pelican-2023-01-03-08-08-09 type zfs (ro,relatime,xattr,posixacl)
$ ls -l /.zfs/snapshot/backup-pelican-2022-12-15-07-36-11
[shows incorrectly the contents of /var/lib]

So it seems to me that there is something very broken about the automounting of snapshots. They're mounting in /.zfs and ignoring their filesystem mountpoint, giving incorrect contents, and probably causing the symbolic link problem.

EDIT: I can get correct and incorrect contents in the same location like this:

rb@plover:/.zfs/snapshot$ ls backup-pelican-2023-01-03-08-08-09
AccountsService      flatpak         [and more contents of /var/lib]
rb@plover:/.zfs/snapshot$ sudo umount backup-pelican-2023-01-03-08-08-09
rb@plover:/.zfs/snapshot$ ls backup-pelican-2023-01-03-08-08-09
bin   etc  [and more contents of /]

Does this warrant another issue? Showing the wrong contents at the /.zfs/snapshot path seems like it might be.

[rptb1]

Possibly relevant

zfs/module/os/linux/zfs/zfs_ctldir.c

Lines 60 to 62 in c935fe2

	* All mounts are handled automatically by an user mode helper which invokes
	* the mount procedure. Unmounts are handled by allowing the mount
	* point to expire so the kernel may automatically unmount it.

and I note that this is different to the FreeBSD module.

The call to the user mode helper (mount.zfs) is constructed here:

zfs/module/os/linux/zfs/zfs_ctldir.c

Lines 1095 to 1102 in c935fe2

	/*
	* Construct a mount point path from sb of the ctldir inode and dirent
	* name, instead of from d_path(), so that chroot'd process doesn't fail
	* on mount.zfs(8).
	*/
	snprintf(full_path, MAXPATHLEN, "%s/.zfs/snapshot/%s",
	zfsvfs->z_vfs->vfs_mntpoint ? zfsvfs->z_vfs->vfs_mntpoint : "",
	dname(dentry));

Note that this gives up and uses "" in the conditional, which might cause everything to be mounted at /.zfs. The gun is smoking. I am suspicious!

[rptb1]

Note that this gives up and uses "" in the conditional, which might cause everything to be mounted at /.zfs. The gun is smoking. I am suspicious!

I have tried to confirm this is happening by enabling debug logging in the hope that I would see a message from

zfs/module/os/linux/zfs/zfs_ctldir.c

Line 1125 in c935fe2

dprintf("mount; name=%s path=%s\n", full_name, full_path);

but I suspect my zfs module is not built with ZFS_DEBUG and so this is disabled.

In case someone with a debug build can do it, here is what I tried:

cd /var/lib/.zfs/snapshot
umount * /.zfs/snapshot/*
# clear debug log
echo 0 > /proc/spl/kstat/zfs/dbgmsg
# enable debug messages
echo 1 >> /sys/module/zfs/parameters/zfs_flags
# provoke "too many symbolic links"
ls backup-pelican-2023-01-03-08-08-09
# stop debug messages
echo 0 >> /sys/module/zfs/parameters/zfs_flags
# examine debug log
cat /proc/spl/kstat/zfs/dbgmsg

I see messages from zfs_dbgmsg calls but not dprintf.

[nachtgeist]

So it seems to me that there is something very broken about the automounting of snapshots. They're mounting in /.zfs and ignoring their filesystem mountpoint, giving incorrect contents, and probably causing the symbolic link problem.

Yup. Got another case:

• Debian Bullseye with ZFS 2.1.7, systemd
• rootfs on ZFS
• no bind mounts
• zfs get mountpoint -r tank -t filesystem gives:

NAME                        PROPERTY    VALUE               SOURCE
tank                        mountpoint  /                   local
tank/home                   mountpoint  /home               inherited from tank
tank/opt                    mountpoint  /opt                inherited from tank
tank/rootfs                 mountpoint  /                   local
tank/root                   mountpoint  /root               inherited from tank
tank/srv                    mountpoint  /srv                inherited from tank

Listing and auto-mounting of snapshots via $MNTPOINT/.zfs/snapshot/* works everywhere EXCEPT in /.
When I realized the connection explained below, there were snapshots with identical names present for different datasets - date-time-strings, actually - which had
been created by sanoid via cron. So let's say there are identically named snapshots of tank/rootfs and tank/root like so:

tank/rootfs@a
tank/rootfs@b

tank/root@a
tank/root@b

This leads to:

• Acessing /root/.zfs/snapshot/* works, i.e. ls, cd, cat, cp...
• let's do an umount /root/.zfs/snapshot/* (It'll become clear why in a second)
• ls /.zfs/snapshot/ gives the dreaded "too many levels of symbolic links"...but oh look!
• cd /root/.zfs/snapshot/a; ls now displays the contents of snapshot "a" of tank/rootfs@a, NOT tank/root@a!

This got me thinking and I tinkered with the init script from initramfs-tools and just applied this patch:

diff -Naur /usr/share/initramfs-tools/init.orig /usr/share/initramfs-tools/init
--- /usr/share/initramfs-tools/init.orig        2023-01-13 02:30:14.717469685 +0100
+++ /usr/share/initramfs-tools/init     2023-01-13 02:30:09.565173985 +0100
@@ -57,7 +57,7 @@
 export break=
 export init=/sbin/init
 export readonly=y
-export rootmnt=/root
+export rootmnt=/rfs
 mkdir "$rootmnt"
 export debug=
 export panic=

The only thing this changes is the name of the mount point where the rootfs-on-zfs gets initially mounted during boot by the initramfs. Re-build the initramfs and reboot...

Now, ls /.zfs/snapshot nicely lists a and b, ls /.zfs/snapshot/* yields empty directories, however. And nothing got mounted.

Let's mkdir -p /rfs/.zfs/snapshot and ls /.zfs/snapshot/a.

No error printed, again empty directories and nothing got mounted.

Now let's mkdir /rfs/.zfs/snapshot/a and ls /.zfs/snapshot/a again yields "ls: cannot access '/.zfs/snapshot/b': Too many levels of symbolic links",

BUT

ls /rfs/.zfs/snapshot/a shows the expected snapshot content.

Under /rfs. The mountpoint where the initramfs's init mounted tank/rfs to before pivot-rooting it to /.

Note: This did NOT happen on a rootfs-on-zfs setup on Debian Buster with ZFS 2.0.3 from buster-backports. However, that buster machine still ran sysvinit instead of systemd.

I'll try the steps described here on bullseye-sysvinit and buster-systemd machines in the coming days...
#9461 (comment) seems to describe the root cause

[rptb1]

I'm not sure why this issue is still closed. It's clearly not fixed.

[cjthompson]

I'm not sure why this issue is still closed. It's clearly not fixed.

I just got this error today as well (zfs v2.1.9-2ubuntu1)

[ceastus]

I just upgraded from Ubuntu 22.04 to 23.04 (reusing my pool) and I see the same issue, just as nachtgeist described.
ZFS 2.1.9-2ubuntu1.1
6.2.0-23-generic #23-Ubuntu

Listing and auto-mounting of snapshots via $MNTPOINT/.zfs/snapshot/* works everywhere EXCEPT in /.

The snaps for / are mounted under /root/.zfs/snapshot

root@xander:/.zfs# ls -l snapshot/pre-rebuild-2023-06-22/
ls: cannot access 'snapshot/pre-rebuild-2023-06-22/': Too many levels of symbolic links
2 root@xander:/.zfs# mount |grep xander.ubuntu
xander/ubuntu on / type zfs (rw,nodev,noatime,xattr,posixacl)
xander/ubuntu@pre-rebuild-2023-06-22 on /root/.zfs/snapshot/pre-rebuild-2023-06-22 type zfs (ro,relatime,xattr,posixacl)

I don't see this issue when booting from the ISO, so that meshes.
I can manually mount snaps to the correct locations.

[GregorKopka]

@ahrens @behlendorf
please reopen.

[behlendorf]

Reopening. The issue here is further described in #9461 (comment)

[mariaa144]

I had faced this problem on NixOS using ZFS as root. It was upsetting when I couldn't get to my snapshots quickly to restore my Firefox session I accidentally deleted.

A simple work around which allowed me to access my snapshots is to manually mount the snapshot in another directory, instead of using the .zfs directory.

I did the following:

# Find my snapshot
zfs list -t snapshot

# create a mount directory
sudo mkdir /mnt/snapshot_test

# mount the snapshot
sudo mount -t zfs rpool/nixos/home@zfs-auto-snap_daily-2024-04-16-08h01 /mnt/snapshot_test/

This allowed me to copy my files from the snapshot in the directory /mnt/snapshot_test.

[pbek]

I have the same issue when I try to use a .zfs/snapshot as source for creating a backup with restic inside a docker container. The snapshots were done by sanoid on NixOS.

This e.g. also yields the error inside the docker container:

# "/backup" is the mount inside the docker container
ls /backup/home/.zfs/snapshot/autosnap_2024-05-04_11:00:06_hourly

Outside the docker container, this doesn't yield the error:

ls /home/.zfs/snapshot/autosnap_2024-05-04_11:00:06_hourly

[maxz]

I'm still experiencing this.

I'm not using Docker or any other containerization.

Kernel 6.10.14, ZFS 2.2.6.

[amotin]

The problem might not have an easy solution inside a container. Otherwise see #16646 (it should be slightly updated soon).

[maxz]

I'm eager to test it, but I'm not using an initrd to mount my ZFS volume and it is not mounted as the root volume either.
So I have some doubts whether this will fix my issue.