Fail with "Error: No such file or directory (os error 2)"

[ikus060]

I'm trying to use py-spy on a python process already running py-spy dump -p 219292.

That process make use of user name space. That probably explain why py-spy is failing.

1. I'm question if it's possible to use py-spy cross namespace ?
2. Would it be possible to handle the exception and continue the "spying" event if some file are not found ?

Running with strace:

# strace -e trace=open py-spy dump -p 219292
open("/proc/219292/maps", O_RDONLY|O_CLOEXEC) = 3
open("/proc/219292/exe", O_RDONLY|O_CLOEXEC) = 3
open("/proc/219292/root/tmp/minarca-jail-ku6ln2cg/opt/rdiff-backup-2.0/libpython3.7m.so.1.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
Error: No such file or directory (os error 2)
+++ exited with 1 +++

/tmp/minarca-jail-ku6ln2cg/opt/rdiff-backup-2.0/libpython3.7m.so.1.0 doesn't exists. It's probably only mounted in the namespace

[Jongy]

py-spy accesses the files in the mount (mount, not user) namespace of the target process (that's the /proc/219292/root prefix). It works just fine for containerized processes.

In this case py-spy decided that libpython.so is needed (because the Python is built with --enable-shared, and thus the core of Python is implemented in libpython.so, not in the executable). The file is indeed needed.
Any chance the file is deleted? Can you please check inside the container?

[ikus060]

Not exists "/proc/219292/root/tmp/minarca-jail-ku6ln2cg/opt/rdiff-backup-2.0/libpython3.7m.so.1.0"
Exists "/proc/282195/root/opt/rdiff-backup-2.0/libpython3.7m.so.1.0"

"/tmp/minarca-jail-ku6ln2cg" the the root of the container

Looking at /proc/:

exe is a symlink to the executable.
exe -> /tmp/minarca-jail-g5905ukq/opt/rdiff-backup-2.0/rdiff-backup

root is also a symlink to a path.
root -> /tmp/minarca-jail-g5905ukq

So the real file is

sharelib = /proc/219292/root/tmp/minarca-jail-ku6ln2cg/opt/rdiff-backup-2.0/libpython3.7m.so.1.0
root ="/tmp/minarca-jail-g5905ukq"
/proc/282195/root/ + (sharelib[len(root):])

[ikus060]

@Jongy Is the information provided help you write a fix ?

[Jongy]

Not quite yet but we're getting there.

py-spy attempts to access the path based on what it reads from /proc/pid/maps which contains the absolute path of the libpython, so no symlink resolving mistakes should / could occur. What might happens here, based on root pointing to non /, is py-spy mishandling a chrooted container. Possibly the correct handling is to remove the prefix of root from the path, if the maps file gives it when reading maps. I was under the impression it will not.

Let's try to walk it one by one - to have all data. Please run all these commands so we can see the exact files & directories along the way:

ls -ld /proc/219292/cwd/
ls -ld /proc/219292/root/
ls -ld /proc/219292/root/opt/rdiff-backup-2.0/
ls -ld /proc/219292/root/opt/rdiff-backup-2.0/libpython3.7m.so.1.0
ls -ld /proc/219292/root/tmp/
ls -ld /proc/219292/root/tmp/minarca-jail-ku6ln2cg/
ls -ld /proc/219292/root/tmp/minarca-jail-ku6ln2cg/opt/
ls -ld /proc/219292/root/tmp/minarca-jail-ku6ln2cg/opt/rdiff-backup-2.0/
ls -ld /proc/219292/root/tmp/minarca-jail-ku6ln2cg/opt/rdiff-backup-2.0/libpython3.7m.so.1.0

replace 219292 with the PID of the process, if it changed.

[ikus060]

# ls -ld /proc/982626/cwd/
    drwx------ 10 minarca minarca 10 Mar  6 07:40 /proc/982626/cwd/
# ls -ld /proc/982626/root/
    drwx------ 10 minarca minarca 10 Mar  6 07:40 /proc/982626/root/
# ls -ld /proc/982626/root/opt/rdiff-backup-2.0/
    drwxr-xr-x 5 root root 22 Jan 19 16:36 /proc/982626/root/opt/rdiff-backup-2.0/
# ls -ld /proc/982626/root/opt/rdiff-backup-2.0/libpython3.7m.so.1.0
    -rw-r--r-- 1 root root 3437576 Dec 20 06:53 /proc/982626/root/opt/rdiff-backup-2.0/libpython3.7m.so.1.0
# ls -ld /proc/982626/root/tmp/
    ls: cannot access '/proc/982626/root/tmp/': No such file or directory
# ls -ld /proc/982626/root/tmp/minarca-jail-*/
    ls: cannot access '/proc/982626/root/tmp/minarca-jail-*/': No such file or directory
# ls -ld /proc/982626/root/tmp/minarca-jail-*/opt/
    ls: cannot access '/proc/982626/root/tmp/minarca-jail-*/opt/': No such file or directory
# ls -ld /proc/982626/root/tmp/minarca-jail-*/opt/rdiff-backup-2.0/
    ls: cannot access '/proc/982626/root/tmp/minarca-jail-*/opt/rdiff-backup-2.0/': No such file or directory
# ls -ld /proc/982626/root/tmp/minarca-jail-*/opt/rdiff-backup-2.0/libpython3.7m.so.1.0
    ls: cannot access '/proc/982626/root/tmp/minarca-jail-*/opt/rdiff-backup-2.0/libpython3.7m.so.1.0': No such file or directory

Here the extra command I also ran:

root@ranculos:~# ls -la /proc/982626/root
lrwxrwxrwx 1 minarca minarca 0 Mar  6 07:42 /proc/982626/root -> /tmp/minarca-jail-okxuc76l

[Jongy]

I managed to reproduce the problem locally. Interesting.

Possibly the correct handling is to remove the prefix of root from the path, if the maps file gives it when reading maps. I was under the impression it will not.

This is the fix that needs to be implemented in py-spy. It can be generalized to - instead of stripping the / that is usually the readlink(/proc/pid/root), you need to strip the entire readlink result, because that part is included in the /proc/pid/maps when viewed outside of the process.

I might have the time to fix it in py-spy soon but cannot promise.

[Jongy]

I implemented a fix, pushed a PR - @ikus060 please try building it and see if it solves your problem :) I explained in the PR the steps I took and the way I see that it solved the issue.

[ikus060]

Ho, I forgot about this issue.
Will try to test this week

[ikus060]

Sorry for the long delay, this issue slip my mind and I only need py-spy once in a while.

Thans changes you made seams accurate with my analysis too and is working for my requirement.

Thanks