Infinite loop with multiprocessing

[samtygier-stfc]

Describe the bug
I have hit this when using pyfakefs with a test that uses multiprocessing.shared_memory.

platform linux -- Python 3.10.13, pytest-8.0.0, pluggy-1.4.0 -- /home/sam/mambaforge/envs/test/bin/python3.10
cachedir: .pytest_cache
rootdir: /home/sam
plugins: pyfakefs-5.3.5
collecting ... collected 1 item
test_proc2.py::ExampleTestCase::test_shared_memory Traceback (most recent call last):
  File "/home/sam/mambaforge/envs/test/lib/python3.10/multiprocessing/resource_tracker.py", line 199, in main
    cmd, name, rtype = line.strip().decode('ascii').split(':')
UnicodeDecodeError: 'ascii' codec can't decode byte 0xed in position 3: ordinal not in range(128)
Traceback (most recent call last):
  File "/home/sam/mambaforge/envs/test/lib/python3.10/multiprocessing/resource_tracker.py", line 199, in main
    cmd, name, rtype = line.strip().decode('ascii').split(':')
UnicodeDecodeError: 'ascii' codec can't decode byte 0xf4 in position 0: ordinal not in range(128)
Traceback (most recent call last):
  File "/home/sam/mambaforge/envs/test/lib/python3.10/multiprocessing/resource_tracker.py", line 199, in main
    cmd, name, rtype = line.strip().decode('ascii').split(':')
UnicodeDecodeError: 'ascii' codec can't decode byte 0xde in position 1: ordinal not in range(128)

This then requires a SIGKILL to stop.

This is probably related to #570

How To Reproduce
I can reproduce with:

from pyfakefs.fake_filesystem_unittest import TestCase
from multiprocessing import shared_memory

class ExampleTestCase(TestCase):
    def setUp(self):
        self.setUpPyfakefs()

    def test_shared_memory(self):
        mem = shared_memory.SharedMemory(name="test", create=True, size=128)
        mem.close()
        mem.unlink()

Your environment

Linux-6.5.0-17-generic-x86_64-with-glibc2.35
Python 3.10.13 | packaged by conda-forge | (main, Dec 23 2023, 15:36:39) [GCC 12.3.0]
pyfakefs 5.3.5
pytest 8.0.0

[mrbean-bremen]

Unfortunately, this won't work, at least not out of the box, due to the limitations of pyfakefs. Multiprocessing is one of the modules not playing nice with pyfakefs due to the use of system C functions to access the file system. In this case, it uses posixshmem, which is a C module, to access the shared memory. This will work in the real filesystem, so either the file is not found, or if the file exists there, a real file descriptor is returned, which is not found in the fake filesystem.

I could not reproduce the inifinite loop so far, but I got another error related to the mentioned problem. Currently, I don't have a solution for this (other than patching it away).

[samtygier-stfc]

Thanks for having a look.
Is there a way to allow this to pass through to the real filesystem? I tried self.fs.add_real_directory("/dev/shm/",read_only=False) which did not help.
It looks like the point in resource_tracker.py where this fails is with a pipe in a spawned process.

[mrbean-bremen]

Not really. The problem is the mixture of faked on non-faked calls, e.g. the usage of faked descriptors in the real fs and vice verse. Even if you would map this into the fake fs, the returned descriptors would still be from the real fs (due to the C calls), which do not exist in the fake fs (or if they do exist, they point to something else, which could lead to all kinds of undefined behavior).
Sometimes it makes sense to pause patching in a test, execute something in the real fs, and resume patching again, but this won't work in this case.
It could be possible to write a fake version of posixshmem and patch the usage accordingly, but I think this wouldn't work either because of other problems in multiprocessing. The biggest problem is probably that the fake fs will not work in spawed processes in principle - and as these are the point of multiprocessing, I see no real way to fix this - sorry!