Cannot import file from fakefs

[Aran-Fey]

Describe the bug
Files that only exist on the fake file system cannot be imported, neither with a regular import statement nor via importlib.

How To Reproduce

import pytest

import importlib.util
import sys
from pathlib import Path


def test_import(fs):
    module_path = Path('heya.py')
    module_path.write_text('x = 1234')
    
    spec = importlib.util.spec_from_file_location('heya', module_path)
    module = importlib.util.module_from_spec(spec)

    sys.modules['heya'] = module
    spec.loader.exec_module(module)  # Crashes with FileNotFoundError


if __name__ == '__main__':
    pytest.main([__file__])

Your environment

Windows-11-10.0.22631-SP0
Python 3.12.0 (tags/v3.12.0:0fb18b0, Oct 2 2023, 13:03:39) [MSC v.1935 64 bit (AMD64)]
pyfakefs 5.7.1
pytest 7.3.2

[mrbean-bremen]

The reason for this is that we usually want to be able to import real modules in tests - most tests depend on this.

That being said, there is a possibility to patch open_code (which is the function used to load code during import) via the patch-open-code argument. The default setting is OFF, as this is what the vast maojority of tests will need, but you could also set it to AUTO (any module found in the fake filesystem is opend from there) or ON (all open_code calls are patched). For your case (and most cases that do something like that), AUTO is probably the better solution, e.g. something like this:

import importlib.util
import sys
from pathlib import Path

import pytest

from pyfakefs.fake_filesystem import PatchMode
from pyfakefs.fake_filesystem_unittest import Patcher

@pytest.fixture
def fs_patch_open():
    with Patcher(patch_open_code=PatchMode.AUTO) as p:
        yield p.fs

def test_import(fs_patch_open):
    module_path = Path('heya.py')
    module_path.write_text('x = 1234')
...

should work for you.

Note that this is not a common use case and basically not tested - I would not exclude all sorts of problems and side effects if using this for tests in a larger code base.

[Aran-Fey]

That works, thanks. Unfortunately I've had to rewrite my code to use importlib.import_module, and that still doesn't work even with PatchMode.ON:

def test_import(fs_patch_open):
    module_path = Path('heya.py')
    module_path.write_text('x = 1234')
    
    sys.path.insert(0, str(module_path.parent))
    importlib.import_module(module_path.stem)  # throws ModuleNotFoundError

Is there an easy workaround for this as well?

[mrbean-bremen]

Sorry, I see no way to fix this easily. The other one could work because you have manually inserted the module into sys.modules, but in this case the import mechanism will try to load the module directly.

In principle, it could be possible to add the code that you used in your first example, e.g. this part:

    spec = importlib.util.spec_from_file_location('heya', module_path)
    module = importlib.util.module_from_spec(spec)
    sys.modules['heya'] = module

in the dynamic patcher in case patch_open is switched on. I'm not sure if this would defy the purpose of the test, but it probably would behave like the first test.
I'm also not sure of the possible implications if circumventing the import mechanism this way, but I can see that dynamically generating and importing modules may be a use case worth testing (though uncommon).

What do you think? If that would make sense for you, I could have a go at it.

[Aran-Fey]

I actually realized that I don't need to use import_module after all. It's my first time implementing dynamic imports, and I'm figuring things out as I go, sorry!

That said, it would be great if it was possible to make this work with the import statement. This is because the module I'm importing may want to import a sibling module, like this:

def test_import(fs):
    module_path = Path('heya.py')
    module_path.write_text('import aloha')

    Path('aloha.py').write_text('x = 1234')
    
    sys.path.insert(0, str(module_path.parent))
    importlib.import_module(module_path.stem)

I'm also not too familiar with the internals of the import mechanism, and I'm sure there may be minor differences if a part of it is monkeypatched, but so far this code has emulated regular imports closely enough for me. If it's not too much trouble, please give it a shot.

[mrbean-bremen]

Sure, I'll see what I can do. No promises though...