python 3.9.18 + pyfakefs + pathlib + class variable: test failed

[khiker]

Describe the bug
following test failed under python 3.9.18. but 3.12.1 success.

import pathlib
class ClassVariable():
    PATH = pathlib.Path("/tmp/foobar.txt")
    def __init__(self):
        pass
    def exists(self):
        return ClassVariable.PATH.is_file()

import pytest
def test_classvariable(fs):
    fs.create_file(ClassVariable.PATH)
    assert ClassVariable().exists() == True

How To Reproduce
run above script.

Result (3.9.18):

$ pytest -v ./test.py
============================================================================================== test session starts ==============================================================================================
platform linux -- Python 3.9.18, pytest-7.4.4, pluggy-1.3.0 -- /home/kiyoto/.anyenv/envs/pyenv/versions/3.9.18/bin/python3.9
cachedir: .pytest_cache
rootdir: /tmp/yyy
plugins: pyfakefs-5.3.4
collected 1 item

test.py::test_classvariable FAILED                                                                                                                                                                        [100%]

=================================================================================================== FAILURES ====================================================================================================
______________________________________________________________________________________________ test_classvariable _______________________________________________________________________________________________

fs = <pyfakefs.fake_filesystem.FakeFilesystem object at 0x7fa88069bbe0>

    def test_classvariable(fs):
        fs.create_file(ClassVariable.PATH)
>       assert ClassVariable().exists() == True
E       assert False == True
E        +  where False = <bound method ClassVariable.exists of <test.ClassVariable object at 0x7fa88070a400>>()
E        +    where <bound method ClassVariable.exists of <test.ClassVariable object at 0x7fa88070a400>> = <test.ClassVariable object at 0x7fa88070a400>.exists
E        +      where <test.ClassVariable object at 0x7fa88070a400> = ClassVariable()

/tmp/yyy/test.py:12: AssertionError

Result (3.12.1)

$ poetry run pytest -vvv test.py
============================================================================================== test session starts ==============================================================================================
platform linux -- Python 3.12.1, pytest-7.4.4, pluggy-1.3.0 -- /home/kiyoto/.cache/pypoetry/virtualenvs/xxx-76ftnvza-py3.12/bin/python
cachedir: .pytest_cache
rootdir: /tmp/xxx
plugins: pyfakefs-5.3.4
collected 1 item

test.py::test_classvariable PASSED

Your environment

$ python -c "import platform; print(platform.platform())"
Linux-6.6.11-amd64-x86_64-with-glibc2.37
$ python -c "import sys; print('Python', sys.version)"
Python 3.9.18 (main, Jan 20 2024, 10:42:53)
[GCC 13.2.0]
$ python -c "from pyfakefs import __version__; print('pyfakefs', __version__)"
pyfakefs 5.3.4
$ python -c "import pytest; print('pytest', pytest.__version__)"
pytest 7.4.4

[mrbean-bremen]

Thanks for the report! This is a known limitation, related to the fact that class objects are essentially global objects initialized at load time when no patching has occurred yet.
That it works in Python 3.12 is due to some internal restructuring of pathlib in that version, that makes patching work differently (and actually makes it easier). I will check if the problem is generally no longer present in Python 3.12, and add a respective note in the documentation.

I will have a closer look if something can be done nevertheless in earlier Python versions, as this is a pattern now found frequently, but can't promise anything.

[khiker]

Thanks for the report! This is a known limitation, related to the fact that class objects are essentially global objects initialized at load time when no patching has occurred yet.
That it works in Python 3.12 is due to some internal restructuring of pathlib in that version, that makes patching work differently (and actually makes it easier). I will check if the problem is generally no longer present in Python 3.12, and add a respective note in the documentation.

I understand. thank you for your explanation.

I will have a closer look if something can be done nevertheless in earlier Python versions, as this is a pattern now found frequently, but can't promise anything.

using mocker.patch.object seemed to be workaround. I think I will try to use that in 3.9.18.

import pathlib
class ClassVariable():
    PATH = pathlib.Path("/tmp/foobar.txt")
    def __init__(self):
        pass
    def exists(self):
        return ClassVariable.PATH.is_file()

import pytest
@pytest.fixture(autouse=True)
def replace_object(fs, mocker):
    mocker.patch.object(ClassVariable, "PATH", pathlib.Path(ClassVariable.PATH))

def test_classvariable(fs):
    fs.create_file(ClassVariable.PATH)
    assert ClassVariable().exists() is True

Result

$ pytest -v ./test.py
============================================================================================= test session starts ==============================================================================================
platform linux -- Python 3.9.18, pytest-7.4.4, pluggy-1.3.0 -- /home/kiyoto/.anyenv/envs/pyenv/versions/3.9.18/bin/python3.9
cachedir: .pytest_cache
rootdir: /home/kiyoto/project/python/pyfakefs/yyy
plugins: pyfakefs-5.3.4, mock-3.12.0
collected 1 item

test.py::test_classvariable PASSED

[mrbean-bremen]

using mocker.patch.object seemed to be workaround

True. Using mock that way is usually a possibility to work around some of the limitations of pyfakefs. Another would be to reload the tested module inside the test, as mentioned in the documentation (not possible in your minimal example, as the test lives in the some module). Nevertheless, I will have another look, maybe I can think of something...

[mrbean-bremen]

Ok, I had another look here, and I really don't see a possibility to fix in Python < 3.11 without using a mock as shown in your comment, or reloading the code containing the class variable path. I have played around a bit trying it, but found no way that is not quite complicated and slow.

The problem is that in these Python versions, an accessor was used to access the file system, a reference to which was saved in every Path instance. This accessor was overwritten in pyfakefs, but if the path has been created before the patching, it remained in the path as before.
Accessing the path like it was done in the example (PATH.is_file()) directly accesses this non-patched accessor, and patching that would involve looking for all global and class variables in any module that are paths and patch them, which is what I tried but gave up.

In Python 3.11 pathlib was restructured, and instead of using an accessor, the os functions where directly called (in 3.12 it was restructured again, but that part did not change). As these are patched inside the test, the problem does not happen anymore.

So I will probably add a note to te documentation that this specific problem does no longer happen with newer Python versions (EDIT: done) and leave it at that. I will convert this issue into a discussion issue for reference.