Fake directory missing on local test, but adding it breaks the pipeline with "already exists" error

[hyler]

Hey good people, I'm sorry for the seemingly bad report, I'm not sure if this is a real bug, but I'm seeing some strange behavior on macOS. I'll try to be succinct. Please assist me in delivering more information, I am new and inexperienced.

I have a method under testing that at some point does something like:

with open('/tmp/filename.txt') as f:
    ...  # Yada, yada

In my test method, I provide the fs fixture, and I make /tmp like this:

fs.create_dir('/tmp')  # Can't always rely on the OS having a real /tmp

and then I check that the file got written, etc. It all works wonderfully.

When this package is in the build pipeline (we use ADO), I get this error:

        fs.create_dir('/resources/testing')
>       fs.create_dir('/tmp')  # Can't always rely on the OS having a real /tmp
...
>       raise OSError(err_no, message, filename)
E       FileExistsError: [Errno 17] File exists in the fake filesystem: '/tmp'

However, if I remove the creation of /tmp, my local test (with python3 -m pytest) fails with the same error (which is why I added it). Furthermore, apparently the package has been building just fine in the pipeline so far without explicitly creating /tmp. (Yes, I know this implies we haven't been running pytest locally... I made changes to this package for the first time today in a long time after others had worked on it.)

I also noticed that as soon as I pass the fs fixture to my test method, calls to os.path.exists() start returning False for directories that are present on macOS, such as /usr, etc. I believe this is to be expected, since (by my logic) by providing the fs fixture I'm asking for a fake FS to "overlay" the real one.

However, I am not sure what is going on in the build pipeline. I think ADO uses Linux (either Debian or Ubuntu) to run their builds. I also have noticed that on macOS /tmp is actually a link (tmp@ -> private/tmp) but I'm not sure of the implications of that.

I can trigger the "File exists" error locally by duplicating the line:

fs.create_dir('/tmp')  # Can't always rely on the OS having a real /tmp
fs.create_dir('/tmp')  # This breaks local testing

Here are a few other details I have:

• I'm using macOS Ventura, 13.2.1
• I'm using Python 3.10.10 installed with Homebrew
• I'm using pytest 7.2.2 and pyfakefs 5.0.0 (but it's the same with 5.1.0)
• The project in question uses Python 3.9 (specifically, python:3.9-slim in the Dockerfile), but I somehow don't think this is the issue here

Anyway, any help is appreciated, this is driving me nuts. I'll be more than happy to provide you with any information you need.

Thank you!

[mrbean-bremen]

I moved this to discussions, as this may be interesting to others.

As described in the documentation, pyfakefs has an almost empty file system at start, except for the temporary directory, which is expected to exist by other modules (like tempfile). As you have noticed, under macOS the temp directory as actually located in a subdirectoty of /private/tmp, while under Linux it is /tmp.
To make this work under both systems (and under Windows, if needed), you have two possibilities:

• create the directory only if it does not exist in the fake filesystem, for example:

  from pathlib import Path
  ...
  Path("/tmp").mkdir(exist_ok=True)

• use the actual temp directory in your code, which will always exist:

  import tempdir
  ...
  with open(os.path.join(tempfile.gettempdir(), 'filename.txt')) as f:
      ...  # Yada, yada

With the second approach you would have to modify the tested code, but it is actually cleaner. It would also work correctly under Windows.

[mrbean-bremen]

There is no semantic difference between fs.create_dir() and os.makedirs() - both also create subdirectories if needed (which os.mkdir() does not). create_dir is there only for compatibility and not really needed anymore. And you could also use

os.makedirs('/tmp', exist_ok=True)

so you won't need the check for existence.

[hyler]

I have one last question and I'll drop it, I promise. I've read some of the discussions around what exactly is created by default when the fixture sets up its stuff, and now I have also read the docs (far more thoroughly this time). But it seems to me that Apple's engineers have anticipated people wanting to use /tmp from the get go, and that directory does exist on macOS, even if it does ultimately link to somewhere else.

So with that in mind, why not have pyfakefs perform on macOS the way it does on Linux and other UNIXes? As far as I know, macOS is certified to be POSIX compliant, and from what I can find online its overall certifications wound guarantee that /tmp is there. Therefore, it would seem like it's okay—on a macOS host—to have the fixture behave similarly, and make /tmp available transparently like the OS does.

I'm not actually presenting an argument that pyfakefs should be changed—I'm way too junior of a... anything to do that, just wondering if there's any good engineering reason I'm not aware of. After all, getting such questions answered is how I learn.

[mrbean-bremen]

Actually, I have been thinking along the same lines. Currently, the temp directories are just taken as returned from the real os (using tempfile.gettempdir()), and created in the fake fs - this being the only exception for the filesystem being empty at start. It could make sense though to create that /tmp link specifically for macOS, for convenience. The downside is that that the link would take some disk space, what makes testing for disk space less convenient (and currently falis some tests), so I'm not decided yet if I want to really do this.

And BTW, questioning the current state should always be encouraged - nothing about being too junior here...

[hyler]

Ah, I see the reasoning behind the approach you took. Yes, on my Mac for example it returns:

>>> import tempfile
>>> tempfile.gettempdir()
'/var/folders/pl/6fhh67ks0nbdl5yk66ykms900000gp/T'

Interestingly, Python's own tempfile.gettempdir() looks for a bunch of things first before it decides to return /tmp, one of them being TMPDIR, which is defined on macOS. A deeper sleuthing around the web reveals some interesting info:

TMPDIR as defined in OSX is only accessible by yourself [...]

and

/tmp/: system-wide
TMPDIR: per-user

Courtesy of this super helpful StackExchange question.

So yes, it seems like relying on /tmp to be there on macOS is just as valid (coding best practices notwithstanding) as it is on other UNIX-likes, whether you're accessing the file system manually or programmatically. I actually wish tempfile.gettempdir() would return /tmp, it seems to (a noob like) me the more "expected" behavior. (Though I suppose one of the defining pillars of noobdom is that it seems to the noob that things should be a certain way through some intangible feeling on the inside.)

[mrbean-bremen]

Security-wise, I think it makes sense to default to a user-specific temp dir as done in macOS and Windows.

[mrbean-bremen]

Update:
I adapted the code to create the /tmp link in the fake filesystem under macOS, so in the future it shall not be needed to create this directory. I resolved the problem of the changed file system size by ignoring the link in the size calculation (actually resetting the used size after creating it), which avoids having to account for it in any tests related to disk space.