shutil.which() can return non-executable path starting with 3.12 on Windows

[lazka]

Bug report

Bug description:

Starting with Python 3.12 if a file is in PATH that does not end with PATHEXT, but happens to be named the same as the searched command it is returned instead of the real command later in PATH which includes PATHEXT.

# Assume:
# PATH=C:\foo;C:\WINDOWS\system32
# "C:\foo\cmd" exists
# "C:\WINDOWS\system32\cmd.exe" exists

import shutil
print(shutil.which('cmd'))
# Actual: C:\foo\cmd
# Expected: C:\WINDOWS\system32\cmd.exe (3.11 result)

I have verified that this change was introduced with #103179 and reverting it fixes the issue.

@csm10495 ^

Downstream context: mesonbuild/meson#13886

CPython versions tested on:

3.12

Operating systems tested on:

Windows

Linked PRs

• gh-127001: Fix PATHEXT issues in shutil.which() on Windows #127035
• [3.13] gh-127001: Fix PATHEXT issues in shutil.which() on Windows (GH-127035) #127156
• [3.12] gh-127001: Fix PATHEXT issues in shutil.which() on Windows (GH-127035) #127158

[zooba]

Reverting the entire change is clearly not going to happen - the logic was incorrect before, and apparently is still incorrect, so let's figure out what correct logic for handling PATHEXT would be.

[serhiy-storchaka]

Should not it be fixed by #109590?

[serhiy-storchaka]

Ah, it still checks for file without extension.

[lazka]

Should not it be fixed by #109590?

That only handles the "in the same directory" case, and not in different directories, from what I see.

But, that issue mentioned Get-Command cmd in powershell, which I now tried, and that also returns the wrong path as "CommandType=Application" here, even though the docs say that it only returns files ending with PATHEXT (https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/get-command?view=powershell-7.4#-commandtype) so I don't know what's "right" anymore :)

PS C:\Users\user> echo $env:PATH
C:\msys64\usr\bin;C:\Windows\System32;C:\Windows
PS C:\Users\user> echo $env:PATHEXT
.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC;.CPL
PS C:\Users\user> Get-Command cmd

CommandType     Name                                               Version    Source
-----------     ----                                               -------    ------
Application     cmd                                                0.0.0.0    C:\msys64\usr\bin\cmd

PS C:\Users\user> Get-Command -All cmd

CommandType     Name                                               Version    Source
-----------     ----                                               -------    ------
Application     cmd                                                0.0.0.0    C:\msys64\usr\bin\cmd
Application     cmd.exe                                            10.0.2610… C:\Windows\System32\cmd.exe

[zooba]

The right behaviour ought to be "whatever CreateProcess("cmd") would run". Terminals all layer their own logic on top, but Python tries to generally stick to the closest public API that implements the functionality.

(One explicit exception we make to CreateProcess behaviour is that we don't search the application directory first with which, because if we did, it wouldn't be useful. By skipping that somewhat strange edge case, it is possible to use which to override the application directory. This mainly only matters for python.exe - CreateProcess("python") will always launch sys._base_executable due to how the API works, but CreateProcess(shutil.which("python")) follows PATH, which is likely what was intended if a user provided the command name.)

[lazka]

The right behaviour ought to be "whatever CreateProcess("cmd") would run". Terminals all layer their own logic on top, but Python tries to generally stick to the closest public API that implements the functionality.

Also CreateProcess prefers "C:\Windows\System32" over PATH, like curl/tar/bash that are in there, which is another can of worms :(

Since my initial presented scenario occurs in MSYS2, and it's not quite clear why we have that "cmd" bash script in PATH in the first place, I'm trying to get rid of it instead now :/

[csm10495]

Ouch. Sorry for the regression folks. I wish this was less complicated than it is. I see @serhiy-storchaka has already started on a fix. Thanks all.

[serhiy-storchaka]

There was other bug introduced in #109995: full match did not work for files with multcomponent extension in PATHEXT, (e.g. .foo.bar).

Other corner case was only discussed in comments: a dot (.) in PATHEXT.

[lazka]

I can confirm that this fixes things for my use case. Thanks everyone!