On Windows, shutil.which incorrectly resolves to non-executable extensionless files (python >= 3.12.0b1)

[dairiki]

Bug report

Bug description:

This change (935aa45, PR #103179) changed the behavior of shutil.which so that it checks for the extensionless filename first, before checking extensions from PATHEXT. Previously, the extensionless name was not checked at all.

This is causing problems because, at least in my case, there is both an extensionless and extensioned version of the program, and the extensionless version is a bash script (which apparently can't be executed directly on Windows.)

---

Full Description

On a GitHub Windows shared runner, node.js is installed.

Run Get-ChildItem -Path "C:\Program Files\nodejs"

    Directory: C:\Program Files\nodejs

Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
d----           9/10/2023 11:53 PM                node_modules
[... elided ...]
-a---            8/8/2023 11:20 PM       71207576 node.exe
-a---           6/23/2023  6:02 AM            702 nodevars.bat
-a---            8/8/2023 10:22 PM           1365 npm
-a---           6/23/2023  6:01 AM            483 npm.cmd
-a---            8/8/2023 10:22 PM           1567 npx
-a---           6/23/2023  6:01 AM            539 npx.cmd

Here, npm is a bash script, while npm.cmd is a cmd.exe script.

In python 3.11 (and before), shutil.which("npm") returns "C:\\Program Files\\nodejs\\npm.CMD". That can be executed successfully by subprocess.run.

In python 3.12rc3, shutil.which("npm") returns "C:\\Program Files\\nodejs\\npm", which, being a bash script, appears not to be by executable by subprocess.

subprocess.run((shutil.which("npm"), "install"))

fails with OSError: [WinError 193] %1 is not a valid Win32 application on python 3.12rc3.

Notes

On the system in question Get-Command npm returns C:\Program Files\nodejs\npm.cmd. So PowerShell gets it right.

(PATHEXT is .COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC;.CPL.)

---

I'm not absolutely convinced that this is a bug. But again, I'm also not sure how to work around it without essentially reimplementing shutil.which.

CPython versions tested on:

3.12

Operating systems tested on:

Windows

Linked PRs

• gh-109590: Fix incorrect resolution of paths without extensions in shutil.which on win32. #109797
• gh-109590 - Update shutil.which on win32 to only give back PATHEXT matches by default #109995
• [3.12] gh-109590: Update shutil.which on Windows to prefer a PATHEXT extension on executable files (GH-109995) #110202

[dairiki]

More research as to the history of the change:

The original name, without added extension from PATHEXT, was added to the search list in #103179 to address #75586:

On windows, shutil.which does not match the semantics of built-in command lookup. If you pass the name of a script like foo.py and the PATHEXT variable doesn't include .py it will search for foo.py.exe, foo.py.bat, foo.py.cmd, etc. but not foo.py, which should be the first name checked.

But, later in #75586, there is this:

Note that CMD will only try to execute an extensionless file if "." is in PATHEXT. (In the Windows file namespace, "name" and "name." are equivalent.) [...]

Assuming that is true, then the original name should only be prepended to the search list if it has an extension. Extensionless names should not be considered when searching for an executable.

[csm10495]

@eryksun can you give some thoughts here?

I'm a bit nervous about this type of case hitting others in 3.12 and wondering your thoughts.

We could add logic to shutil.which to do:

if there is a directory (/ or \\) in the cmd OR there is a dot in it (denoting an extension), check the given cmd first. 

OR 

IF '.' is in PATHEXT, also check cmd first if it doesn't have a dot.

I think this gets it in-line with what CMD would do, but I'm not 100% on it.

[csm10495]

Some testing with CMD:

C:\Users\csm10495\Desktop\cpython>PCbuild\amd64\<TAB>
_testembed.exe     python.exe         pyw.exe            venvwlauncher.exe
py.exe             pythonw.exe        venvlauncher.exe
C:\Users\csm10495\Desktop\cpython>PCbuild\amd64\python
Python 3.13.0a0 (heads/customization_modules_test:cbad68d4d2, Sep 23 2023, 21:14:32) [MSC v.1936 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> exit()

C:\Users\csm10495\Desktop\cpython>echo "TEST" > PCbuild\amd64\python

C:\Users\csm10495\Desktop\cpython>PCbuild\amd64\python
Python 3.13.0a0 (heads/customization_modules_test:cbad68d4d2, Sep 23 2023, 21:14:32) [MSC v.1936 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> exit()

C:\Users\csm10495\Desktop\cpython>PCbuild\amd64\python.
'PCbuild\amd64\python.' is not recognized as an internal or external command,
operable program or batch file.

C:\Users\csm10495\Desktop\cpython>type PCbuild\amd64\python
"TEST"

C:\Users\csm10495\Desktop\cpython>echo %PATHEXT%
.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC;.PY;.PYW

So i guess if you give a directory component, it still consults PATHEXT first.

[csm10495]

I implemented the '.' in pathext to check in this draft pr: #109797
.. but i couldn't replicate the described behavior in CMD in Win 10.0.19045.3448:

I cleared C:\Program Files\nodejs\npm to be an empty file. If it was called it shouldn't really do anything.. but it kept hitting npm.cmd instead.

C:\Windows\System32>where npm
C:\Program Files\nodejs\npm
C:\Program Files\nodejs\npm.cmd

C:\Windows\System32>echo %PATHEXT%
.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC;.PY;.PYW

C:\Windows\System32>type "C:\Program Files\nodejs\npm"

C:\Windows\System32>npm --version
8.5.2

C:\Windows\System32>set PATHEXT=%PATHEXT%;.;

C:\Windows\System32>echo %PATHEXT%
.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC;.PY;.PYW;.;

C:\Windows\System32>where npm
C:\Program Files\nodejs\npm
C:\Program Files\nodejs\npm.cmd

C:\Windows\System32>type "C:\Program Files\nodejs\npm"

C:\Windows\System32>npm --version
8.5.2

C:\Windows\System32>set PATHEXT=.;%PATHEXT%;.;

C:\Windows\System32>npm --version
8.5.2

C:\Windows\System32>echo %PATHEXT%
.;.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC;.PY;.PYW;.;;.;

I'll wait for @eryksun 's thoughts.

[csm10495]

One last thing before bed :)

Maybe it does work but only if the rest of PATHEXT can't resolve something first (as in: it ignores the order in PATHEXT. Even if the dot is first, it will do all the other extensions before checking for an extensionless file).

I renamed npm.cmd to .npm.cmd, then:

C:\Windows\System32>npm
'npm' is not recognized as an internal or external command,
operable program or batch file.

C:\Windows\System32>echo %PATHEXT%
.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC;.PY;.PYW

C:\Windows\System32>set PATHEXT=%PATHEXT%;.

C:\Windows\System32>npm
<opened the npm file which led to the 'how should windows open this dialog'>

C:\Windows\System32>dir "C:\Program Files\nodejs"
 Volume in drive C has no label.
 Volume Serial Number is CAC3-BECF

 Directory of C:\Program Files\nodejs

09/23/2023  11:16 PM    <DIR>          .
09/23/2023  11:16 PM    <DIR>          ..
02/02/2022  06:00 AM               483 .npm.cmd
01/10/2022  04:22 PM               335 corepack
01/10/2022  04:22 PM               218 corepack.cmd
02/02/2022  06:00 AM             3,032 install_tools.bat
03/10/2022  06:31 PM        62,530,184 node.exe
10/14/2021  12:30 AM               702 nodevars.bat
03/15/2022  04:07 PM    <DIR>          node_modules
09/23/2023  11:09 PM                11 npm
02/02/2022  06:00 AM             1,552 npx
02/02/2022  06:00 AM               539 npx.cmd
               9 File(s)     62,537,056 bytes
               3 Dir(s)  390,529,925,120 bytes free

C:\Windows\System32>

If that's the desired behavior, it wouldn't be too hard to update the PR.

[csm10495]

@zooba any thoughts on this?

[zooba]

Assuming that is true, then the original name should only be prepended to the search list if it has an extension. Extensionless names should not be considered when searching for an executable.

This sounds reasonable at face value.

CMD is not the hallmark of consistent behaviour. I wouldn't want to set it as our standard. If you want CMD behaviour, pass shell=True (no, don't really).

The closer we get to the underlying API behaviour, the happier I'll be - in this case, it's SearchPathW.