On Windows `scandir` handles paths different from other filesystem APIs

[ChrisDenton]

Python has to append \\* to the path before calling FindFirstFileW. This slightly changes how paths are parsed by the OS.

I think our policy of "as close to the OS behaviour as is portable" still applies here, but because we're the ones appending \\* to the path for scandir, trimming spaces first isn't unreasonable. I'd rather use an API that doesn't involve us modifying the path, but if that doesn't exist, then we don't have a lot of choice.

But since the original issue was opened for "what's the general policy on this" and that's been answered, let's start a new issue specifically for scandir.

Originally posted by @zooba in #115104

To fix this, GetFullPathNameW could be used before appending \\* to the path and calling FindFirstFileW. This will create a normalized absolute path without changing the meaning of the path (as far as the OS is concerned). So it'll be consistent with other filesystem APIs.

Linked PRs

• gh-150880: Normalize Windows scandir wildcard paths #152906
• [3.15] gh-150880: Normalize paths on Windows before appending wildcard (GH-152906) #152963
• [3.14] gh-150880: Normalize paths on Windows before appending wildcard (GH-152906) #152964
• [3.13] gh-150880: Normalize paths on Windows before appending wildcard (GH-152906) #152965

[serhiy-storchaka]

What does os.listdir() return for such path?

[ChrisDenton]

That throws FileNotFoundError, the same as scandir.

[vstinner]

I don't understand well which paths are causing troubles. Can you provide an example? Maybe a script reproducing the issue?

[zooba]

I don't understand well which paths are causing troubles

Normally Windows will ignore trailing spaces on a path, so e.g. open("file.txt ") will succeed, because the OS ignores the space.

But it doesn't ignore trailing spaces within the path, so if you do os.scandir("C:\dir ") then you might think the trailing space will be ignored, but internally we do + "\*" (because FindFirstFileW requires a wildcard) which then means the space isn't trailing anymore. It doesn't get stripped, and "dir " doesn't actually exist (even though "dir" does), so it fails.

[zooba]

I would say that os.path shouldn't mess with trailing spaces, because those are manipulation functions, and we want to let the user control it as much as possible. (Concretely, I don't want os.path.join to start stripping its arguments.)

Equally, anything that we pass through to the OS (like most of the os functions) should pass through directly (excepting cases that are API mismatches, such as embedded nulls in strings).

The grey area is where we modify the user's value before passing it to the OS. On one hand, we can reasonably assume that since "C:\dir " won't ever exist, we should trim the space before we join with *. On the other hand, there are at least a few ways for the caller to say that they meant for the space to be there (the most prominent being "\\?\C:\dir ") and in that case we shouldn't trim the space. I don't really like having to make choices at that level of "we know exactly how the OS is going to interpret this", so the question is should we try?

[ChrisDenton]

I do think GetFullPathNameW is the correct API to ask the OS to trim the path for us if that's the route we go down. Paths starting with \\?\ are the only special case Python would need to handle itself (by not doing anything special with them).

[zooba]

Okay, yeah, having now looked at the implementation of GetFullPathNameW and in particular, the comments around this behaviour, we should be using that API to resolve any paths before we manipulate and then use them.

In terms of a PR, I think that means we should probably be updating our joinpath implementation in fileutils.c, and then making sure that posixmodule.c is using that rather than directly manipulating strings. I'm going from memory on those, so not totally sure how appropriate that is and it may turn out to be the wrong idea, but we should already have fairly central native path manipulation that is separate from ntpath.py/os.path, and since scandir (and listdir) are doing path manipulation they should be using it.

[zainnadeem786]

Hi everyone,

I investigated this on the current main branch and was able to reproduce the behavior on Windows.

From my testing:

• open(), os.stat(), and os.path.exists() all allow Windows to normalize a trailing space in the final path component.
• os.listdir(), os.scandir(), and consequently pathlib.Path.iterdir() do not. They append a wildcard before calling FindFirstFileW(), so the trailing space becomes an internal path component and Windows no longer trims it.

I also verified that the same behavior occurs for UNC paths.

While investigating, I confirmed that calling GetFullPathNameW() before appending the wildcard normalizes ordinary paths as expected, whereas \\?\ extended paths intentionally preserve trailing spaces. That seems consistent with the discussion above.

My current understanding is that the inconsistency comes from the wildcard being appended before normalization rather than from FindFirstFileW() itself.

One thing I wanted to check before preparing a PR:

Would the preferred direction be to normalize only non-extended paths before constructing the enumeration pattern (while preserving \\?\ semantics), or is there another implementation approach you would prefer?

I'm happy to prepare a PR once the intended direction is confirmed.

[zooba]

I put the direction in my last message - extend _Py_add_relfile or _Py_join_relfile to use GetFullPathNameW on the base directory before appending, and make sure we're using that everywhere.

[zainnadeem786]

Thanks for the clarification!

That makes sense. I'll follow that approach by updating _Py_add_relfile / _Py_join_relfile to normalize the base directory with GetFullPathNameW() before appending the wildcard, while preserving the existing \\?\ path semantics, and then update the affected callers accordingly.

I'll prepare a PR based on that direction. Thanks!