Add `pathlib.Path.metadata()` and soft-deprecate `info`

[barneygale]

Feature or enhancement

Proposal:

I propose we add a pathlib.Path.metadata() method that requires a keyword-only cached argument, e.g.:

>>> m = pathlib.Path('README.txt').metadata(cached=True)
>>> m
Metadata('README.txt')
>>> m.is_file()
True

If cached is True, the method may return a Metadata object that was previously fetched and cached. If False, the return value is always up-to-date.

pathlib.Path.info should be soft-deprecated; its implementation should call self.metadata(cached=True)

Has this already been discussed elsewhere?

I have already discussed this feature proposal on Discourse

Links to previous discussion of this feature:

https://discuss.python.org/t/ergonomics-of-new-pathlib-path-scandir/71721/70

[barneygale]

An alternative name for the "cached" argument might be "sync", with the opposite meaning.

[cmaloney]

Is the metadata object required to be filled in at return or could it be more lazily fetched? Recently there is work ongoing to add statx and that has a lot more flags for selecting specific desired subsets of information. If the Metadata object needs to be complete up front then would need a way to say "At least this information is required" (otherwise might end up with multiple statx calls required)

[merwok]

I’m not a fan that a method name uses a noun rather than a verb, but also that repeated calls to this method would return the same cached info 🤔 It seems that a property would be more idiomatic here, but then there’s no way to pass parameters. Two properties?

[barneygale]

Is the metadata object required to be filled in at return or could it be more lazily fetched? Recently there is work ongoing to add statx and that has a lot more flags for selecting specific desired subsets of information. If the Metadata object needs to be complete up front then would need a way to say "At least this information is required" (otherwise might end up with multiple statx calls required)

I think it's worth exploring. What API would you propose for specifying the minimum information to query up-front? (Presumably not os.STATX_*?)

If we did this, I think we'd need to add a follow_symlinks argument to metadata(), which probably entails that exists(), is_dir() etc must not accept a follow_symlinks argument. Therefore Path.info wouldn't be equivalent to Path.metadata(cached=True). A bigger change but maybe worthwhile.

I’m not a fan that a method name uses a noun rather than a verb, but also that repeated calls to this method would return the same cached info 🤔 It seems that a property would be more idiomatic here, but then there’s no way to pass parameters. Two properties?

Maybe a metadata() method and a cached_metadata property? (If we add follow_symlinks to metadata() I guess we'd need an additional cached_symlink_metadata property)

FWIW Rust has fs::metadata() and fs::symlink_metadata()

[cmaloney]

Re: API, my thought is specifying what members are needed / required ("ensured" to be set) then it being up to the metadata object how to get that information effectively (whether a stat, DirEntry, statx, or GetFileInformationByName, etc.). If all the data is already available then no calls made; otherwise calculate the set needed + what underlying call(s) could provide on the particular platform.

[barneygale]

Makes sense. So something like this perhaps?

r = pathlib.MetadataRequest(size=True, permissions=True)
m = pathlib.Path('blah').metadata(request=r)

[merwok]

That looks a little complicated to access file metadata.

@ncoghlan would you have suggestions for a clean API here?

[cmaloney]

Yea, my thought would be rather than a request object just a Sequence (List / set / tuple) of the members which are expected to work. Open to other ideas though / haven't prototyped and explore design space a lot.

[barneygale]

Some pondering on caching...

In most cases, users can handle caching of metadata themselves by calling metadata = path.metadata() and then stashing metadata somewhere. The only time Path.metadata(cached=True) more than a mere convenience is when you call path.iterdir() and want to retrieve the pre-populated os.DirEntry-based metadata. And perhaps that's the only use case that really matters, which would make Path.metadata(cached=...) an over-generalisation. A couple of more focused alternatives:

1. We could introduce a new Path.iterdir()-like method that generates (Path, Metadata) pairs (edit: or perhaps a triple if symlink metadata is split out)
2. Or we could introduce a subclass of Path called DirectoryPath with additional cached_metadata and cached_symlink_metadata properties, and adjust Path.iterdir() to generate DirectoryPath objects. Though this is complicated by WindowsPath and PosixPath.

Grateful for any advice.

[ncoghlan]

The challenges of clearly expressing metadata queries is a large part of how we ended up with the .info design in the first place: there are a lot of directory processing scripts that don't need to worry about file trees getting changed underneath them, and for those that do, files and subdirectories going away completely (or new ones getting added) is more likely to cause problems than the state of existing files changing. The .info design accordingly ended up treating "no parallel tree mutation" as the typical case, and leaving it to users to design their own custom caching (or skip any caching entirely) if handling parallel tree mutation is a requirement.

The enhancement that I think may make sense to narrow the gap between the two usage models would be something like a refresh_info or ensure_info method that invalidated the currently cached metadata, and accepted an optional specification of cached values that should be immediately looked up. The presence of that method would then help highlight that info is a metadata cache scoped to the particular Path instance. Such an enhancement still wouldn't be enough to rid directory traversal code of TOCTOU (time of check vs time of use) bugs though, any more than bypassing the caching layer entirely does (only appropriate exception handling can do that)