Document `PyCF_ALLOW_INCOMPLETE_INPUT`

[johnslavik]

Original title: Should PyCF_ALLOW_INCOMPLETE_INPUT remain undocumented?

Since _IncompleteInputError is undocumented on purpose -- according to GH-119521 -- should then PyCF_ALLOW_INCOMPLETE_INPUT remain an undocumented macro in the C API as well? I'm coming from GH-141004.
CC @pablogsal @JelleZijlstra @encukou

CC @ZeroIntensity mentorship docs topic-C-API

[johnslavik]

Mentioning the private exception would be essential to the documentation of this macro... unless I just say it's a "private exception handled by the insiders 🕵 Who knows, knows"

[ZeroIntensity]

Personally, I'm not a fan of pretending that something is private by not documenting it. People love to read CPython's source to find what kind of things we're hiding from them (C API users especially), so it would not surprise me if PyCF_ALLOW_INCOMPLETE_INPUT is already in use somewhere. If we don't want people to use it, we should go through the formal deprecation process.

[johnslavik]

it would not surprise me if PyCF_ALLOW_INCOMPLETE_INPUT is already in use somewhere

It is.

[pablogsal]

Since _IncompleteInputError is undocumented on purpose -- according to GH-119521 -- should then PyCF_ALLOW_INCOMPLETE_INPUT remain an undocumented macro in the C API as well? I'm coming from GH-141004.

Yes, it can change at any point and is an implementation detail for some internal modules. So either we explicitly state that or we leave it undocumented.

If we don't want people to use it, we should go through the formal deprecation process.

I disagree. If is not documented it's not supported (obviously reality is not that easy when a lot of critical projects decide to use it anyway). This is an implementation detail and always have been and we can remove it or change its semantics. That's how it was designed and it must be kept that way.

[ZeroIntensity]

This is an implementation detail and always have been and we can remove it or change its semantics. That's how it was designed and it must be kept that way.

Ah, I was unclear in my first message, sorry!

I'm not asking for this to be "supported". I just think we should add something to the docs that says "Hey, here's this cool thing, but don't you dare use it, use this other thing instead". I agree that this should stay an implementation detail.

[johnslavik]

"Hey, here's this cool thing, but don't you dare use it, use this other thing instead".

I think that the convention of not documenting unsupported things is just to (1) save unnecessary maintenance elsewhere than in the code itself and (2) not present the reader with something that for sure is irrelevant and unactionable to them.

I see why it makes sense to document something only if we intend people to use it at some point in time. There for sure are private parts of the standard library which are undocumented, even though they have been around for a very long time and some critical projects do rely on it, e.g. typing._eval_type.

We do not intend anybody to use PyCF_ALLOW_INCOMPLETE_INPUT, so if we ever remove it in the code, the documentation won't have to be additionally maintained to reflect that internal change, which was never really a relevant thing any C API user.

[johnslavik]

I guess that the bad part of this is that there is no obvious clue in the name of PyCF_ALLOW_INCOMPLETE_INPUT that it should not (or even must not) be used, such as in _IncompleteInputError. _IncompleteInputError is "private" by convention. Perhaps a different naming convention would be more helpful in the future.

[ZeroIntensity]

I think that the convention of not documenting unsupported things is just to (1) save unnecessary maintenance elsewhere than in the code itself and (2) not present the reader with something that for sure is irrelevant and unactionable to them.

It's usually more complicated than that. Again, people love using our internal APIs, especially when they don't realize they're internal. The documentation would primarily be for existing codebases that found this in our source code a while ago and began using it. Through the docs, we can ask them not to use it and tell them what to use instead. If we don't do anything, they'll keep using it and complain to us when we break it someday.

Perhaps a different naming convention would be more helpful in the future.

Yeah, it should have definitely been _PyCF. I added a CI job preventing this from happening in the future.

[johnslavik]

Okay, I think that the general conclusion is that we should not document this at the moment. I've got the guidance I needed -- thank y'all for the quick and thoughtful responses!

Please reopen if you think that this needs further discussion.

[ZeroIntensity]

I think this needs more discussion. I am strongly -1 on leaving C APIs undocumented to try to hide them; for APIs without a _Py prefix, omitting documentation is not enough to prevent people from using them.

I'd be fine with taking an approach similar to #141634, where the flag is documented as "do not use this, use this other thing". That said, I don't see a great alternative for this flag, so if this is supposed to be private, what should people use instead?