[C API] Deprecate _Py_Identifier API

[vstinner]

I propose to deprecate the private _Py_Identifier API.

• Deprecate private _PyObject_CallMethodId(), _PyObject_GetAttrId(), _PyUnicode_FromId().
• Remove immediately internal functions using _Py_Identifier.

See also the discussions:

• Deprecate private C API functions #141049
• https://discuss.python.org/t/c-api-how-much-private-is-the-private-py-identifier-api/29190 (2023).
• https://mail.python.org/archives/list/python-dev@python.org/thread/DNMZAMB4M6RVR76RDZMUK2WRLI6KAAYS/ (2022)

Since 2022, CPython has been modified to replace _Py_IDENTIFIER() with _Py_STR(). Python no longer uses the _Py_IDENTIFIER() API anymore.

See also issue #141049 "Deprecate private C API functions".

Linked PRs

• gh-142217: Remove internal _Py_Identifier functions #142219
• gh-142217: Deprecate the private _Py_Identifier C API #142221
• gh-142217: Recommend PyUnicode_InternFromString() to replace _PyUnicode_FromId() #142746

[cdce8p]

I just tested the latest cpython build against mypy. #142221 is causing a lot of errors with -Werror,-Wdeprecated-declarations. Yes it's possible to replace most _Py_IDENTIFIER calls with PyUnicode_FromString, see python/mypy@f287f7d, but I'm not sure this is the right call here. The discuss thread mentions the need / desire for an API to get/create immortal strings, has there been any progress for it?

cpython itself uses &_Py_ID in a lot of places. These could actually also be used by mypy in most cases though I don't think it's intended that way. python/mypy@dbbc992

[encukou]

The discuss thread mentions the need / desire for an API to get/create immortal strings, has there been any progress for it?

Not really.
Note that CPython's immortal strings (_Py_STR) are statically allocated, in a way that requires code generation with lots of knowledge about the internals, and taking advantage of the fact that there are no conflicts.
We unfortunately can't provide that exact API to extensions -- if two of those intern the same string, only one can “win”.

AFAIK, we can't really make _Py_IDENTIFIER substantially better than users calling if(!x) {x = PyUnicode_InternFromString("...");} on init, and Py_CLEAR(x) on teardown.

The advantage of _Py_Identifier is the ease of use -- in particular, the teardown is automatic. But it comes at the cost of an elaborate (and largely untested) mechanism to make everything work correctly.

---

cpython itself uses &_Py_ID in a lot of places. These could actually also be used by mypy in most cases though I don't think it's intended that way.

Don't do that, this uses internal ABI that will break in patch releases. As in: it does break in practice in patch releases, it's not a CYA theoretical possibility.

---

@vstinner: would PyUnicode_InternFromString be a better suggested replacement than PyUnicode_FromString?

[vstinner]

@vstinner: would PyUnicode_InternFromString be a better suggested replacement than PyUnicode_FromString?

Oh, good idea: I wrote #142746 to update the documentation.

[vstinner]

Yes it's possible to replace most _Py_IDENTIFIER calls with PyUnicode_FromString, see python/mypy@f287f7d, but I'm not sure this is the right call here.

Yes, it's a safe replacement. To avoid negative impact on performance, you might try to only decode bytes strings (char*) from UTF-8 once by storing the object (PyObject*) in a module state, if possible.

I close the issue since the APIs are now deprecated.