_testcapi mishandles Py_fopen() failures

[serhiy-storchaka]

Bug report

Py_fopen() sets an exception and returns NULL on error, but several _testcapi helpers ignore both halves of that contract.

Modules/_testcapi/object.c does not check the result at all. All four helpers pass it straight to PyObject_Print():

    fp = Py_fopen(filename, "w+");

    if (PyObject_Print(object, fp, flags) < 0) {

so a NULL file pointer reaches fprintf() and the interpreter crashes:

>>> import _testcapi
>>> _testcapi.call_pyobject_print('x', '/nonexistent-dir/out.txt', False)
Segmentation fault

The same happens with pyobject_print_null(), pyobject_print_noref_object() and pyobject_print_os_error().

Modules/_testcapimodule.c sets a second exception. The six pymarshal_* helpers do:

    fp = Py_fopen(filename, "rb");
    if (fp == NULL) {
        PyErr_SetFromErrno(PyExc_OSError);
        return NULL;
    }

so OSError is instantiated while Py_fopen()'s exception is still pending:

>>> _testcapi.pymarshal_read_object_from_file('nonexistent')
SystemError: <class 'OSError'> returned a result with an exception set

On a debug build it aborts instead: Objects/call.c:342: _PyObject_Call: Assertion `!_PyErr_Occurred(tstate)' failed.

Both date from f89e5e2 (gh-127350), which converted these helpers from fopen() to Py_fopen(). They affect 3.14 and later.

Related cleanup: these helpers also close the file with fclose() rather than Py_fclose(), which the documentation requires -- 8 sites in object.c, 6 in _testcapimodule.c and 14 in run.c. It is harmless today, since Py_fclose() is just a wrapper.

All other Py_fopen() call sites in the tree check the result correctly.

Linked PRs

• gh-155905: Fix error handling in _testcapi helpers #155906
• [3.15] gh-155905: Fix error handling in _testcapi helpers (GH-155906) #156580
• [3.14] gh-155905: Fix error handling in _testcapi helpers (GH-155906) #156581