Mod Python causing segmentation faults when compiled with Python3.9

[rohan-97]

Hello,

I have recently upgraded from python3.7 to python3.9 and compiled mod_pyhton with python3.9.
After the upgrade, I see apache process crashing with segmentation fault intemittently.

I generated core dump and following is the traceback which I see.

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
--Type <RET> for more, q to quit, c to continue without paging--l
Core was generated by `/usr/sbin/apache2 -d /var/service_dispatcher -X -k start'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  release_sentinel (wr_raw=0x7fd5ee538810) at ../Modules/_threadmodule.c:1246
1246    ../Modules/_threadmodule.c: No such file or directory.
[Current thread is 1 (Thread 0x7fd5e47f8700 (LWP 62175))]
(gdb) bt
#0  release_sentinel (wr_raw=0x7fd5ee538810) at ../Modules/_threadmodule.c:1246
#1  0x00007fd5f17f0a2a in release_interpreter (idata=0x564a091fcb40) at mod_python.c:306
#2  python_handler (req=0x7fd5c496f0a0, phase=<optimized out>) at mod_python.c:1573
#3  0x0000564a08e0c2d0 in ap_run_fixups ()
#4  0x0000564a08e0ea3b in ap_process_request_internal ()
#5  0x0000564a08e30978 in ap_process_async_request ()
#6  0x0000564a08e30bce in ap_process_request ()
#7  0x0000564a08e2ca44 in ?? ()
#8  0x0000564a08e216e0 in ap_run_process_connection ()
#9  0x00007fd5f18522d3 in ?? () from /usr/lib/apache2/modules/mod_mpm_worker.so
#10 0x00007fd5f1b30ea7 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#11 0x00007fd5f1a50aef in clone () from /lib/x86_64-linux-gnu/libc.so.6
(gdb)

The traceback points to following line in mod_python.c,

following is the code block for reference

static void release_interpreter(interpreterdata *idata)
{
    PyThreadState *tstate = PyThreadState_Get();
#ifdef WITH_THREAD
    PyThreadState_Clear(tstate);
    if (idata)
        APR_ARRAY_PUSH(idata->tstates, PyThreadState *) = tstate;
    else
        PyThreadState_Delete(tstate);
    PyEval_ReleaseThread(tstate);
#else
    if (!idata) PyThreadState_Delete(tstate);
#endif
}

Looking at the code, we first create a PyThreadState object,
then make a call to PyThreadState_Clear(tstate) and if idata is empty, we make call to PyThreadState_Delete(tstate).

but in python3.9, there is a change in behaviour of PyThreadState_Clear and PyThreadState_Delete methods.

If we check the official documenatation of python, it states

void PyThreadState_Clear(PyThreadState *tstate)
Part of the Stable ABI.
Reset all information in a thread state object. The global interpreter lock must be held.

Changed in version 3.9: This function now calls the PyThreadState.on_delete callback. Previously, that happened in PyThreadState_Delete().

however when PyThreadState.on_delete callback is called, we see segmentation fault.

Following is the pull request which introduced this change,

python/cpython#18296

In the changed files we can see tstate->on_delete hook call is moved from PyThreadState_Delete to PyThreadState_Clear,

In order to avoid call to this hook,
I changed tstate->on_delete to point to null and following is th change which I did in mod_python.c's release_interpreter code block

static void release_interpreter(interpreterdata *idata)
{
    PyThreadState *tstate = PyThreadState_Get();
#ifdef WITH_THREAD
    tstate->on_delete = NULL;
    PyThreadState_Clear(tstate);
    if (idata)
        APR_ARRAY_PUSH(idata->tstates, PyThreadState *) = tstate;
    else
        PyThreadState_Delete(tstate);
    PyEval_ReleaseThread(tstate);
#else
    if (!idata) PyThreadState_Delete(tstate);
#endif
}

After this change, mod_python and apache is working completely fine,
However I am not sure if this is right fix or not.

Let me know if there are other ways to fix this issue and make mod_python compatible with python3.9

Otherwise I can raise a Pull request with that change.

Attaching core dump.

Thank You

[rohan-97]

core-apache2-62165-1671776305.zip
Attaching Core Dump

[grisha]

@rohan-97 Thanks for looking into this.

I think you'll need to dig deeper - setting tstate->on_delete to NULL prevents whatever it was from being executed, but it was there for a reason.

The traceback from gdb points to ../Modules/_threadmodule.c:1246 - which looks like it's a comment, at least looking at the 3.9 tag:

https://github.com/python/cpython/blob/3.9/Modules/_threadmodule.c#L1246

...which doesn't make sense, most likely your Python is from a slightly different version, perhaps patched by the distribution or something like that - you might want to try to track down exactly which line of code it is that causes the segfault.