Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Conflict with interpreter init versus PLPython on Python 3.12 #60

Open
mfenniak opened this issue May 29, 2024 · 6 comments
Open

Conflict with interpreter init versus PLPython on Python 3.12 #60

mfenniak opened this issue May 29, 2024 · 6 comments
Assignees

Comments

@mfenniak
Copy link
Collaborator

I think generally multicorn2 works well with Python 3.12, but the test suite cannot be executed in completeness because of the use of plpython in the write_filesystem.sql test.

$$ language plpython3u;

When this test is executed, the backend crashes. When loaded in a debugger, it is crashing while reporting the error: PyImport_AppendInittab() may not be called after Py_Initialize()", with a stack trace indicating it is being invoked from PLy_initialize in PLPython.

#0  __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
#1  0x00007ffff70a2f63 in __pthread_kill_internal (signo=6, threadid=<optimized out>) at pthread_kill.c:78
#2  0x00007ffff7052e86 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#3  0x00007ffff703b935 in __GI_abort () at abort.c:79
#4  0x00007fffeb303e93 in fatal_error_exit (status=-1) at Python/pylifecycle.c:2735
#5  fatal_error (fd=<optimized out>, header=header@entry=1,
    prefix=prefix@entry=0x7fffeb6958f0 <__func__.10.lto_priv.1> "PyImport_AppendInittab",
    msg=msg@entry=0x7fffeb6d3c30 "PyImport_AppendInittab() may not be called after Py_Initialize()", status=status@entry=-1)
    at Python/pylifecycle.c:2846
#6  0x00007fffeb35be43 in _Py_FatalErrorFunc (func=0x7fffeb6958f0 <__func__.10.lto_priv.1> "PyImport_AppendInittab",
    msg=0x7fffeb6d3c30 "PyImport_AppendInittab() may not be called after Py_Initialize()") at Python/pylifecycle.c:2932
#7  0x00007fffeb35f4f8 in PyImport_AppendInittab (name=<optimized out>, initfunc=<optimized out>) at Python/import.c:1499
#8  PyImport_AppendInittab (name=name@entry=0x7ffff6ca4340 "plpy", initfunc=<optimized out>) at Python/import.c:1494
#9  0x00007ffff6c9cb76 in PLy_initialize () at plpy_main.c:129
#10 PLy_initialize () at plpy_main.c:107
... snip ...

Neither multicorn2 nor PLPython call PyImport_AppendInittab after calling Py_Initialize... multicorn2's order is correct:

multicorn2/src/multicorn.c

Lines 136 to 144 in aebb975

PyImport_AppendInittab("plpy", PyInit_plpy);
need_import_plpy = true;
}
PG_CATCH();
{
need_import_plpy = false;
}
PG_END_TRY();
Py_Initialize();

And Postgres' order is correct:

https://github.com/postgres/postgres/blob/8fea1bd5411b793697a4c9087c403887e050c4ac/src/pl/plpython/plpy_main.c#L116-L117

So I'm left with the initial impression that the two interpreters loaded into the backend are conflicting on each other, which frankly makes sense. There is a change in Python 3.12 (python/cpython#99402) that introduced this error condition:

https://github.com/python/cpython/pull/99402/files#diff-28cfc3e2868980a79d93d2ebdc8747dcb9231f3dd7f2caef96e74107d1ea3bf3R2691-R2694

I don't have any initial thoughts on how to avoid this conflict. Multicorn basically tries to mirror PLPython's initialization (doing a PyImport_AppendInittab("plpy") just because PLPython does it too), and it would be possible for it to detect if the runtime is already initialized and skip it. However, it's equally possible for PLPython to be init'd second (as in this case), in which no changes in Multicorn would prevent the error.

Maybe if Multicorn could runtime detect the presence of the PLy_initialize symbol and invoke it... which stores a static bool for whether it has initialized the runtime... that would avoid the problem. But as it's a static function and this is C, I'm not sure that would be very feasible.

I'll do some more research... but in the short-term this is the problem description at least.

@mfenniak mfenniak self-assigned this May 29, 2024
@ergo70
Copy link

ergo70 commented Sep 25, 2024

I can confirm this behavior. Currently multicorn2 and PL/Python3 are mutual exclusive with Python 3.12. Once one is enabled, the second sends the server into recovery mode, regardless of the order the extensions are CREATEd.

@ergo70
Copy link

ergo70 commented Sep 25, 2024

Wrapping in Py_IsInitialized() seems to fix this.

 /* Try to load plpython3 with its own module */ 
        PG_TRY();
        {
        void * PyInit_plpy = load_external_function("plpython3", "PyInit_plpy", true, NULL);
        if (!Py_IsInitialized()) {      
              PyImport_AppendInittab("plpy", PyInit_plpy);
        }
        need_import_plpy = true;
        }
        PG_CATCH();
        {
                need_import_plpy = false;
        }
        PG_END_TRY();
        if (!Py_IsInitialized()) {
                Py_Initialize();
        }

multicorn.c

and

 /* The rest should only be done once per session */
        if (inited)
                return;
          
        if (!Py_IsInitialized()) {
                PyImport_AppendInittab("plpy", PyInit_plpy);
                Py_Initialize();
        }

plpy_main.c

I have no further knowledge of using Python in C, but those changes apparently allow both extensions to co-exist, regardless of load order.

@luss
Copy link
Contributor

luss commented Sep 26, 2024 via email

@luss
Copy link
Contributor

luss commented Sep 26, 2024 via email

@luss
Copy link
Contributor

luss commented Jan 8, 2025

I winder if we need to do something like what's proposed in #69 to handle this problem.

@mbachry
Copy link
Contributor

mbachry commented Jan 9, 2025

My pull request #69 won't help with this issue. This one is a bit hopeless. The only way I see is forcing plypython3 to initialize before multicorn carries on with its own initialization. It's not easy because everything directly useful is static in plpython3. Here's a hacky way how I was able to do it and make tests run with Python 3.12:

diff --git a/src/multicorn.c b/src/multicorn.c
index 22bf7914..fc3968f8 100644
--- a/src/multicorn.c
+++ b/src/multicorn.c
@@ -130,9 +130,19 @@ _PG_init()
        /* Try to load plpython3 with its own module */
        PG_TRY();
        {
-       void * PyInit_plpy = load_external_function("plpython3", "PyInit_plpy", true, NULL);
-       PyImport_AppendInittab("plpy", PyInit_plpy);
-       need_import_plpy = true;
+              PGFunction ply_call = load_external_function("plpython3", "plpython3_call_handler", true, NULL);
+
+              /* Call plpython3_call_handler postgres function using
+                 psql calling convention. We don't count on it doing
+                 anything useful except initializing plpython3 */
+              LOCAL_FCINFO(fcinfo, 1);
+              struct FmgrInfo flinfo = {.fn_oid = 0};
+              InitFunctionCallInfoData(*fcinfo, &flinfo, 1, InvalidOid, NULL, NULL);
+              fcinfo->args[0].value = 0;
+              fcinfo->args[0].isnull = false;
+              ply_call(fcinfo);
+
+              need_import_plpy = true;
        }
        PG_CATCH();
        {

It exploits the fact that plpython3_call_handler is exported in the DLL and the first thing it does is calling PLy_initialize (https://github.com/postgres/postgres/blob/master/src/pl/plpython/plpy_main.c#L198).

# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants