GC-related crash on PyPy3.11 7.3.20 when memcached server exits before the program

[mgorny]

I've been debugging a crash in pytest-services test suite, when running with PyPy 3.11 7.3.20. I've been able to reduce it to the following test case:

import os
import os.path
import subprocess
import time

import pylibmc


def test_memcached():
    memcached_socket = "/tmp/memcached.sock"

    if os.path.exists(memcached_socket):
        os.unlink(memcached_socket)

    watcher = subprocess.Popen(["memcached", "-s", memcached_socket])

    while not os.path.exists(memcached_socket):
        assert watcher.returncode is None
        time.sleep(1)

    mc = pylibmc.Client([memcached_socket])
    mc.set("some", 1)

    watcher.terminate()
    watcher.communicate()

$ pypy3.11 -m pytest test.py
========================================================= test session starts =========================================================
platform linux -- Python 3.11.13[pypy-7.3.20-final], pytest-8.4.1, pluggy-1.6.0
rootdir: /tmp/foo
collected 1 item                                                                                                                      

test.py .                                                                                                                       [100%]

========================================================== 1 passed in 1.03s ==========================================================
pypy3.11: libmemcached/do.cc:96: memcached_return_t memcached_vdo(memcached_instance_st*, libmemcached_io_vector_st*, size_t, bool): Assertion `memcached_last_error(instance->root) == MEMCACHED_SUCCESS' failed.
Aborted                    (core dumped) pypy3.11 -m pytest test.py

What's interesting, I can reproduce it only when running the case via pytest, but that's probably incidental — my educated guess is that the code somewhere is relying on a specific GC order. That said, I can't rule out if this isn't an actual bug in libmemcached.

Backtrace:

(gdb) bt
#0  0x00007fcf68958dbc in ?? () from /usr/lib64/libc.so.6
#1  0x00007fcf689008e6 in raise () from /usr/lib64/libc.so.6
#2  0x00007fcf688e834b in abort () from /usr/lib64/libc.so.6
#3  0x00007fcf688e82b5 in ?? () from /usr/lib64/libc.so.6
#4  0x00007fcf66123439 in memcached_vdo (instance=0x5648f58c9ea0, vector=0x7ffd7c93c7f0, count=1, with_flush=<optimized out>)
    at libmemcached/do.cc:96
#5  0x00007fcf66138fec in (anonymous namespace)::send_quit_message (instance=0x5648f58c9ea0) at libmemcached/quit.cc:67
#6  memcached_quit_server (instance=0x5648f58c9ea0, io_death=io_death@entry=false) at libmemcached/quit.cc:115
#7  0x00007fcf66139036 in send_quit (memc=memc@entry=0x5648f58801b0) at libmemcached/quit.cc:143
#8  0x00007fcf661376b0 in __memcached_free (ptr=0x5648f58801b0, release_st=true) at libmemcached/memcached.cc:134
#9  0x00007fcf66137873 in memcached_free (ptr=<optimized out>) at libmemcached/memcached.cc:312
#10 0x00007fcf683582ed in PylibMC_ClientType_dealloc (self=0x5648f58b2a00) at src/_pylibmcmodule.c:116
#11 0x00005648ed013ef8 in ?? ()
[…]
#171 0x00005648eda32418 in ?? ()
#172 0x00007fcf688ea4ae in ?? () from /usr/lib64/libc.so.6
#173 0x00007fcf688ea569 in __libc_start_main () from /usr/lib64/libc.so.6
#174 0x00005648ec67e195 in _start ()

This is libmemcached 1.0.18, pylibmc 1.6.3.

[mgorny]

It's been pointed out to me that this may be an actual bug in libmemcached — fixed in its fork @ awesomized/libmemcached@9973d38.

If that's the case, then it might be a good idea to switch the project over.