Segmentation fault in combination with certain libraries

[niboshi]

import _hashlib
import mysql.connector
import pfio

dest_path = 'hdfs:///my/hdfs/file'
with pfio.open(dest_path, 'wb') as file_out:
    file_out.write(b'data')

In my environment, the above code results in a segmentation fault:

python: Relink `/usr/local/lib/python3.6/site-packages/pyarrow/libarrow.so.17' with `/lib/x86_64-linux-gnu/librt.so.1' for IFUNC symbol `clock_gettime'
Segmentation fault (core dumped)

• You can replace _hashlib with any libraries that internally depend on hashlib, including itself.
• Removing any of the first two imports would resolve the error.

Some environment info:

$ which python
/usr/local/bin/python

$ python --version
Python 3.6.9

$ pip freeze | grep mysql
mysql-connector-python==8.0.21

$ pip freeze | grep pfio
pfio==1.0.0

$ ldd /usr/local/lib/python3.6/lib-dynload/_hashlib.cpython-36m-x86_64-linux-gnu.so
        linux-vdso.so.1 (0x00007ffd4e7ce000)
        libcrypto.so.1.1 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 (0x00007fa2138dc000)
        libpython3.6m.so.1.0 => /usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0 (0x00007fa213231000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fa212e40000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fa212c3c000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fa212a1d000)
        libexpat.so.1 => /lib/x86_64-linux-gnu/libexpat.so.1 (0x00007fa2127eb000)
        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fa2125ce000)
        libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007fa2123cb000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fa21202d000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fa213fae000)

[niboshi]

The order matters:

NG

import _hashlib
import mysql.connector
import pfio

import _hashlib
import pfio
import mysql.connector

import pfio
import _hashlib
import mysql.connector

OK

import mysql.connector
import _hashlib
import pfio

import mysql.connector
import pfio
import _hashlib

import pfio
import mysql.connector
import _hashlib

[belldandyxtq]

Thank you for the report, let me look into it. It seems the problem is in the pyarrow

[kuenishi]

The version of PyArrow isn't clear at the description, but I think it's not the latest 3.0.0, which is currently fixed for PFIO 1.1.0. Do you have any chance to try reproduction with PFIO 1.1.0? @niboshi