[Python][FlightRPC] Tests segfault on OSX in conda-forge

[h-vetinari]

Describe the bug, including details regarding any error messages, version, and platform.

#14832 switched on the test suite within the conda-builds, which itself synced with conda-forge/arrow-cpp-feedstock#875.

Then as of conda-forge/arrow-cpp-feedstock#975 (synced back in #34652), we needed to add

arrow/dev/tasks/conda-recipes/arrow-cpp/meta.yaml

Lines 398 to 399 in 5e1c112

	# segfaults on OSX: to investigate ASAP
	{% set tests_to_skip = tests_to_skip + " or test_flight" %} # [osx]

This is a pretty serious issue because potentially the whole flight module is broken/unusable on osx. I've raised this point in several discussions around the conda testing here, but never got around to posting an issue.

I've retried with arrow 13.0.0 recently and the segfault remains, already in the first test of the test_flight.py module

test_feather.py .............................ss......................... [ 27%]
.......ss....x........s.....s..                                          [ 27%]
test_filesystem.py .....                                                 [ 27%]
Fatal Python error: Segmentation fault

Thread 0x0000000109600e00 (most recent call first):
  File "$PREFIX/lib/python3.8/site-packages/pyarrow/tests/test_flight.py", line 1000 in test_flight_client_close
  File "$PREFIX/lib/python3.8/site-packages/_pytest/python.py", line 194 in pytest_pyfunc_call
  File "$PREFIX/lib/python3.8/site-packages/pluggy/_callers.py", line 80 in _multicall
  File "$PREFIX/lib/python3.8/site-packages/pluggy/_manager.py", line 112 in _hookexec
  File "$PREFIX/lib/python3.8/site-packages/pluggy/_hooks.py", line 433 in __call__
  File "$PREFIX/lib/python3.8/site-packages/_pytest/python.py", line 1788 in runtest
  File "$PREFIX/lib/python3.8/site-packages/_pytest/runner.py", line 169 in pytest_runtest_call
  File "$PREFIX/lib/python3.8/site-packages/pluggy/_callers.py", line 80 in _multicall
  File "$PREFIX/lib/python3.8/site-packages/pluggy/_manager.py", line 112 in _hookexec
  File "$PREFIX/lib/python3.8/site-packages/pluggy/_hooks.py", line 433 in __call__
  File "$PREFIX/lib/python3.8/site-packages/_pytest/runner.py", line 262 in <lambda>
  File "$PREFIX/lib/python3.8/site-packages/_pytest/runner.py", line 341 in from_call
  File "$PREFIX/lib/python3.8/site-packages/_pytest/runner.py", line 261 in call_runtest_hook
  File "$PREFIX/lib/python3.8/site-packages/_pytest/runner.py", line 222 in call_and_report
  File "$PREFIX/lib/python3.8/site-packages/_pytest/runner.py", line 133 in runtestprotocol
  File "$PREFIX/lib/python3.8/site-packages/_pytest/runner.py", line 114 in pytest_runtest_protocol
  File "$PREFIX/lib/python3.8/site-packages/pluggy/_callers.py", line 80 in _multicall
  File "$PREFIX/lib/python3.8/site-packages/pluggy/_manager.py", line 112 in _hookexec
  File "$PREFIX/lib/python3.8/site-packages/pluggy/_hooks.py", line 433 in __call__
  File "$PREFIX/lib/python3.8/site-packages/_pytest/main.py", line 349 in pytest_runtestloop
  File "$PREFIX/lib/python3.8/site-packages/pluggy/_callers.py", line 80 in _multicall
  File "$PREFIX/lib/python3.8/site-packages/pluggy/_manager.py", line 112 in _hookexec
  File "$PREFIX/lib/python3.8/site-packages/pluggy/_hooks.py", line 433 in __call__
  File "$PREFIX/lib/python3.8/site-packages/_pytest/main.py", line 324 in _main
  File "$PREFIX/lib/python3.8/site-packages/_pytest/main.py", line 270 in wrap_session
  File "$PREFIX/lib/python3.8/site-packages/_pytest/main.py", line 317 in pytest_cmdline_main
  File "$PREFIX/lib/python3.8/site-packages/pluggy/_callers.py", line 80 in _multicall
  File "$PREFIX/lib/python3.8/site-packages/pluggy/_manager.py", line 112 in _hookexec
  File "$PREFIX/lib/python3.8/site-packages/pluggy/_hooks.py", line 433 in __call__
  File "$PREFIX/lib/python3.8/site-packages/_pytest/config/__init__.py", line 166 in main
  File "$PREFIX/lib/python3.8/site-packages/_pytest/config/__init__.py", line 189 in console_main
  File "$PREFIX/bin/pytest", line 10 in <module>
/Users/runner/miniforge3/conda-bld/apache-arrow_1690235820790/test_tmp/run_test.sh: line 9: 43620 Segmentation fault: 11  pytest -rfEs -k "not (_not_a_real_test or test_cuda or test_cpp_extension_in_python or test_cython_api or test_visit_strings)"

To fix this, the line quoted above should be removed, and the conda tests still pass.

Component(s)

C++, FlightRPC, Packaging, Python

[raulcd]

I haven't seen this issue on our current CI jobs. From what I've seen we are using a pretty old version of ABSL:

ARROW_ABSL_BUILD_VERSION=20211102.0

I've created a testing PR to see if we can reproduce this issue if I try upgrading it: #36909

[raulcd]

It doesn't seem to be related as for the wheels generation we use vcpkg and we are already using the newer abseil version:

 The following packages will be built and installed:
  * abseil[core]:arm64-osx -> 20230125.0#1
    abseil[core]:arm64-osx-static-release -> 20230125.0#1

or

   * abseil[core]:x64-osx -> 20230125.0#1
    abseil[core]:amd64-osx-static-release -> 20230125.0#1

[h-vetinari]

When I was looking into this at the time of conda-forge/arrow-cpp-feedstock#975, I noted:

It seems this is segfaulting in test_flight.py::test_flight_list_flights on osx-64 both for arrow 10 & 11, might be related to conda-forge/grpc-cpp-feedstock#281, which I haven't had time to look into yet...

and

Looking at the source of that test and the utility class ConstantFlightServer, it notes:

See ARROW-4796: this server implementation will segfault if Flight
does not properly hold a reference to the Table object.

[h-vetinari]

I think the cautionary comment from the test code is probably still the best hint for now:

See ARROW-4796: this server implementation will segfault if Flight
does not properly hold a reference to the Table object.

Did someone follow this lead already? If not, could someone from flight side have a look?

[kou]

@lidavidm Could you take a look at this?

[lidavidm]

I think that comment is a red herring.

Is this x64 only, or does arm also crash? (I only have an arm mac and I'm reluctant to set up a rosetta-d dev environment on my personal machine, though I suppose I may have no choice)