Python: Driver manager not using entrypoint from manifest

[ianmcook]

What happened?

Steps to reproduce (on macOS):

Download libduckdb.dylib from https://github.com/duckdb/duckdb/releases/download/v1.3.2/libduckdb-osx-universal.zip and put it somewhere where dlopen can find it.

Install a nightly build of the driver manager:

pip install --pre --extra-index-url https://repo.fury.io/arrow-adbc-nightlies adbc-driver-manager

Create a manifest foo.toml containing this:

[Driver]
entrypoint = 'duckdb_adbc_init'
shared = 'duckdb'

In the same directory where foo.toml is, run:

import pyarrow as pa
from adbc_driver_manager import dbapi
import os

os.environ['ADBC_CONFIG_PATH'] = os.getcwd()

with dbapi.connect(
    driver="foo",
) as con, con.cursor() as cursor:
    cursor.execute("SELECT 1;")
    cursor.fetch_arrow_table()

Stack Trace

Traceback (most recent call last):
  File "<python-input-18>", line 1, in <module>
    with dbapi.connect(
         ~~~~~~~~~~~~~^
        driver="foo"
        ^^^^^^^^^^^^
    ) as con, con.cursor() as cursor:
    ^
  File "/Users/ian/adbc-examples/python-duckdb/python-duckdb-4/.venv/lib/python3.13/site-packages/adbc_driver_manager/dbapi.py", line 227, in connect
    db = _lib.AdbcDatabase(**db_kwargs)
  File "adbc_driver_manager/_lib.pyx", line 569, in adbc_driver_manager._lib.AdbcDatabase.__init__
  File "adbc_driver_manager/_lib.pyx", line 261, in adbc_driver_manager._lib.check_error
adbc_driver_manager.InternalError: INTERNAL: [Driver Manager] dlsym(AdbcDriverInit) failed: dlsym(0x722fcba0, AdbcDriverInit): symbol not found

How can we reproduce the bug?

No response

Environment/Setup

No response

[ianmcook]

@zeroshade any clue what's happening here?

[ianmcook]

Matt repro'd this using the nightly build of the driver manager, but couldn't repro it when building the driver manager from head of main. @lidavidm are the nightly builds stale or broken?

[eitsupi]

Side note, I tried out the driver manager nightly build on R-universe (https://apache.r-universe.dev/adbcdrivermanager) on Linux, and it worked flawlessly.
(It was my first time trying the manifest file, and I'm really impressed with how well it worked.)

[lidavidm]

@ianmcook what is your pip freeze?

[lidavidm]

Actually yeah, it seems nothing's been uploaded in quite a while. Frankly, I'm surprised it still exists

https://manage.fury.io/arrow-adbc-nightlies/python:adbc-driver-manager/versions

[lidavidm]

The pipeline itself seems to still "work", so I wonder if it's just that we ran out of space on Gemfury.

[lidavidm]

I went and deleted a few by hand and I'll re-run the pipeline.

I used to have a step that would delete old versions. Unfortunately a year or so ago it broke, apparently Gemfury had silently sunsetted the API. I'm not going to consider manually deleting packages a sustainable solution so we may want to find alternative hosting if anyone has ideas.

[lidavidm]

It seems all-new packages are now up.

[lidavidm]

It does seem they have a new API. I'd have to check in an API token as a secret, though

[ianmcook]

Thanks. I can confirm this issue is solved using the fresh nightly build.

I opened a separate issue for the Gemfury delete fix at #3294. I'll close this one.