GH-47128: [Python] Numba-CUDA interop with NVIDIA bindings (#47150)

### Rationale for this change

Testing with [Numba-CUDA](https://github.com/NVIDIA/numba-cuda) which uses the [NVIDIA CUDA Python bindings](https://github.com/NVIDIA/cuda-python) by default identified that PyArrow Numba interop has an incompatibility with Numba / Numba-CUDA using the NVIDIA bindings. See Issue #47128.

### What changes are included in this PR?

The fix is to get device pointer values from their `device_pointer_value` property, which is consistent across the ctypes and NVIDIA bindings in Numba.

I also attempted to update the CI config to install Numba-CUDA. I think some of the comments in `docker-compose.yml` were a bit out of sync with changes to it, so I also updated comments that appeared to be relevant to reflect what I had to run locally. I could have got the CI changes all wrong - happy to change these, as they're not really the critical part of this PR.

Fixes #47128.

### Are these changes tested?

Yes, by the existing `test_cuda_numba_interop.py` and the CI changes in this PR.

### Are there any user-facing changes?

No.
* GitHub Issue: #47128

Authored-by: Graham Markall <[email protected]>
Signed-off-by: Sutou Kouhei <[email protected]>

Author: gmarkall

.env

@@ -67,6 +67,7 @@ LLVM=18
 MAVEN=3.8.7
 NODE=20
 NUMBA=latest
+NUMBA_CUDA=latest
 NUMPY=latest
 PANDAS=latest
 PYTHON=3.9

ci/docker/linux-apt-python-3.dockerfile

@@ -32,9 +32,10 @@ RUN python3 -m venv ${ARROW_PYTHON_VENV} && \
       -r arrow/python/requirements-test.txt
 
 ARG numba
+ARG numba_cuda
 COPY ci/scripts/install_numba.sh /arrow/ci/scripts/
 RUN if [ "${numba}" != "" ]; then \
-        /arrow/ci/scripts/install_numba.sh ${numba} \
+        /arrow/ci/scripts/install_numba.sh ${numba} ${numba_cuda} \
     ; fi
 
 ENV ARROW_ACERO=ON \

ci/scripts/install_numba.sh

@@ -19,8 +19,8 @@
 
 set -e
 
-if [ "$#" -ne 1 ]; then
-  echo "Usage: $0 <numba version>"
+if [ "$#" -ne 1 ] && [ "$#" -ne 2 ]; then
+  echo "Usage: $0 <numba version> [numba-cuda version]"
   exit 1
 fi
 
@@ -41,3 +41,17 @@ elif [ "${numba}" = "latest" ]; then
 else
   pip install "numba==${numba}"
 fi
+
+if [ "$#" -eq 1 ]; then
+  exit 0
+fi
+
+numba_cuda=$2
+
+if [ "${numba_cuda}" = "master" ]; then
+  pip install https://github.com/NVIDIA/numba-cuda/archive/main.tar.gz#egg=numba-cuda
+elif [ "${numba_cuda}" = "latest" ]; then
+  pip install numba-cuda
+else
+  pip install "numba-cuda==${numba_cuda}"
+fi

docker-compose.yml

@@ -560,10 +560,11 @@ services:
 
   ubuntu-cuda-cpp:
     # Usage:
-    #   docker compose build cuda-cpp
-    #   docker compose run --rm cuda-cpp
+    #   docker compose build ubuntu-cuda-cpp
+    #   docker compose run --rm ubuntu-cuda-cpp
     # Parameters:
     #   ARCH: amd64
+    #   UBUNTU: 22.04, 24.04
     #   CUDA: <depends on your nvidia driver, should match system CUDA>
     image: ${REPO}:${ARCH}-ubuntu-${UBUNTU}-cuda-${CUDA}-cpp
     build:
@@ -946,13 +947,15 @@ services:
 
   ubuntu-cuda-python:
     # Usage:
-    #   docker compose build cuda-cpp
-    #   docker compose build cuda-python
-    #   docker compose run --rm cuda-python
+    #   docker compose build ubuntu-cuda-cpp
+    #   docker compose build ubuntu-cuda-python
+    #   docker compose run --rm ubuntu-cuda-python
     # Parameters:
     #   ARCH: amd64
     #   CUDA: <depends on your nvidia driver, should match system CUDA>
     #   UBUNTU: 22.04, 24.04
+    #   NUMBA: master, latest, <version>
+    #   NUMBA_CUDA: master, latest, <version>
     image: ${REPO}:${ARCH}-ubuntu-${UBUNTU}-cuda-${CUDA}-python-3
     build:
       context: .
@@ -962,6 +965,7 @@ services:
       args:
         base: ${REPO}:${ARCH}-ubuntu-${UBUNTU}-cuda-${CUDA}-cpp
         numba: ${NUMBA}
+        numba_cuda: ${NUMBA_CUDA}
     shm_size: *shm-size
     environment:
       <<: [*common, *ccache, *sccache]

python/pyarrow/_cuda.pyx

@@ -451,9 +451,9 @@ cdef class CudaBuffer(Buffer):
           Device buffer as a view of numba MemoryPointer.
         """
         ctx = Context.from_numba(mem.context)
-        if mem.device_pointer.value is None and mem.size==0:
+        if mem.device_pointer_value is None and mem.size==0:
             return ctx.new_buffer(0)
-        return ctx.foreign_buffer(mem.device_pointer.value, mem.size, base=mem)
+        return ctx.foreign_buffer(mem.device_pointer_value, mem.size, base=mem)
 
     def to_numba(self):
         """Return numba memory pointer of CudaBuffer instance.