[BUG] Intermittent failures in test_lock #787
Description
A failure that sometimes seems to occur in test_lock. E.g. in https://github.com/NVIDIA/numba-cuda/actions/runs/21905724736/job/63246007708?pr=786
The short version of the traceback looks like:
Traceback (most recent call last):
File "cuda/core/_memory/_buffer.pyx", line 140, in cuda.core._memory._buffer.Buffer.close
File "cuda/core/_memory/_buffer.pyx", line 377, in cuda.core._memory._buffer.Buffer_close
File "C:\actions-runner\_work\_tool\Python\3.12.10\x64\Lib\site-packages\cuda\core\cu13\_memory\_legacy.py", line 105, in deallocate
File "cuda/core/_stream.pyx", line 228, in cuda.core._stream.Stream.sync
File "cuda/core/_utils/cuda_utils.pyx", line 63, in cuda.core._utils.cuda_utils.HANDLE_RETURN
File "cuda/core/_utils/cuda_utils.pyx", line 81, in cuda.core._utils.cuda_utils._check_driver_error
cuda.core._utils.cuda_utils.CUDAError: CUDA_ERROR_INVALID_CONTEXT: This most frequently indicates that there is no context bound to the current thread. This can also be returned if the context passed to an API call is not a valid handle (such as a context that has had ::cuCtxDestroy() invoked on it). This can also be returned if a user mixes different API versions (i.e. 3010 context with 3020 API calls). See ::cuCtxGetApiVersion() for more details. This can also be returned if the green context passed to an API call was not converted to a ::CUcontext using ::cuCtxFromGreenCtx API.
Captured traceback and related info
__________________________ TestDispatcher.test_lock ___________________________
[gw6] win32 -- Python 3.12.10 C:\actions-runner\_work\_tool\Python\3.12.10\x64\python.exe
> ???
cuda/core/_memory/_buffer.pyx:140:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
cuda/core/_memory/_buffer.pyx:377: in cuda.core._memory._buffer.Buffer_close
???
..\..\..\_tool\Python\3.12.10\x64\Lib\site-packages\cuda\core\cu13\_memory\_legacy.py:105: in deallocate
stream.sync()
cuda/core/_stream.pyx:228: in cuda.core._stream.Stream.sync
???
cuda/core/_utils/cuda_utils.pyx:63: in cuda.core._utils.cuda_utils.HANDLE_RETURN
???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> ???
E cuda.core._utils.cuda_utils.CUDAError: CUDA_ERROR_INVALID_CONTEXT: This most frequently indicates that there is no context bound to the current thread. This can also be returned if the context passed to an API call is not a valid handle (such as a context that has had ::cuCtxDestroy() invoked on it). This can also be returned if a user mixes different API versions (i.e. 3010 context with 3020 API calls). See ::cuCtxGetApiVersion() for more details. This can also be returned if the green context passed to an API call was not converted to a ::CUcontext using ::cuCtxFromGreenCtx API.
cuda/core/_utils/cuda_utils.pyx:81: CUDAError
The above exception was the direct cause of the following exception:
cls = <class '_pytest.runner.CallInfo'>
func = <function call_and_report.<locals>.<lambda> at 0x00000205589BFB00>
when = 'call'
reraise = (<class '_pytest.outcomes.Exit'>, <class 'KeyboardInterrupt'>)
@classmethod
def from_call(
cls,
func: Callable[[], TResult],
when: Literal["collect", "setup", "call", "teardown"],
reraise: type[BaseException] | tuple[type[BaseException], ...] | None = None,
) -> CallInfo[TResult]:
"""Call func, wrapping the result in a CallInfo.
:param func:
The function to call. Called without arguments.
:type func: Callable[[], _pytest.runner.TResult]
:param when:
The phase in which the function is called.
:param reraise:
Exception or exceptions that shall propagate if raised by the
function, instead of being wrapped in the CallInfo.
"""
excinfo = None
instant = timing.Instant()
try:
> result: TResult | None = func()
^^^^^^
..\..\..\_tool\Python\3.12.10\x64\Lib\site-packages\_pytest\runner.py:344:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
..\..\..\_tool\Python\3.12.10\x64\Lib\site-packages\_pytest\runner.py:246: in <lambda>
lambda: runtest_hook(item=item, **kwds), when=when, reraise=reraise
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
..\..\..\_tool\Python\3.12.10\x64\Lib\site-packages\pluggy\_hooks.py:512: in __call__
return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
..\..\..\_tool\Python\3.12.10\x64\Lib\site-packages\pluggy\_manager.py:120: in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
..\..\..\_tool\Python\3.12.10\x64\Lib\site-packages\_pytest\logging.py:850: in pytest_runtest_call
yield
..\..\..\_tool\Python\3.12.10\x64\Lib\site-packages\_pytest\capture.py:900: in pytest_runtest_call
return (yield)
^^^^^
..\..\..\_tool\Python\3.12.10\x64\Lib\site-packages\_pytest\skipping.py:263: in pytest_runtest_call
return (yield)
^^^^^
..\..\..\_tool\Python\3.12.10\x64\Lib\site-packages\_pytest\unraisableexception.py:158: in pytest_runtest_call
collect_unraisable(item.config)
..\..\..\_tool\Python\3.12.10\x64\Lib\site-packages\_pytest\unraisableexception.py:79: in collect_unraisable
raise errors[0]
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
config = <_pytest.config.Config object at 0x000002052F9C69C0>
def collect_unraisable(config: Config) -> None:
pop_unraisable = config.stash[unraisable_exceptions].pop
errors: list[pytest.PytestUnraisableExceptionWarning | RuntimeError] = []
meta = None
hook_error = None
try:
while True:
try:
meta = pop_unraisable()
except IndexError:
break
if isinstance(meta, BaseException):
hook_error = RuntimeError("Failed to process unraisable exception")
hook_error.__cause__ = meta
errors.append(hook_error)
continue
msg = meta.msg
try:
> warnings.warn(pytest.PytestUnraisableExceptionWarning(msg))
E pytest.PytestUnraisableExceptionWarning: Exception ignored in: 'cuda.core._memory._buffer.Buffer.__dealloc__'
E Enable tracemalloc to get traceback where the object was allocated.
E See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.
..\..\..\_tool\Python\3.12.10\x64\Lib\site-packages\_pytest\unraisableexception.py:67: PytestUnraisableExceptionWarning
---------------------------- Captured stderr call -----------------------------
Traceback (most recent call last):
File "cuda/core/_memory/_buffer.pyx", line 140, in cuda.core._memory._buffer.Buffer.close
File "cuda/core/_memory/_buffer.pyx", line 377, in cuda.core._memory._buffer.Buffer_close
File "C:\actions-runner\_work\_tool\Python\3.12.10\x64\Lib\site-packages\cuda\core\cu13\_memory\_legacy.py", line 105, in deallocate
File "cuda/core/_stream.pyx", line 228, in cuda.core._stream.Stream.sync
File "cuda/core/_utils/cuda_utils.pyx", line 63, in cuda.core._utils.cuda_utils.HANDLE_RETURN
File "cuda/core/_utils/cuda_utils.pyx", line 81, in cuda.core._utils.cuda_utils._check_driver_error
cuda.core._utils.cuda_utils.CUDAError: CUDA_ERROR_INVALID_CONTEXT: This most frequently indicates that there is no context bound to the current thread. This can also be returned if the context passed to an API call is not a valid handle (such as a context that has had ::cuCtxDestroy() invoked on it). This can also be returned if a user mixes different API versions (i.e. 3010 context with 3020 API calls). See ::cuCtxGetApiVersion() for more details. This can also be returned if the green context passed to an API call was not converted to a ::CUcontext using ::cuCtxFromGreenCtx API.
=========================== short test summary info ===========================
FAILED cudapy/test_dispatcher.py::TestDispatcher::test_lock - pytest.PytestUnraisableExceptionWarning: Exception ignored in: 'cuda.core._memory._buffer.Buffer.__dealloc__'
Enable tracemalloc to get traceback where the object was allocated.
See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.
===== 1 failed, 1918 passed, 83 skipped, 13 xfailed in 124.72s (0:02:04) ======