Error during segment -- cudaErrorIllegalAddress

[pzumbo]

Hi,

I get an error during segment step

segment(
    trained_model,
    dm_predict,
    save_dir=segmentation_output_dir,
    seg_tag='segger_output',
    transcript_file=xenium_data_dir / 'transcripts.parquet',  
    receptive_field=receptive_field,
    min_transcripts=5,
    cell_id_col='segger_cell_id',
    use_cc=False,
    knn_method='kd_tree',
    verbose=True,
    save_cell_masks=False
)

The error is:

Processing Train batches: 6%|████▍ Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/workspace/segger_dev/src/segger/prediction/predict_parquet.py", line 570, in segment predict_batch( File "/workspace/segger_dev/src/segger/prediction/predict_parquet.py", line 375, in predict_batch assignments["score"] = cp.asnumpy(belongs) # Move back to CPU ^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/dist-packages/cupy/init.py", line 807, in asnumpy return a.get(stream=stream, order=order, out=out, blocking=blocking) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "cupy/_core/core.pyx", line 1873, in cupy._core.core._ndarray_base.get File "cupy/_core/core.pyx", line 1978, in cupy._core.core._ndarray_base.get File "cupy/cuda/memory.pyx", line 683, in cupy.cuda.memory.MemoryPointer.copy_to_host_async File "cupy_backends/cuda/api/runtime.pyx", line 940, in cupy_backends.cuda.api.runtime.streamIsCapturing File "cupy_backends/cuda/api/runtime.pyx", line 146, in cupy_backends.cuda.api.runtime.check_status cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorIllegalAddress: an illegal memory access was encountered

This only is occurring on some xenium runs. The training completed without error.

[EliHei2]

Hey @pzumbo, are you using CPU for this step? in that case this is expected, we assumed that the users already have GPUs, we're gonna release a version soon that does not need a GPU for this step.

[AdnanAbouelela]

Hi @EliHei2, I'm encountering the same error during the segment step, I'm running it on an A100. It does not seem to be an issue with a particular training batch. I'd appreciate your input!