How to fix: CUDA error(719)

[mkrzywda]

How can I fix CUDA error(719)?

[Hint: 'cudaErrorLaunchFailure'. An exception occurred on the device while executing a kernel. Common causes include dereferencing an invalid device pointerand accessing out of bounds shared memory. Less common cases can be system specific - more information about these cases canbe found in the system specific user guide. This leaves the process in an inconsistent state and any further CUDA work willreturn the same error. To continue using CUDA, the process must be terminated and relaunched.] (at /paddle/paddle/phi/backends/gpu/cuda/cuda_info.cc:265)

Traceback:

Traceback (most recent call last): File "/app/server.py", line 66, in PerformOcr text, details = extract_ocr_data(request.screenshot_content) File "/app/ocr_service.py", line 88, in extract_ocr_data result = ocr.ocr(image_data, slice=slice_params) File "/usr/local/lib/python3.10/dist-packages/paddleocr/paddleocr.py", line 760, in ocr dt_boxes, rec_res, _ = self.__call__(img, cls, slice) File "/usr/local/lib/python3.10/dist-packages/paddleocr/tools/infer/predict_system.py", line 94, in __call__ dt_boxes, elapse = self.text_detector(slice_crop, use_slice=True) File "/usr/local/lib/python3.10/dist-packages/paddleocr/tools/infer/predict_det.py", line 396, in __call__ dt_boxes, elapse = self.predict(img) File "/usr/local/lib/python3.10/dist-packages/paddleocr/tools/infer/predict_det.py", line 258, in predict output = output_tensor.copy_to_cpu() OSError: (External) CUDA error(719), unspecified launch failure. [Hint: 'cudaErrorLaunchFailure'. An exception occurred on the device while executing a kernel. Common causes include dereferencing an invalid device pointerand accessing out of bounds shared memory. Less common cases can be system specific - more information about these cases canbe found in the system specific user guide. This leaves the process in an inconsistent state and any further CUDA work willreturn the same error. To continue using CUDA, the process must be terminated and relaunched.] (at /paddle/paddle/phi/backends/gpu/cuda/cuda_info.cc:265)

ocr = PaddleOCR(
    use_angle_cls=False,
    lang="en",
    use_gpu=True,
    use_space_char=True,
    det_db_thresh=0.3,
    det_db_box_thresh=0.5,
    cls=False,
    bin=False,
    inv=False,
    alpha_color=False,
    max_text_length=100,
)

Cuda Params:

NVIDIA GeForce RTX 4090  
NVIDIA-SMI 535.216.03
Driver Version: 535.216.03
CUDA Version: 12.2 

paddlepaddle library

paddlepaddle-gpu==2.6.2
paddleocr==2.9.0

[GreatV]

The error CUDA error(719): unspecified launch failure typically indicates an issue where a CUDA kernel crashes due to an illegal memory access, such as:

• Dereferencing an invalid device pointer
• Accessing out-of-bounds shared memory
• GPU memory corruption from a previous operation

Since you're using PaddleOCR with CUDA 12.2 and an RTX 4090, follow these steps to debug and potentially fix the issue:

1. Restart the Process

Since error 719 puts the CUDA context in an invalid state, a restart is required to continue using CUDA. Restart your Python script or manually reset the GPU:

sudo nvidia-smi --gpu-reset

Then restart your script.

2. Check for GPU Compatibility Issues

Your CUDA 12.2 + NVIDIA-SMI 535.216.03 + cudnn 9.3.0 setup may not be fully compatible with paddlepaddle-gpu==2.6.2.

• Try downgrading CUDA to 11.8 and cuDNN to 8.6.0, as these are known to work better with PaddleOCR.

To verify your CUDA setup:

nvcc --version
nvidia-smi

If needed, reinstall CUDA and cuDNN.

3. Update PaddleOCR and PaddlePaddle

Your versions:

paddlepaddle-gpu==2.6.2
paddleocr==2.9.0

Try updating both:

pip install --upgrade paddlepaddle-gpu paddleocr

Alternatively, reinstall with explicit compatible versions:

pip install paddlepaddle-gpu==2.6.3 paddleocr==2.9.1

4. Set CUDA Environment Variables

Manually setting CUDA environment variables can sometimes resolve errors:

export CUDA_LAUNCH_BLOCKING=1
export CUDA_VISIBLE_DEVICES=0  # Adjust based on your device
export PYTHONUNBUFFERED=1

Try running PaddleOCR with:

CUDA_LAUNCH_BLOCKING=1 python server.py

This forces synchronous execution, which helps debug CUDA errors.

5. Reduce Your Computation Load

Your detection threshold values:

det_db_thresh=0.3,
det_db_box_thresh=0.5,

Try adjusting them higher (e.g., 0.5 and 0.7), as lower values can increase the workload, potentially causing memory issues.

6. Avoid TensorRT (If Enabled)

A common reported issue involves TensorRT compatibility problems when using PaddleOCR in English mode (lang='en').

Check if you're using TensorRT:

ocr = PaddleOCR(use_tensorrt=True, use_gpu=True, lang="en", ...)

If so, disable it and test:

ocr = PaddleOCR(use_tensorrt=False, use_gpu=True, lang="en", ...)

Refer to this related Issue:
➡ use_tensorrt=True does not work with lang='en'

7. Check GPU Memory Usage

If your RTX 4090 runs out of memory, crashes may occur. Monitor memory usage while running:

watch -n 1 nvidia-smi

If usage spikes to near 100%, you may need to reduce batch sizes or image resolution.

8. Test with a Minimal Example

Run this minimal script to see if the error reproduces:

from paddleocr import PaddleOCR

ocr = PaddleOCR(use_gpu=True)  
result = ocr.ocr('test.jpg')
print(result)

If this works but your main script fails, something in your image handling or additional parameters is triggering the CUDA issue.

9. Ensure PaddlePaddle is Built for the Correct CUDA Version

Check if PaddlePaddle is compiled for your CUDA version:

import paddle
print(paddle.utils.run_check())

If there's a mismatch, install a compatible version.

---

Final Steps

1. Restart your process and reset the GPU.
2. Check CUDA/PaddlePaddle compatibility.
3. Disable TensorRT (use_tensorrt=False).
4. Try CUDA 11.8 + cuDNN 8.6.0 for better stability.
5. Monitor nvidia-smi for memory issues.

Let me know if you need further debugging! 🚀

---

Response generated by 🤖 feifei-bot | chatgpt-4o-latest

⚠️ This is an AI-generated response. Please verify any critical information.

[GreatV]

You can try updating the paddle version to 3.0 rc1. Refer to https://www.paddlepaddle.org.cn/install/quick