Deploy YOLOv8 with TensorRT and DeepStream SDK | Seeed Studio Wiki

[giscus[bot]]

Deploy YOLOv8 with TensorRT and DeepStream SDK | Seeed Studio Wiki

Deploy YOLOv8 on NVIDIA Jetson using TensorRT and DeepStream SDK - Data Label, AI Model Train, AI Model Deploy

https://wiki.seeedstudio.com/YOLOv8-DeepStream-TRT-Jetson/

[asduban]

Hello,

I have this error at the end that I can't resolve: "CUDA Failure: CUDA Driver version is insufficient for CUDA Runtime version in file yoloplugins.cpp. Aborted (core dumped)"
I work with the Jetson AGX Orin developpement kit.
Do you have any idea to solve my problem ?

Astrid Duban

[lakshanthad]

Hello,

Can I know the JetPack version and the DeepStream version you have installed?

[faizan1234567]

can we use it with deepstream 6.4.0 becasue i am getting an error when I use it with yolov7.

[vodan37]

What about calibration for int8?
did you do this? How?

[ParmeetSinghKhatat]

I have reComputer J3011 JETSON ORIN. It has tensorrt version 10.3.0
I also have created a .engine file from my .pt file to run my model . Following is my script to run the Instance segmentation model from YOLO on a video stream

import cv2
import numpy as np
import ctypes
import pycuda.driver as cuda
import pycuda.autoinit
import tensorrt as trt
from typing import List, Optional

Assuming you have the HostDeviceMem, allocate_buffers, do_inference from your original script

Load the TensorRT engine from the .engine file

def load_engine(trt_logger, engine_path: str) -> trt.ICudaEngine:
with open(engine_path, 'rb') as f:
engine_data = f.read()
runtime = trt.Runtime(trt_logger)
engine = runtime.deserialize_cuda_engine(engine_data)
return engine

def allocate_buffers(engine: trt.ICudaEngine):
inputs = []
outputs = []
bindings = []
stream = cuda.Stream()

# Loop over bindings and allocate memory
for binding in engine:
    tensor_shape = engine.get_tensor_shape(binding)
    dtype = trt.nptype(engine.get_tensor_dtype(binding))  # Get dtype of tensor
    size = trt.volume(tensor_shape) * dtype.itemsize  # Calculate buffer size

    # Allocate device memory
    if engine.binding_is_input(binding):
        input_mem = cuda.mem_alloc(size)
        inputs.append(input_mem)
        bindings.append(int(input_mem))
    else:
        output_mem = cuda.mem_alloc(size)
        outputs.append(output_mem)
        bindings.append(int(output_mem))

return inputs, outputs, bindings, stream

Function to process a frame from the video and perform inference

def process_frame(frame: np.ndarray, context, engine, inputs, outputs, bindings, stream):
# Preprocess the frame (resize, normalize, etc.) based on model's input shape
input_shape = (1, 3, 224, 224) # Adjust based on your model's input shape
frame_resized = cv2.resize(frame, (224, 224)) # Resize frame to match input shape
frame_normalized = frame_resized.astype(np.float32) / 255.0 # Normalize the frame
frame_input = np.transpose(frame_normalized, (2, 0, 1)) # Convert to (C, H, W)

# Copy input data to host buffer
inputs[0].host = frame_input.flatten()

# Run inference
outputs = do_inference(context, engine, bindings, inputs, outputs, stream)

# Post-process output (get predicted class, etc.)
output_array = outputs[0].flatten()
predicted_class = np.argmax(output_array)
return predicted_class, outputs[0]

Video processing loop

def run_inference_on_video(engine_path: str, video_path: str):
trt_logger = trt.Logger(trt.Logger.WARNING)
engine = load_engine(trt_logger, engine_path)
context = engine.create_execution_context()

# Allocate memory buffers for inputs and outputs
inputs, outputs, bindings, stream = allocate_buffers(engine)

# Open video using OpenCV
cap = cv2.VideoCapture(video_path)
if not cap.isOpened():
    print("Error opening video file.")
    return

while cap.isOpened():
    ret, frame = cap.read()
    if not ret:
        break  # End of video

    # Process the frame and run inference
    predicted_class, output = process_frame(frame, context, engine, inputs, outputs, bindings, stream)

    # Display frame with predicted class
    cv2.putText(frame, f"Predicted Class: {predicted_class}", (20, 30), cv2.FONT_HERSHEY_SIMPLEX, 1, (0, 255, 0), 2)
    cv2.imshow('Video Inference', frame)

    if cv2.waitKey(1) & 0xFF == ord('q'):
        break  # Exit loop if 'q' is pressed

# Clean up and release video capture object
cap.release()
cv2.destroyAllWindows()

# Free allocated buffers
free_buffers(inputs, outputs, stream)

Run the inference on a video

if name == "main":
engine_path = # Path to .engine file
video_path = # Path to video file
run_inference_on_video(engine_path, video_path)

But i am facing the following error continously
raceback (most recent call last):

File "/home/eric/1.py", line 106, in
run_inference_on_video(engine_path, video_path)
File "/home/eric/1.py", line 71, in run_inference_on_video
inputs, outputs, bindings, stream = allocate_buffers(engine)
File "/home/eric/1.py", line 29, in allocate_buffers
size = trt.volume(tensor_shape) * dtype.itemsize # Calculate buffer size
TypeError: unsupported operand type(s) for *: 'int' and 'getset_descriptor'
[01/25/2025-19:09:25] [TRT] [E] [graphContext.h::~MyelinGraphContext::84] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:435] Error 709 destroying stream '0xaaaaef228060'.)
[01/25/2025-19:09:25] [TRT] [E] [graphContext.h::~MyelinGraphContext::84] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:435] Error 709 destroying stream '0xaaaaef26d050'.)
[01/25/2025-19:09:25] [TRT] [E] [defaultAllocator.cpp::deallocate::64] Error Code 1: Cuda Runtime (invalid device context)
[01/25/2025-19:09:25] [TRT] [E] [defaultAllocator.cpp::deallocate::64] Error Code 1: Cuda Runtime (invalid device context)
[01/25/2025-19:09:25] [TRT] [E] [defaultAllocator.cpp::deallocate::64] Error Code 1: Cuda Runtime (invalid device context)
[01/25/2025-19:09:25] [TRT] [E] [scopedCudaResources.cpp::~ScopedCudaStream::55] Error Code 1: Cuda Runtime (invalid device context)
[01/25/2025-19:09:25] [TRT] [E] [scopedCudaResources.cpp::~ScopedCudaEvent::32] Error Code 1: Cuda Runtime (invalid device context)
[01/25/2025-19:09:25] [TRT] [E] [scopedCudaResources.cpp::~ScopedCudaEvent::32] Error Code 1: Cuda Runtime (invalid device context)
[01/25/2025-19:09:25] [TRT] [E] [scopedCudaResources.cpp::~ScopedCudaEvent::32] Error Code 1: Cuda Runtime (invalid device context)
[01/25/2025-19:09:25] [TRT] [E] [scopedCudaResources.cpp::~ScopedCudaEvent::32] Error Code 1: Cuda Runtime (invalid device context)
[01/25/2025-19:09:25] [TRT] [E] [scopedCudaResources.cpp::~ScopedCudaEvent::32] Error Code 1: Cuda Runtime (invalid device context)
[01/25/2025-19:09:25] [TRT] [E] [scopedCudaResources.cpp::~ScopedCudaEvent::32] Error Code 1: Cuda Runtime (invalid device context)
[01/25/2025-19:09:25] [TRT] [E] [scopedCudaResources.cpp::~ScopedCudaEvent::32] Error Code 1: Cuda Runtime (invalid device context)
[01/25/2025-19:09:25] [TRT] [E] [scopedCudaResources.cpp::~ScopedCudaEvent::32] Error Code 1: Cuda Runtime (invalid device context)
[01/25/2025-19:09:25] [TRT] [E] [scopedCudaResources.cpp::~ScopedCudaEvent::32] Error Code 1: Cuda Runtime (invalid device context)
[01/25/2025-19:09:25] [TRT] [E] [scopedCudaResources.cpp::~ScopedCudaEvent::32] Error Code 1: Cuda Runtime (invalid device context)
[01/25/2025-19:09:25] [TRT] [E] [scopedCudaResources.cpp::~ScopedCudaEvent::32] Error Code 1: Cuda Runtime (invalid device context)
[01/25/2025-19:09:25] [TRT] [E] [scopedCudaResources.cpp::~ScopedCudaEvent::32] Error Code 1: Cuda Runtime (invalid device context)
[01/25/2025-19:09:25] [TRT] [E] [scopedCudaResources.cpp::~ScopedCudaEvent::32] Error Code 1: Cuda Runtime (invalid device context)
[01/25/2025-19:09:25] [TRT] [E] [scopedCudaResources.cpp::~ScopedCudaEvent::32] Error Code 1: Cuda Runtime (invalid device context)
[01/25/2025-19:09:25] [TRT] [E] [scopedCudaResources.cpp::~ScopedCudaEvent::32] Error Code 1: Cuda Runtime (invalid device context)
01/25/2025-19:09:25] [TRT] [E] [scopedCudaResources.cpp::~ScopedCudaEvent::32] Error Code 1: Cuda Runtime (invalid device context)
[01/25/2025-19:09:25] [TRT] [E] [scopedCudaResources.cpp::~ScopedCudaEvent::32] Error Code 1: Cuda Runtime (invalid device context)
[01/25/2025-19:09:25] [TRT] [E] [scopedCudaResources.cpp::~ScopedCudaEvent::32] Error Code 1: Cuda Runtime (invalid device context)
[01/25/2025-19:09:25] [TRT] [E] [scopedCudaResources.cpp::~ScopedCudaEvent::32] Error Code 1: Cuda Runtime (invalid device context)
[01/25/2025-19:09:25] [TRT] [E] [scopedCudaResources.cpp::~ScopedCudaEvent::32] Error Code 1: Cuda Runtime (invalid device context)
[01/25/2025-19:09:25] [TRT] [E] [scopedCudaResources.cpp::~ScopedCudaEvent::32] Error Code 1: Cuda Runtime (invalid device context)
[01/25/2025-19:09:25] [TRT] [E] [scopedCudaResources.cpp::~ScopedCudaEvent::32] Error Code 1: Cuda Runtime (invalid device context)
[01/25/2025-19:09:25] [TRT] [E] [scopedCudaResources.cpp::~ScopedCudaEvent::32] Error Code 1: Cuda Runtime (invalid device context)
[01/25/2025-19:09:25] [TRT] [E] [scopedCudaResources.cpp::~ScopedCudaEvent::32] Error Code 1: Cuda Runtime (invalid device context)
[01/25/2025-19:09:25] [TRT] [E] [scopedCudaResources.cpp::~ScopedCudaEvent::32] Error Code 1: Cuda Runtime (invalid device context)
[01/25/2025-19:09:25] [TRT] [E] [scopedCudaResources.cpp::~ScopedCudaEvent::32] Error Code 1: Cuda Runtime (invalid device context)
[01/25/2025-19:09:25] [TRT] [E] [scopedCudaResources.cpp::~ScopedCudaEvent::32] Error Code 1: Cuda Runtime (invalid device context)
[01/25/2025-19:09:25] [TRT] [E] [scopedCudaResources.cpp::~ScopedCudaEvent::32] Error Code 1: Cuda Runtime (invalid device context)
[01/25/2025-19:09:25] [TRT] [E] [scopedCudaResources.cpp::~ScopedCudaEvent::32] Error Code 1: Cuda Runtime (invalid device context)
[01/25/2025-19:09:25] [TRT] [E] [scopedCudaResources.cpp::~ScopedCudaEvent::32] Error Code 1: Cuda Runtime (invalid device context)
[01/25/2025-19:09:25] [TRT] [E] [scopedCudaResources.cpp::~ScopedCudaEvent::32] Error Code 1: Cuda Runtime (invalid device context)
[01/25/2025-19:09:25] [TRT] [E] [cudaDriverHelpers.cpp::operator()::106] Error Code 1: Cuda Driver (context is destroyed)
[01/25/2025-19:09:25] [TRT] [E] [cudaDriverHelpers.cpp::operator()::106] Error Code 1: Cuda Driver (context is destroyed)
[01/25/2025-19:09:25] [TRT] [E] [cudaDriverHelpers.cpp::operator()::106] Error Code 1: Cuda Driver (context is destroyed)
[01/25/2025-19:09:25] [TRT] [E] [defaultAllocator.cpp::deallocate::64] Error Code 1: Cuda Runtime (invalid device context)
[01/25/2025-19:09:25] [TRT] [E] [defaultAllocator.cpp::deallocate::64] Error Code 1: Cuda Runtime (invalid device context)
[01/25/2025-19:09:25] [TRT] [E] [defaultAllocator.cpp::deallocate::64] Error Code 1: Cuda Runtime (invalid device context)
[01/25/2025-19:09:25] [TRT] [E] [cudaDriverHelpers.cpp::operator()::106] Error Code 1: Cuda Driver (invalid resource handle)
[01/25/2025-19:09:25] [TRT] [E] [defaultAllocator.cpp::deallocate::64] Error Code 1: Cuda Runtime (invalid device context)
[01/25/2025-19:09:25] [TRT] [E] [defaultAllocator.cpp::deallocate::64] Error Code 1: Cuda Runtime (invalid device context)
[01/25/2025-19:09:25] [TRT] [E] [defaultAllocator.cpp::deallocate::64] Error Code 1: Cuda Runtime (invalid device context)
[01/25/2025-19:09:25] [TRT] [E] [defaultAllocator.cpp::deallocate::64] Error Code 1: Cuda Runtime (invalid device context)
[01/25/2025-19:09:25] [TRT] [E] [cudaDriverHelpers.cpp::operator()::106] Error Code 1: Cuda Driver (context is destroyed)
[01/25/2025-19:09:25] [TRT] [E] [cudaDriverHelpers.cpp::operator()::106] Error Code 1: Cuda Driver (context is destroyed)
[01/25/2025-19:09:25] [TRT] [E] [cudaDriverHelpers.cpp::operator()::106] Error Code 1: Cuda Driver (context is destroyed)
[01/25/2025-19:09:25] [TRT] [E] [cudaDriverHelpers.cpp::operator()::106] Error Code 1: Cuda Driver (context is destroyed)
[01/25/2025-19:09:25] [TRT] [E] [cudaDriverHelpers.cpp::operator()::106] Error Code 1: Cuda Driver (invalid resource handle)
[01/25/2025-19:09:25] [TRT] [E] [cudaDriverHelpers.cpp::operator()::106] Error Code 1: Cuda Driver (context is destroyed)
[01/25/2025-19:09:25] [TRT] [E] [cudaDriverHelpers.cpp::operator()::106] Error Code 1: Cuda Driver (invalid resource handle)

can someone go through my script try to run on their JETSON device and tell me a solution

[naps22]

Very practical and well-documented guide! Deploying YOLOv8 on NVIDIA Jetson using TensorRT and DeepStream makes real-time inference highly efficient, especially for edge AI applications. The integration of YOLOv8 and DeepStream not only boosts performance but also enables scalable video analytics with minimal latency. It’s great to see how powerful and lightweight object detection can become with this setup.