-
Notifications
You must be signed in to change notification settings - Fork 153
Description
i'm trying to build an application that takes a video feed from a camera (with gstreamer) and makes and inference (object detection) with yolov5 using the NPU, i need 4 istances of this application (for 4 camera in total)
the part of gstreamer works flawlessly because with 4 terminal running the gstreamer pipeline (with hardware acceleration) it doesn't run into a kernel panic.
the problem is when i also run the inference:
- with one inference (one camera) it runs without any problem
- with two inferences (or more) it goes straight to a kernel panic
the only solution i found on it is to limit the RAM of the board to 4GB, but for me is too little for what i have to do (in the future there will be other things beside the inference)
i'm using rknpu2 to use the NPU.
i know that the issue is generated by the RGA (this explains the weird reason of the 4GB of memory) but i don't know how to fix it. Maybe some kind of dma32 allocation but i have no idea for how to implement it.
is there a fix for my issue?
here are some info:
`
#pragma once
#include
#include
#include "yolov5.h"
#include "image_utils.h"
#include "file_utils.h"
#include "image_drawing.h"
#include "dma_alloc.hpp"
class YoloModel {
public:
bool loaded = false;
rknn_app_context_t rknn_app_ctx;
YoloModel(std::string model_path) {
int ret;
memset(&rknn_app_ctx, 0, sizeof(rknn_app_context_t));
init_post_process();
ret = init_yolov5_model(model_path.c_str(), &rknn_app_ctx);
if (ret != 0)
{
printf("init_yolov5_model fail! ret=%d model_path=%s\n", ret, model_path);
return;
}
loaded = true;
}
bool infer(cv::Mat cvImage) {
if(!loaded)
return false;
int ret;
image_buffer_t src_image;
memset(&src_image, 0, sizeof(image_buffer_t));
//ret = read_image("../yolov5/model/bus.jpg", &src_image);
/*ret = dma_buf_alloc(DMA_HEAP_DMA32_PATCH, src_image.size, &rknn_app_ctx.img_dma_buf.dma_buf_fd,
(void **) & (rknn_app_ctx.img_dma_buf.dma_buf_virt_addr));
memcpy(rknn_app_ctx.img_dma_buf.dma_buf_virt_addr, src_image.virt_addr, src_image.size);
dma_sync_cpu_to_device(rknn_app_ctx.img_dma_buf.dma_buf_fd);
free(src_image.virt_addr);
src_image.virt_addr = (unsigned char *)rknn_app_ctx.img_dma_buf.dma_buf_virt_addr;
src_image.fd = rknn_app_ctx.img_dma_buf.dma_buf_fd;
rknn_app_ctx.img_dma_buf.size = src_image.size;*/
src_image.width = cvImage.cols;
src_image.height = cvImage.rows;
src_image.format = IMAGE_FORMAT_RGB888;
src_image.virt_addr = cvImage.data;
object_detect_result_list od_results;
ret = inference_yolov5_model(&rknn_app_ctx, &src_image, &od_results);
if (ret != 0)
{
printf("init_yolov5_model fail! ret=%d\n", ret);
return false;
}
// print
char text[256];
for (int i = 0; i < od_results.count; i++)
{
object_detect_result *det_result = &(od_results.results[i]);
printf("\t%s @ (%d %d %d %d) %.3f\n", coco_cls_to_name(det_result->cls_id),
det_result->box.left, det_result->box.top,
det_result->box.right, det_result->box.bottom,
det_result->prop);
int x1 = det_result->box.left;
int y1 = det_result->box.top;
int x2 = det_result->box.right;
int y2 = det_result->box.bottom;
draw_rectangle(&src_image, x1, y1, x2 - x1, y2 - y1, COLOR_BLUE, 3);
// sprintf(text, "%s %.1f%%", coco_cls_to_name(det_result->cls_id), det_result->prop * 100);
// draw_text(&src_image, text, x1, y1 - 20, COLOR_RED, 10);
}
/*dma_buf_free(rknn_app_ctx.img_dma_buf.size, &rknn_app_ctx.img_dma_buf.dma_buf_fd,
rknn_app_ctx.img_dma_buf.dma_buf_virt_addr);*/
return true;
}
};
`
- board = orange pi 5 plus
- version = ubuntu jammy (joshua)
- kernel = 5.10 (joshua)
as you can see i've tried to use the dma allocation from the example but it gave me this error:

i'm also available to modify the source code of the kernel if needed

