Error issue when running "efficientAD.onnx" with onnxruntime.

[B1SH0PP]

I tried to run the efficient.onnx file exported from the Anomalib library on C++ ONNXRunTime, but encountered a model structure issue during execution. I'm not sure if this is due to a problem with how I wrote the ONNXRunTime code or if it's caused by the network structure.

[B1SH0PP]

This is the code for setting up my ONNXRunTime inference environment:

cv::Mat det1, det2;
cv::resize(input, det1, cv::Size(256, 256), cv::INTER_AREA);
det1.convertTo(det1, CV_32FC3);
InferencerOfEfficientAD::preProcess(det1, det2);
cv::Mat blob = cv::dnn::blobFromImage(det2, 1., cv::Size(256, 256), cv::Scalar(0, 0, 0), false, true);  //Creates 4-dimensional blob from image

auto memory_info = Ort::MemoryInfo::CreateCpu(OrtAllocatorType::OrtArenaAllocator, OrtMemType::OrtMemTypeDefault);
vector<Ort::Value> input_tensors;
input_tensors.push_back(Ort::Value::CreateTensor<float>(memory_info, blob.ptr<float>(), blob.total(), input_dims.data(), input_dims.size()));
	
const char* ch_in = "input";
const char* const* p_in = &ch_in;
const char* ch_out = "output";
const char* const* p_out = &ch_out;

vector<Ort::Value> output_tensors;
try {
	output_tensors = session->Run(Ort::RunOptions{ nullptr }, p_in, input_tensors.data(), 1, p_out, 1);
}
catch(const Ort::Exception& e){
	std::cerr << "ONNX Runtime error: " << e.what() << std::endl;
}

[B1SH0PP]

Experts, please help me, this is really important to me.

[blaz-r]

Hello. I'm not familiar with ONNX engine and OpenCV in C++ but could you please also share the InferencerOfEfficientAD::preProcess function?

It is important that values for EfficientAd are scaled to intervals [0, 1] (if you didn't change the normalization: none option in config). Looking at blobFromImage docs, it accepts scaleFactor so I think you could use that and set it to 1/255. Additionally you'll need to make sure that channel order is RGB, in your case I think you have swapRB set to false so the channel order won't be swapped.

I don't think any of these things is really the reason that the inference doesn't work, but these things should be fixed to get valid results once you figure out the problem.

But maybe if you use the blobFromImage to scale, channel swap and it seems it also sets type to CV_32F, maybe you can avoid problems in your own pre-process code. I think the call to achieve this would be:

cv::Mat blob = cv::dnn::blobFromImage(det1, 1.0 / 255, cv::Size(256, 256), cv::Scalar(0, 0, 0), true, true);

[B1SH0PP]

Thank you very much for your help!
here is the InferencerOfEfficientAD::preProcess function:

void InferencerOfEfficientAD::preProcess(const cv::Mat& image, cv::Mat& image_blob)
{
	cv::Mat input;
	image.copyTo(input);

	std::vector<cv::Mat> channels, channel_p;
	split(input, channels);
	cv::Mat R, G, B;
	B = channels.at(0);
	G = channels.at(1);
	R = channels.at(2);

	//ToFloat
	B = (B / 255.);
	G = (G / 255.);
	R = (R / 255.);

	channel_p.push_back(R);
	channel_p.push_back(G);
	channel_p.push_back(B);

	cv::Mat outt;
	merge(channel_p, outt);
	image_blob = outt;
}

And after I modified it according to what you said, the error still hasn't changed.
I would like to ask you, currently, if I want to apply the network for inference on an industrial C++ platform, which would be a better choice: ONNXRunTime, TensorRT, or something else?

[blaz-r]

Hi. It seems like you are doing all the required preproccesing so that should be fine. I'd say that since blobFromImage can do that you just use it instead of writing custom code, but at the end it's just important that all these steps are applied.

I'm not sure what the problem would be. I will try to use efficientad onnx in python today and I'll report back to you if it's a problem in model conversion.

Regarding the platform, I'm not that familiar with C++ tools, maybe you can also try OpenVINO, but I sadly can't help much in this regard.

[blaz-r]

Hi @B1SH0PP I tested EfficientAD exported onnx model with python using OpenVINO inferencer and it works, therefore I believe that the problem is somewhere in your code.

Looking at your problem again, I found a following issue microsoft/onnxruntime#8545 and microsoft/onnxruntime#10486 that have a similar error, which might be a CUDA issue.
Sadly I can't offer much more advice as I am not that familiar with C++ Onnxruntime.

Hope this helped and that you manage to resolve the issue.

[B1SH0PP]

Thank you again for your answer. I have not yet solved the problem, but I will continue to try to resolve it. If I manage to solve it, I will update the information below.
Thank you again!

[samet-akcay]

@B1SH0PP, not sure if it would help, but @ashwinvaidya17 created a OpenVINO C++ inference inference capabilities here.

This might give you some ideas?