CPU inference performance of RFDETRSegNano (~200 ms @312x312) – expected?

[NiccoloBalestrieri]

Hi, first of all thanks for releasing RFDETR, great work.

I’m testing RFDETRSegNano for a CPU real-time use case (cause i want compare the result with yolo26) and I wanted to check whether the inference performance I’m seeing is expected, or if I’m missing a recommended optimization path.

Environment

• Model: RFDETRSegNano (pretrained)
• OS: Windows 11
• CPU: - CPU: Intel(R) Core(TM) Ultra 7 155U @ 1.70 GHz
• PyTorch: 2.8.0
• ONNX Runtime: 1.23.2
• Execution provider: CPUExecutionProvider
• Batch size: 1
• Input resolution: 312 × 312

Export & inference setup

PyTorch export
model = RFDETRSegNano() model.optimize_for_inference(dtype=torch.float32) model.export()
ONNX inference
`sess = rt.InferenceSession(
"inference_model_optimized.onnx",
providers=["CPUExecutionProvider"]
)

outputs = sess.run(output_names, {input_name: img})`

Observed performance

• Average inference time: ~200 ms
• Outputs:
  • Boxes: (1, 100, 4)
  • Class scores: (1, 100, 91)
  • Feature / mask output: (1, 100, 78, 78)

FP16 export was also tested, but ONNX Runtime CPU executes in FP32.

### Question
Is ~200 ms CPU inference at 312×312 the expected performance for RFDETRSegNano?
If not:

• are there recommended CPU-specific optimizations?
• is the third output (78×78) required for pure detection?
• is OpenVINO / INT8 the intended deployment path for CPU?

Any guidance would be appreciated, Thanks!

[akonanykhin]

I am getting similar results across multiple hardware targets: CPU (ONNX and OpenVINO), ARM (TFLite), 3 Jetsons (ONNX with TRT), GPU (ONNX with CUDA/TRT).
Pre-trained RF-DETR nano is approx 10x slower than YOLOv8 nano across all of them.
It is positioned around YOLOv8 medium latency-wise, which makes sense given parameter count and file size.
So looks like those magical benchmarks are somehow only working on T4 GPU.

[NiccoloBalestrieri]

Tks, I think the same. I'm using YOLOv26-n and there is no match at the moment. I hope for some improvements in the future !

[isaacrob]

@akonanykhin we see the same latency comparisons on jetson, l4, whatever GPU we measure. We do however see performance issues running specifically with ONNX and using older TRT versions which ship with jetpack 5. I'd recommend upgrading to 6.2 to use TRT 10.3.

@NiccoloBalestrieri we haven't done benchmarking work on CPU but it isn't super surprising we'd be disproportionately slow, as transformers are less efficient on CPU than GPUs which are excellent for matrix multiplication. I would love if you could provide the script you're using to compare vs YOLO and the results you see on your setup!

[isaacrob]

@akonanykhin you should be able to reproduce our results using this repo https://github.com/roboflow/single_artifact_benchmarking if you can't let me know

[akonanykhin]

@akonanykhin you should be able to reproduce our results using this repo https://github.com/roboflow/single_artifact_benchmarking if you can't let me know

So I tried newer TRT containers out of curiosity. While results were better, RF-DETR nano is still significantly slower than Yolov8.
On RTX 3080Ti TRT 10.9: 6.9ms >> 2.3ms
On Jetson Orin Nano Super TRT 10.3: 17.2ms >> 4.7ms

Admittedly I didn't adhere to your methodology, 200ms break and all.
But we are not talking about 20% difference here.
If there is some instability, then both models should be affected evenly.
If somehow your methodology only improves RF-DETR results by several orders, then it discredits the methodology if anything, as it's not representative of how people use models in real world.

Thus, I remain skeptical of the benchmarks you published.
As currently seen, RF-DETR nano model is not the same weight class to Yolo nano model in latency on vast majority of hardware targets.
So comparing them is disingenoius.

[isaacrob]

Can you confirm you're enabling CUDA graphs and confirm the resolution you're running the models at?

Just to be clear, are you suggesting that we faked our results that were peer reviewed at ICLR?

[isaacrob]

also @akonanykhin can you confirm you're including NMS timing in measuring the YOLO latency?