MLPerf Inference Status Dashboard

[arjunsuresh]

MLPerf Inference Benchmark Status Dashboard

Last Updated: 2025-09-02

This dashboard provides a high-level overview of the status of various MLPerf inference benchmarks across different frameworks on CPU and GPU hardware.

---

Status Key

• Frameworks:
  • TF: TensorFlow
  • PyT: PyTorch
  • ONNX: ONNX Runtime
  • TRT: NVIDIA TensorRT
• Icons:
  • ✅: Running Fine - Benchmark passes functional and performance tests.
  • ⚠️: Issues Found - Benchmark runs but has performance regressions or minor errors.
  • 🔄: In Progress - Currently under testing or implementation.
  • ❌: Failed / Not Supported - Benchmark fails to run or is not implemented.

---

Benchmark Status

Benchmark	Model	CPU Status	GPU Status
Image Classification	ResNet-50	TF: ✅, ONNX: ✅	TRT: ✅, ONNX: ✅, TF: ✅
Object Detection	Retinanet	PyT: ✅, ONNX: ✅	TRT: ✅, PyT: ✅,
Natural Language	BERT	TF: ✅, PyT: ✅, ONNX: ✅	TRT: ✅, PyT: 🔄 , TF: 🔄
Recommendation	DLRMv2	PyT: ✅	TRT: ✅, PyT: ✅
Medical Imaging	3D-UNet	PyT: 🔄	TRT: ✅, PyT: ✅
Large Language Model	llama3-8b	vLLM: ✅	TRT: ✅, vLLM: ✅
Large Language Model	llama3-80b	vLLM:❌	TRT: ✅, vLLM: ✅
Large Language Model	llama3-405b	vLLM: ❌	TRT: ✅, vLLM: ✅

---

Summary & Notes

• CPU: Broad support for established models like ResNet-50 and BERT is stable. Newer or more complex models like 3D-UNet and GPT-J have limited or no support. ONNX Runtime shows a performance warning for BERT that needs investigation.
• GPU: The ecosystem is mature, with NVIDIA's TensorRT providing excellent coverage and performance across all key benchmarks.
• This dashboard is a snapshot. Status can change based on framework versions, hardware drivers, and model updates.