Inference v5.1

What's Changed

• Mixtral fix: match reference with standalone script by @pgmpablo157321 in #2054
• Add automotive submission checker by @pgmpablo157321 in #2051
• Edit mlperf.conf for pointpainting - automotive by @anandhu-eng in #2052
• Automotive benchmark table by @pgmpablo157321 in #2050
• Update backend_pytorch_native.py | Fixes #2056 by @arjunsuresh in #2057
• Update truncate_accuracy_log.py | Remove a wrong ERROR message in logs by @arjunsuresh in #2061
• Changes for final report generation - PointPainting by @anandhu-eng in #2063
• Fix mlperf.conf link for equal issue mode by @anandhu-eng in #2069
• Update results.cc | Add another significant digit to percentile laten… by @arjunsuresh in #2066
• Create benchmark checklist for pointpainting by @anandhu-eng in #2068
• Fix np.memmap usage, add flag to force not using memmap by @nv-alicheng in #2081
• Add device map - pointpainting automotive by @anandhu-eng in #2087
• PointPainting Documentation update by @rod409 in #2089
• Update verify_performance.py | Support 99.9 percentile latency in TEST01 for pointpainting by @arjunsuresh in #2071
• Match server scenario to standalone implementation by @pgmpablo157321 in #2086
• Add reference model details by @anandhu-eng in #2084
• [405B] Set max_tokens to 2k by @attafosu in #2088
• Add parameter number and FLOPs values by @anandhu-eng in #2090
• Update benchmark-checklist.md - PointPainting by @anandhu-eng in #2083
• Report improvement - support output of IDs to a json file by @arjunsuresh in #2059
• Fixed GPTJ accuracy checker by @nvzhihanj in #2093
• Fix SDXL, Retinanet and GPTJ accuracy checker by @nvzhihanj in #2094
• Update auto-update-dev.yml | update docs as well by @arjunsuresh in #2096
• Update submission_checker.py | Prevent empty accuracy in open division by @arjunsuresh in #2097
• Added information about GitHub tests currently live by @anandhu-eng in #2091
• Docs update, fix download links for llama models by @arjunsuresh in #2055
• Add MLC Automation commands by @anandhu-eng in #2115
• Remove llama3.1 user conf unnecessary and misleading lines by @pgmpablo157321 in #2114
• Update docs by @arjunsuresh in #2118
• Update verify_performance.py | Fix compliance test for extra percenti… by @arjunsuresh in #2120
• Update default version of final report script by @pgmpablo157321 in #2124
• Update accuracy_igbh.py | Fixes 2119 by @arjunsuresh in #2123
• 🔄 synced file(s) with mlcommons/power-dev by @mlcommons-bot in #2125
• Final report cosmetic fix by @pgmpablo157321 in #2134
• Update loadgen package name in classification_and_detection setup by @annietllnd in #2131
• Add exception for github-actions[bot] to cla.yml by @nathanwasson in #2135
• Final report cosmetic fix by @pgmpablo157321 in #2141
• Update submission_checker.py | Fix open model unit in Results by @arjunsuresh in #2144
• Add Llama 3.1 to special unit dict by @pgmpablo157321 in #2150
• [Post Mortem] Log number of errors in detail log by @pgmpablo157321 in #2164
• Docs update by @nathanwasson in #2137
• [Post Mortem] Check all systems and measurements folders have results by @pgmpablo157321 in #2166
• [Post Mortem] Add calibration check to submission checker by @pgmpablo157321 in #2185
• Add find peak performance documentation by @pgmpablo157321 in #2186
• [Post Mortem] Check equal issue for open division + check accuracy run covers all the dataset by @pgmpablo157321 in #2170
• add deepseek-r1 multi-backend reference implementation by @viraatc in #2198
• fix: update sglang docker for deepseek-r1 by @viraatc in #2201
• fix: update eval_accuracy to handle mlperf_log_accuracy.json by @viraatc in #2202
• Use existing submission generation workflow from mlperf-automations repo by @anandhu-eng in #2199
• Add deepseek configuration + v5.1 Readme by @pgmpablo157321 in #2203
• Updated readme with mlc commands for model,dataset,accuracy and submission generation by @anandhu-eng in #2143
• Docs - Update disk space for reference implementation by @anandhu-eng in #2159
• Update Waymo access instructions by @nathanwasson in #2148
• Update verify_performance.py | Refactor the code by @arjunsuresh in #2073
• Update mlperf.conf with final deepseek configuration by @pgmpablo157321 in #2208
• Add whisper reference implementation by @pgmpablo157321 in #2193
• Llama3.1-8b reference implementation by @pgmpablo157321 in #2190
• Skip Imagenet calibration dataset download in GH actions by @anandhu-eng in #2209
• Add v5.1 submission checker by @pgmpablo157321 in #2204
• Update Llama 3.1 model access instructions by @nathanwasson in #2149
• Rename audit.conf to audit.config by @arjunsuresh in #2127
• Quick fix: correct metrics by @pgmpablo157321 in #2211
• [Whisper] Updating dataset for repacked dev-all. by @keithachorn-intel in #2212
• Partial fix for compliance TEST01 update by @keithachorn-intel in #2215
• Completion of compliance TEST01 fix by @keithachorn-intel in #2217
• Fix typo in automation command by @anandhu-eng in #2219
• Fix Readme for inference v5.1 by @pgmpablo157321 in #2216

New Contributors

• @annietllnd made their first contribution in #2131

Full Changelog: v5.0.1...v5.1