Inference v5.1
What's Changed
- Mixtral fix: match reference with standalone script by @pgmpablo157321 in #2054
- Add automotive submission checker by @pgmpablo157321 in #2051
- Edit mlperf.conf for pointpainting - automotive by @anandhu-eng in #2052
- Automotive benchmark table by @pgmpablo157321 in #2050
- Update backend_pytorch_native.py | Fixes #2056 by @arjunsuresh in #2057
- Update truncate_accuracy_log.py | Remove a wrong ERROR message in logs by @arjunsuresh in #2061
- Changes for final report generation - PointPainting by @anandhu-eng in #2063
- Fix mlperf.conf link for equal issue mode by @anandhu-eng in #2069
- Update results.cc | Add another significant digit to percentile laten… by @arjunsuresh in #2066
- Create benchmark checklist for pointpainting by @anandhu-eng in #2068
- Fix np.memmap usage, add flag to force not using memmap by @nv-alicheng in #2081
- Add device map - pointpainting automotive by @anandhu-eng in #2087
- PointPainting Documentation update by @rod409 in #2089
- Update verify_performance.py | Support 99.9 percentile latency in TEST01 for pointpainting by @arjunsuresh in #2071
- Match server scenario to standalone implementation by @pgmpablo157321 in #2086
- Add reference model details by @anandhu-eng in #2084
- [405B] Set max_tokens to 2k by @attafosu in #2088
- Add parameter number and FLOPs values by @anandhu-eng in #2090
- Update benchmark-checklist.md - PointPainting by @anandhu-eng in #2083
- Report improvement - support output of IDs to a json file by @arjunsuresh in #2059
- Fixed GPTJ accuracy checker by @nvzhihanj in #2093
- Fix SDXL, Retinanet and GPTJ accuracy checker by @nvzhihanj in #2094
- Update auto-update-dev.yml | update docs as well by @arjunsuresh in #2096
- Update submission_checker.py | Prevent empty accuracy in open division by @arjunsuresh in #2097
- Added information about GitHub tests currently live by @anandhu-eng in #2091
- Docs update, fix download links for llama models by @arjunsuresh in #2055
- Add MLC Automation commands by @anandhu-eng in #2115
- Remove llama3.1 user conf unnecessary and misleading lines by @pgmpablo157321 in #2114
- Update docs by @arjunsuresh in #2118
- Update verify_performance.py | Fix compliance test for extra percenti… by @arjunsuresh in #2120
- Update default version of final report script by @pgmpablo157321 in #2124
- Update accuracy_igbh.py | Fixes 2119 by @arjunsuresh in #2123
- 🔄 synced file(s) with mlcommons/power-dev by @mlcommons-bot in #2125
- Final report cosmetic fix by @pgmpablo157321 in #2134
- Update loadgen package name in classification_and_detection setup by @annietllnd in #2131
- Add exception for github-actions[bot] to cla.yml by @nathanwasson in #2135
- Final report cosmetic fix by @pgmpablo157321 in #2141
- Update submission_checker.py | Fix open model unit in Results by @arjunsuresh in #2144
- Add Llama 3.1 to special unit dict by @pgmpablo157321 in #2150
- [Post Mortem] Log number of errors in detail log by @pgmpablo157321 in #2164
- Docs update by @nathanwasson in #2137
- [Post Mortem] Check all systems and measurements folders have results by @pgmpablo157321 in #2166
- [Post Mortem] Add calibration check to submission checker by @pgmpablo157321 in #2185
- Add find peak performance documentation by @pgmpablo157321 in #2186
- [Post Mortem] Check equal issue for open division + check accuracy run covers all the dataset by @pgmpablo157321 in #2170
- add deepseek-r1 multi-backend reference implementation by @viraatc in #2198
- fix: update sglang docker for deepseek-r1 by @viraatc in #2201
- fix: update eval_accuracy to handle mlperf_log_accuracy.json by @viraatc in #2202
- Use existing submission generation workflow from mlperf-automations repo by @anandhu-eng in #2199
- Add deepseek configuration + v5.1 Readme by @pgmpablo157321 in #2203
- Updated readme with mlc commands for model,dataset,accuracy and submission generation by @anandhu-eng in #2143
- Docs - Update disk space for reference implementation by @anandhu-eng in #2159
- Update Waymo access instructions by @nathanwasson in #2148
- Update verify_performance.py | Refactor the code by @arjunsuresh in #2073
- Update mlperf.conf with final deepseek configuration by @pgmpablo157321 in #2208
- Add whisper reference implementation by @pgmpablo157321 in #2193
- Llama3.1-8b reference implementation by @pgmpablo157321 in #2190
- Skip Imagenet calibration dataset download in GH actions by @anandhu-eng in #2209
- Add v5.1 submission checker by @pgmpablo157321 in #2204
- Update Llama 3.1 model access instructions by @nathanwasson in #2149
- Rename audit.conf to audit.config by @arjunsuresh in #2127
- Quick fix: correct metrics by @pgmpablo157321 in #2211
- [Whisper] Updating dataset for repacked dev-all. by @keithachorn-intel in #2212
- Partial fix for compliance TEST01 update by @keithachorn-intel in #2215
- Completion of compliance TEST01 fix by @keithachorn-intel in #2217
- Fix typo in automation command by @anandhu-eng in #2219
- Fix Readme for inference v5.1 by @pgmpablo157321 in #2216
New Contributors
- @annietllnd made their first contribution in #2131
Full Changelog: v5.0.1...v5.1