Skip to content

Commit 7373829

Browse files
Akshat-Tripathipgmpablo157321v-shobhitgithub-actions[bot]nathanwasson
authored
Added Wan2.2-T2V to submission checker (mlcommons#2457)
* Added Wan2.2-T2V to submission checker * Updated acc_pattern * Add accuracy_sample_count (mlcommons#2414) * add accuracy_sample_count * cap count to QSL->TotalSampleCount() * [Automated Commit] Format Codebase * empty commit to re-trigger test * [Automated Commit] Format Codebase * rm newline * rm test05 lines * add accuracy_sample_count to submission_checker * [Automated Commit] Format Codebase * fix check * [Automated Commit] Format Codebase * empty commit to trigger test * add gpt-oss-120b loadgen settings to mlperf.conf * Remove Rclone download instructions for datasets supported by R2 Downloader (mlcommons#2358) * Remove Rclone instructions from README.md * Remove Rclone download instructions from README.md * Tweak README.md * Switch from Rclone to R2 Downloader in README.md * Switch from Rclone to R2 Downloader in README.md * Switch from Rclone to R2 Downloader in README.md * Switch Rclone for R2 Downloader in README.md * Switch Rclone for R2 Downloader in README.md * Use r2 downloader for gpt j model download (mlcommons#2365) * Provide r2 download commands for mixtral model and datasets (mlcommons#2364) * Replace MLCFlow RClone command for criteo dataset with R2 (mlcommons#2363) * Deprecate MLCFlow rclone download command with r2 (mlcommons#2362) * Add instruction to download DeepSeek model through MLCflow (mlcommons#2361) * [Automated Commit] Format Codebase * Trigger cla-check * [Automated Commit] Format Codebase * Update build_wheels.yml * [Automated Commit] Format Codebase * Add dtypes to README.md --------- Co-authored-by: ANANDHU S <[email protected]> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Arjun Suresh <[email protected]> Co-authored-by: Pablo Gonzalez <[email protected]> Co-authored-by: Pablo Gonzalez <[email protected]> * empty commit to trigger test * Update seeds for inference v6.0 (mlcommons#2437) * use total_sample_count for default of accuracy_sample_count * revert changes to .github * Add accuracy_sample_count to modularized submission checker * [Automated Commit] Format Codebase * empty commit to trigger test --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Nathan Wasson <[email protected]> Co-authored-by: ANANDHU S <[email protected]> Co-authored-by: Arjun Suresh <[email protected]> Co-authored-by: Pablo Gonzalez <[email protected]> Co-authored-by: Pablo Gonzalez <[email protected]> * Increment version to 6.0.6 * Change dataset access link to new URL (mlcommons#2458) Updated dataset access link in README.md. * MLCFlow automation: Add model and dataset download commands (mlcommons#2455) Co-authored-by: Arjun Suresh <[email protected]> * Update v6.0 benchmark table (mlcommons#2460) Co-authored-by: hanyunfan <[email protected]> * Temporary turn off retinanet GitHub action (mlcommons#2467) * [DLRMv3] Add compliance test (mlcommons#2453) * compliance test for dlrmv3 * add test08 * update * update * squash changes --------- Co-authored-by: hanyunfan <[email protected]> * Fix PyTorch version issue for Llama2-70b and Mixtral-8x7b (mlcommons#2466) Update torch installation from unavailable nightly build (torch==2.2.0.dev20231006+cpu) to stable release (torch==2.2.0) which is available on the PyTorch CPU wheel index. Co-authored-by: Arjun Suresh <[email protected]> * Add yolo run commands to inference docs (mlcommons#2461) * Quick fix: Add user conf to yolo reference implementation (mlcommons#2446) * Quick fix: Add user conf to yolo reference implementation * Add model_name to yolo implementation * Fix default name * Fix logging for YOLO benchmark (mlcommons#2450) * [Automated Commit] Format Codebase * fix formatting * Remove duplicate log tracing argument Removed duplicate argument for enabling log tracing. * Add performance_sample_count_override for yolo * Update version check and filter scenarios for 6.0 * Remove min_query_count - interfering with runs Remove minimum query count requirement for performance mode. * Update models from submission checker constants (mlcommons#2464) Removed 'stable-diffusion-xl' and 'dlrm-v3' from scenarios. * Generate final report: Update filter scenarios for version 6.0 (mlcommons#2465) * Generate final report: Update filter scenarios for version 6.0 * Update mlperf.conf --------- Co-authored-by: ANANDHU S <[email protected]> Co-authored-by: hanyunfan <[email protected]> Co-authored-by: Arjun Suresh <[email protected]> * Increment version to 6.0.7 --------- Co-authored-by: Pablo Gonzalez <[email protected]> Co-authored-by: v-shobhit <[email protected]> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Nathan Wasson <[email protected]> Co-authored-by: ANANDHU S <[email protected]> Co-authored-by: Arjun Suresh <[email protected]> Co-authored-by: Pablo Gonzalez <[email protected]> Co-authored-by: pgmpablo157321 <[email protected]> Co-authored-by: hanyunfan <[email protected]> Co-authored-by: Linjian Ma <[email protected]> Co-authored-by: SamareshSingh <[email protected]> Co-authored-by: arjunsuresh <[email protected]> Co-authored-by: Miro <[email protected]>
1 parent 6520e39 commit 7373829

File tree

2 files changed

+15
-6
lines changed

2 files changed

+15
-6
lines changed

‎tools/submission/submission_checker/constants.py‎

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,7 @@
3939
"deepseek-r1": ["Offline"],
4040
"gpt-oss-120b": ["Offline"],
4141
"qwen3-vl-235b-a22b": ["Server", "Offline"],
42+
"wan-2.2-t2v-a14b": ["Offline", "SingleStream"],
4243
},
4344
"optional-scenarios-datacenter": {
4445
"llama2-70b-99": ["Interactive", "Server"],
@@ -188,6 +189,7 @@
188189
"dlrm-v3": ("AUC", 78.663 * 0.99), # TODO: Placeholder for now
189190
"yolo-95": ("mAP", 53.4 * 0.95),
190191
"yolo-99": ("mAP", 53.4 * 0.99),
192+
"wan-2.2-t2v-a14b": ("vbench_score", 70.48 * 0.99),
191193
},
192194
"accuracy-upper-limit": {
193195
"stable-diffusion-xl": (
@@ -232,6 +234,7 @@
232234
# TODO: Need to add accuracy sample count checkers as well (4395)
233235
"gpt-oss-120b": 6396,
234236
"qwen3-vl-235b-a22b": 48289,
237+
"wan-2.2-t2v-a14b": 247,
235238
"dlrm-v3": 34996,
236239
"yolo-95": 5000,
237240
"yolo-99": 5000,
@@ -262,6 +265,7 @@
262265
# TODO: Need to add accuracy sample count checkers as well (4395)
263266
"gpt-oss-120b": 6396,
264267
"qwen3-vl-235b-a22b": 48289,
268+
"wan-2.2-t2v-a14b": 247,
265269
"dlrm-v3": 34996,
266270
"yolo-95": 1525,
267271
"yolo-99": 1525,
@@ -338,6 +342,7 @@
338342
"gpt-oss-120b": {"SingleStream": 1024, "Server": 270336, "Offline": 1},
339343
"qwen3-vl-235b-a22b": {"SingleStream": 1024, "Server": 270336, "Offline": 1},
340344
"dlrm-v3": {"Server": 270336, "Offline": 1},
345+
"wan-2.2-t2v-a14b": {"SingleStream": 247, "Offline": 1},
341346
"yolo-95": {"SingleStream": 1024, "MultiStream": 270336, "Offline": 1},
342347
"yolo-99": {"SingleStream": 1024, "MultiStream": 270336, "Offline": 1},
343348
},
@@ -354,13 +359,15 @@
354359
"rgat",
355360
"pointpainting",
356361
"whisper",
362+
"wan-2.2-t2v-a14b",
357363
"yolo-99",
358364
"yolo-95",
359365
],
360366
"models_TEST04": [
361367
"resnet",
362368
"stable-diffusion-xl",
363369
"pointpainting",
370+
"wan-2.2-t2v-a14b"
364371
],
365372
"models_TEST06": [
366373
"llama2-70b-99",
@@ -1378,7 +1385,8 @@
13781385
"FID_SCORE": r".*'FID_SCORE':\s+'?([\d.]+).*",
13791386
"gsm8k_accuracy": r".*'gsm8k':\s([\d.]+).*",
13801387
"mbxp_accuracy": r".*'mbxp':\s([\d.]+).*",
1381-
"exact_match": r".*'exact_match':\s([\d.]+).*"
1388+
"exact_match": r".*'exact_match':\s([\d.]+).*",
1389+
"vbench_score": r".*'vbench_score':\s([\d.]+).*",
13821390
}
13831391

13841392
SYSTEM_DESC_REQUIRED_FIELDS = [

‎tools/submission/submission_checker_old.py‎

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,7 @@
11
"""A checker for MLPerf Inference submissions from v5.0 onwards (for checking older submissions please use the submission checker from the respective release)
22
"""
33

4-
from __future__ import division
5-
from __future__ import print_function
6-
from __future__ import unicode_literals
4+
from __future__ import division, print_function, unicode_literals
75

86
import argparse
97
import datetime
@@ -12,7 +10,6 @@
1210
import os
1311
import re
1412
import sys
15-
1613
from glob import glob
1714

1815
from log_parser import MLPerfLog
@@ -67,7 +64,7 @@
6764
"deepseek-r1": ["Offline"],
6865
"gpt-oss-120b": ["Offline"],
6966
"qwen3-vl-235b-a22b": ["Server", "Offline"],
70-
"dlrm-v3": ["Server", "Offline"],
67+
"wan-2.2-t2v-a14b": ["Offline", "SingleStream"],
7168
},
7269
"optional-scenarios-datacenter": {
7370
"llama2-70b-99": ["Interactive", "Server"],
@@ -216,6 +213,7 @@
216213
"dlrm-v3": ("AUC", 78.663 * 0.99), # TODO: Placeholder for now
217214
"yolo-95": ("mAP", 53.4 * 0.95),
218215
"yolo-99": ("mAP", 53.4 * 0.99),
216+
"wan-2.2-t2v-a14b": ("vbench", 70.48 * 0.99),
219217
},
220218
"accuracy-upper-limit": {
221219
"stable-diffusion-xl": (
@@ -259,6 +257,7 @@
259257
"whisper": 1633,
260258
"gpt-oss-120b": 6396,
261259
"qwen3-vl-235b-a22b": 48289,
260+
"wan-2.2-t2v-a14b": 247,
262261
"dlrm-v3": 34996,
263262
"yolo-95": 5000,
264263
"yolo-99": 5000,
@@ -288,6 +287,7 @@
288287
"whisper": 1633,
289288
"gpt-oss-120b": 4395,
290289
"qwen3-vl-235b-a22b": 48289,
290+
"wan-2.2-t2v-a14b": 247,
291291
"dlrm-v3": 34996,
292292
"yolo-95": 1525,
293293
"yolo-99": 1525,
@@ -366,6 +366,7 @@
366366
"dlrm-v3": {"Server": 270336, "Offline": 1},
367367
"yolo-95": {"SingleStream": 1024, "MultiStream": 270336, "Offline": 1},
368368
"yolo-99": {"SingleStream": 1024, "MultiStream": 270336, "Offline": 1},
369+
"wan-2.2-t2v-a14b": {"SingleStream": 247, "Offline": 1}
369370
},
370371
},
371372
"v5.1": {

0 commit comments

Comments
 (0)