Skip to content

Commit 9b77f35

Browse files
pgmpablo157321hanyunfan
pgmpablo157321
and
hanyunfan
authored
Folder structure fixes to checker and samples (mlcommons#2433)
* Folder structure fixes to checker and samples * Add structure changes to preprocess_submssion script --------- Co-authored-by: hanyunfan <[email protected]>
1 parent 26864ba commit 9b77f35

File tree

14 files changed

+28
-21
lines changed

14 files changed

+28
-21
lines changed

‎.github/workflows/test-resnet50.yml‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,10 @@ jobs:
3535
python-version: ${{ matrix.python-version }}
3636
- name: Install dependencies
3737
run: |
38-
python3 -m pip install mlc-scripts
38+
python3 -m pip install mlcflow
39+
mlc pull repo mlcommons@mlperf-automations --branch=subcheckmod_change
40+
#python3 -m pip install mlc-scripts
3941
- name: Test Resnet50 and end to end submission generation
4042
run: |
41-
mlcr run,mlperf,inference,generate-run-cmds,_submission,_short --quiet --submitter="MLCommons" --hw_name=default --model=resnet50 --implementation=reference --backend=${{ matrix.backend }} --device=cpu --scenario=Offline --test_query_count=500 --adr.compiler.tags=gcc --adr.inference-src.tags=_branch.${{ github.event.pull_request.head.ref }},_repo.${{ github.event.pull_request.head.repo.html_url }} --adr.inference-src.version=custom --adr.inference-src-loadgen.tags=_branch.${{ github.event.pull_request.head.ref }},_repo.${{ github.event.pull_request.head.repo.html_url }} --adr.inference-src-loadgen.version=custom --adr.loadgen.version=custom ${{ matrix.loadgen-flag }}
43+
mlcr run,mlperf,inference,generate-run-cmds,_submission,_short,_r6.0 --quiet --submitter="MLCommons" --hw_name=default --model=resnet50 --implementation=reference --backend=${{ matrix.backend }} --device=cpu --scenario=Offline --test_query_count=500 --adr.compiler.tags=gcc --adr.inference-src.tags=_branch.submission_checker_refactor --adr.inference-src.version=custom --adr.inference-src-loadgen.tags=_branch.${{ github.event.pull_request.head.ref }},_repo.${{ github.event.pull_request.head.repo.html_url }} --adr.inference-src-loadgen.version=custom --adr.loadgen.version=custom ${{ matrix.loadgen-flag }}
44+
#mlcr run,mlperf,inference,generate-run-cmds,_submission,_short --quiet --submitter="MLCommons" --hw_name=default --model=resnet50 --implementation=reference --backend=${{ matrix.backend }} --device=cpu --scenario=Offline --test_query_count=500 --adr.compiler.tags=gcc --adr.inference-src.tags=_branch.${{ github.event.pull_request.head.ref }},_repo.${{ github.event.pull_request.head.repo.html_url }} --adr.inference-src.version=custom --adr.inference-src-loadgen.tags=_branch.${{ github.event.pull_request.head.ref }},_repo.${{ github.event.pull_request.head.repo.html_url }} --adr.inference-src-loadgen.version=custom --adr.loadgen.version=custom ${{ matrix.loadgen-flag }}

‎.github/workflows/test-retinanet.yml‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -33,7 +33,10 @@ jobs:
3333
python-version: ${{ matrix.python-version }}
3434
- name: Install dependencies
3535
run: |
36-
python3 -m pip install mlc-scripts
36+
python3 -m pip install mlcflow
37+
mlc pull repo mlcommons@mlperf-automations --branch=subcheckmod_change
38+
#python3 -m pip install mlc-scripts
3739
- name: Test Retinanet and end to end submission generation
3840
run: |
39-
mlcr run,mlperf,inference,generate-run-cmds,_submission,_short --quiet --submitter="MLCommons" --hw_name=default --model=retinanet --implementation=reference --backend=${{ matrix.backend }} --device=cpu --scenario=Offline --test_query_count=10 --adr.compiler.tags=gcc --adr.inference-src.version=custom --adr.inference-src.tags=_repo.${{ github.event.pull_request.head.repo.html_url }},_branch.${{ github.event.pull_request.head.ref }} --adr.inference-src-loadgen.tags=_branch.${{ github.event.pull_request.head.ref }},_repo.${{ github.event.pull_request.head.repo.html_url }} --adr.inference-src-loadgen.version=custom --adr.loadgen.version=custom
41+
mlcr run,mlperf,inference,generate-run-cmds,_submission,_short,_r6.0 --quiet --submitter="MLCommons" --hw_name=default --model=retinanet --implementation=reference --backend=${{ matrix.backend }} --device=cpu --scenario=Offline --test_query_count=10 --adr.compiler.tags=gcc --adr.accuracy-check-src.tags=_branch.submission_checker_refactor --adr.inference-src.version=custom --adr.inference-src.tags=_branch.submission_checker_refactor --adr.inference-src-loadgen.tags=_branch.${{ github.event.pull_request.head.ref }},_repo.${{ github.event.pull_request.head.repo.html_url }} --adr.inference-src-loadgen.version=custom --adr.loadgen.version=custom
42+
#mlcr run,mlperf,inference,generate-run-cmds,_submission,_short --quiet --submitter="MLCommons" --hw_name=default --model=retinanet --implementation=reference --backend=${{ matrix.backend }} --device=cpu --scenario=Offline --test_query_count=10 --adr.compiler.tags=gcc --adr.inference-src.version=custom --adr.inference-src.tags=_repo.${{ github.event.pull_request.head.repo.html_url }},_branch.${{ github.event.pull_request.head.ref }} --adr.inference-src-loadgen.tags=_branch.${{ github.event.pull_request.head.ref }},_repo.${{ github.event.pull_request.head.repo.html_url }} --adr.inference-src-loadgen.version=custom --adr.loadgen.version=custom

‎.github/workflows/test-rgat.yml‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -32,7 +32,10 @@ jobs:
3232
python-version: ${{ matrix.python-version }}
3333
- name: Install dependencies
3434
run: |
35-
python3 -m pip install mlc-scripts
35+
python3 -m pip install mlcflow
36+
mlc pull repo mlcommons@mlperf-automations --branch=subcheckmod_change
37+
#python3 -m pip install mlc-scripts
3638
- name: Test R-GAT and end to end submission generation
3739
run: |
38-
mlcr run,mlperf,inference,generate-run-cmds,_submission,_short --quiet --submitter="MLCommons" --category=datacenter --hw_name=default --model=rgat --implementation=reference --backend=${{ matrix.backend }} --device=cpu --scenario=Offline --test_query_count=500 --adr.compiler.tags=gcc --adr.inference-src.tags=_branch.$PR_HEAD_REF,_repo.${{ github.event.pull_request.head.repo.html_url }} --adr.inference-src-loadgen.tags=_branch.$PR_HEAD_REF,_repo.${{ github.event.pull_request.head.repo.html_url }} --adr.inference-src-loadgen.version=custom --adr.loadgen.version=custom ${{ matrix.loadgen-flag }}
40+
mlcr run,mlperf,inference,generate-run-cmds,_submission,_short,_r6.0 --quiet --submitter="MLCommons" --category=datacenter --hw_name=default --model=rgat --implementation=reference --backend=${{ matrix.backend }} --device=cpu --scenario=Offline --test_query_count=500 --adr.compiler.tags=gcc --adr.inference-src.tags=_branch.submission_checker_refactor --adr.inference-src-loadgen.tags=_branch.$PR_HEAD_REF,_repo.${{ github.event.pull_request.head.repo.html_url }} --adr.inference-src-loadgen.version=custom --adr.loadgen.version=custom ${{ matrix.loadgen-flag }}
41+
#mlcr run,mlperf,inference,generate-run-cmds,_submission,_short --quiet --submitter="MLCommons" --category=datacenter --hw_name=default --model=rgat --implementation=reference --backend=${{ matrix.backend }} --device=cpu --scenario=Offline --test_query_count=500 --adr.compiler.tags=gcc --adr.inference-src.tags=_branch.$PR_HEAD_REF,_repo.${{ github.event.pull_request.head.repo.html_url }} --adr.inference-src-loadgen.tags=_branch.$PR_HEAD_REF,_repo.${{ github.event.pull_request.head.repo.html_url }} --adr.inference-src-loadgen.version=custom --adr.loadgen.version=custom ${{ matrix.loadgen-flag }}

‎.github/workflows/test-submission-generation.yml‎

Lines changed: 7 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
1-
# This workflow will test the submission generation using MLPerf Automation
1+
# This workflow will test the submission generation using MLPerf Automation.
2+
# This test calls the test-mlperf-inference-submission-generation.yml file from automation repo
23

34
name: MLC based Submission Generation
45

@@ -12,7 +13,9 @@ on:
1213

1314
jobs:
1415
run-tests:
15-
uses: mlcommons/mlperf-automations/.github/workflows/test-mlperf-inference-submission-generation.yml@dev
16+
uses: mlcommons/mlperf-automations/.github/workflows/test-mlperf-inference-submission-generation.yml@subcheckmod_change
1617
with:
17-
ref: ${{ github.event.pull_request.head.ref }}
18-
repo-url: ${{ github.event.pull_request.head.repo.html_url }}
18+
inf-branch: ${{ github.event.pull_request.head.ref }}
19+
inf-repo-url: ${{ github.event.pull_request.head.repo.html_url }}
20+
automation-repo: mlcommons@mlperf-automations
21+
automation-repo-branch: subcheckmod_change

tools/submission/directory_structure_samples/sample_1/closed/AMD/code/llama2-70b-99.9/README.md renamed to tools/submission/directory_structure_samples/sample_1/closed/AMD/src/llama2-70b-99.9/README.md

File renamed without changes.

tools/submission/directory_structure_samples/sample_1/closed/AMD/code/llama2-70b-99/README.md renamed to tools/submission/directory_structure_samples/sample_1/closed/AMD/src/llama2-70b-99/README.md

File renamed without changes.

tools/submission/directory_structure_samples/sample_1/closed/AMD/code/mixtral-8x7b/README.md renamed to tools/submission/directory_structure_samples/sample_1/closed/AMD/src/mixtral-8x7b/README.md

File renamed without changes.

tools/submission/directory_structure_samples/sample_1/closed/AMD/code/stable-diffusion-xl/README.md renamed to tools/submission/directory_structure_samples/sample_1/closed/AMD/src/stable-diffusion-xl/README.md

File renamed without changes.

tools/submission/directory_structure_samples/sample_1/open/NVIDIA/results/GB200-NVL72_GB200-186GB_aarch64x72_Dynamo/llama3.1-405b/Interactive/GB200-NVL72_GB200-186GB_aarch64x72_Dynamo_Interactive.json renamed to tools/submission/directory_structure_samples/sample_1/open/NVIDIA/results/GB200-NVL72_GB200-186GB_aarch64x72_Dynamo/llama3.1-405b/Interactive/measurements.json

File renamed without changes.

tools/submission/directory_structure_samples/sample_1/open/NVIDIA/results/Jetson-Thor_TRT/llama3.1-8b-edge/SingleStream/Jetson-Thor_TRT.json renamed to tools/submission/directory_structure_samples/sample_1/open/NVIDIA/results/Jetson-Thor_TRT/llama3.1-8b-edge/SingleStream/measurements.json

File renamed without changes.

0 commit comments

Comments
 (0)