Merge pull request #18 from ctuning/master

sync

Author: gfursin

.github/workflows/test-cmxr-mlperf-inference-rgat.yml

@@ -0,0 +1,55 @@
+name: MLPerf inference R-GAT via CMXR
+
+on:
+  pull_request_target:
+    branches: [ "master", "main", "dev" ]
+    paths:
+      - '.github/workflows/test-mlperf-inference-rgat.yml'
+      - '**'
+      - '!**.md'
+
+jobs:
+  build:
+    runs-on: ${{ matrix.os }}
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ubuntu-latest]
+        python-version: [ "3.12" ]
+        backend: [ "pytorch" ]
+        implementation: [ "python" ]
+
+    steps:
+    - uses: actions/checkout@v3
+    - name: Set up Python ${{ matrix.python-version }}
+      uses: actions/setup-python@v3
+      with:
+        python-version: ${{ matrix.python-version }}
+    - name: Install CMX
+      run: |
+        pip install cmind
+    - name: Install mlcflow
+      run: |
+        pip install mlcflow
+        pip install tabulate
+    - name: Pull MLOps repository
+      run: |
+        cmx pull repo ${{ github.event.pull_request.head.repo.html_url }} --branch=${{ github.event.pull_request.head.ref }}
+    - name: Test MLPerf Inference R-GAT using ${{ matrix.backend }} on ${{ matrix.os }}
+      run: |
+        cmxr --tags=run,mlperf,inference,generate-run-cmds,_submission,_short --adr.inference-src.tags=_branch.dev --pull_changes=yes --pull_inference_changes=yes  --submitter="MLC" --hw_name=gh_${{ matrix.os }}_x86 --model=rgat --implementation=${{ matrix.implementation }} --backend=${{ matrix.backend }} --device=cpu --scenario=Offline --test_query_count=500 --adr.compiler.tags=gcc --category=datacenter --quiet  --v --target_qps=1
+  
+    - name: Push Test MLPerf Results (only for cTuning dev branches)
+      if: github.repository_owner == 'ctuning'
+      env:
+          USER: "GitHub Action"
+          EMAIL: "admin@cTuning.org"
+          GITHUB_TOKEN: ${{ secrets.TEST_RESULTS_GITHUB_TOKEN }}
+      run: |
+        git config --global user.name "mlcommons-bot"
+        git config --global user.email "mlcommons-bot@users.noreply.github.com"
+        git config --global credential.https://github.com.helper ""
+        git config --global credential.https://github.com.helper "!gh auth git-credential"
+        git config --global credential.https://gist.github.com.helper ""
+        git config --global credential.https://gist.github.com.helper "!gh auth git-credential"
+        cmxr --tags=push,github,mlperf,inference,submission --repo_url=https://github.com/mlcommons/mlperf_inference_test_submissions_v5.0 --repo_branch=auto-update --commit_message="Results from R50 GH action on ${{ matrix.os }}" --quiet

README.md

@@ -18,16 +18,16 @@ in the most efficient and cost-effective way across diverse models, data sets, s
 
 It includes the following sub-projects.
 
-### Common Metadata eXchange (CMX)
+### Common Metadata eXchange (CMX, 2025)
 
 The [CMX framework](https://github.com/mlcommons/ck/tree/master/cmx) 
 facilitates the decomposition of complex software systems and benchmarks such as MLPerf
 into portable, reusable, and interconnected automation recipes for MLOps and DevOps. 
 These recipes are developed and continuously improved by the community.
 
 ***Starting in 2025, CMX V4.0.0 serves as drop-in, backward-compatible replacement 
-   for the earlier [Collective Mind framework (CM)](https://github.com/mlcommons/ck/tree/master/cm),
-   while providing a simpler and more robust interface.***
+   for the earlier [Collective Mind framework (CM)](https://github.com/mlcommons/ck/tree/master/cm)
+   and other MLCommons automation prototypes, while providing a simpler and more robust interface.***
 
 CMX is a lightweight, Python-based toolset that provides a unified
 command-line interface (CLI), a Python API, and minimal dependencies.
@@ -60,10 +60,11 @@ while keeping backward compatibility.
 
 See the [online documentation](https://docs.mlcommons.org/inference) 
 at MLCommons to run MLPerf inference benchmarks across diverse systems using CMX.
-Just install `pip install cmind` and substitute the following commands with `cmx`:
+Just install `pip install cmind` and substitute the following commands and flags:
 * `cm` -> `cmx`
-* `mlc` -> `cmx run mlc`
-* `mlcr` -> `cmx run mlcr`
+* `mlc` -> `cmx`
+* `mlcr` -> `cmxr`
+* `-v` -> `--v`
 
 ### MLPerf results visualization
 
@@ -93,9 +94,9 @@ and support reproducibility efforts at ML and systems conferences.
 
 ## Legacy projects 
 
-* [CM](https://github.com/mlcommons/ck/tree/master/cm)
-* [CM-MLOps](https://github.com/mlcommons/ck/tree/master/cm-mlops)
-* [CM4MLOps](https://github.com/mlcommons/cm4mlops)
+* [CM](https://github.com/mlcommons/ck/tree/master/cm) (2022-2024)
+* [CM-MLOps](https://github.com/mlcommons/ck/tree/master/cm-mlops) (2021)
+* [CM4MLOps](https://github.com/mlcommons/cm4mlops) (2022-2024)
 * [CK automation framework v1 and v2](https://github.com/mlcommons/ck/tree/master/ck)
 
 

cm/CHANGES.md

@@ -1,3 +1,6 @@
+## CMX V4.0.1
+   - provided backward compatibility with various MLCommons tools
+
 ## CMX V4.0.0
    - added CMX interface for new tools
 

cm/README.md

@@ -16,8 +16,8 @@ into portable, reusable, and interconnected automation recipes for MLOps and Dev
 These recipes are developed and continuously improved by the community.
 
 ***Starting in 2025, CMX V4.0.0 serves as drop-in, backward-compatible replacement 
-   for the earlier [Collective Mind framework (CM)](https://github.com/mlcommons/ck/tree/master/cm),
-   while providing a simpler and more robust interface.***
+   for the earlier [Collective Mind framework (CM)](https://github.com/mlcommons/ck/tree/master/cm)
+   and other MLCommons automation prototypes, while providing a simpler and more robust interface.***
 
 CMX is a lightweight, Python-based toolset that provides a unified
 command-line interface (CLI), a Python API, and minimal dependencies.
@@ -59,10 +59,11 @@ while keeping backward compatibility.
 
 See the [online documentation](https://docs.mlcommons.org/inference) 
 at MLCommons to run MLPerf inference benchmarks across diverse systems using CMX.
-Just install `pip install cmind` and substitute the following commands with `cmx`:
+Just install `pip install cmind` and substitute the following commands and flags:
 * `cm` -> `cmx`
-* `mlc` -> `cmx run mlc`
-* `mlcr` -> `cmx run mlcr`
+* `mlc` -> `cmx`
+* `mlcr` -> `cmxr`
+* `-v` -> `--v`
 
 ### MLPerf results visualization
 
@@ -125,6 +126,7 @@ To learn more about the motivation behind this project, please explore the follo
 * [White paper](https://arxiv.org/abs/2406.16791)
 * [CMX architecture](https://github.com/mlcommons/ck/tree/master/docs/specs/cm-diagram-v3.5.1.png)
 * [CMX installation GUI](https://access.cknowledge.org/playground/?action=install)
+* [CMX Python API](https://cKnowledge.org/docs/cm)
 
 *TBD*
 

cm/cmind/__init__.py

@@ -9,7 +9,7 @@
 # White paper: https://arxiv.org/abs/2406.16791
 # Project contributors: https://github.com/mlcommons/ck/blob/master/CONTRIBUTING.md
 
-__version__ = "4.0.0"
+__version__ = "4.0.1"
 
 from cmind.core import access
 from cmind.core import x

cm/cmind/cli.py

@@ -1,4 +1,4 @@
-# Collective Mind command line wrapper
+# Collective Mind and Common Metadata eXchange command line wrapper
 #
 # Author(s): Grigori Fursin
 # Contributor(s):
@@ -122,6 +122,32 @@ def run_script(argv = None):
 
     return run(['run', 'script'] + argv)
 
+############################################################
+def runx_script(argv = None):
+    """
+    Shortcut to "cmx run script ..." or "mlcr ..."
+
+    CMX command line format:
+
+    Args:
+        argv (list | string): command line arguments
+
+    Returns: 
+        (CM return dict):
+
+        * return (int): return code == 0 if no error and >0 if error
+        * (error) (str): error string if return>0
+
+        * Output from a CM automation action
+
+    """
+
+    # Access CMX
+    if argv is None:
+        argv = sys.argv[1:]
+
+    return runx(['run', 'mlcr'] + argv)
+
 ############################################################
 def docker_script(argv = None):
     """