Merge pull request #643 from mlcommons/main

restoring deleted files that broke various projects including our reproducibility initiatives at conferences such as ACM/IEEE MICRO 2023

Author: gfursin

.github/workflows/format.yml

@@ -26,22 +26,16 @@ jobs:
           python3 -m pip install autopep8
           for FILE in $(git diff --name-only ${{ github.event.before }} | grep -E '.*\.py$')
           do
-            # Check if the file still exists in the working tree
-            if [ -f "$FILE" ]; then
-              autopep8 --in-place -a "$FILE"
-              git add "$FILE"
-            fi
+            autopep8 --in-place -a $FILE
+            git add $FILE
           done
 
       - name: Format modified C++ files
         run: |
           for FILE in $(git diff --name-only ${{ github.event.before }} | grep -E '.*\.(cc|cpp|h|hpp)$')
           do
-            # Check if the file still exists in the working tree
-            if [ -f "$FILE" ]; then
-              clang-format -i -style=file $FILE
-              git add $FILE
-            fi
+            clang-format -i -style=file $FILE
+            git add $FILE
           done
 
       - name: Commit and create PR

README.md

@@ -2,7 +2,7 @@
 
 [![License](https://img.shields.io/badge/License-Apache%202.0-green)](LICENSE.md)
 [![Python Version](https://img.shields.io/badge/python-3+-blue.svg)](https://github.com/mlcommons/ck/tree/master/cm/cmind)
-[![Powered by CM](https://img.shields.io/badge/Powered_by-MLCommons%20CM-blue)](https://github.com/mlcommons/ck).
+[![Powered by CM](https://img.shields.io/badge/Powered_by-MLCommons%20CM-blue)](https://pypi.org/project/cmind).
 [![Downloads](https://static.pepy.tech/badge/cm4mlops)](https://pepy.tech/project/cm4mlops)
 
 [![CM script automation features test](https://github.com/mlcommons/cm4mlops/actions/workflows/test-cm-script-features.yml/badge.svg)](https://github.com/mlcommons/cm4mlops/actions/workflows/test-cm-script-features.yml)
@@ -13,16 +13,23 @@
 [![Test QAIC Software kit Compilation](https://github.com/mlcommons/cm4mlops/actions/workflows/test-qaic-software-kit.yml/badge.svg)](https://github.com/mlcommons/cm4mlops/actions/workflows/test-qaic-software-kit.yml)
 
 
-# Collective Mind (CM)
+# CM4MLOps repository
 
-**Collective Mind (CM)** is a Python package with a CLI and API designed for creating and managing automations. Two key automations developed using CM are **Script** and **Cache**, which streamline machine learning (ML) workflows, including managing Docker runs. Both Script and Cache automations are part of the **cm4mlops** repository.
+**CM4MLOps** repository is powered by the [Collective Mind automation framework](https://github.com/mlcommons/ck/tree/master/cm), 
+a [Python package](https://pypi.org/project/cmind/) with a CLI and API designed for creating and managing automations. 
 
-The CM scripts, also housed in the **cm4mlops** repository, consist of hundreds of modular Python-wrapped scripts accompanied by `yaml` metadata, enabling the creation of robust and flexible ML workflows.
+Two key automations developed using CM are **Script** and **Cache**, which streamline machine learning (ML) workflows, 
+including managing Docker runs. Both Script and Cache automations are part of the **cm4mlops** repository.
+
+The CM scripts, also housed in this repository, consist of hundreds of modular Python-wrapped scripts accompanied 
+by `yaml` metadata, enabling the creation of robust and flexible ML workflows.
 
 - **CM Scripts Documentation**: [https://docs.mlcommons.org/cm4mlops/](https://docs.mlcommons.org/cm4mlops/)
 - **CM CLI Documentation**: [https://docs.mlcommons.org/ck/specs/cm-cli/](https://docs.mlcommons.org/ck/specs/cm-cli/)  
 
-The `mlperf-branch` of the **cm4mlops** repository is dedicated to developments specific to MLPerf Inference. Please submit any pull requests (PRs) to this branch. For more information about using CM for MLPerf Inference, refer to the [MLPerf Inference Documentation](https://docs.mlcommons.org/inference/).
+The `mlperf-branch` of the **cm4mlops** repository is dedicated to developments specific to MLPerf Inference. 
+Please submit any pull requests (PRs) to this branch. For more information about using CM for MLPerf Inference, 
+refer to the [MLPerf Inference Documentation](https://docs.mlcommons.org/inference/).
 
 ## News
 
@@ -34,22 +41,12 @@ The `mlperf-branch` of the **cm4mlops** repository is dedicated to developments
 
 ## CM concepts
 
-Check our [ACM REP'23 keynote](https://doi.org/10.5281/zenodo.8105339).
-
-## Authors
-
-[Grigori Fursin](https://cKnowledge.org/gfursin) and [Arjun Suresh](https://www.linkedin.com/in/arjunsuresh)
-
-## Major script developers
-
-Arjun Suresh, Anandhu S, Grigori Fursin
+Check our [ACM REP'23 keynote](https://doi.org/10.5281/zenodo.8105339) and the [white paper](https://arxiv.org/abs/2406.16791).
 
-## Funding
+## CM script developers
 
-We thank [cKnowledge.org](https://cKnowledge.org), [cTuning foundation](https://cTuning.org)
-and [MLCommons](https://mlcommons.org) for sponsoring this project!
+Arjun Suresh, Anandhu Sooraj, Grigori Fursin
 
-## Acknowledgments
+## Parent project
 
-We thank all [volunteers, collaborators and contributors](https://github.com/mlcommons/cm4mlops/graphs/contributors) 
-for their support, fruitful discussions, and useful feedback! 
+Visit the [parent Collective Knowledge project](https://github.com/mlcommons/ck) for further details.

automation/script/README-extra.md

@@ -2,7 +2,6 @@
 
 # CM "script" automation
 
-
 <details>
 <summary>Click here to see the table of contents.</summary>
 

automation/script/module.py

@@ -1311,7 +1311,7 @@ def _run(self, i):
 
                     r = self._call_run_deps(prehook_deps, self.local_env_keys, local_env_keys_from_meta, env, state, const, const_state, add_deps_recursive,
                                             recursion_spaces + extra_recursion_spaces,
-                                            remembered_selections, variation_tags_string, True, debug_script_tags, verbose, show_time, extra_recursion_spaces, run_state)
+                                            remembered_selections, variation_tags_string, found_cached, debug_script_tags, verbose, show_time, extra_recursion_spaces, run_state)
                     if r['return'] > 0:
                         return r
 
@@ -1372,7 +1372,7 @@ def _run(self, i):
 
                         r = self._call_run_deps(posthook_deps, self.local_env_keys, clean_env_keys_post_deps, env, state, const, const_state, add_deps_recursive,
                                                 recursion_spaces + extra_recursion_spaces,
-                                                remembered_selections, variation_tags_string, True, debug_script_tags, verbose, show_time, extra_recursion_spaces, run_state)
+                                                remembered_selections, variation_tags_string, found_cached, debug_script_tags, verbose, show_time, extra_recursion_spaces, run_state)
                         if r['return'] > 0:
                             return r
 
@@ -1383,7 +1383,7 @@ def _run(self, i):
                         # Check chain of post dependencies on other CM scripts
                         r = self._call_run_deps(post_deps, self.local_env_keys, clean_env_keys_post_deps, env, state, const, const_state, add_deps_recursive,
                                                 recursion_spaces + extra_recursion_spaces,
-                                                remembered_selections, variation_tags_string, True, debug_script_tags, verbose, show_time, extra_recursion_spaces, run_state)
+                                                remembered_selections, variation_tags_string, found_cached, debug_script_tags, verbose, show_time, extra_recursion_spaces, run_state)
                         if r['return'] > 0:
                             return r
 
@@ -1605,82 +1605,6 @@ def _run(self, i):
                     if r['return'] > 0:
                         return r
 
-            # Prepare common input to prepare and run script
-            run_script_input = {
-                'path': path,
-                'bat_ext': bat_ext,
-                'os_info': os_info,
-                'const': const,
-                'state': state,
-                'const_state': const_state,
-                'reuse_cached': reuse_cached,
-                'recursion': recursion,
-                'recursion_spaces': recursion_spaces,
-                'remembered_selections': remembered_selections,
-                'tmp_file_run_state': self.tmp_file_run_state,
-                'tmp_file_run_env': self.tmp_file_run_env,
-                'tmp_file_state': self.tmp_file_state,
-                'tmp_file_run': self.tmp_file_run,
-                'local_env_keys': self.local_env_keys,
-                'local_env_keys_from_meta': local_env_keys_from_meta,
-                'posthook_deps': posthook_deps,
-                'add_deps_recursive': add_deps_recursive,
-                'remembered_selections': remembered_selections,
-                'found_script_tags': found_script_tags,
-                'variation_tags_string': variation_tags_string,
-                'found_cached': False,
-                'debug_script_tags': debug_script_tags,
-                'verbose': verbose,
-                'meta': meta,
-                'self': self
-            }
-
-            # Check if pre-process and detect
-            if str(meta.get('predeps', 'True')).lower() not in ["0", "false", "no"] and os.path.isfile(
-                    path_to_customize_py):  # possible duplicate execution - needs fix
-                r = utils.load_python_module(
-                    {'path': path, 'name': 'customize'})
-                if r['return'] > 0:
-                    return r
-
-                customize_code = r['code']
-
-                customize_common_input = {
-                    'input': i,
-                    'automation': self,
-                    'artifact': script_artifact,
-                    'customize': script_artifact.meta.get('customize', {}),
-                    'os_info': os_info,
-                    'recursion_spaces': recursion_spaces,
-                    'script_tags': script_tags,
-                    'variation_tags': variation_tags
-                }
-                run_script_input['customize_code'] = customize_code
-                run_script_input['customize_common_input'] = customize_common_input
-
-                if repro_prefix != '':
-                    run_script_input['repro_prefix'] = repro_prefix
-                if ignore_script_error:
-                    run_script_input['ignore_script_error'] = True
-                if 'predeps' in dir(customize_code) and not fake_run:
-
-                    logging.debug(
-                        recursion_spaces +
-                        '  - Running preprocess ...')
-
-                    run_script_input['run_state'] = run_state
-
-                    ii = copy.deepcopy(customize_common_input)
-                    ii['env'] = env
-                    ii['state'] = state
-                    ii['meta'] = meta
-                    # may need to detect versions in multiple paths
-                    ii['run_script_input'] = run_script_input
-
-                    r = customize_code.predeps(ii)
-                    if r['return'] > 0:
-                        return r
-
             # Check chain of dependencies on other CM scripts
             if len(deps) > 0:
                 logging.debug(recursion_spaces +
@@ -1702,8 +1626,6 @@ def _run(self, i):
             # Clean some output files
             clean_tmp_files(clean_files, recursion_spaces)
 
-            # Repeated code
-            '''
             # Prepare common input to prepare and run script
             run_script_input = {
                 'path': path,
@@ -1733,7 +1655,6 @@ def _run(self, i):
                 'meta': meta,
                 'self': self
             }
-            '''
             if os.path.isfile(
                     path_to_customize_py):  # possible duplicate execution - needs fix
                 r = utils.load_python_module(

challenge/add-derived-metrics-to-mlperf-inference/README.md

@@ -0,0 +1,32 @@
+### Challenge
+
+Check past MLPerf inference results in [this MLCommons repository](https://github.com/mlcommons/cm4mlperf-results)
+and add derived metrics such as result/No of cores, power efficiency, device cost, operational costs, etc.
+
+Add clock speed as a third dimension to graphs and improve Bar graph visualization.
+
+Read [this documentation](https://github.com/mlcommons/ck/blob/master/docs/mlperf/inference/README.md) 
+to run reference implementations of MLPerf inference benchmarks 
+using the CM automation language and use them as a base for your developments.
+
+Check [this ACM REP'23 keynote](https://doi.org/10.5281/zenodo.8105339) to learn more about our open-source project and long-term vision.
+
+
+### Prizes
+
+* *All contributors will receive 1 point for submitting valid results for 1 complete benchmark on one system.*
+* *All contributors will receive an official MLCommons Collective Knowledge contributor award (see [this example](https://ctuning.org/awards/ck-award-202307-zhu.pdf)).*
+
+
+### Organizers
+
+* [MLCommons](https://cKnowledge.org/mlcommons-taskforce)
+* [cTuning.org](https://www.linkedin.com/company/ctuning-foundation)
+* [cKnowledge.org](https://www.linkedin.com/company/cknowledge)
+
+### Results
+
+All accepted results will be publicly available in the CM format with derived metrics 
+in this [MLCommons repository](https://github.com/mlcommons/cm4mlperf-results),
+in [MLCommons Collective Knowledge explorer](https://access.cknowledge.org/playground/?action=experiments) 
+and at official [MLCommons website](https://mlcommons.org).

challenge/add-derived-metrics-to-mlperf-inference/_cm.json

@@ -0,0 +1,22 @@
+{
+  "alias": "add-derived-metrics-to-mlperf-inference",
+  "automation_alias": "challenge",
+  "automation_uid": "3d84abd768f34e08",
+  "date_close_extension": true,
+  "date_open": "20240204",
+  "points": 2,
+  "tags": [
+    "modularize",
+    "optimize",
+    "reproduce",
+    "replicate",
+    "benchmark",
+    "automate",
+    "derived-metrics",
+    "mlperf-inference",
+    "mlperf-inference-derived-metrics"
+  ],
+  "title": "Add derived metrics to MLPerf inference benchmarks (power efficiency, results / No of cores, costs, etc)",
+  "trophies": true,
+  "uid": "c65b56d7770946ee"
+}

challenge/automate-mlperf-inference-v3.1-and-v4.0-2024/README.md

@@ -0,0 +1,4 @@
+20240220: 
+* A prototype of a GUI to generate CM commands to run MLPerf inference benchmarks is ready: [link](https://access.cknowledge.org/playground/?action=howtorun&bench_uid=39877bb63fb54725)
+* A prototype of the infrastructure to reproduce MLPerf inference benchmark results is ready: [link](https://access.cknowledge.org/playground/?action=reproduce)
+* On-going efforts: https://github.com/mlcommons/ck/issues/1052

challenge/automate-mlperf-inference-v3.1-and-v4.0-2024/_cm.yaml

@@ -0,0 +1,21 @@
+alias: automate-mlperf-inference-v3.1-and-v4.0-2024
+uid: f89f152fc2614240
+
+automation_alias: challenge
+automation_uid: 3d84abd768f34e08
+
+title: Add MLCommons CM workflows and unifed interface to automate MLPerf inference v3.1 and v4.0 benchmarks (Intel, Nvidia, Qualcomm, Arm64, TPU ...)
+
+date_open: '20231215'
+date_close: '20240315'
+
+hot: true
+
+tags:
+- automate
+- mlperf-inference-v3.1-and-v4.0
+- 2024
+
+experiments:
+- tags: mlperf-inference,v3.1
+- tags: mlperf-inference,v4.0

challenge/compose-high-performance-and-cost-efficient-ai-systems-based-on-mlperf-4.0-2024/README.md

@@ -0,0 +1,10 @@
+This challenge is under preparation. You can read about the motivation behind this challenge in our [invited talk at MLPerf-Bench @ HPCA'24](https://doi.org/10.5281/zenodo.10786893).
+
+We plan to extend [MLCommons CM framework](https://github.com/mlcommons/ck) 
+to automatically compose high-performance and cost-efficient AI systems
+based on MLPerf inference v4.0 results and [CM automation recipes](https://access.cknowledge.org/playground/?action=scripts).
+
+* A prototype of a GUI to generate CM commands to run MLPerf inference benchmarks is ready: [link](https://access.cknowledge.org/playground/?action=howtorun&bench_uid=39877bb63fb54725)
+* A prototype of the infrastructure to reproduce MLPerf inference benchmark results is ready: [link](https://access.cknowledge.org/playground/?action=reproduce)
+
+Contact the [MLCommons Task Force on Automation and Reproducibility](https://github.com/mlcommons/ck/blob/master/docs/taskforce.md) for more details.

challenge/compose-high-performance-and-cost-efficient-ai-systems-based-on-mlperf-4.0-2024/_cm.yaml

@@ -0,0 +1,25 @@
+alias: compose-high-performance-and-cost-efficient-ai-systems-based-on-mlperf-4.0-2024
+uid: 7c983102d89e4869
+
+automation_alias: challenge
+automation_uid: 3d84abd768f34e08
+
+title: "Compose high-performance and cost-efficint AI systems using MLCommons' Collective Mind and MLPerf inference"
+
+date_open: '20240101'
+
+tags:
+- compose
+- ai
+- systems
+- mlperf-inference-v4.0
+- cm
+- mlcommons-cm
+- mlperf
+- v4.0
+- performance
+- energy
+- cost
+
+experiments:
+- tags: mlperf-inference,v4.0