Skip to content

Commit 2ac968b

Browse files
arjunsuresharjunsuresh
committed
Added back the script action docs
1 parent 9c1e977 commit 2ac968b

File tree

5 files changed

+372
-134
lines changed

5 files changed

+372
-134
lines changed

‎docs/targets/script/execution-flow.md‎

Lines changed: 85 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,85 @@
1+
# MLC Script Execution
2+
3+
## Understanding MLC scripts
4+
5+
* An MLC script is identified by a set of tags and by an unique ID.
6+
* Further each MLC script can have multiple variations and they are identified by variation tags which are treated in the same way as tags and identified by a `_` prefix.
7+
8+
### MLC script execution flow
9+
```mermaid
10+
graph TD
11+
MLC -->|env = incoming env + env_from_meta| B[Script]
12+
B -->|env - local_env_keys| C[List of Dependencies]
13+
C --> D[Preprocess]
14+
D -->|env - local_env_keys| E[Prehook dependencies]
15+
E -->F[Run script]
16+
F -->|env - clean_env_keys_post_deps| G[Posthook dependencies]
17+
G --> H[Postprocess]
18+
H -->|env - clean_env_keys_post_deps| I[Post dependencies]
19+
I -->|"env(new_env_keys)"| J[Script return]
20+
```
21+
22+
* When an MLC script is invoked (either by tags or by unique ID), its `meta.yaml` is processed first which will check for any `deps` script and if there are, then they are executed in order.
23+
* Once all the `deps` scripts are executed, `customize.py` file is checked and if existing `preprocess` function inside it is executed if present.
24+
* Then any `prehook_deps` scripts mentioned in `meta.yaml` are executed similar to `deps`
25+
* After this, keys in `env` dictionary is exported as `ENV` variables and `run` file if exists is executed.
26+
* Once run file execution is done, any `posthook_deps` scripts mentioned in `meta.yaml` are executed similar to `deps`
27+
* Then `postprocess` function inside customize.py is executed if present.
28+
* After this stage any `post_deps` scripts mentioned in `meta.yaml` is executed.
29+
30+
*If a script is already cached, then the `preprocess`, `run file` and `postprocess` executions won't happen and only the dependencies marked as `dynamic` will be executed from `deps`, `prehook_deps`, `posthook_deps` and `postdeps`.*
31+
32+
### Input flags
33+
When we run an MLC script we can also pass inputs to it and any input added in `input_mapping` dictionary inside `meta.yaml` gets converted to the corresponding `ENV` variable.
34+
35+
### Conditional execution of any `deps`, `post_deps`
36+
We can use `skip_if_env` dictionary inside any `deps`, `prehook_deps`, `posthook_deps` or `post_deps` to make its execution conditional
37+
38+
### Versions
39+
We can specify any specific version of a script using `version`. `version_max` and `version_min` are also possible options.
40+
41+
* When `version_min` is given, any version above this if present in the cache or detected in the system can be chosen. If nothing is detected `default_version` if present and if above `version_min` will be used for installation. Otherwise `version_min` will be used as `version`.
42+
43+
* When `version_max` is given, any version below this if present in the cache or detected in the system can be chosen. If nothing is detected `default_version` if present and if below `version_max` will be used for installation. Otherwise `version_max_usable` (additional needed input for `version_max`) will be used as `version`.
44+
45+
### Variations
46+
* Variations are used to customize MLC script and each unique combination of variations uses a unique cache entry. Each variation can turn on `env` keys also any other meta including dependencies specific to it. Variations are turned on like tags but with a `_` prefix. For example, if a script is having tags `"get,myscript"`, to call the variation `"test"` inside it, we have to use tags `"get,myscript,_test"`.
47+
48+
#### Variation groups
49+
`group` is a key to map variations into a group and at any time only one variation from a group can be used in the variation tags. For example, both `cpu` and `cuda` can be two variations under the `device` group, but user can at any time use either `cpu` or `cuda` as variation tags but not both.
50+
51+
#### Dynamic variations
52+
Sometimes it is difficult to add all variations needed for a script like say `batch_size` which can take many different values. To handle this case, we support dynamic variations using '#' where '#' can be dynamically replaced by any string. For example, `"_batch_size.8"` can be used as a tag to turn on the dynamic variation `"_batch_size.#"`.
53+
54+
### ENV flow during MLC script execution
55+
56+
57+
* During a given script execution incoming `env` dictionary is saved `(saved_env)` and all the updates happens on a copy of it.
58+
* Once a script execution is over (which includes all the dependent script executions as well), newly created keys and any updated keys are merged with the `saved_env` provided the keys are mentioned in `new_env_keys`
59+
* Same behaviour applies to `state` dictionary.
60+
61+
#### Special env keys
62+
* Any env key with a prefix `MLC_TMP_*` and `MLC_GIT_*` are not passed by default to any dependency. These can be force passed by adding the key(s) to the `force_env_keys` list of the concerned dependency.
63+
* Similarly we can avoid any env key from being passed to a given dependency by adding the prefix of the key in the `clean_env_keys` list of the concerned dependency.
64+
* `--input` is automatically converted to `MLC_INPUT` env key
65+
* `version` is converted to `MLC_VERSION`, ``version_min` to `MLC_VERSION_MIN` and `version_max` to `MLC_VERSION_MAX`
66+
* If `env['MLC_GH_TOKEN']=TOKEN_VALUE` is set then git URLs (specified by `MLC_GIT_URL`) are changed to add this token.
67+
* If `env['MLC_GIT_SSH']=yes`, then git URLs are changed to SSH from HTTPS.
68+
69+
### Script Meta
70+
#### Special keys in script meta
71+
* TBD: `reuse_version`, `inherit_variation_tags`, `update_env_tags_from_env`
72+
73+
### How cache works?
74+
* If `cache=true` is set in a script meta, the result of the script execution is cached for further use.
75+
* For a cached script, `env` and `state` updates are done using `new_env` and `new_state` dictionaries which are stored in the `cm-cached.json` file inside the cached folder.
76+
* By using `--new` input, a new cache entry can be forced even when an old one exist.
77+
* By default no depndencies are run for a cached entry unless `dynamic` key is set for it.
78+
79+
80+
Please see [here](https://github.com/mlcommons/mlperf-automations/blob/main/docs/getting-started.md) for trying MLC scripts.
81+
82+
83+
84+
85+
&copy; 2022-25 [MLCommons](https://mlcommons.org)<br>

‎docs/targets/script/index.md‎

Lines changed: 170 additions & 63 deletions
Original file line numberDiff line numberDiff line change
@@ -1,85 +1,192 @@
1-
# MLC script
2-
3-
## Understanding MLC scripts
4-
5-
* An MLC script is identified by a set of tags and by an unique ID.
6-
* Further each MLC script can have multiple variations and they are identified by variation tags which are treated in the same way as tags and identified by a `_` prefix.
7-
8-
### MLC script execution flow
9-
```mermaid
10-
graph TD
11-
MLC -->|env = incoming env + env_from_meta| B[Script]
12-
B -->|env - local_env_keys| C[List of Dependencies]
13-
C --> D[Preprocess]
14-
D -->|env - local_env_keys| E[Prehook dependencies]
15-
E -->F[Run script]
16-
F -->|env - clean_env_keys_post_deps| G[Posthook dependencies]
17-
G --> H[Postprocess]
18-
H -->|env - clean_env_keys_post_deps| I[Post dependencies]
19-
I -->|"env(new_env_keys)"| J[Script return]
1+
# **Script Action**
2+
3+
The following actions are supported for managing scripts in **MLCFlow**.
4+
5+
---
6+
7+
## **Syntax Variations**
8+
9+
MLC scripts can be identified in different ways:
10+
11+
1. **Using tags:** `--tags=<comma-separated-tags>` (e.g., `--tags=detect,os`)
12+
2. **Using alias:** `<script_alias>` (e.g., `detect-os`)
13+
3. **Using UID:** `<script_uid>` (e.g., `5b4e0237da074764`)
14+
4. **Using both alias and UID:** `<script_alias>,<script_uid>` (e.g., `detect-os,5b4e0237da074764`)
15+
16+
!!! note
17+
For simplicity, syntax variations are only shown for the `find` action, but similar options apply to all other actions.
18+
19+
---
20+
21+
## **Find**
22+
23+
The `find` action retrieves the path of scripts available in MLC repositories.
24+
25+
### **Syntax Variations**
26+
27+
| Command Format | Example Usage |
28+
|---------------|--------------|
29+
| `mlc find script --tags=<tags>` | `mlc find script --tags=detect,os` |
30+
| `mlc find script <script_alias>` | `mlc find script detect-os` |
31+
| `mlc find script <script_uid>` | `mlc find script 5b4e0237da074764` |
32+
| `mlc find script <script_alias>,<script_uid>` | `mlc find script detect-os,5b4e0237da074764` |
33+
34+
<details>
35+
<summary><strong>Example Output</strong> 📌</summary>
36+
37+
```bash
38+
arjun@intel-spr-i9:~$ mlc find script --tags=detect,os -j
39+
[2025-02-14 02:55:12,999 main.py:1686 INFO] - Item path: /home/arjun/MLC/repos/gateoverflow@mlperf-automations/script/detect-os
40+
```
41+
</details>
42+
43+
🔹 **Example usage:** [GitHub Workflow](https://github.com/mlcommons/mlcflow/blob/d0269b47021d709e0ffa7fe0db8c79635bfd9dff/.github/workflows/test-mlc-core-actions.yaml)
44+
45+
---
46+
47+
## **Show**
48+
49+
Retrieves the path and metadata of scripts in MLC repositories.
50+
51+
**Example Command:**
52+
```bash
53+
mlc show script --tags=detect,os
2054
```
2155

22-
* When an MLC script is invoked (either by tags or by unique ID), its `meta.yaml` is processed first which will check for any `deps` script and if there are, then they are executed in order.
23-
* Once all the `deps` scripts are executed, `customize.py` file is checked and if existing `preprocess` function inside it is executed if present.
24-
* Then any `prehook_deps` scripts mentioned in `meta.yaml` are executed similar to `deps`
25-
* After this, keys in `env` dictionary is exported as `ENV` variables and `run` file if exists is executed.
26-
* Once run file execution is done, any `posthook_deps` scripts mentioned in `meta.yaml` are executed similar to `deps`
27-
* Then `postprocess` function inside customize.py is executed if present.
28-
* After this stage any `post_deps` scripts mentioned in `meta.yaml` is executed.
56+
<details>
57+
<summary><strong>Example Output</strong> 📌</summary>
58+
59+
```bash
60+
arjun@intel-spr-i9:~$ mlc show script --tags=detect,os
61+
[2025-02-14 02:56:16,604 main.py:1404 INFO] - Showing script with tags: detect,os
62+
Location: /home/arjun/MLC/repos/gateoverflow@mlperf-automations/script/detect-os:
63+
Main Script Meta:
64+
uid: 863735b7db8c44fc
65+
alias: detect-os
66+
tags: ['detect-os', 'detect', 'os', 'info']
67+
new_env_keys: ['MLC_HOST_OS_*', '+MLC_HOST_OS_*', 'MLC_HOST_PLATFORM_*', 'MLC_HOST_PYTHON_*', 'MLC_HOST_SYSTEM_NAME', 'MLC_RUN_STATE_DOCKER', '+PATH']
68+
new_state_keys: ['os_uname_*']
69+
......................................................
70+
For full script meta, see meta file at /home/arjun/MLC/repos/gateoverflow@mlperf-automations/script/detect-os/meta.yaml
71+
```
72+
</details>
73+
74+
---
75+
76+
## **Add**
77+
78+
Creates a new script in a registered MLC repository.
79+
80+
**Example Command:**
81+
```bash
82+
mlc add script <user@repo>:new_script --tags=benchmark
83+
```
2984

30-
*If a script is already cached, then the `preprocess`, `run file` and `postprocess` executions won't happen and only the dependencies marked as `dynamic` will be executed from `deps`, `prehook_deps`, `posthook_deps` and `postdeps`.*
85+
**Options:**
86+
- `--template_tags`: A comma-separated list of tags to create a new MLC script based on existing templates.
3187

32-
### Input flags
33-
When we run an MLC script we can also pass inputs to it and any input added in `input_mapping` dictionary inside `meta.yaml` gets converted to the corresponding `ENV` variable.
88+
<details>
89+
<summary><strong>Example Output</strong> 📌</summary>
3490

35-
### Conditional execution of any `deps`, `post_deps`
36-
We can use `skip_if_env` dictionary inside any `deps`, `prehook_deps`, `posthook_deps` or `post_deps` to make its execution conditional
91+
```bash
92+
arjun@intel-spr-i9:~$ mlc add script gateoverflow@mlperf-automations --tags=benchmark --template_tags=app,mlperf,inference
93+
More than one script found for None:
94+
1. /home/arjun/MLC/repos/gateoverflow@mlperf-automations/script/app-mlperf-inference-mlcommons-python
95+
2. /home/arjun/MLC/repos/gateoverflow@mlperf-automations/script/app-mlperf-inference-ctuning-cpp-tflite
96+
3. /home/arjun/MLC/repos/gateoverflow@mlperf-automations/script/app-mlperf-inference
97+
4. /home/arjun/MLC/repos/gateoverflow@mlperf-automations/script/app-mlperf-inference-mlcommons-cpp
98+
Select the correct one (enter number, default=1): 1
99+
[2025-02-14 02:58:33,453 main.py:664 INFO] - Folder successfully copied from /home/arjun/MLC/repos/gateoverflow@mlperf-automations/script/app-mlperf-inference-mlcommons-python to /home/arjun/MLC/repos/gateoverflow@mlperf-automations/script/gateoverflow@mlperf-automations
100+
```
101+
</details>
37102

38-
### Versions
39-
We can specify any specific version of a script using `version`. `version_max` and `version_min` are also possible options.
103+
---
40104

41-
* When `version_min` is given, any version above this if present in the cache or detected in the system can be chosen. If nothing is detected `default_version` if present and if above `version_min` will be used for installation. Otherwise `version_min` will be used as `version`.
105+
## **Move (`mv`)**
42106

43-
* When `version_max` is given, any version below this if present in the cache or detected in the system can be chosen. If nothing is detected `default_version` if present and if below `version_max` will be used for installation. Otherwise `version_max_usable` (additional needed input for `version_max`) will be used as `version`.
107+
Transfers a script between repositories or renames a script within the same repository.
44108

45-
### Variations
46-
* Variations are used to customize MLC script and each unique combination of variations uses a unique cache entry. Each variation can turn on `env` keys also any other meta including dependencies specific to it. Variations are turned on like tags but with a `_` prefix. For example, if a script is having tags `"get,myscript"`, to call the variation `"test"` inside it, we have to use tags `"get,myscript,_test"`.
47-
48-
#### Variation groups
49-
`group` is a key to map variations into a group and at any time only one variation from a group can be used in the variation tags. For example, both `cpu` and `cuda` can be two variations under the `device` group, but user can at any time use either `cpu` or `cuda` as variation tags but not both.
109+
**Example Command:**
110+
```bash
111+
mlc mv script <user@source_repo>:script <user@target_repo>:script
112+
```
113+
114+
---
115+
116+
## **Copy (`cp`)**
117+
118+
Duplicates a script between repositories or within the same repository.
119+
120+
**Example Command:**
121+
```bash
122+
mlc cp script <user@source_repo>:script <user@target_repo>:script
123+
```
124+
125+
---
126+
127+
## **Remove (`rm`)**
128+
129+
Deletes one or more scripts from MLC repositories.
50130

51-
#### Dynamic variations
52-
Sometimes it is difficult to add all variations needed for a script like say `batch_size` which can take many different values. To handle this case, we support dynamic variations using '#' where '#' can be dynamically replaced by any string. For example, `"_batch_size.8"` can be used as a tag to turn on the dynamic variation `"_batch_size.#"`.
131+
**Example Command:**
132+
```bash
133+
mlc rm script --tags=detect,os -f
134+
```
135+
136+
---
53137

54-
### ENV flow during MLC script execution
138+
## **Run**
55139

140+
Executes a script from an MLC repository.
56141

57-
* During a given script execution incoming `env` dictionary is saved `(saved_env)` and all the updates happens on a copy of it.
58-
* Once a script execution is over (which includes all the dependent script executions as well), newly created keys and any updated keys are merged with the `saved_env` provided the keys are mentioned in `new_env_keys`
59-
* Same behaviour applies to `state` dictionary.
142+
**Example Command:**
143+
```bash
144+
mlc run script --tags=detect,os -j
145+
```
60146

61-
#### Special env keys
62-
* Any env key with a prefix `MLC_TMP_*` and `MLC_GIT_*` are not passed by default to any dependency. These can be force passed by adding the key(s) to the `force_env_keys` list of the concerned dependency.
63-
* Similarly we can avoid any env key from being passed to a given dependency by adding the prefix of the key in the `clean_env_keys` list of the concerned dependency.
64-
* `--input` is automatically converted to `MLC_INPUT` env key
65-
* `version` is converted to `MLC_VERSION`, ``version_min` to `MLC_VERSION_MIN` and `version_max` to `MLC_VERSION_MAX`
66-
* If `env['MLC_GH_TOKEN']=TOKEN_VALUE` is set then git URLs (specified by `MLC_GIT_URL`) are changed to add this token.
67-
* If `env['MLC_GIT_SSH']=yes`, then git URLs are changed to SSH from HTTPS.
147+
**Options:**
148+
- `-j`: Shows the output in a JSON format
149+
- `mlcr` can be used as a shortcut to `mlc run script --tags=`
150+
- `--input`:
151+
- `--path`:
152+
- `--outdirname`:
153+
- `--new`:
154+
- `--force_cache`:
155+
- `--version`:
156+
- `--version_max`:
157+
- `--version_min`:
158+
- `--quiet`:
159+
- `--silent`:
160+
- `-v`:
161+
- *`<Individual script inputs>`: In addition to the above options an `mlcr` command also takes any input specified with in a script meta in `input_mappings` as its input.
68162

69-
### Script Meta
70-
#### Special keys in script meta
71-
* TBD: `reuse_version`, `inherit_variation_tags`, `update_env_tags_from_env`
163+
<details>
164+
<summary><strong>Example Output</strong> 📌</summary>
72165

73-
### How cache works?
74-
* If `cache=true` is set in a script meta, the result of the script execution is cached for further use.
75-
* For a cached script, `env` and `state` updates are done using `new_env` and `new_state` dictionaries which are stored in the `cm-cached.json` file inside the cached folder.
76-
* By using `--new` input, a new cache entry can be forced even when an old one exist.
77-
* By default no depndencies are run for a cached entry unless `dynamic` key is set for it.
166+
---
78167

168+
## **Docker Execution**
79169

80-
Please see [here](https://github.com/mlcommons/mlperf-automations/blob/main/docs/getting-started.md) for trying MLC scripts.
170+
Runs scripts inside a **containerized environment**.
81171

172+
📌 Please refer to the [Docker README](#) for the full list of Docker options for MLC scripts.
173+
174+
**Example Command:**
175+
```bash
176+
mlc docker script --tags=detect,os -j
177+
```
82178

179+
---
180+
181+
## **Test**
182+
183+
Validates scripts configured with a `tests` section in `meta.yaml`.
184+
185+
**Example Command:**
186+
```bash
187+
mlc test script --tags=benchmark
188+
```
83189

190+
🔹 **Example of test configuration:** [Meta.yaml Example](https://github.com/mlcommons/mlperf-automations/blob/0e647d7126e610d010a21dbfccca097febe80af9/script/get-generic-sys-util/meta.yaml#L24)
84191

85-
&copy; 2022-25 [MLCommons](https://mlcommons.org)<br>
192+
---

0 commit comments

Comments
 (0)