You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
@@ -34,16 +36,19 @@ We use [uv](https://docs.astral.sh/uv/) to develop NeMo Run. The following steps
34
36
If all tests passed, then you should be good to get started with the development of NeMo-Run.
35
37
36
38
## Code Structure
39
+
37
40
The repository is home to flexible Python modules, sample scripts, tests, and more.
38
41
Here is a brief overview of where everything lives:
42
+
39
43
-[docker](docker/) - Dockerfiles to build NeMo with NeMo Run.
40
44
-[docs](docs/) - Walkthroughs and guides the library.
41
45
-[examples](examples/) - Examples for how users may want to use NeMo Run.
42
46
-[src](src/) -
43
-
-[nemo_run](src/nemo_run/) - The source code for NeMo Run.
47
+
-[nemo_run](src/nemo_run/) - The source code for NeMo Run.
44
48
-[test](test/) - Unit tests.
45
49
46
50
## Examples and Documentation
51
+
47
52
Examples provide an easy way for users to see how the NeMo Run works in action.
48
53
They should be incredibly lightweight and rely mostly on `nemo_run` for their functionality
49
54
Most should be designed for a user to get up and running on their local machines, but examples on remote clusters are welcomed if it makes sense.
@@ -54,35 +59,49 @@ It should include both an explanation of the API, and how it's used in its corre
54
59
The documentation should also cover potential pitfalls and caveats.
55
60
This existing examples and documentation should serve as a good reference to what is expected.
56
61
62
+
### Building Documentation
63
+
64
+
Run the following commands to switch to the project documentation folder and generate HTML output.
65
+
66
+
```sh
67
+
cd docs/
68
+
uv run --group docs sphinx-build source/ _build/html
69
+
```
70
+
71
+
The resulting HTML files are generated in a `_build/html` folder created under the project `docs/` folder.
72
+
57
73
## Python style
58
-
We use [``ruff``](https://docs.astral.sh/ruff/) for linting and formatting. To lint and format your code, you can run `uv run --group lint -- ruff check` and `uv run --group lint -- ruff format` respectively.
74
+
75
+
We use [`ruff`](https://docs.astral.sh/ruff/) for linting and formatting. To lint and format your code, you can run `uv run --group lint -- ruff check` and `uv run --group lint -- ruff format` respectively.
59
76
60
77
## Unit tests
78
+
61
79
Unit tests should be simple and fast.
62
80
Developers should be able to run them frequently while developing without any slowdown.
63
81
64
82
## Pull Requests (PR) Guidelines
65
83
66
84
**Send your PRs to the `main` or `dev` branch**
67
85
68
-
1) Make sure your PR does one thing. Have a clear answer to "What does this PR do?".
69
-
2) Read General Principles and style guide above
70
-
3) Make sure you sign off your commits. E.g. use ``git commit --signoff`` when committing.
71
-
4) Make sure all unit tests finish successfully before sending PR
72
-
5) Send your PR and request a review
86
+
1. Make sure your PR does one thing. Have a clear answer to "What does this PR do?".
87
+
2. Read General Principles and style guide above
88
+
3. Make sure you sign off your commits. E.g. use `git commit --signoff` when committing.
89
+
4. Make sure all unit tests finish successfully before sending PR
90
+
5. Send your PR and request a review
73
91
74
92
The `dev` branch is for active development and may be unstable. Unit tests are expected to pass before merging into `dev` or `main`.
75
93
Every release `dev` and `main` will sync to be the same.
76
94
77
95
## Sign Your Work
78
96
79
-
* We require that all contributors "sign-off" on their commits. This certifies that the contribution is your original work, or you have rights to submit it under the same license, or a compatible license.
97
+
- We require that all contributors "sign-off" on their commits. This certifies that the contribution is your original work, or you have rights to submit it under the same license, or a compatible license.
98
+
99
+
- Any contribution which contains commits that are not Signed-Off will not be accepted.
80
100
81
-
* Any contribution which contains commits that are not Signed-Off will not be accepted.
101
+
- To sign off on a commit you simply use the `--signoff` option when committing your changes:
82
102
83
-
* To sign off on a commit you simply use the `--signoff` option when committing your changes:
### **Q:** I made a change locally in my git repo and tested it using the local executor. However, the change is not reflected in the remote job.
86
86
87
-
**A**: This is most likely because you haven't committed the changes. See details about `GitArchivePackager`[here](./execution.md#packagers) to learn more.
87
+
**A**: This is most likely because you haven't committed the changes. See details about `GitArchivePackager`[here](guides/execution.md#packagers) to learn more.
88
88
89
89
### **Q:** I made a change locally _outside_ my git repo and tested it using the local executor. However, the change is not reflected in the remote job.
Copy file name to clipboardExpand all lines: docs/source/guides/management.md
+35-31Lines changed: 35 additions & 31 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -2,40 +2,42 @@
2
2
3
3
The central component for management of tasks in NeMo-Run is the `Experiment` class. It allows you to define, launch, and manage complex workflows consisting of multiple tasks. This guide provides an overview of the `Experiment` class, its methods, and how to use it effectively.
4
4
5
-
**Creating an Experiment**
6
-
---------------------------
5
+
## **Creating an Experiment**
7
6
8
7
To create an experiment, you can instantiate the `Experiment` class by passing in a descriptive title:
8
+
9
9
```python
10
10
exp = Experiment("My Experiment")
11
11
```
12
+
12
13
When executed, it will automatically generate a unique experiment ID for you, which represents one unique run of the experiment.
13
14
14
-
> [!NOTE]
15
-
> `Experiment` is a context manager and `Experiment.add` and `Experiment.run` methods can currently only be used after entering the context manager.
15
+
> [!NOTE] > `Experiment` is a context manager and `Experiment.add` and `Experiment.run` methods can currently only be used after entering the context manager.
16
16
17
-
**Adding Tasks**
18
-
-----------------
17
+
## **Adding Tasks**
19
18
20
19
You can add tasks to an experiment using the `add` method. This method supports tasks of the following kind:
21
20
22
-
1. A single task which is an instance of either `run.Partial` or `run.Script`, along with its executor.
23
-
```python
24
-
with exp:
25
-
exp.add(task_1, executor=run.LocalExecutor())
26
-
```
21
+
- A single task which is an instance of either `run.Partial` or `run.Script`, along with its executor.
27
22
28
-
2. A list of tasks, each of which is an instance of either `run.Partial` or `run.Script`, along with a single executor or a list of executors for each task in the group. Currently, all tasks in the group will be executed in parallel.
- A list of tasks, each of which is an instance of either `run.Partial` or `run.Script`, along with a single executor or a list of executors for each task in the group. Currently, all tasks in the group will be executed in parallel.
You can specify a descriptive name for the task using the `name` keyword argument.
35
36
36
37
`add` also takes in a list of plugins, each an instance of `run.Plugin`. Plugins are used to make changes to the task and executor together, which is useful in some cases - for example, to enable a config option in the task and set an environment variable in the executor related to the config option.
37
38
38
39
`add` returns a unique id for the task/job. This unique id can be used to define complex dependencies between a group of tasks as follows:
40
+
39
41
```python
40
42
with run.Experiment("dag-experiment", log_level="INFO") as exp:
@@ -48,30 +50,31 @@ with run.Experiment("dag-experiment", log_level="INFO") as exp:
48
50
)
49
51
```
50
52
51
-
**Launching an Experiment**
52
-
---------------------------
53
+
## **Launching an Experiment**
53
54
54
55
Once you have added all tasks to an experiment, you can launch it using the `run` method. This method takes several optional arguments, including `detach`, `sequential`, and `tail_logs` and `direct`:
55
56
56
-
*`detach`: If `True`, the experiment will detach from the process executing it. This is useful when launching an experiment on a remote cluster, where you may want to end the process after scheduling the tasks in that experiment.
57
-
*`sequential`: If `True`, all tasks will be executed sequentially. This is only applicable when the individual tasks do not have any dependencies on each other.
58
-
*`tail_logs`: If `True`, logs will be displayed in real-time.
59
-
*`direct`: If `True`, each task in the experiment will be executed directly in the same process on your local machine. This does not support task/job groups.
57
+
-`detach`: If `True`, the experiment will detach from the process executing it. This is useful when launching an experiment on a remote cluster, where you may want to end the process after scheduling the tasks in that experiment.
58
+
-`sequential`: If `True`, all tasks will be executed sequentially. This is only applicable when the individual tasks do not have any dependencies on each other.
59
+
-`tail_logs`: If `True`, logs will be displayed in real-time.
60
+
-`direct`: If `True`, each task in the experiment will be executed directly in the same process on your local machine. This does not support task/job groups.
You can check the status of an experiment using the `status` method:
71
+
71
72
```python
72
73
exp.status()
73
74
```
75
+
74
76
This method will display information the status of each task in the experiment. The following is a sample output from the status of experiment in [hello_scripts.py](../../../examples/hello-world/hello_scripts.py):
77
+
75
78
```bash
76
79
Experiment Status for experiment_with_scripts_1730761155
77
80
@@ -94,24 +97,24 @@ Task 2: simple.add.add_object
94
97
- Local Directory: /home/your_user/.nemo_run/experiments/experiment_with_scripts/experiment_with_scripts_1730761155/simple.add.add_object
95
98
```
96
99
97
-
**Canceling a Task**
98
-
---------------------
100
+
## **Canceling a Task**
99
101
100
102
You can cancel a task using the `cancel` method:
103
+
101
104
```python
102
105
exp.cancel("task_id")
103
106
```
104
107
105
-
**Viewing Logs**
106
-
-----------------
108
+
## **Viewing Logs**
107
109
108
110
You can view the logs of a task using the `logs` method:
111
+
109
112
```python
110
113
exp.logs("task_id")
111
114
```
112
115
113
-
**Experiment output**
114
-
-----------------
116
+
## **Experiment output**
117
+
115
118
Once an experiment is run, NeMo-Run displays information on ways to inspect and reproduce past experiments. This allows you to check logs, sync artifacts (in the future), cancel running tasks, and rerun an old experiment.
116
119
117
120
```python
@@ -129,6 +132,7 @@ nemorun experiment status experiment_with_scripts_1720556256
0 commit comments