Skip to content

Commit ea2cae8

Browse files
daniel-thomdaniel-thom
committed
Add nushell tutorial
1 parent fe004d7 commit ea2cae8

File tree

4 files changed

+308
-26
lines changed

4 files changed

+308
-26
lines changed

‎README.md‎

Lines changed: 26 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -83,32 +83,32 @@ For detailed documentation, see the [docs](docs/) directory.
8383
## Architecture
8484

8585
```
86-
┌─────────────────────────────────────────────────────────────┐
87-
│ Torc Server │
88-
│ ┌────────────────────────────────────────────────────┐ │
89-
│ │ REST API (Tokio 1-thread) │ │
90-
│ │ /workflows /jobs /files /user_data /results │ │
91-
│ └───────────────────┬────────────────────────────────┘ │
92-
│ │ │
93-
│ ┌───────────────────▼────────────────────────────────┐ │
94-
│ │ SQLite Database (WAL) │ │
95-
│ │ • Workflow state • Job dependencies │ │
96-
│ │ • Resource tracking • Execution results │ │
97-
│ └────────────────────────────────────────────────────┘ │
98-
└─────────────────────────────────────────────────────────────┘
99-
▲
100-
│ HTTP/REST
101-
│
102-
┌────────────────────┼────────────────────┐
103-
│ │ │
104-
┌───────▼────────┐ ┌────────▼────────┐ ┌───────▼────────┐
105-
│ Torc CLI │ │ Job Runner 1 │ │ Job Runner N │
106-
│ │ │ (compute-01) │ │ (compute-nn) │
107-
│ • Create │ │ │ │ │
108-
│ workflows │ │ • Poll for jobs │ │ • Poll for jobs│
109-
│ • Submit specs │ │ • Execute tasks │ │ • Execute tasks│
110-
│ • Monitor │ │ • Report results│ │ • Report results│
111-
└────────────────┘ └─────────────────┘ └────────────────┘
86+
┌───────────────────────────────────────────────────────────┐
87+
│ Torc Server │
88+
│ ┌─────────────────────────────────────────────────────┐ │
89+
│ │ REST API (Tokio 1-thread) │ │
90+
│ │ /workflows /jobs /files /user_data /results │ │
91+
│ └──────────────────────┬──────────────────────────────┘ │
92+
│ │ │
93+
│ ┌──────────────────────▼──────────────────────────────┐ │
94+
│ │ SQLite Database (WAL) │ │
95+
│ │ • Workflow state • Job dependencies │ │
96+
│ │ • Resource tracking • Execution results │ │
97+
│ └─────────────────────────────────────────────────────┘ │
98+
└───────────────────────────────────────────────────────────┘
99+
▲
100+
│ HTTP/REST
101+
│
102+
┌───────────────────┼───────────────────┐
103+
│ │ │
104+
┌───────▼───────┐ ┌────────▼────────┐ ┌───────▼───────┐
105+
│ Torc CLI │ │ Job Runner 1 │ │ Job Runner N │
106+
│ │ │ (compute-01) │ │ (compute-nn) │
107+
│ • Create │ │ │ │ │
108+
│ workflows │ │ • Poll for jobs │ │ • Poll for jobs│
109+
│ • Submit specs│ │ • Execute tasks │ │ • Execute tasks│
110+
│ • Monitor │ │ • Report results│ │ • Report results│
111+
└───────────────┘ └─────────────────┘ └───────────────┘
112112
```
113113

114114
## Command-Line Interface

‎docs/src/SUMMARY.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -66,6 +66,7 @@
6666
- [Advanced Parameterization](./tutorials/advanced-params.md)
6767
- [Multi-Stage Workflows with Barriers](./tutorials/multi-stage-barrier.md)
6868
- [Map Python functions across workers](./tutorials/map_python_function_across_workers.md)
69+
- [Filtering CLI Output with Nushell](./tutorials/filtering-with-nushell.md)
6970

7071
---
7172

‎docs/src/tutorials/README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,7 @@ This section contains learning-oriented lessons to help you get started with Tor
1414
8. [Advanced Parameterization](./advanced-params.md) - Multi-dimensional hyperparameter grid search
1515
9. [Multi-Stage Workflows with Barriers](./multi-stage-barrier.md) - Scale to thousands of jobs efficiently
1616
10. [Map Python Functions](./map_python_function_across_workers.md) - Distribute Python functions across workers
17+
11. [Filtering CLI Output with Nushell](./filtering-with-nushell.md) - Filter jobs, results, and user data with readable queries
1718

1819
Start with the Configuration Files tutorial to set up your environment, then try the Dashboard Deployment tutorial if you want to use the web interface.
1920

‎docs/src/tutorials/filtering-with-nushell.md‎

Lines changed: 280 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,280 @@
1+
# Tutorial 11: Filtering CLI Output with Nushell
2+
3+
This tutorial teaches you how to filter and analyze Torc CLI output using [Nushell](https://www.nushell.sh/), a modern shell with powerful structured data capabilities.
4+
5+
## Learning Objectives
6+
7+
By the end of this tutorial, you will:
8+
9+
- Understand why Nushell is useful for filtering Torc output
10+
- Know how to filter jobs by status, name, and other fields
11+
- Be able to analyze results and find failures quickly
12+
- Create complex queries combining multiple conditions
13+
14+
## Prerequisites
15+
16+
- Torc CLI installed and configured
17+
- A workflow with jobs (ideally one with various statuses)
18+
19+
## Why Nushell?
20+
21+
Torc's CLI can output JSON with the `-f json` flag. While tools like `jq` can process JSON, Nushell offers a more readable, SQL-like syntax that's easier to learn and use interactively.
22+
23+
Compare filtering failed jobs:
24+
25+
```bash
26+
# jq (cryptic syntax)
27+
torc jobs list 123 -f json | jq '.jobs[] | select(.status == "failed")'
28+
29+
# Nushell (readable, SQL-like)
30+
torc jobs list 123 -f json | from json | get jobs | where status == "failed"
31+
```
32+
33+
Nushell is:
34+
- **Cross-platform**: Works on Linux, macOS, and Windows
35+
- **Readable**: Uses intuitive commands like `where`, `select`, `sort-by`
36+
- **Interactive**: Tab completion and helpful error messages
37+
- **Powerful**: Built-in support for JSON, YAML, CSV, and more
38+
39+
## Installing Nushell
40+
41+
Install Nushell from [nushell.sh/book/installation](https://www.nushell.sh/book/installation.html):
42+
43+
```bash
44+
# macOS
45+
brew install nushell
46+
47+
# Windows
48+
winget install nushell
49+
50+
# Linux (various methods available)
51+
cargo install nu
52+
```
53+
54+
After installation, run `nu` to start a Nushell session. You can use Nushell interactively or run individual commands with `nu -c "command"`.
55+
56+
## Basic Filtering
57+
58+
### Setup: Get JSON Output
59+
60+
All examples assume you have a workflow ID. Replace `$WORKFLOW_ID` with your actual ID:
61+
62+
```nu
63+
# In Nushell, set your workflow ID
64+
let wf = 123
65+
```
66+
67+
### List All Jobs
68+
69+
```nu
70+
torc jobs list $wf -f json | from json | get jobs
71+
```
72+
73+
This parses the JSON and extracts the `jobs` array into a table.
74+
75+
### Filter by Status
76+
77+
Find all failed jobs:
78+
79+
```nu
80+
torc jobs list $wf -f json | from json | get jobs | where status == "failed"
81+
```
82+
83+
Find jobs that are ready or running:
84+
85+
```nu
86+
torc jobs list $wf -f json | from json | get jobs | where status in ["ready", "running"]
87+
```
88+
89+
### Filter by Name Pattern
90+
91+
Find jobs with "train" in the name:
92+
93+
```nu
94+
torc jobs list $wf -f json | from json | get jobs | where name =~ "train"
95+
```
96+
97+
The `=~` operator performs substring/regex matching.
98+
99+
### Combine Conditions
100+
101+
Find failed jobs with "process" in the name:
102+
103+
```nu
104+
torc jobs list $wf -f json | from json | get jobs | where status == "failed" and name =~ "process"
105+
```
106+
107+
Find jobs that failed or were canceled:
108+
109+
```nu
110+
torc jobs list $wf -f json | from json | get jobs | where status == "failed" or status == "canceled"
111+
```
112+
113+
## Selecting and Formatting Output
114+
115+
### Select Specific Columns
116+
117+
Show only name and status:
118+
119+
```nu
120+
torc jobs list $wf -f json | from json | get jobs | select name status
121+
```
122+
123+
### Sort Results
124+
125+
Sort by name:
126+
127+
```nu
128+
torc jobs list $wf -f json | from json | get jobs | sort-by name
129+
```
130+
131+
Sort failed jobs by ID (descending):
132+
133+
```nu
134+
torc jobs list $wf -f json | from json | get jobs | where status == "failed" | sort-by id -r
135+
```
136+
137+
### Count Results
138+
139+
Count jobs by status:
140+
141+
```nu
142+
torc jobs list $wf -f json | from json | get jobs | group-by status | transpose status jobs | each { |row| { status: $row.status, count: ($row.jobs | length) } }
143+
```
144+
145+
Or more simply, count failed jobs:
146+
147+
```nu
148+
torc jobs list $wf -f json | from json | get jobs | where status == "failed" | length
149+
```
150+
151+
## Analyzing Results
152+
153+
### Find Jobs with Non-Zero Return Codes
154+
155+
```nu
156+
torc results list $wf -f json | from json | get results | where return_code != 0
157+
```
158+
159+
### Find Results with Specific Errors
160+
161+
```nu
162+
torc results list $wf -f json | from json | get results | where return_code != 0 | select job_id return_code
163+
```
164+
165+
### Join Jobs with Results
166+
167+
Get job names for failed results:
168+
169+
```nu
170+
let jobs = (torc jobs list $wf -f json | from json | get jobs)
171+
let results = (torc results list $wf -f json | from json | get results | where return_code != 0)
172+
$results | each { |r|
173+
let job = ($jobs | where id == $r.job_id | first)
174+
{ name: $job.name, return_code: $r.return_code, job_id: $r.job_id }
175+
}
176+
```
177+
178+
## Working with User Data
179+
180+
### List User Data Entries
181+
182+
```nu
183+
torc user-data list $wf -f json | from json | get user_data
184+
```
185+
186+
### Filter by Key
187+
188+
Find user data with a specific key:
189+
190+
```nu
191+
torc user-data list $wf -f json | from json | get user_data | where key =~ "config"
192+
```
193+
194+
### Parse JSON Values
195+
196+
User data values are JSON strings. Parse and filter them:
197+
198+
```nu
199+
torc user-data list $wf -f json | from json | get user_data | each { |ud|
200+
{ key: $ud.key, value: ($ud.value | from json) }
201+
}
202+
```
203+
204+
## Practical Examples
205+
206+
### Example 1: Debug Failed Jobs
207+
208+
Find failed jobs and get their result details:
209+
210+
```nu
211+
# Get failed job IDs
212+
let failed_ids = (torc jobs list $wf -f json | from json | get jobs | where status == "failed" | get id)
213+
214+
# Show results for those jobs
215+
torc results list $wf -f json | from json | get results | where job_id in $failed_ids | select job_id return_code
216+
```
217+
218+
### Example 2: Find Stuck Jobs
219+
220+
Find jobs that have been running for a long time (status is "running"):
221+
222+
```nu
223+
torc jobs list $wf -f json | from json | get jobs | where status == "running" | select id name
224+
```
225+
226+
### Example 3: Parameter Sweep Analysis
227+
228+
For a parameterized workflow, find which parameter values failed:
229+
230+
```nu
231+
torc jobs list $wf -f json | from json | get jobs | where status == "failed" and name =~ "lr" | get name
232+
```
233+
234+
### Example 4: Export to CSV
235+
236+
Export failed jobs to CSV for further analysis:
237+
238+
```nu
239+
torc jobs list $wf -f json | from json | get jobs | where status == "failed" | to csv | save failed_jobs.csv
240+
```
241+
242+
## Quick Reference
243+
244+
| Operation | Nushell Command |
245+
|-----------|-----------------|
246+
| Parse JSON | `from json` |
247+
| Get field | `get jobs` |
248+
| Filter rows | `where status == "failed"` |
249+
| Select columns | `select name status id` |
250+
| Sort | `sort-by name` |
251+
| Sort descending | `sort-by id -r` |
252+
| Count | `length` |
253+
| Substring match | `where name =~ "pattern"` |
254+
| Multiple conditions | `where status == "failed" and name =~ "x"` |
255+
| In list | `where status in ["ready", "running"]` |
256+
| Group by | `group-by status` |
257+
| Save to file | `save output.json` |
258+
| Convert to CSV | `to csv` |
259+
260+
## Tips
261+
262+
1. **Use `nu` interactively**: Start a Nushell session to explore data step by step
263+
2. **Tab completion**: Nushell provides completions for commands and field names
264+
3. **Pipeline debugging**: Add `| first 5` to see a sample before processing all data
265+
4. **Save queries**: Create shell aliases or scripts for common filters
266+
267+
## What You Learned
268+
269+
In this tutorial, you learned:
270+
271+
- Why Nushell is a great tool for filtering Torc CLI output
272+
- How to filter jobs by status and name patterns
273+
- How to analyze results and find failures
274+
- How to work with user data
275+
- Practical examples for debugging workflows
276+
277+
## Next Steps
278+
279+
- [Nushell Documentation](https://www.nushell.sh/book/) - Learn more about Nushell's capabilities
280+
- [Torc CLI Reference](../reference/cli.md) - Full list of CLI commands and their JSON output

0 commit comments

Comments
 (0)