Skip to content

Commit a171374

Browse files
rzmkrzmk
committed
feat: add lesson 3
1 parent 8302a54 commit a171374

File tree

9 files changed

+273
-8
lines changed

9 files changed

+273
-8
lines changed

‎_toc.yml‎

Lines changed: 10 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -5,11 +5,13 @@ format: jb-book
55
root: index
66
title: Introduction
77
chapters:
8-
- file: exercises-setup
9-
- file: lessons/0/index
10-
title: "Lesson 0: Exploring qsv help messages and syntax"
11-
- file: lessons/1/index
12-
title: "Lesson 1: Displaying file content with qsv table"
13-
- file: lessons/2/index
14-
title: "Lesson 2: Piping commands"
15-
- file: appendix
8+
- file: exercises-setup
9+
- file: lessons/0/index
10+
title: "Lesson 0: Exploring qsv help messages and syntax"
11+
- file: lessons/1/index
12+
title: "Lesson 1: Displaying file content with qsv table"
13+
- file: lessons/2/index
14+
title: "Lesson 2: Piping commands"
15+
- file: lessons/3/index
16+
title: "Lesson 3: qsv and JSON"
17+
- file: appendix

‎lessons/3/buses.csv‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
id,primary_color,secondary_color,length,air_conditioner,amenities
2+
001,black,blue,full,true,"wheelchair ramp, tissue boxes, cup holders, USB ports"
3+
002,black,red,full,true,"wheelchair ramp, tissue boxes, USB ports"
4+
003,white,blue,half,true,"wheelchair ramp, tissue boxes"
5+
004,orange,blue,full,false,"wheelchair ramp, tissue boxes, USB ports"
6+
005,black,blue,full,true,"wheelchair ramp, tissue boxes, cup holders, USB ports"

‎lessons/3/exercise.ipynb‎

Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,71 @@
1+
{
2+
"cells": [
3+
{
4+
"cell_type": "markdown",
5+
"metadata": {},
6+
"source": [
7+
"# Exercise 3: qsv and JSON\n",
8+
"\n",
9+
"Generate CSV data from `flowers.json` and pipe the output into `qsv table`."
10+
]
11+
},
12+
{
13+
"cell_type": "code",
14+
"execution_count": null,
15+
"metadata": {},
16+
"outputs": [],
17+
"source": [
18+
"qsv json --help"
19+
]
20+
},
21+
{
22+
"cell_type": "markdown",
23+
"metadata": {},
24+
"source": [
25+
"We have nested data in `flowers_nested.json`. Generate CSV data from the file with only roses data and pipe the output into `qsv table`."
26+
]
27+
},
28+
{
29+
"cell_type": "code",
30+
"execution_count": null,
31+
"metadata": {
32+
"vscode": {
33+
"languageId": "plaintext"
34+
}
35+
},
36+
"outputs": [],
37+
"source": []
38+
},
39+
{
40+
"cell_type": "markdown",
41+
"metadata": {},
42+
"source": [
43+
"Generate JSON data from `buses.csv`."
44+
]
45+
},
46+
{
47+
"cell_type": "code",
48+
"execution_count": null,
49+
"metadata": {},
50+
"outputs": [],
51+
"source": [
52+
"qsv slice --help"
53+
]
54+
}
55+
],
56+
"metadata": {
57+
"kernelspec": {
58+
"display_name": "Bash",
59+
"language": "bash",
60+
"name": "bash"
61+
},
62+
"language_info": {
63+
"codemirror_mode": "shell",
64+
"file_extension": ".sh",
65+
"mimetype": "text/x-sh",
66+
"name": "bash"
67+
}
68+
},
69+
"nbformat": 4,
70+
"nbformat_minor": 4
71+
}

‎lessons/3/flowers.json‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
[
2+
{
3+
"name": "tulip",
4+
"primary_color": "purple",
5+
"available": true,
6+
"quantity": 4
7+
},
8+
{
9+
"name": "rose",
10+
"primary_color": "red",
11+
"available": true,
12+
"quantity": 6
13+
},
14+
{
15+
"name": "sunflower",
16+
"primary_color": "yellow",
17+
"available": false,
18+
"quantity": 0
19+
}
20+
]

‎lessons/3/flowers_nested.json‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
{
2+
"tulips": [
3+
{
4+
"color": "purple",
5+
"quantity": 2
6+
},
7+
{
8+
"color": "white",
9+
"quantity": 1
10+
},
11+
{
12+
"color": "orange",
13+
"quantity": 1
14+
}
15+
],
16+
"roses": [
17+
{
18+
"color": "red",
19+
"quantity": 4
20+
},
21+
{
22+
"color": "white",
23+
"quantity": 1
24+
},
25+
{
26+
"color": "pink",
27+
"quantity": 1
28+
}
29+
]
30+
}

‎lessons/3/index.md‎

Lines changed: 136 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,136 @@
1+
---
2+
jupytext:
3+
text_representation:
4+
extension: .md
5+
format_name: myst
6+
kernelspec:
7+
display_name: Bash
8+
language: bash
9+
name: bash
10+
---
11+
12+
# Lesson 3: qsv and JSON
13+
14+
![Examples of generating JSON and CSV data with qsv](./media/how-to-convert-csv-to-json-and-vice-versa-with-qsv.png)
15+
16+
A common file format that's often used for sharing data is JSON. qsv includes various capabilities to interact between CSV data and JSON data, so we'll cover two common use cases in this lesson.
17+
18+
## JSON to CSV
19+
20+
![Example of generating JSON data from CSV data using qsv](./media/json-to-csv.png)
21+
22+
The [`qsv json`](https://github.com/dathere/qsv/blob/master/src/cmd/json.rs) command can help with generating CSV data from JSON data. The help message includes criteria necessary for generating JSON data from CSV data, along with multiple useful flags including:
23+
24+
- `--jaq` for filtering JSON data using [jq](https://jqlang.github.io/jq/)-like syntax (based on [jaq](https://github.com/01mf02/jaq)), where jq is a common JSON command-line tool
25+
- `--select` for selecting, reordering, or dropping columns for output
26+
27+
```{code-cell}
28+
:tags: ["scroll-output"]
29+
qsv json -h
30+
```
31+
32+
## CSV to JSON
33+
34+
![Example of generating CSV data from JSON data using qsv](./media/csv-to-json.png)
35+
36+
The [`qsv slice`](https://github.com/dathere/qsv/blob/master/src/cmd/slice.rs) command has a `--json` flag that can help generate JSON data from CSV data.
37+
38+
```{code-cell}
39+
:tags: ["scroll-output"]
40+
qsv slice -h
41+
```
42+
43+
## Exercise 3: qsv and JSON
44+
45+
[![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/dathere/100.dathere.com/main?labpath=lessons%2F3%2Fexercise.ipynb)
46+
47+
Using `qsv json`, `qsv slice`, and their options, complete each of the following tasks on the `flowers.json`, `flowers_nested.json`, and `buses.csv` files:
48+
49+
1. Generate CSV data from `flowers.json` and pipe the output into `qsv table`.
50+
2. We have nested data in `flowers_nested.json`. Generate CSV data from the file with only roses data and pipe the output into `qsv table`.
51+
3. Generate JSON data from `buses.csv`.
52+
53+
> Here we show the usage text of `qsv json` for your reference. Solve this exercise using [Thebe](exercises-setup:thebe), [Binder](exercises-setup:binder) or [locally](exercises-setup:local).
54+
55+
```{code-cell}
56+
:tags: ["scroll-output"]
57+
qsv json --help
58+
```
59+
60+
::::{admonition} Solution for task 1
61+
:class: dropdown seealso
62+
63+
You may first want to identify the structure of `flowers.json` and whether it adheres to the criteria necessary for `qsv json` to work as intended by default.
64+
65+
Here's the criteria for your reference:
66+
67+
The JSON data is expected to be non-empty and non-nested as either:
68+
69+
1. An array of objects where:
70+
- A. All objects are non-empty, have non-empty and unique keys, and the same keys are in each object.
71+
- B. Values are not objects or arrays.
72+
2. An object where values are not objects or arrays and the object is as described above.
73+
74+
The data in `flowers.json` is non-empty, non-nested, and fits the criteria for an array of objects in point 1, therefore we do not need to run a filter using `--jaq` to generate CSV data from it when using `qsv json`.
75+
76+
The following command should generate the expected output:
77+
78+
```bash
79+
qsv json flowers.json | qsv table
80+
```
81+
82+
The output should be:
83+
84+
```
85+
name primary_color available quantity
86+
tulip purple true 4
87+
rose red true 6
88+
sunflower yellow false 0
89+
```
90+
91+
::::
92+
93+
::::{admonition} Solution for task 2
94+
:class: dropdown seealso
95+
96+
The `flowers_nested.json` file includes nested data and the task only requires data about roses. Observing the `flowers_nested.json` file's data, we see that it is an array of flowers where there are two keys that have their own array of objects. However, the criteria for running `qsv json` the data to not be nested.
97+
98+
The two keys are `tulips` and `roses`. Since we only want roses data, we need to specify the roses data for our output. We can use the `--jaq` option to filter the JSON data for only roses data by specifying the `.roses` filter which runs a filter on the JSON data to only process the value of the `roses` key for generating CSV data.
99+
100+
The following command should generate the expected output:
101+
102+
```bash
103+
qsv json flowers_nested.json --jaq .roses | qsv table
104+
```
105+
106+
The output should be:
107+
108+
```
109+
color quantity
110+
red 4
111+
white 1
112+
pink 1
113+
```
114+
115+
::::
116+
117+
::::{admonition} Solution for task 3
118+
:class: dropdown seealso
119+
120+
We can use `qsv slice` with its `--json` flag to generate JSON data from the given CSV.
121+
122+
The following command should generate the expected output:
123+
124+
```bash
125+
qsv slice buses.csv --json
126+
```
127+
128+
The output should be:
129+
130+
```json
131+
[{"id":"001","primary_color":"black","secondary_color":"blue","length":"full","air_conditioner":"true","amenities":"wheelchair ramp, tissue boxes, cup holders, USB ports"},{"id":"002","primary_color":"black","secondary_color":"red","length":"full","air_conditioner":"true","amenities":"wheelchair ramp, tissue boxes, USB ports"},{"id":"003","primary_color":"white","secondary_color":"blue","length":"half","air_conditioner":"true","amenities":"wheelchair ramp, tissue boxes"},{"id":"004","primary_color":"orange","secondary_color":"blue","length":"full","air_conditioner":"false","amenities":"wheelchair ramp, tissue boxes, USB ports"},{"id":"005","primary_color":"black","secondary_color":"blue","length":"full","air_conditioner":"true","amenities":"wheelchair ramp, tissue boxes, cup holders, USB ports"}]
132+
```
133+
134+
If you have jaq installed then you could pipe this output into `jaq .` to view the data in a prettier format.
135+
136+
::::

‎lessons/3/media/csv-to-json.png‎

84.1 KB
Loading

‎lessons/3/media/how-to-convert-csv-to-json-and-vice-versa-with-qsv.png‎

154 KB
Loading

‎lessons/3/media/json-to-csv.png‎

84.3 KB
Loading

0 commit comments

Comments
 (0)