Skip to content

Commit 9dbdc09

Browse files
gerrykougerrykou
authored
Merge branch 'DataExpert-io:main' into main
2 parents 4350cd6 + 715f229 commit 9dbdc09

24 files changed

+406
-9
lines changed

‎README.md

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -181,7 +181,9 @@ Here's the mostly comprehensive list of data engineering creators:
181181
| Subhankar | | [Subhankar](https://www.linkedin.com/in/subhankarumass/) (5k+) | | | |
182182
| Ankur Ranjan | [Big Data Show](https://www.youtube.com/@TheBigDataShow) (100k+) | [Ankur Ranjan](https://www.linkedin.com/in/thebigdatashow/) (48k+) | | | |
183183
| Lenny | | [Lenny A](https://www.linkedin.com/in/lennyardiles/) (6k+) | | | |
184-
184+
| Mehdi Ouazza | [Mehdio DataTV](https://www.youtube.com/@mehdio) (3k+) | [Mehdi Ouazza](https://www.linkedin.com/in/mehd-io/) (20k+) | [mehd_io](https://x.com/mehd_io) | | [@mehdio_datatv](https://www.tiktok.com/@mehdio_datatv) |
185+
| ITVersity | [ITVersity](https://www.youtube.com/@itversity) (67k+) | [Durga Gadiraju](https://www.linkedin.com/in/durga0gadiraju/) (48k+) | | |
186+
| Arnaud Milleker | | [Arnaud Milleker](https://www.linkedin.com/in/arnaudmilleker/) (7k+) | | | |
185187

186188
### Great Podcasts
187189

‎bootcamp/introduction.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,8 @@ This will be six weeks of curricula
88
- Day 1 Lab is [here](https://www.dataexpert.io/lesson/dimensional-data-modeling-lab-day-1-yt)
99
- Day 2 Lecture is [here](https://www.dataexpert.io/lesson/dimensional-data-modeling-day-2-lecture-yt)
1010
- Day 2 Lab is [here](https://www.dataexpert.io/lesson/dimensional-data-modeling-day-2-lab-yt)
11+
- Day 3 Lecture is [here](https://www.dataexpert.io/lesson/dimensional-data-modeling-day-3-lecture-yt)
12+
- Day 3 Lab is [here](https://www.dataexpert.io/lesson/dimensional-data-modeling-day-3-lab-yt)
1113
- Fact Data Modeling
1214
- Homework is (to be added)
1315
- Data Quality (analytics)

‎bootcamp/materials/1-dimensional-data-modeling/Makefile

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -7,21 +7,21 @@ up:
77
cp example.env .env; \
88
exit 1; \
99
fi
10-
docker-compose up -d;
10+
docker compose up -d;
1111

1212
.PHONY: down
1313
down:
14-
docker-compose down -v
14+
docker compose down -v
1515
@if [[ "$(docker ps -q -f name=${DOCKER_CONTAINER})" ]]; then \
1616
echo "Terminating running container..."; \
1717
docker rm ${DOCKER_CONTAINER}; \
1818
fi
1919

2020
.PHONY: restart
2121
restart:
22-
docker-compose down -v; \
22+
docker compose down -v; \
2323
sleep 5; \
24-
docker-compose up -d;
24+
docker compose up -d;
2525

2626
.PHONY: logs
2727
logs:

‎bootcamp/materials/1-dimensional-data-modeling/README.md

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -47,7 +47,7 @@ There are two methods to get Postgres running locally.
4747
2. Run this command after replacing **`<computer-username>`** with your computer's username:
4848
4949
```bash
50-
psql -U <computer-username> postgres < data.dump
50+
pg_restore -U <computer-username> postgres data.dump
5151
```
5252
5353
3. Set up DataGrip, DBeaver, or your VS Code extension to point at your locally running Postgres instance.
@@ -115,6 +115,10 @@ There are two methods to get Postgres running locally.
115115
- If the test connection is successful, click "Finish" or "Save" to save the connection. You should now be able to use the database client to manage your PostgreSQL database locally.
116116
117117
## **🚨 Tables not loading!? 🚨**
118+
- If you're seeing errors about `error: invalid command \N`, you should use `pg_restore` to load `data.dump`.
119+
```bash
120+
pg_restore -U $POSTGRES_USER -d $POSTGRES_DB data.dump
121+
```
118122
- If you are on Windows and used **`docker compose up`**, table creation and data load will not take place with container creation. Once you have docker container up and verified that you are able to connect to empty postgres database with your own choice of client, follow the following steps:
119123
1. On Docker desktop, connect to my-postgres-container terminal.
120124
2. Run:
@@ -123,7 +127,7 @@ There are two methods to get Postgres running locally.
123127
-v ON_ERROR_STOP=1 \
124128
--username $POSTGRES_USER \
125129
--dbname $POSTGRES_DB \
126-
< /docker-entrypoint-initdb.d/data.dump>
130+
< /docker-entrypoint-initdb.d/data.dump
127131
```
128132
- → This will run the file `data.dump` from inside your docker container.
129133

‎bootcamp/materials/1-dimensional-data-modeling/lecture-lab/players.sql

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@
1818
draft_round TEXT,
1919
draft_number TEXT,
2020
seasons season_stats[],
21-
scorer_class scoring_class,
21+
scoring_class scoring_class,
2222
years_since_last_active INTEGER,
2323
is_active BOOLEAN,
2424
current_season INTEGER,

‎bootcamp/materials/1-dimensional-data-modeling/sql/load_players_table_day2.sql

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -63,7 +63,7 @@ SELECT
6363
WHEN (seasons[CARDINALITY(seasons)]::season_stats).pts > 15 THEN 'good'
6464
WHEN (seasons[CARDINALITY(seasons)]::season_stats).pts > 10 THEN 'average'
6565
ELSE 'bad'
66-
END::scorer_class AS scorer_class,
66+
END::scoring_class AS scoring_class,
6767
w.season - (seasons[CARDINALITY(seasons)]::season_stats).season as years_since_last_active,
6868
w.season,
6969
(seasons[CARDINALITY(seasons)]::season_stats).season = season AS is_active

‎bootcamp/materials/1-dimensional-data-modeling/visual notes/01__Dimensional Data Modeling.png

4.51 MB
Loading

‎bootcamp/materials/1-dimensional-data-modeling/visual notes/02__Idempotency_SCD.png

4.76 MB
Loading

‎bootcamp/materials/2-fact-data-modeling/.gitignore

Lines changed: 138 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,138 @@
1+
# Byte-compiled / optimized / DLL files
2+
__pycache__/
3+
*.py[cod]
4+
*$py.class
5+
6+
# C extensions
7+
*.so
8+
9+
# Distribution / packaging
10+
.Python
11+
build/
12+
develop-eggs/
13+
dist/
14+
downloads/
15+
eggs/
16+
.eggs/
17+
lib/
18+
lib64/
19+
parts/
20+
sdist/
21+
var/
22+
wheels/
23+
pip-wheel-metadata/
24+
share/python-wheels/
25+
*.egg-info/
26+
.installed.cfg
27+
*.egg
28+
MANIFEST
29+
30+
# PyInstaller
31+
# Usually these files are written by a python script from a template
32+
# before PyInstaller builds the exe, so as to inject date/other infos into it.
33+
*.manifest
34+
*.spec
35+
36+
# Installer logs
37+
pip-log.txt
38+
pip-delete-this-directory.txt
39+
40+
# Unit test / coverage reports
41+
htmlcov/
42+
.tox/
43+
.nox/
44+
.coverage
45+
.coverage.*
46+
.cache
47+
nosetests.xml
48+
coverage.xml
49+
*.cover
50+
*.py,cover
51+
.hypothesis/
52+
.pytest_cache/
53+
54+
# Translations
55+
*.mo
56+
*.pot
57+
58+
# Django stuff:
59+
*.log
60+
local_settings.py
61+
db.sqlite3
62+
db.sqlite3-journal
63+
64+
# Flask stuff:
65+
instance/
66+
.webassets-cache
67+
68+
# Scrapy stuff:
69+
.scrapy
70+
71+
# Sphinx documentation
72+
docs/_build/
73+
74+
# PyBuilder
75+
target/
76+
77+
# Jupyter Notebook
78+
.ipynb_checkpoints
79+
80+
# IPython
81+
profile_default/
82+
ipython_config.py
83+
84+
# pyenv
85+
.python-version
86+
87+
# pipenv
88+
# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
89+
# However, in case of collaboration, if having platform-specific dependencies or dependencies
90+
# having no cross-platform support, pipenv may install dependencies that don't work, or not
91+
# install all needed dependencies.
92+
#Pipfile.lock
93+
94+
# PEP 582; used by e.g. github.com/David-OConnor/pyflow
95+
__pypackages__/
96+
97+
# Celery stuff
98+
celerybeat-schedule
99+
celerybeat.pid
100+
101+
# SageMath parsed files
102+
*.sage.py
103+
104+
# Environments
105+
.env
106+
.venv
107+
env/
108+
venv/
109+
ENV/
110+
env.bak/
111+
venv.bak/
112+
113+
# Spyder project settings
114+
.spyderproject
115+
.spyproject
116+
117+
# Rope project settings
118+
.ropeproject
119+
120+
# mkdocs documentation
121+
/site
122+
123+
# mypy
124+
.mypy_cache/
125+
.dmypy.json
126+
dmypy.json
127+
128+
# Pyre type checker
129+
.pyre/
130+
131+
dump.sql
132+
133+
# Personal workspace files
134+
.idea/*
135+
.vscode/*
136+
137+
postgres-data/*
138+
homework/your_username

‎bootcamp/materials/2-fact-data-modeling/README.md

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
# Week 2 Fact Data Modeling
2+
3+
This repo follows the same setup as week 1. Please go to the dimensional data modeling [README](../1-dimensional-data-modeling/README.md) for instructions.

0 commit comments

Comments
 (0)