Skip to content

Commit 232dba5

Browse files
jade-codesjade-codes
committed
Merge remote-tracking branch 'origin/main' into pg-admin-docker-compose
2 parents 171cbf3 + 6f1d07e commit 232dba5

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

54 files changed

+1344239
-15
lines changed

‎README.md

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -31,7 +31,7 @@ Top 3 must read books are:
3131
### Great [list of over 10 communities to join](communities.md):
3232

3333
Top must-join communities for DE:
34-
- [EcZachly Data Engineering Discord](https://discord.gg/JGumAXncAK)
34+
- [DataExpert.io Community Discord](https://discord.gg/JGumAXncAK)
3535
- [Data Talks Club Slack](https://datatalks.club/slack)
3636
- [Data Engineer Things Community](https://www.dataengineerthings.org/aboutus/)
3737

@@ -80,7 +80,9 @@ Top must-join communities for ML:
8080
- [Looker Studio](https://lookerstudio.google.com/overview)
8181
- [Tableau](https://www.tableau.com/)
8282
- [Power BI](https://powerbi.microsoft.com/)
83+
- [Hex](https://hex.ai/)
8384
- [Apache Superset](https://superset.apache.org/)
85+
- [Evidence](https://evidence.dev)
8486
- Data Integration
8587
- [Cube](https://cube.dev)
8688
- [Fivetran](https://www.fivetran.com)
@@ -178,7 +180,13 @@ Here's the mostly comprehensive list of data engineering creators:
178180
| Constantin Lungu | | [Constantin Lungu](https://www.linkedin.com/in/constantin-lungu-668b8756) (10k+) | | | |
179181
| Ijaz Ali | | [Ijaz Ali](https://www.linkedin.com/in/ijaz-ali-6aaa87122/) (24K+)
180182
| Subhankar | | [Subhankar](https://www.linkedin.com/in/subhankarumass/) (5k+) | | | |
181-
183+
| Ankur Ranjan | [Big Data Show](https://www.youtube.com/@TheBigDataShow) (100k+) | [Ankur Ranjan](https://www.linkedin.com/in/thebigdatashow/) (48k+) | | | |
184+
| Lenny | | [Lenny A](https://www.linkedin.com/in/lennyardiles/) (6k+) | | | |
185+
| Mehdi Ouazza | [Mehdio DataTV](https://www.youtube.com/@mehdio) (3k+) | [Mehdi Ouazza](https://www.linkedin.com/in/mehd-io/) (20k+) | [mehd_io](https://x.com/mehd_io) | | [@mehdio_datatv](https://www.tiktok.com/@mehdio_datatv) |
186+
| ITVersity | [ITVersity](https://www.youtube.com/@itversity) (67k+) | [Durga Gadiraju](https://www.linkedin.com/in/durga0gadiraju/) (48k+) | | |
187+
| Arnaud Milleker | | [Arnaud Milleker](https://www.linkedin.com/in/arnaudmilleker/) (7k+) | | | |
188+
| Soumil Shah | [Soumil Shah] (https://www.youtube.com/@SoumilShah) (50k) | [Soumil Shah](https://www.linkedin.com/in/shah-soumil/) (8k+) | | | |
189+
| Ananth Packkildurai | | [Ananth Packkildurai](https://www.linkedin.com/in/ananthdurai/) (18k+) | | | |
182190

183191
### Great Podcasts
184192

‎bootcamp/introduction.md

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,12 @@ This will be six weeks of curricula
44

55
- Dimensional Data Modeling
66
- Homework is [here](materials/1-dimensional-data-modeling/homework/homework.md)
7+
- Day 1 Lecture is [here](https://www.dataexpert.io/lesson/dimensional-data-modeling-lecture-day-1-yt)
8+
- Day 1 Lab is [here](https://www.dataexpert.io/lesson/dimensional-data-modeling-lab-day-1-yt)
9+
- Day 2 Lecture is [here](https://www.dataexpert.io/lesson/dimensional-data-modeling-day-2-lecture-yt)
10+
- Day 2 Lab is [here](https://www.dataexpert.io/lesson/dimensional-data-modeling-day-2-lab-yt)
11+
- Day 3 Lecture is [here](https://www.dataexpert.io/lesson/dimensional-data-modeling-day-3-lecture-yt)
12+
- Day 3 Lab is [here](https://www.dataexpert.io/lesson/dimensional-data-modeling-day-3-lab-yt)
713
- Fact Data Modeling
814
- Homework is (to be added)
915
- Data Quality (analytics)

‎bootcamp/materials/1-dimensional-data-modeling/Makefile

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -7,21 +7,21 @@ up:
77
cp example.env .env; \
88
exit 1; \
99
fi
10-
docker-compose up -d;
10+
docker compose up -d;
1111

1212
.PHONY: down
1313
down:
14-
docker-compose down -v
14+
docker compose down -v
1515
@if [[ "$(docker ps -q -f name=${DOCKER_CONTAINER})" ]]; then \
1616
echo "Terminating running container..."; \
1717
docker rm ${DOCKER_CONTAINER}; \
1818
fi
1919

2020
.PHONY: restart
2121
restart:
22-
docker-compose down -v; \
22+
docker compose down -v; \
2323
sleep 5; \
24-
docker-compose up -d;
24+
docker compose up -d;
2525

2626
.PHONY: logs
2727
logs:

‎bootcamp/materials/1-dimensional-data-modeling/README.md

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
# 📅 Data Modeling
22

33
This repository contains the setup for the data modeling modules in Weeks 1 and 2.
4-
4+
55
:wrench: **Tech Stack**
66

77
- Git
@@ -47,7 +47,7 @@ There are two methods to get Postgres running locally.
4747
2. Run this command after replacing **`<computer-username>`** with your computer's username:
4848
4949
```bash
50-
psql -U <computer-username> postgres < data.dump
50+
pg_restore -U <computer-username> postgres data.dump
5151
```
5252
5353
3. Set up DataGrip, DBeaver, or your VS Code extension to point at your locally running Postgres instance.
@@ -124,6 +124,10 @@ Where:
124124
- If the test connection is successful, click "Finish" or "Save" to save the connection. You should now be able to use the database client to manage your PostgreSQL database locally.
125125
126126
## **🚨 Tables not loading!? 🚨**
127+
- If you're seeing errors about `error: invalid command \N`, you should use `pg_restore` to load `data.dump`.
128+
```bash
129+
pg_restore -U $POSTGRES_USER -d $POSTGRES_DB data.dump
130+
```
127131
- If you are on Windows and used **`docker compose up`**, table creation and data load will not take place with container creation. Once you have docker container up and verified that you are able to connect to empty postgres database with your own choice of client, follow the following steps:
128132
1. On Docker desktop, connect to my-postgres-container terminal.
129133
2. Run:
@@ -132,7 +136,7 @@ Where:
132136
-v ON_ERROR_STOP=1 \
133137
--username $POSTGRES_USER \
134138
--dbname $POSTGRES_DB \
135-
< /docker-entrypoint-initdb.d/data.dump>
139+
< /docker-entrypoint-initdb.d/data.dump
136140
```
137141
- → This will run the file `data.dump` from inside your docker container.
138142

‎bootcamp/materials/1-dimensional-data-modeling/docker-compose.yml

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -5,13 +5,12 @@ services:
55
container_name: ${DOCKER_CONTAINER}
66
env_file:
77
- .env
8-
- example.env
98
environment:
109
- POSTGRES_DB=${POSTGRES_SCHEMA}
1110
- POSTGRES_USER=${POSTGRES_USER}
1211
- POSTGRES_PASSWORD=${POSTGRES_PASSWORD}
1312
ports:
14-
- "${CONTAINER_PORT}:5432"
13+
- "${HOST_PORT}:5432"
1514
volumes:
1615
- ./:/bootcamp/
1716
- ./data.dump:/docker-entrypoint-initdb.d/data.dump

‎bootcamp/materials/1-dimensional-data-modeling/lecture-lab/players.sql

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,8 @@
1818
draft_round TEXT,
1919
draft_number TEXT,
2020
seasons season_stats[],
21-
scoring_class scorer_class,
21+
scoring_class scoring_class,
22+
years_since_last_active INTEGER,
2223
is_active BOOLEAN,
2324
current_season INTEGER,
2425
PRIMARY KEY (player_name, current_season)

‎bootcamp/materials/1-dimensional-data-modeling/sql/load_players_table_day2.sql

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -63,7 +63,7 @@ SELECT
6363
WHEN (seasons[CARDINALITY(seasons)]::season_stats).pts > 15 THEN 'good'
6464
WHEN (seasons[CARDINALITY(seasons)]::season_stats).pts > 10 THEN 'average'
6565
ELSE 'bad'
66-
END::scorer_class AS scorer_class,
66+
END::scoring_class AS scoring_class,
6767
w.season - (seasons[CARDINALITY(seasons)]::season_stats).season as years_since_last_active,
6868
w.season,
6969
(seasons[CARDINALITY(seasons)]::season_stats).season = season AS is_active

‎bootcamp/materials/1-dimensional-data-modeling/visual notes/01__Dimensional Data Modeling.png

4.51 MB
Loading

‎bootcamp/materials/1-dimensional-data-modeling/visual notes/02__Idempotency_SCD.png

4.76 MB
Loading

‎bootcamp/materials/2-fact-data-modeling/.gitignore

Lines changed: 138 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,138 @@
1+
# Byte-compiled / optimized / DLL files
2+
__pycache__/
3+
*.py[cod]
4+
*$py.class
5+
6+
# C extensions
7+
*.so
8+
9+
# Distribution / packaging
10+
.Python
11+
build/
12+
develop-eggs/
13+
dist/
14+
downloads/
15+
eggs/
16+
.eggs/
17+
lib/
18+
lib64/
19+
parts/
20+
sdist/
21+
var/
22+
wheels/
23+
pip-wheel-metadata/
24+
share/python-wheels/
25+
*.egg-info/
26+
.installed.cfg
27+
*.egg
28+
MANIFEST
29+
30+
# PyInstaller
31+
# Usually these files are written by a python script from a template
32+
# before PyInstaller builds the exe, so as to inject date/other infos into it.
33+
*.manifest
34+
*.spec
35+
36+
# Installer logs
37+
pip-log.txt
38+
pip-delete-this-directory.txt
39+
40+
# Unit test / coverage reports
41+
htmlcov/
42+
.tox/
43+
.nox/
44+
.coverage
45+
.coverage.*
46+
.cache
47+
nosetests.xml
48+
coverage.xml
49+
*.cover
50+
*.py,cover
51+
.hypothesis/
52+
.pytest_cache/
53+
54+
# Translations
55+
*.mo
56+
*.pot
57+
58+
# Django stuff:
59+
*.log
60+
local_settings.py
61+
db.sqlite3
62+
db.sqlite3-journal
63+
64+
# Flask stuff:
65+
instance/
66+
.webassets-cache
67+
68+
# Scrapy stuff:
69+
.scrapy
70+
71+
# Sphinx documentation
72+
docs/_build/
73+
74+
# PyBuilder
75+
target/
76+
77+
# Jupyter Notebook
78+
.ipynb_checkpoints
79+
80+
# IPython
81+
profile_default/
82+
ipython_config.py
83+
84+
# pyenv
85+
.python-version
86+
87+
# pipenv
88+
# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
89+
# However, in case of collaboration, if having platform-specific dependencies or dependencies
90+
# having no cross-platform support, pipenv may install dependencies that don't work, or not
91+
# install all needed dependencies.
92+
#Pipfile.lock
93+
94+
# PEP 582; used by e.g. github.com/David-OConnor/pyflow
95+
__pypackages__/
96+
97+
# Celery stuff
98+
celerybeat-schedule
99+
celerybeat.pid
100+
101+
# SageMath parsed files
102+
*.sage.py
103+
104+
# Environments
105+
.env
106+
.venv
107+
env/
108+
venv/
109+
ENV/
110+
env.bak/
111+
venv.bak/
112+
113+
# Spyder project settings
114+
.spyderproject
115+
.spyproject
116+
117+
# Rope project settings
118+
.ropeproject
119+
120+
# mkdocs documentation
121+
/site
122+
123+
# mypy
124+
.mypy_cache/
125+
.dmypy.json
126+
dmypy.json
127+
128+
# Pyre type checker
129+
.pyre/
130+
131+
dump.sql
132+
133+
# Personal workspace files
134+
.idea/*
135+
.vscode/*
136+
137+
postgres-data/*
138+
homework/your_username

0 commit comments

Comments
 (0)