DataTalks Data Engineering ZoomCamp

This repo contains my notes and homework assignments for the 2024 Data Talks Data Engineering Zoomcamp plus additional notes on orchestrators from other years - Airflow (2022), Prefect (2023), and Kestra (2025) The final project for the 2024 course can be found in this repo NYC Collisions Analytics

NOTES

MODULE 1: Docker & SQL • Terraform & GCP
MODULE 2A: Orchestration with Mage
MODULE 2B: Orchestration with Airflow
MODULE 2C: Orchestration with Prefect
MODULE 2D: Orchestration with Kestra
MODULE 3: Data Warehouses with BigQuery
MODULE 4: Analytics Engineering with dbt
MODULE 5: Batch with Spark
MODULE 6: Streaming with Kafka

RESOURCES

GIT

• GIT Beginner
• GIT Cheatsheet
• GIT Markdown Cheatsheet
• GIT Extended Special commit cases
• GIT Readme Adding HTML and CSS to GIT Readme
• GIT Merge Divergent Branches

Terraform

• Terraform State Files

Docker

• Docker Commands - DockerLabs
• Docker Commands - Linux Handbook

PostgreSQL

• Postgres PostgreSQL official documentation

PGAdmin

• PG Admin

Vizualization

• Vizualize in Python

Google Cloud Platform

• CLI Auth Login

Mage

• Mage Dox

dbt - data build tool

• dbt resources

dlt - data loader tool

• dlt practice api sources
• Building a dbt Pipeline
• dlt Demos
• dlt Blog

Duck DB

• duck DB
• Video - Why should you care about DuckDB

Batch Processing - Apache Spark

• Apache Spark
• Spark 101
• Spark - Stand Alone Mode
• Deploy Spark Cluster on AWS
• MapReduce vs. Spark
• PySpark Job in DataProc
• Spark Context vs Spark Session
• Spark Pros Cons
• Cluster vs Client Mode
• Stand Alone Internals

Streaming - Kafka

• Kafka

Potential Project Data Sources

• US Energy Use