Merge branch 'DataExpert-io:main' into main

Author: damaradiprabowo

README.md

@@ -2,55 +2,39 @@
 
 This repo has all the resources you need to become an amazing data engineer!
 
-Make sure to check out the [projects](projects.md) section for more hands-on examples!
+## Getting started
+
+If you are new to data engineering, start by following this [2024 breaking into data engineering roadmap](https://blog.dataengineer.io/p/the-2024-breaking-into-data-engineering)
+
+For more applied learning:
+- Check out the [projects](projects.md) section for more hands-on examples!
+- Check out the [interviews](interviews.md) section for more advice on how to pass data engineering interviews!
+- Check out the [books](books.md) section for a list of high quality data engineering books
+- Check out the [communities](communities.md) section for a list of high quality data engineering communities to join
+- Check out the [newsletter](newsletters.md) section to learn via email 
 
-Make sure to check out the [interviews](interviews.md) section for more advice on how to pass data engineering interviews!
 
 ## Resources
 
-Great books:
+### Great [list of over 25 books](books.md)
 
+Top 3 must read books are:
 - [Fundamentals of Data Engineering](https://www.amazon.com/Fundamentals-Data-Engineering-Robust-Systems/dp/1098108302/)
 - [Designing Data-Intensive Applications](https://www.amazon.com/Designing-Data-Intensive-Applications-Reliable-Maintainable/dp/1449373321/)
 - [Designing Machine Learning Systems](https://www.amazon.com/Designing-Machine-Learning-Systems-Production-Ready/dp/1098107969)
-- [The Hundred Page Machine Learning Book](https://www.amazon.com/Hundred-Page-Machine-Learning-Book/dp/199957950X)
-- [Kimball - The Data Warehouse Toolkit](https://ia801609.us.archive.org/14/items/the-data-warehouse-toolkit-kimball/The%20Data%20Warehouse%20Toolkit%20-%20Kimball.pdf)
-- [Data Mesh](https://www.oreilly.com/library/view/data-mesh/9781492092384/)
-- [Machine Learning System Design Interview](https://www.amazon.com/Machine-Learning-System-Design-Interview/dp/1736049127)
-- [Streaming Systems](https://www.amazon.com/Streaming-Systems-Where-Large-Scale-Processing/dp/1491983876)
-- [High Performance Spark](https://www.amazon.com/High-Performance-Spark-Practices-Optimizing/dp/1491943203)
-- [Building Evolutionary Architectures, 2nd Edition](https://www.oreilly.com/library/view/building-evolutionary-architectures/9781492097532/)
-- [Data Management at Scale, 2nd Edition](https://www.oreilly.com/library/view/data-management-at/9781098138851/)
-- [Deciphering Data Architectures](https://www.oreilly.com/library/view/deciphering-data-architectures/9781098150754/)
-- [97 Things Every Data Engineer Should Know: Collective Wisdom from the Experts](https://www.amazon.com/Things-Every-Data-Engineer-Should/dp/1492062413)
-- [Data Governance: The Definitive Guide](https://www.oreilly.com/library/view/data-governance-the/9781492063483/)
-- [Trino: The Definitive Guide](https://trino.io/trino-the-definitive-guide.html)
-- [Delta Lake: The Definitive Guide](https://www.oreilly.com/library/view/delta-lake-the/9781098151935/)
-- [Hadoop: The Definitive Guide](https://www.oreilly.com/library/view/hadoop-the-definitive/9781491901687/)
-- [Modern Data Engineering with Apache Spark: A Hands-On Guide for Building Mission-Critical Streaming Applications](https://www.amazon.com/Modern-Engineering-Apache-Spark-Hands/dp/1484274512)
-- [Data Engineering with dbt: A practical guide to building a dependable data platform with SQL](https://www.amazon.com/Data-Engineering-dbt-cloud-based-dependable-ebook/dp/B0C4LL19G7)
-- [Data Engineering with AWS](https://www.oreilly.com/library/view/data-engineering-with/9781804614426/)
-- [Practical DataOps: Delivering Agile Date Science at Scale](https://www.amazon.com/Practical-DataOps-Delivering-Agile-Science/dp/1484251032)
-- [Data Engineering Design Patterns](https://www.dedp.online/)
-- [Snowflake Data Engineering](https://www.manning.com/books/snowflake-data-engineering)
-- [Unlocking dbt](https://www.amazon.com/Unlocking-dbt-Design-Transformations-Warehouse/dp/1484296990/)
-- [Learning Spark, Second Edition](https://pages.databricks.com/rs/094-YMS-629/images/LearningSpark2.0.pdf)
-
-Communities:
-
-- [Seattle Data Guy Discord](https://discord.gg/ah95MZKkFF)
+
+### Great [list of over 10 communities to join](communities.md):
+
+Top must-join communities for DE:
 - [EcZachly Data Engineering Discord](https://discord.gg/JGumAXncAK)
-- [AdalFlow Discrod (LLM Library)](https://discord.com/invite/ezzszrRZvT)
-- [Chip Huyen MLOps Discord](https://discord.gg/dzh728c5t3)
-- [Data Engineer Things Community](https://www.dataengineerthings.org/aboutus/)
-- [DBT Community](https://www.getdbt.com/community/join-the-community/)
-- [r/dataengineering](https://www.reddit.com/r/dataengineering)
-- [Microsoft Fabric Community](https://community.fabric.microsoft.com/)
-- [r/MicrosoftFabric](https://www.reddit.com/r/MicrosoftFabric/)
 - [Data Talks Club Slack](https://datatalks.club/slack)
-- [Data Engineering Wiki](https://dataengineering.wiki/)
+- [Data Engineer Things Community](https://www.dataengineerthings.org/aboutus/)
+
+Top must-join communities for ML:
+- [AdalFlow Discord](https://discord.com/invite/ezzszrRZvT)
+- [Chip Huyen MLOps Discord](https://discord.gg/dzh728c5t3)
 
-Companies:
+### Companies:
 
 - Orchestration  
   - [Mage](https://www.mage.ai)
@@ -106,8 +90,15 @@ Companies:
   - [DuckDB](https://duckdb.org/)
 - LLM application library
   - [AdalFlow](https://github.com/SylphAI-Inc/AdalFlow)
+  - [LangChain](https://github.com/langchain-ai/langchain)
+  - [LlamaIndex](https://github.com/run-llama/llama_index)
+- Real-Time Data
+  - [Aggregations.io](https://aggregations.io)
+  - [Responsive](https://www.responsive.dev/)
+  - [RisingWave](https://risingwave.com/)
 
-Data Engineering blogs of companies:
+
+### Data Engineering blogs of companies:
 
 - [Netflix](https://netflixtechblog.com/tagged/big-data)
 - [Uber](https://www.uber.com/blog/houston/data/?uclick_id=b2f43229-f3f4-4bae-bd5d-10a05db2f70c)
@@ -120,7 +111,7 @@ Data Engineering blogs of companies:
 - [Meta](https://engineering.fb.com/category/data-infrastructure/)
 - [Onehouse](https://www.onehouse.ai/blog)
 
-Data Engineering Whitepapers:
+### Data Engineering Whitepapers:
 
 - [A Five-Layered Business Intelligence Architecture](https://ibimapublishing.com/articles/CIBIMA/2011/695619/695619.pdf)
 - [Lakehouse:A New Generation of Open Platforms that Unify Data Warehousing and Advanced Analytics](https://www.cidrdb.org/cidr2021/papers/cidr2021_paper17.pdf)
@@ -129,11 +120,15 @@ Data Engineering Whitepapers:
 - [Spark: Cluster Computing with Working Sets](https://dl.acm.org/doi/10.5555/1863103.1863113)
 - [The Google File System](https://research.google/pubs/the-google-file-system/)
 - [Building a Universal Data Lakehouse](https://www.onehouse.ai/whitepaper/onehouse-universal-data-lakehouse-whitepaper)
+- [XTable in Action: Seamless Interoperability in Data Lakes](https://arxiv.org/abs/2401.09621)
+- [MapReduce: Simplified Data Processing on Large Clusters](https://research.google/pubs/mapreduce-simplified-data-processing-on-large-clusters/)
 
-Great YouTube Channels:
+### Great YouTube Channels:
+- *you have to have >10k subscribes to be added*
 
 - 100k+ subscribers
   - [E-learning Bridge](https://www.youtube.com/@shashank_mishra)
+  - [Seattle Data Guy](https://www.youtube.com/c/SeattleDataGuy)
   - [TrendyTech](https://www.youtube.com/c/TrendytechInsights)
   - [Darshil Parmar](https://www.youtube.com/@DarshilParmar)
   - [Andreas Kretz](https://www.youtube.com/c/andreaskayy)
@@ -143,26 +138,86 @@ Great YouTube Channels:
   - [Adam Marczak](https://www.youtube.com/@AdamMarczakYT)
   - [nullQueries](https://www.youtube.com/@nullQueries)
   - [TECHTFQ by Thoufiq](https://www.youtube.com/@techTFQ)
+  - [SQLBI](https://www.youtube.com/@SQLBI)
 - 10k+ subscribers
   - [Data with Zach](https://www.youtube.com/c/datawithzach)
-  - [Seattle Data Guy](https://www.youtube.com/c/SeattleDataGuy)
   - [Azure Lib](https://www.youtube.com/@azurelib-academy)
   - [Advancing Analytics](https://www.youtube.com/@AdvancingAnalytics)
   - [Kahan Data Solutions](https://www.youtube.com/@KahanDataSolutions)
   - [Ankit Bansal](https://youtube.com/@ankitbansal6)
   - [Mr. K Talks Tech](https://www.youtube.com/channel/UCzdOan4AmF65PmLLks8Lmww)
-- 1k+ subscribers
-  - [Eric Roby](https://www.youtube.com/@codingwithroby)
 
-Great Podcasts
+### LinkedIn Voices 
+- *you have to have >5k followers to be added*
+
+- 100k+ Followers
+  - [Zach Wilson](https://www.linkedin.com/in/eczachly)
+  - [Ben Rogojan](https://www.linkedin.com/in/benjaminrogojan)
+  - [Sumit Mittal](https://www.linkedin.com/in/bigdatabysumit/)
+  - [Shashank Mishra](https://www.linkedin.com/in/shashank219/)
+  - [Chip Huyen](https://www.linkedin.com/in/chiphuyen/)
+  - [Alex Xu](https://www.linkedin.com/in/alexxubyte)
+  - [Deepak Goyal](https://www.linkedin.com/in/deepak-goyal-93805a17/)
+  - [Andreas Kretz](https://www.linkedin.com/in/andreas-kretz)
+- 50k+ Followers
+  - [Joe Reis](https://www.linkedin.com/in/josephreis)
+  - [Darshil Parmar](https://www.linkedin.com/in/darshil-parmar/)
+  - [Ankit Bansal](https://www.linkedin.com/in/ankitbansal6/)
+  - [Marc Lamberti](https://www.linkedin.com/in/marclamberti)
+  - [Marco Russo](https://www.linkedin.com/in/sqlbi)
+- 10k+ Followers
+  - [Li Yin](https://www.linkedin.com/in/li-yin-ai/)
+  - [Joseph Machado](https://www.linkedin.com/in/josephmachado1991/)
+  - [Eric Roby](https://www.linkedin.com/in/codingwithroby/)
+  - [Simon Whiteley](https://www.linkedin.com/in/simon-whiteley-uk/)
+  - [Simon Späti](https://www.linkedin.com/in/sspaeti/)
+- 5k+ Followers
+  - [Dipankar Mazumdar](https://www.linkedin.com/in/dipankar-mazumdar/)
+  - [Daniel Ciocirlan](https://www.linkedin.com/in/danielciocirlan)
+  - [Hugo Lu](https://www.linkedin.com/in/hugo-lu-confirmed/)
+  - [Tobias Macey](https://www.linkedin.com/in/tmacey)
+  - [Marcos Ortiz](https://www.linkedin.com/in/mlortiz)
+  - [Julien Hurault](https://www.linkedin.com/in/julienhuraultanalytics/)
+
+### Twitter / X voices
+- *you have to have >5k followers to be added*
+
+- 100k+ followers
+  - [Alex Xu](https://twitter.com/alexxubyte/)
+- 10k+ followers
+  - [Zach Wilson](https://www.twitter.com/EcZachly)
+  - [Seattle Data Guy](https://www.twitter.com/SeattleDataGuy)
+  - [Marco Russo](https://x.com/marcorus)
+  - [Daniel Blanco](https://www.twitter.com/DanielBlancoSWE)
+- 5k+ followers
+  - [Sumit Mittal](https://www.twitter.com/bigdatasumit)
+  - [Joseph Machado](https://twitter.com/startdataeng)
+
+### Instagram creators
+- *you have to have >5k followers to be added*
+
+- 100k+ followers
+  - [Zach Wilson](https://www.instagram.com/eczachly)
+- 5k+ followers
+  - [Andreas Kretz](https://www.instagram.com/learndataengineering)
+
+TikTok
+- *you have to have >10k followers to be added*
+
+- 50k+ followers
+  - [Zach Wilson](https://www.tiktok.com/@eczachly)
+- 10k+ followers
+  - [Alex The Analyst](https://www.tiktok.com/@alex_the_analyst)
+
+### Great Podcasts
 
 - [The Data Engineering Show](https://www.dataengineeringshow.com/)
 - [Data Engineering Podcast](https://www.dataengineeringpodcast.com/)
 - [DataTopics](https://www.datatopics.io/)
 - [The Data Engineering Side Of Data](https://podcasts.apple.com/us/podcast/the-engineering-side-of-data/id1566999533)
 - [DataWare](https://www.ascend.io/dataaware-podcast/)
 - [The Data Coffee Break Podcast](https://www.deezer.com/us/show/5293247)
-- [Thd datastack show](https://datastackshow.com/)
+- [The Datastack show](https://datastackshow.com/)
 - [Intricity101 Data Sharks Podcast](https://www.intricity.com/learningcenter/podcast)
 - [Drill to Detail with Mark Rittman](https://www.rittmananalytics.com/drilltodetail/)
 - [Analytics Power Hour](https://analyticshour.io/)
@@ -178,32 +233,15 @@ Great Podcasts
 - [Monday Morning Data Chat](https://open.spotify.com/show/3Km3lBNzJpc1nOTJUtbtMh)
 - [The Data Chief](https://www.thoughtspot.com/data-chief/podcast)
 
-Newsletters:
+### Great [list of 20+ newsletters](newsletters.md)
 
+Top must follow newsletters for data engineering:
 - [DataEngineer.io Newsletter](https://blog.dataengineer.io)
-- [Seattle Data Guy](https://seattledataguy.substack.com)
 - [Joe Reis](https://joereis.substack.com)
-- [Data Engineering Weekly](https://www.dataengineeringweekly.com)
-- [Data Engineering Central](https://dataengineeringcentral.substack.com)
-- [Dutch Engineer](https://dutchengineer.substack.com)
-- [ByteByteGo](https://blog.bytebytego.com)
 - [Start Data Engineering](https://www.startdataengineering.com)
-- [Developing Dev](https://www.developing.dev)
-- [High Growth Engineer](https://careercutler.substack.com/)
-- [Learn Analytics Engineering](https://learnanalyticsengineering.substack.com/)
-- [Marvelous MLOps](https://marvelousmlops.substack.com/)
-- [medium Data Engineering Newsletter](https://medium.com/data-engineering-weekly)
-- [Benn Stancil](https://benn.substack.com/)
-- [Metadata Weekly](https://metadataweekly.substack.com/)
-- [Technically](https://technically.substack.com/)
-- [Blef.fr Data News](https://www.blef.fr/blog/)
-- [All Hands on Data](https://allhandsondata.substack.com/)
-- [Modern Data 101](https://moderndata101.substack.com/)
-- [SELECT Insights](https://newsletter.ssp.sh/)
-- [Interesting Data Gigs](https://newsletter.interestinggigs.com)
-- [Ju Data Engineering Weekly](https://juhache.substack.com/)
-
-Glossaries:
+- [Data Engineering Weekly](https://www.dataengineeringweekly.com)
+
+### Glossaries:
 - [Data Engineering Vault](https://www.ssp.sh/brain/data-engineering/)
 - [Airbyte Data Glossary](https://glossary.airbyte.com/)
 - [Data Engineering Wiki by Reddit](https://dataengineering.wiki/Index)
@@ -212,75 +250,15 @@ Glossaries:
 - [Airtable Glossary](https://airtable.com/shrGh8BqZbkfkbrfk/tbluZ3ayLHC3CKsDb)
 - [Data Engineering Glossary by Dagster](https://dagster.io/glossary)
 
-LinkedIn
-
-- 100k+ Followers
-  - [Zach Wilson](https://www.linkedin.com/in/eczachly)
-  - [Ben Rogojan](https://www.linkedin.com/in/benjaminrogojan)
-  - [Sumit Mittal](https://www.linkedin.com/in/bigdatabysumit/)
-  - [Shashank Mishra](https://www.linkedin.com/in/shashank219/)
-  - [Chip Huyen](https://www.linkedin.com/in/chiphuyen/)
-  - [Alex Xu](https://www.linkedin.com/in/alexxubyte)
-  - [Deepak Goyal](https://www.linkedin.com/in/deepak-goyal-93805a17/)
-  - [Andreas Kretz](https://www.linkedin.com/in/andreas-kretz)
-- 50k+ Followers
-  - [Joe Reis](https://www.linkedin.com/in/josephreis)
-  - [Darshil Parmar](https://www.linkedin.com/in/darshil-parmar/)
-  - [Ankit Bansal](https://www.linkedin.com/in/ankitbansal6/)
-  - [Marc Lamberti](https://www.linkedin.com/in/marclamberti)
-- 10k+ Followers
-  - [Li Yin](https://www.linkedin.com/in/li-yin-ai/)
-  - [Joseph Machado](https://www.linkedin.com/in/josephmachado1991/)
-  - [Eric Roby](https://www.linkedin.com/in/codingwithroby/)
-  - [Simon Whiteley](https://www.linkedin.com/in/simon-whiteley-uk/)
-  - [Simon Späti](https://www.linkedin.com/in/sspaeti/)
-- 5k+ Followers
-  - [Dipankar Mazumdar](https://www.linkedin.com/in/dipankar-mazumdar/)
-  - [Daniel Ciocirlan](https://www.linkedin.com/in/danielciocirlan)
-  - [Hugo Lu](https://www.linkedin.com/in/hugo-lu-confirmed/)
-  - [Tobias Macey](https://www.linkedin.com/in/tmacey)
-  - [Marcos Ortiz](https://www.linkedin.com/in/mlortiz)
-  - [Julien Hurault](https://www.linkedin.com/in/julienhuraultanalytics/)
-- 1k+ Followers
-  - [Shruti Mantri](https://www.linkedin.com/in/shruti-mantri-88527a67/)
-  - [Volker Janz](https://www.linkedin.com/in/vjanz/)
-
-Twitter / X
-
-- [Zach Wilson](https://www.twitter.com/EcZachly)
-- [Seattle Data Guy](https://www.twitter.com/SeattleDataGuy)
-- [Sumit Mittal](https://www.twitter.com/bigdatasumit)
-- [Joseph Machado](https://twitter.com/startdataeng)
-- [Alex Xu](https://twitter.com/alexxubyte/)
-- [Eric Roby](https://twitter.com/codingwithroby)
-- [Andreas Kretz](https://twitter.com/andreaskayy)
-- [Marc Lamberti](https://twitter.com/marclambertiml)
-- [Dipankar Mazumdar](https://twitter.com/Dipankartnt)
-- [Start Data Engineering](https://twitter.com/startdataeng)
-- [Data Cyborg](https://twitter.com/data_cyborg)
-- [Simon Späti](https://twitter.com/sspaeti)
-- [Marcos Ortiz](https://twitter.com/marcosluis2186)
-
-Instagram
-
-- [Zach Wilson](https://www.instagram.com/eczachly)
-- [Andreas Kretz](https://www.instagram.com/learndataengineering)
-- [Seattle Data Guy](https://www.instagram.com/seattledataguy)
-
-TikTok
-
-- [Zach Wilson](https://www.tiktok.com/@eczachly)
-- [Alex The Analyst](https://www.tiktok.com/@alex_the_analyst)
-- [Marcos Ortiz](https://www.tiktok.com/@marcosluis2186)
 
-Design Patterns
+### Design Patterns
 
-- [Cumulative Table Design](https://www.github.com/EcZachly/cumulative-table-design)
+- [Cumulative Table Design](https://www.github.com/DataExpert-io/cumulative-table-design)
 - [Microbatch Deduplication](https://www.github.com/EcZachly/microbatch-hourly-deduped-tutorial)
 - [The Little Book of Pipelines](https://www.github.com/EcZachly/little-book-of-pipelines)
 - [Data Developer Platform](https://datadeveloperplatform.org/architecture/)
 
-Courses / Academies
+### Courses / Academies
 
 - [DataExpert.io course](https://www.dataexpert.io) use code **HANDBOOK10** for a discount!
 - [LearnDataEngineering.com](https://www.learndataengineering.com)
@@ -293,19 +271,14 @@ Courses / Academies
 - [Data Engineering Zoomcamp by DataTalksClub](https://datatalks.club/)
 - [Efficient Data Processing in Spark](https://josephmachado.podia.com/efficient-data-processing-in-spark)
 - [Scaler](https://www.scaler.com/)
+- [DataTeams - Data Engingeer hiring platform](https://www.datateams.ai/)
+- [Udemy Courses from Daniel Blanco](https://danielblanco.dev/links)
 
-Certifications Courses
+### Certifications Courses
 
 - [Google Cloud Certified - Professional Data Engineer](https://cloud.google.com/certification/data-engineer)
 - [Databricks - Data Engineer Professional](https://www.databricks.com/learn/certification/data-engineer-professional)
 - [Azure Data Engineer Associate](https://learn.microsoft.com/credentials/certifications/azure-data-engineer/)
 - [Microsoft Fabric Analytics Engineer Associate](https://learn.microsoft.com/credentials/certifications/fabric-analytics-engineer-associate/)
 - [Exam DP-203: Data Engineering on Microsoft Azure](https://learn.microsoft.com/en-us/credentials/certifications/exams/dp-203/?tab=tab-learning-paths)
 - [AWS Certified Data Engineer - Associate](https://aws.amazon.com/certification/certified-data-engineer-associate/)
-
-Conferences
-
-- [Trino Summit - December 13-14, 2023 - Virtual](https://www.starburst.io/info/trinosummit2023/)
-- [Data Universe - April 10-11, 2024 - New York City](https://www.datauniverseevent.com/)
-- [Data Nova @ Data Universe - April 10-11, 2024 - New York City](https://www.starburst.io/datanova/)
-- [DataTune Conference - March 8-9, 2024 - Nashville, TN](https://www.datatuneconf.com/)