Skip to content

rohit-ashva900/Dynamic_ETL_Pipeline_Project_with_Azure

BranchesTags

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

5 Commits
1.png
1.png
 
 
2.png
2.png
 
 
README.md
README.md
 
 
nyctaxidata.py
nyctaxidata.py
 
 
roadmap.png
roadmap.png
 
 

Repository files navigation

Dynamic_ETL_Pipeline_Project_with_Azure

Developed a dynamic ETL pipeline using Azure Data Factory to extract, transform, and load over 800k rows from the New York City Taxi API into Azure Data Lake Storage Gen2.

Utilized Databricks for data transformation across Bronze, Silver, and Gold layers, optimizing storage with Parquet and Delta Lake formats.

Implemented advanced features like recursive file lookup and DDL schema management for scalable and efficient data workflows.

About

No description, website, or topics provided.

Resources

Readme
Activity

Stars

0 stars

Watchers

1 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

No packages published

Languages