ETL Pipeline using ADF, Delta Lake, Databricks, Spark, Data Governance (UnityCatalog).
A modern Databricks DLT-based Medallion architecture ETL pipeline implementing Bronze, Silver, and Gold layers for scalable and reliable data processing using Delta Live Tables. Designed for end-to-end data ingestion, transformation, and analytics in a Lakehouse environment.
Development and implementation of an ETL pipeline for processing oil well data using Databricks, Delta Live Tables, Spark Structured Streaming and PySpark. The project focused on automating the ingestion, transformation, and validation of large volumes of well log data.
An end-to-end Netflix data engineering pipeline built on Microsoft Azure. This project ingests raw Netflix data, applies PySpark transformations , enforces data quality with Delta Live Tables, and orchestrates workflows via Azure Data Factory and Databricks.
Built an end-to-end data platform using Azure Data Factory, Databricks, Delta Live Tables, Unity Catalog, and CI/CD.
A metadata-driven, CDC-based Medallion architecture pipeline that ingests a Spotify-style streaming warehouse from Azure SQL, processes it through Bronze → Silver → Gold using Databricks Autoloader and Delta Live Tables, and models it as a Star Schema in Unity Catalog — with SCD Type 2 history, Git-based CI/CD, and automated failure alerting.
End-to-end Azure data engineering pipeline using Spotify dummy data with Medallion Architecture, Databricks, Delta Live Tables, and incremental loading.
End-to-End Azure Data Engineering Project using Azure Data Factory, Azure Databricks, Delta Lake, and Azure Storage.
Add a description, image, and links to the deltalivetables topic page so that developers can more easily learn about it.
To associate your repository with the deltalivetables topic, visit your repo's landing page and select "manage topics."