Skip to content
View dumisanimagagula's full-sized avatar
🎯
Focusing
🎯
Focusing
5 followers · 2 following

Achievements

Achievement: Pair ExtraordinaireAchievement: YOLOAchievement: Pull SharkAchievement: Quickdraw

Achievements

Achievement: Pair ExtraordinaireAchievement: YOLOAchievement: Pull SharkAchievement: Quickdraw

Block or report dumisanimagagula

Block user

Prevent this user from interacting with your repositories and sending you notifications. Learn more about blocking users.

You must be logged in to block users.

Maximum 250 characters. Please don’t include any personal information such as legal names or email addresses. Markdown is supported. This note will only be visible to you.
Report abuse

Contact GitHub support about this user’s behavior. Learn more about reporting abuse.

Report abuse
dumisanimagagula/README.md

Dumisani Magagula — Data Engineer

I build data platforms that prioritise reliability, observability, and long-term ownership.
My work spans cloud-native lakehouse architecture on GCP and Azure, config-driven pipeline frameworks,
and analytics engineering with dbt and BigQuery.

What I Build

Systems with real architectural decisions — not just working code.

  • Lakehouse Platform: Config-driven medallion pipeline (Bronze→Silver→Gold) on Apache Iceberg + Spark + Trino — engineers only touch YAML, zero code changes needed to add new sources or transformations
  • Infrastructure as Code: Terraform modules across GCP (BigQuery, GCS, Dataproc, Composer) and Azure (ADF, ADLS Gen2, Synapse) — deployed across dev ($0), staging ($15/mo), and production tiers
  • Orchestration: Airflow DAGs with SparkSubmitOperator, dynamic task generation, health checks, and environment parameterisation across dev/staging/prod
  • dbt Analytics Layer: Medallion-aligned dbt models with full test coverage (not_null, unique, accepted_values), CI/CD via GitHub Actions, and automated dbt docs generation
  • Data Quality Framework: Three-layer quality checks — schema validation at Bronze, range/referential checks at Silver, business logic tests at Gold

Projects

nyc-taxi-data-ingestion — End-to-End Lakehouse Platform

A complete, locally runnable and cloud-deployable data platform built on open standards.

The engineering decisions that matter:

Decision Choice Why
Table format Apache Iceberg Time-travel, schema evolution, partition pruning without rewriting data
Query layer Trino Federated queries across Bronze/Silver/Gold without data movement
Orchestration Airflow SparkSubmitOperator Native Spark job lifecycle management vs Python operators
Config strategy YAML-driven runtime config Decouple pipeline logic from deployment — same code, different environments
Infrastructure Terraform (GCP + Azure) Reproducible, version-controlled cloud provisioning across both clouds

Stack: Python · Apache Spark · Apache Iceberg · Apache Airflow · dbt · Trino · Apache Superset · Docker · Terraform (GCP + Azure) · MinIO · PostgreSQL

dbt_fundamentals_medallion_architecture — Analytics Engineering with dbt

dbt project implementing the full medallion transformation layer with CI/CD and test coverage.

What it demonstrates:

  • Layered dbt models: staging → intermediate → mart pattern
  • sources.yml with freshness assertions
  • schema.yml with not_null, unique, accepted_values, and custom generic tests
  • GitHub Actions CI running dbt test on every push to main
  • dbt docs generate with full DAG lineage

Stack: dbt Core · SQL · GitHub Actions · YAML

Stack

Languages & Processing Python SQL Apache Spark Apache Kafka PowerShell

Pipelines & Orchestration Apache Airflow SSIS Azure Data Factory dbt

Storage & Warehousing SQL Server BigQuery PostgreSQL Apache Iceberg ADLS Gen2

Analytics & Visualisation Power BI QlikView Trino Apache Superset

Cloud & Infrastructure GCP Azure Docker Terraform GitHub Actions Git

Trajectory

Data Engineer (SQL Server · SSIS · Power BI · GCP · Azure)
         ↓
Analytics Engineer (dbt · BigQuery · Synapse · Dataform)
         ↓
Senior Data Architect / Data Platform Owner

Find My Work

Portfolio LinkedIn Twitter/X

Pinned Loading

  1. nyc-taxi-data-ingestion nyc-taxi-data-ingestion Public

    A production-style data ingestion pipeline for NYC Taxi datasets, covering raw data ingestion, schema validation, deduplication, and optimized loading for downstream analytics. Designed with scalab…

    Python

  2. mercedes-benz-dealership-scraper mercedes-benz-dealership-scraper Public

    An end-to-end Python project that scrapes car dealership data from Cars.com, conducts data analysis and visualization, and provides insights into Mercedes-Benz vehicles in the market.

    Jupyter Notebook 4

  3. dbt_fundamentals_medallion_architecture dbt_fundamentals_medallion_architecture Public

    A dbt (data build tool) project built on the TheLook Ecommerce public BigQuery dataset (bigquery-public-data.thelook_ecommerce), originally from a Udemy dbt bootcamp course. The project transforms …

  4. dumisanimagagula dumisanimagagula Public

    Passionate Data Enthusiast | Python, SQL | Building innovative solutions and exploring data science | Let's connect and collaborate! 🚀

  5. elevating-supply-chain-excellence elevating-supply-chain-excellence Public

    ProVisionary Insights harnesses Power BI for comprehensive supply chain analytics, delivering dynamic visualizations and strategic insights into Just In Time's operations.

    Jupyter Notebook 1

  6. netflix-recommendation-system netflix-recommendation-system Public

    The Netflix Recommendation System is a Python project that uses machine learning to provide personalized movie and TV show recommendations based on user input. It analyzes a dataset of Netflix con…

    Python