Cricket Analytics

End-to-end data engineering on Databricks — Cricket analytics powered by Cricsheet ball-by-ball data.

---

Overview

This project ingests raw Cricsheet JSON match files and transforms them through a medallion architecture into governed gold tables — ready for analytics on player performance, venues, and team results.

Insight area	Examples
Player stats	Career batting & bowling aggregates
Team performance	Home vs away vs neutral win rates
Venues	Canonical venue mapping & match context
Player profiles	Roles, batting/bowling styles, SCD history

---

Table of Contents

• Architecture
• Tech Stack
• Project Structure
• Pipelines
• Key Concepts
• Getting Started
• Data Source

---

Architecture

Medallion layers flow from raw JSON landing in Unity Catalog Volumes to curated gold analytics tables.

flowchart TB
    subgraph Source
        CS[Cricsheet JSON files]
    end

    subgraph Bronze
        BR[matches_raw]
    end

    subgraph Silver
        S1[matches]
        S2[players · SCD Type 2]
        S3[match_players]
        S4[deliveries]
        S5[player_roles]
        S6[dim_venues]
    end

    subgraph Gold
        G1[player_batting_stats]
        G2[player_bowling_stats]
        G3[match_summary]
        G4[match_venues]
        G5[players]
        G6[home_away_performance]
    end

    CS --> BR
    BR --> S1 & S2 & S3 & S4
    S5 --> G5
    S6 --> S1
    S1 & S2 & S3 & S4 --> G1 & G2 & G3 & G4 & G5 & G6

Loading

Text diagram

Cricsheet JSON files
        ↓
    BRONZE
  matches_raw (raw JSON)
        ↓
    SILVER
  matches         — match metadata
  players         — SCD Type 2 player registry
  match_players   — squad per match
  deliveries      — ball-by-ball data
  player_roles    — player role/style (seed data)
  dim_venues      — venue canonical mapping (seed data)
        ↓
    GOLD
  player_batting_stats      — career batting statistics
  player_bowling_stats      — career bowling statistics
  match_summary             — innings-level match summary
  match_venues              — match venue enriched data
  players                   — enriched player profiles
  home_away_performance     — win % by home/away/neutral

---

Tech Stack

Layer	Technology
Platform	Databricks (Free Edition)
Storage	Delta Lake
Governance	Unity Catalog
Language	PySpark + SQL
Orchestration	Lakeflow Jobs
IaC / CI/CD	Databricks Asset Bundles (DABs)
Source control	GitHub
Data source	Cricsheet JSON

---

Project Structure

databricks-cricket-analytics/
├── databricks.yml                         # DAB bundle config
├── resources/
│   ├── main_pipeline.yml                  # Main medallion pipeline
│   ├── players_role_pipeline.yml          # Player roles seed job
│   └── venues_pipeline.yml                # Venues seed job
└── notebooks/
    ├── 00_catalog_setup.ipynb             # Catalog & schema bootstrap
    ├── 01_catalog_permissions.ipynb       # Unity Catalog permissions
    ├── 10_bronze_ingestion.ipynb
    ├── 20_silver_matches.ipynb
    ├── 21_silver_players.ipynb
    ├── 22_silver_deliveries.ipynb
    ├── 23_silver_match_players.ipynb
    ├── 24_silver_player_roles.ipynb
    ├── 25_silver_venues.ipynb
    ├── 30_gold_player_batting_stats.ipynb
    ├── 31_gold_player_bowling_stats.ipynb
    ├── 32_gold_match_summary.ipynb
    ├── 33_gold_match_venues.ipynb
    ├── 34_gold_players.ipynb
    └── 35_gold_home_away_performance.ipynb

---

Pipelines

Three Lakeflow Jobs orchestrate ingestion and transformation. All are defined as code in resources/ and deployed via DABs.

Main Pipeline

Triggered when new Cricsheet JSON files arrive in the landing Volume.

bronze_ingestion
      ↓
silver_matches — silver_players — silver_deliveries — silver_match_players
      ↓
gold_batting_stats — gold_bowling_stats — gold_match_summary — gold_players
      ↓
gold_match_venues — gold_home_away_performance

Player Roles Pipeline

Triggered when new CSV seed files land in the players Volume.

silver_player_roles → gold_players

Venues Pipeline

Triggered when new CSV seed files land in the venues Volume.

silver_venues → silver_matches

---

Key Concepts

Concept	Where it's used
Medallion architecture	Bronze → Silver → Gold layering
SCD Type 2	Historical player name changes
SCD Type 1	Venue and player role corrections
Delta Lake	ACID transactions, time travel, merge
Unity Catalog	Governed tables with access controls
Auto Loader	Incremental JSON ingestion
Lakeflow Jobs	DAG-based pipeline orchestration
File arrival trigger	Event-driven pipeline execution
DABs	Infrastructure as code for dev/prod

---

Getting Started

Prerequisites

• Databricks CLI installed
• Access to a Databricks workspace
• Git

1. Install the Databricks CLI

# macOS
brew tap databricks/tap
brew install databricks

# Windows
winget install Databricks.DatabricksCLI

2. Configure authentication

databricks configure
# Enter your workspace host and personal access token

Or set credentials via environment variables — copy .env.example to .env (never commit .env):

cp .env.example .env

DATABRICKS_HOST=https://your-workspace.azuredatabricks.net
DATABRICKS_TOKEN=your_personal_access_token

# macOS / Linux
export DATABRICKS_HOST=https://your-workspace.azuredatabricks.net
export DATABRICKS_TOKEN=your_personal_access_token

# Windows (PowerShell)
$env:DATABRICKS_HOST="https://your-workspace.azuredatabricks.net"
$env:DATABRICKS_TOKEN="your_personal_access_token"

3. Clone and deploy

git clone https://github.com/piestack-labs/databricks-cricket-analytics.git
cd databricks-cricket-analytics

# Validate the bundle
databricks bundle validate

# Deploy to dev (default target)
databricks bundle deploy --target dev

# Deploy to prod
databricks bundle deploy --target prod

4. Run a pipeline

databricks bundle run main_pipeline --target dev

Tip: Run 00_catalog_setup.ipynb and 01_catalog_permissions.ipynb in your workspace before the first pipeline run if the catalog does not yet exist.

---

Data Source

Match data is sourced from Cricsheet — free, open cricket data in JSON format.

Attribute	Detail
Format	JSON (one file per match)
Coverage	Test, ODI, and T20 matches
Contents	Ball-by-ball deliveries, player registry, match metadata

---

Built by PieStack