Lakehouse Comparison: Early Exploration

This repository contains an early exploration comparing three table formats on the same small analytics workload:

• DuckLake
• Apache Iceberg
• Delta Lake

The work focuses on:

• Loading the same Gapminder dataset into each format
• Running the same query patterns for fairness
• Building equivalent Dash apps for interactive exploration
• Capturing metadata and file layout artifacts for inspection

What Is Implemented

All implementation is in early-exploration/.

• app.py: Dash app backed by DuckLake
• app_iceberg.py: Dash app backed by Apache Iceberg
• app_delta.py: Dash app backed by Delta Lake
• benchmark.py: End-to-end benchmark script across all 3 formats
• benchmark_results.json: Saved benchmark output
• ducklake_catalog_dump/, iceberg_catalog_dump/, delta_catalog_dump/: Catalog and metadata inspection dumps
• data/, iceberg_data/, delta_data/: Data and metadata files generated by the runs

Current Benchmark Snapshot

Source: early-exploration/benchmark_results.json

Using the committed run, each format started with 1704 rows, appended 100 rows, and ended with 1804 rows.

Metric	DuckLake	Iceberg	Delta
Setup (ms)	2443.17	951.22	4538.69
Cold filter (ms)	53.26	7.14	4.53
Warm filter median (ms)	30.48	3.8	3.17
Cold aggregation (ms)	10.99	7.32	4.68
Warm agg median (ms)	6.73	3.65	2.76
Append 100 rows (ms)	12.83	17.23	21.9
Data files before/after	1 -> 2	1 -> 2	1 -> 2
Metadata size after (B)	0	17952	2736
Total size after (B)	38841	57926	49911

Notes:

• Delta queries used DuckDB delta_scan in this run (query_engine: duckdb_delta_scan).
• Results are exploratory and environment-dependent, not a final performance claim.

Benchmark Method

early-exploration/benchmark.py measures:

• T1: cold filter query latency
• T2: warm filter query latency (median of 10 runs)
• T3: cold aggregation query latency
• T4: warm aggregation query latency (median of 10 runs)
• T5: append 100 rows latency
• M1: data file count before and after append
• M2: metadata and total on-disk size

All three formats use the same Gapminder CSV source and equivalent query shapes.

Quick Start

1) Create environment and install dependencies

Example with Python venv:

cd early-exploration
python3 -m venv .venv
source .venv/bin/activate
pip install --upgrade pip
pip install dash plotly duckdb pandas pyarrow deltalake pyiceberg

DuckDB extensions are installed dynamically by scripts:

• ducklake extension for DuckLake app and benchmark section
• delta extension for Delta delta_scan path when available

2) Run one of the Dash apps

From early-exploration/:

python app.py

DuckLake app runs on port 8050.

python app_iceberg.py

Iceberg app runs on port 8051.

python app_delta.py

Delta app runs on port 8052.

3) Run benchmark

From early-exploration/:

python benchmark.py

This prints a comparison table and writes fresh JSON results to benchmark_results.json.

Repository Context

This is an early exploration checkpoint intended to support iterative analysis and thesis comparison work. The committed catalog dumps and table artifacts are included so metadata evolution and layout can be inspected directly.