[RFC] Git-style Branching for DuckLake

[nbalusu]

[RFC] Git-style Branching for DuckLake

Hi DuckLake team,

First, thank you for building DuckLake. The architecture—Parquet files + pluggable metadata backends + time travel—is exactly what data lakes need. I've been using it for financial data workflows and it's been a joy to work with.

I've implemented Git-style branching on top of DuckLake and wanted to share the design to see if it aligns with your roadmap. Happy to make changes, split into smaller PRs, or adjust the approach based on your feedback.

---

Problem Statement

Teams working on shared data need isolated workspaces to:

• Develop and test schema changes without affecting production
• Run what-if scenarios on production data
• Maintain multiple data versions for different consumers

Current workarounds require full data copies—expensive and slow.

Potential Solution

Zero-copy branching at the metadata layer:

• Branch creation is instant (metadata only, no data copied)
• Branches share underlying Parquet files
• Full read/write isolation between branches
• Cross-branch queries via AT (BRANCH => 'name') syntax

-- Create isolated workspace (instant)
SELECT * FROM ducklake_create_branch('catalog', 'dev');
-- Work on dev branch

CALL ducklake_use_branch('catalog', 'dev');

ALTER TABLE trades ADD COLUMN pnl_adjustment DECIMAL;

DELETE FROM trades WHERE region = 'TEST';
-- Production unchanged

SELECT * FROM trades AT (BRANCH => 'main');  -- original data

SELECT * FROM trades AT (BRANCH => 'dev');   -- modified data

---

Schema Changes

New Tables (10)

Core (2):

ducklake_branch (branch_id, branch_name, parent_branch_id, fork_snapshot_id, head_snapshot_id, status, created_at)

ducklake_branch_lineage (branch_id, ancestor_branch_id, max_visible_snapshot)

Deletion tracking (8): Same pattern for files, tables, schemas, views, columns, macros, partitions, delete_files:

ducklake_branch_*_deletion (branch_id, ancestor_branch_id, object_id, deleted_at_snapshot)

Modified Tables (24)

All existing metadata tables gain branch_id BIGINT NOT NULL DEFAULT 0:

Table	Change
ducklake_snapshot	PK → (branch_id, snapshot_id)
ducklake_data_file	PK → (branch_id, data_file_id)
ducklake_delete_file	Added data_file_branch_id to track target file's branch
ducklake_schema	branch_id identifies owning branch
ducklake_table	branch_id identifies owning branch
ducklake_column	branch_id identifies owning branch
...	(24 tables total, all with DEFAULT 0 for backward compat)

---

Open Questions

I'd appreciate your feedback on:

1. Does this align with your roadmap? Is branching something you've considered for DuckLake?

2. Merge support: Should we support merging branches?

   • Fast-forward only
   • Three-way merge with conflict detection
   • Leave merging out for now

3. Garbage collection: When can shared files be deleted?

   • Only when no branch references them?
   • Need a GC protocol?

4. Migration path: For existing catalogs, should we:

   • Auto-migrate on attach?
   • Require explicit upgrade command?

5. Naming conventions: Are function names clear, or should they match existing DuckLake patterns more closely?

---

Summary

This implementation adds branching with:

• 10 new tables, 24 modified tables, 20 new indexes
• Pre-computed lineage for O(1) visibility checks
• Deletion records for branch-local object hiding
• Full isolation for DML, DDL, and compaction
• 13 test files covering edge cases

I'm grateful for DuckLake and excited about the possibility of contributing. Happy to:

• Split this into smaller, incremental PRs
• Adjust the design based on your feedback
• Add more tests or documentation

Looking forward to your thoughts!

[YuweiXiao]

Interesting idea! so branching (in the RFC) is essentially a deep copy of the metadata?

I am thinking about an alternative way to branching. Instead of implementing the idea in DuckLake, could we branch the PostgreSQL directly, based on Neon branch, which enables a full catalog fork (instant and copy-on-write). This way, no modifications (other than GC I guess) would be required in DuckLake side.

[nbalusu]

Thanks! that is very nice approach! I actually did a POC with Neon first.

The Postgres-branching options right now:

• Neon — acquired by Databricks, open-source last updated Sept 2025. Enterprise pricing is per-branch which gets murky for our use case
• DoltgreSQL — love the concept, but ~5x slower than Postgres in my testing. Plain Dolt is MySQL-only and DuckLake docs recommend against MySQL for metadata
• Database Lab Engine — too much infra overhead unless you're already in that ecosystem. we need to implement our own scale to zero.

So yeah, ended up here instead.

Re: deep copy — actually no, that's what the lineage table solves. Branching B from A just writes ~3 rows:

(B, B, MAX_INT)        -- own snapshots
(B, A, fork_snap)      -- see A up to fork point
(B, grandparent, cap)  -- inherited visibility

All the actual metadata (schemas, tables, files) stays in place — branches just see it through the lineage join. Only things copied are lineage rows, deletion records, and stats. Everything else is shared.

this is also the bigger reason I moved away from Postgres-level branching. Neon's merge wouldn't understand DuckLake semantics.

Consider: branch A creates a table → gets table_id=4. Meanwhile on a Neon fork, a different table also auto-increments to id=4. Postgres-level merge sees no conflict — same column, same value — but these are completely different DuckLake objects. And it gets worse with schema evolution: both branches ALTER the same column differently, Neon has no idea that's a conflict.

With the lineage approach, merge logic lives in DuckLake where it belongs. We can do proper conflicts resolution, and handle ID allocation correctly. No new infra, no vendor lock-in, just tables and refs.

The tradeoff is more code to maintain in DuckLake itself — but that feels right for something this core to catalog semantics.

Appreciate your feedback! Thanks!!

[YuweiXiao]

Very sound reasoning. A bottom-up design is indeed more appropriate here.