Skip to content

ducklake -> my_ducklake #170

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
szarnyasg merged 1 commit into from
Sep 22, 2025
Merged

ducklake -> my_ducklake #170

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 9 additions & 9 deletions _posts/2025-09-17-ducklake-03.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ ATTACH '' AS iceberg_datalake (
CLIENT_SECRET 'password',
ENDPOINT 'http://127.0.0.1:8181'
);
ATTACH 'ducklake:my_ducklake.ducklake' AS ducklake (DATA_PATH 'data/');
ATTACH 'ducklake:my_ducklake.ducklake' AS my_ducklake (DATA_PATH 'data/');
```
</details>

Expand All @@ -58,7 +58,7 @@ Now you can **copy from Iceberg to DuckLake.**
CREATE SCHEMA iceberg_datalake.default;
CREATE TABLE iceberg_datalake.default.iceberg_table AS
SELECT a FROM range(4) t(a);
COPY FROM DATABASE iceberg_datalake TO ducklake;
COPY FROM DATABASE iceberg_datalake TO my_ducklake;
```

**Copying from DuckLake to Iceberg** also works, given that the schemas are already created in Iceberg.
Expand All @@ -67,9 +67,9 @@ COPY FROM DATABASE iceberg_datalake TO ducklake;
-- Assuming Iceberg catalog is empty since the COPY command does
-- not replace tables
CREATE SCHEMA iceberg_datalake.main;
CREATE TABLE ducklake.default.ducklake_table AS
CREATE TABLE my_ducklake.default.ducklake_table AS
SELECT a FROM range(4) t(a);
COPY FROM DATABASE ducklake TO iceberg_datalake;
COPY FROM DATABASE my_ducklake TO iceberg_datalake;
```

These examples are data copies (i.e., deep copies) of the latest snapshot, which means that only data is ported from Iceberg to DuckLake and vice versa. Metadata-only copies are also supported from Iceberg to DuckLake. The main difference is that metadata only copies do not copy over the underlying data, only the metadata, including all the snapshot history. This means that you can query previous snapshots of an Iceberg table as if it was a DuckLake table.
Expand All @@ -86,7 +86,7 @@ CALL iceberg_to_ducklake('iceberg_datalake', 'ducklake');
Now you can query any version of the iceberg table as if it was a DuckLake table.

```sql
FROM ducklake.default.iceberg_table AT (VERSION => 0);
FROM my_ducklake.default.iceberg_table AT (VERSION => 0);
```

```text
Expand Down Expand Up @@ -148,7 +148,7 @@ FROM Stock;

### CHECKPOINT Statement

DuckLake now also supports the [`CHECKPOINT` statement]({% link docs/stable/duckdb/maintenance/checkpoint.md %}). In DuckLake, this statement runs a series of maintenance functions in a sequential order. This includes flushing inlined data, expiring snapshots, compacting files, and rewriting heavily deleted files as well as cleaning up old or orphaned files. The `CHECKPOINT` statement can be configured via some global options that can be set via the `ducklake.set_option` function.
DuckLake now also supports the [`CHECKPOINT` statement]({% link docs/stable/duckdb/maintenance/checkpoint.md %}). In DuckLake, this statement runs a series of maintenance functions in a sequential order. This includes flushing inlined data, expiring snapshots, compacting files, and rewriting heavily deleted files as well as cleaning up old or orphaned files. The `CHECKPOINT` statement can be configured via some global options that can be set via the `my_ducklake.set_option` function.

```sql
ATTACH 'ducklake:my_ducklake.ducklake' AS my_ducklake;
Expand All @@ -164,13 +164,13 @@ In the following benchmark we did find ~25% improvement when enabling `per_threa

```sql
.timer on
ATTACH 'ducklake:my_ducklake.ducklake' AS ducklake;
ATTACH 'ducklake:my_ducklake.ducklake' AS my_ducklake;
CREATE TABLE sample_table AS SELECT * FROM range(1_000_000_000);
-- 4.5 seconds
CREATE TABLE slow_copy AS SELECT * FROM sample_table;

-- enable the option
CALL ducklake.set_option('per_thread_output', true);
CALL my_ducklake.set_option('per_thread_output', true);

-- 3.4 seconds
CREATE TABLE fast_copy AS SELECT * FROM sample_table;
Expand All @@ -188,7 +188,7 @@ Geometry types are now supported in DuckLake. This means that you can now use mo
INSTALL spatial;
LOAD spatial;

ATTACH 'ducklake:my_ducklake.ducklake' AS ducklake;
ATTACH 'ducklake:my_ducklake.ducklake' AS my_ducklake;
CREATE TABLE geometry_table (polygons GEOMETRY);
INSERT INTO geometry_table VALUES ('POLYGON((0 0, 0 1, 1 1, 1 0, 0 0))');
SELECT
Expand Down