-
Notifications
You must be signed in to change notification settings - Fork 313
Description
Description
When using DuckLake as a catalog type with an S3 data path, SQLMesh fails to read source tables that have HivePartition enabled. The error appears to be caused by incorrect URL encoding of the = character in HivePartition paths (id=xxx becomes id%3D2012 in the S3 URL).
Configuration
gateways:
dp_gateway:
connection:
type: duckdb
catalogs:
s3_data:
type: ducklake
path: ducklake:postgres:dbname=ducklake host=xxxx
data_path: s3://xxxxxxxx/ducklake/
extensions:
- ducklake
- httpfs
secrets:
- type: s3
region: ap-northeast-1
key_id: xxxx
secret: xxxx
url_compatibility_mode: true
endpoint: s3.ap-northeast-1.amazonaws.com
state_connection:
type: postgres
xxxx: ......Error
[2025-11-21, 00:45:57] INFO - **Failed models**:
[2025-11-21, 00:45:57] INFO - * `"s3_data"."xxxxx"."xxxxxxxxxxx"`:
[2025-11-21, 00:45:57] INFO - ```
[2025-11-21, 00:45:57] INFO - IOException:
[2025-11-21, 00:45:57] INFO - IO Error: SSL connection failed error for HTTP GET to:
[2025-11-21, 00:45:57] INFO - 'https://xxxxxxx.s3.ap-northeast-1.amazonaws.com/ducklake/xxx/xxxx/id%3D22012/ducklake-019a9c27-de92-7:
Notice the id%3D22012 in the URL - the = character is being URL-encoded as %3D.
Working Cases
The following configurations do not produce errors:
- When HivePartition is disabled on the source table (data path remains S3)
- When data_path is changed to local storage (with HivePartition enabled)
Hypothesis
It appears that DuckLake or SQLMesh is not correctly handling URL encoding for HivePartition paths when using S3 as the data path. The = character in partition keys (e.g., id=2012) should not be URL-encoded when constructing S3 paths for HivePartition-style layouts.
Environment
- SQLMesh version: 0.224.0
- DuckDB version: 1.4.2
- Operating System: Linux
- Execution environment: AWS Private Cloud
Expected Behavior
SQLMesh should correctly construct S3 paths for HivePartition tables without URL-encoding the partition key separators (=).
Additional Context
This issue only occurs when all three conditions are met:
- Using S3 as the data_path
- Source table has HivePartition enabled
- Reading from the DuckLake catalog