Skip to content

SSL connection error when using S3 data_path with HivePartition-enabled source tables - URL encoding issue #5606

@KL-Iguchi-Naoto

Description

@KL-Iguchi-Naoto

Description

When using DuckLake as a catalog type with an S3 data path, SQLMesh fails to read source tables that have HivePartition enabled. The error appears to be caused by incorrect URL encoding of the = character in HivePartition paths (id=xxx becomes id%3D2012 in the S3 URL).

Configuration

gateways:
  dp_gateway:
    connection:
      type: duckdb
      catalogs:
        s3_data:
          type: ducklake
          path: ducklake:postgres:dbname=ducklake host=xxxx
          data_path: s3://xxxxxxxx/ducklake/
      extensions:
        - ducklake
        - httpfs

    secrets:
      - type: s3
        region: ap-northeast-1
        key_id: xxxx
        secret: xxxx
        url_compatibility_mode: true
        endpoint: s3.ap-northeast-1.amazonaws.com

state_connection:
  type: postgres
  xxxx: ......

Error

[2025-11-21, 00:45:57] INFO - **Failed models**:
[2025-11-21, 00:45:57] INFO - * `"s3_data"."xxxxx"."xxxxxxxxxxx"`:
[2025-11-21, 00:45:57] INFO -   ```
[2025-11-21, 00:45:57] INFO -     IOException:
[2025-11-21, 00:45:57] INFO -     IO Error: SSL connection failed error for HTTP GET to:
[2025-11-21, 00:45:57] INFO - 'https://xxxxxxx.s3.ap-northeast-1.amazonaws.com/ducklake/xxx/xxxx/id%3D22012/ducklake-019a9c27-de92-7:

Notice the id%3D22012 in the URL - the = character is being URL-encoded as %3D.

Working Cases

The following configurations do not produce errors:

  1. When HivePartition is disabled on the source table (data path remains S3)
  2. When data_path is changed to local storage (with HivePartition enabled)

Hypothesis

It appears that DuckLake or SQLMesh is not correctly handling URL encoding for HivePartition paths when using S3 as the data path. The = character in partition keys (e.g., id=2012) should not be URL-encoded when constructing S3 paths for HivePartition-style layouts.

Environment

  • SQLMesh version: 0.224.0
  • DuckDB version: 1.4.2
  • Operating System: Linux
  • Execution environment: AWS Private Cloud

Expected Behavior

SQLMesh should correctly construct S3 paths for HivePartition tables without URL-encoding the partition key separators (=).

Additional Context

This issue only occurs when all three conditions are met:

  1. Using S3 as the data_path
  2. Source table has HivePartition enabled
  3. Reading from the DuckLake catalog

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions