Skip to content

dialect conflict issue using external models with duckdb/tsql #5611

@emragins

Description

@emragins

sqlmesh version: 0.227.1

CONFIG

gateways:
  local_db:
    connection:
      type: duckdb
      catalogs:
      # The catalogue IS the name of the database. These MUST match.
        meshy:
          type: duckdb
          path: meshy.db
        afrs_workday:
          type: sqlite
          path: '../afrs-to-workday/afrs_workday.db'

default_gateway: local_db

model_defaults:
  dialect: tsql

snippet of generated external_models.yaml

- name: '[afrs_workday].[main].[workday_transactions]'
  columns:
    id: BIGINT
    transaction_type: VARCHAR(MAX)
  gateway: local_db

models\workday_transactions.sql

MODEL (
    name main.workday_transactions,
    KIND FULL
);

SELECT * FROM afrs_workday.main.workday_transactions;

ERROR: Running sqlmesh plan

  "meshy"."main"."workday_transactions"

    ParserException:
      Parser Error: Expected an integer constant as type modifier

It seems this is due to the generated SQL code,

CREATE OR REPLACE TABLE "meshy"."sqlmesh__dbo"."dbo__workday_transactions__3081438768" AS
SELECT
    CAST("id" AS BIGINT) AS "id",
    CAST("transaction_type" AS TEXT (MAX)) AS "transaction_type"
FROM (
    SELECT
        "workday_transactions"."id" AS "id",
        "workday_transactions"."transaction_type" AS "transaction_type"
    FROM "afrs_workday"."main"."workday_transactions" AS "workday_transactions"
) AS "_subquery";

ChatGPT informs me that "DuckDB doesn’t support a length modifier on TEXT. TEXT(MAX) is invalid because it is already max length".

When I remove the (MAX) from VARCHAR(MAX) in the external models, it works.

Ideally, I would love to be able to use more precision for my types in my models since production will be MS SQL Server, and I'm just using duckdb for sketching things out and local development. Maybe that is my fault for not using the tool correctly, but it would be nice to be able to cleanly move between databases.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions