Skip to content

Conversation

evertlammerts
Copy link
Collaborator

See #98

We have our own Polars IO plugin to create lazy polars dataframes. We try to push as many predicates down into DuckDB as we can, for which we try to map Polars expressions (including datatypes) to SQL expressions. To do this we depend on Polar's Expr.meta.serialize.

None of this is very stable. Polars IO source plugins are marked @unstable and Expr.meta.serialize says "Serialization is not stable across Polars versions".

In this case the problems seems to come from Polars requiring an explicit scale to be set for decimals (this pr). The serialized format seems to have changed into:

{
  "expr": {
    "Literal": {
      "Scalar": {
        "Decimal": [
          1,
          38, // This now includes the scale
          0
        ]
      }
    }
  },
  "dtype": {
    "Literal": {
      "Decimal": [
        38, // this was already there
        0
      ]
    }
  },
  "options": "Strict"
}

Interestingly, even if we explicitly set precision to e.g. 20, the relevant part of the serialized expression looks as follows:

{
  "expr": {
    "Literal": {
      "Scalar": {
        "Decimal": [
          1,
          38, // Still 38?
          0
        ]
      }
    }
  },
  "dtype": {
    "Literal": {
      "Decimal": [
        20, // This is correct
        0
      ]
    }
  },
  "options": "Strict"
}

This PR allows for both a 2 and 3 item list for decimals.

@evertlammerts evertlammerts changed the title V1.4 andium Fix Polars expr pushdown Oct 4, 2025
@evertlammerts evertlammerts marked this pull request as ready for review October 4, 2025 19:26
@evertlammerts evertlammerts merged commit c1ed4fa into duckdb:v1.4-andium Oct 4, 2025
32 of 33 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants