-
Notifications
You must be signed in to change notification settings - Fork 1.8k
Description
Is this a new bug in dbt-core?
- I believe this is a new bug in dbt-core
- I have searched the existing issues, and I could not find an existing issue for this bug
Current Behavior
When using the --sample flag with dbt, the event time filtering doesn't respect the column-level quote settings defined in sources.yml. This causes SQL compilation errors when using reserved keywords as column names, even when they are explicitly marked as quote: true. The errors arrive when the column name is in camel case or with spaces.
Expected Behavior
The generated SQL should respect the quote: true setting and use the column name with quotes:
-- Expected SQL filter
where "createdAt" >= '2024-01-01' and "createdAt" < '2024-01-31'
Steps To Reproduce
sources.yml
version: 2
sources:
- name: source_name
tables:
- name: table_name
columns:
- name: "createdAt"
quote: true
config:
event_time: "createdAt"
dbt command:
dbt build --target sampling --sample '1 year'
Relevant log output
21:26:36 Failure in model model_name (models/model_name.sql)
21:26:36 Database Error in model model_name (models/model_name.sql)
column "createdat" does not exist
LINE 24: ... * from "testdb"."source_name"."table_name" where createdAt ...
^
HINT: Perhaps you meant to reference the column "table_name.createdAt".
compiled code at target/run/test_dbt_sampling/models/model_name.sql
Compiled code:
create table "testdb"."sampling"."table_na__dbt_tmp"
as
(
select * from (select * from "testdb"."source_name"."table_name" where createdAt >= '2024-07-24 21:26:36.233275+00:00' and createdAt < '2025-07-24 21:26:36.233275+00:00') _dbt_et_filter_subq_table_name;
Environment
- OS: MacOS
- Python: 3.11.13
I've seen the issue both on dbt-core and dbt Cloud.
dbt-core version: 1.10.4
dbt Cloud version: 2025.7.24+00c7a0e
Adapter type: PostgreSQL
Adapter Version: 1.9.1post20+f492c919d3bd415bf5065b3cd8cd1af23562feb0
Which database adapter are you using with dbt?
postgres
Additional Context
By modifying the core/dbt/context/providers.py
, specifically the resolve_event_time_filter
method from BaseResolver
class, I managed to get the appropriate compilation.
def resolve_event_time_filter(self, target: ManifestNode) -> Optional[EventTimeFilter]:
event_time_filter = None
sample_mode = getattr(self.config.args, "sample", None) is not None
# TODO The number of branches here is getting rough. We should consider ways to simplify
# what is going on to make it easier to maintain
# Only do event time filtering if the base node has the necessary event time configs
if (
isinstance(target.config, (NodeConfig, SeedConfig, SourceConfig))
and target.config.event_time
and isinstance(self.model, (ModelNode, SnapshotNode))
):
quote_config = target.config._extra.get('quote', None)
if quote_config:
field_name = f'"{target.config.event_time}"'
else:
field_name = target.config.event_time
# Handling of microbatch models
if (
isinstance(self.model, ModelNode)
and self.model.config.materialized == "incremental"
and self.model.config.incremental_strategy == "microbatch"
and self.manifest.use_microbatch_batches(project_name=self.config.project_name)
and self.model.batch is not None
):
# Sample mode microbatch models
if sample_mode:
start = (
self.config.args.sample.start
if self.config.args.sample.start > self.model.batch.event_time_start
else self.model.batch.event_time_start
)
end = (
self.config.args.sample.end
if self.config.args.sample.end < self.model.batch.event_time_end
else self.model.batch.event_time_end
)
event_time_filter = EventTimeFilter(
field_name=field_name,
start=start,
end=end,
)
# Regular microbatch models
else:
event_time_filter = EventTimeFilter(
field_name=field_name,
start=self.model.batch.event_time_start,
end=self.model.batch.event_time_end,
)
# Sample mode _non_ microbatch models
elif sample_mode:
event_time_filter = EventTimeFilter(
field_name=field_name,
start=self.config.args.sample.start,
end=self.config.args.sample.end,
)
return event_time_filter
Note the retrieval of the quote configuration and the rough double quoting at the top of the method, which provides the appropriate compilation.