Skip to content

Conversation

SkyFan2002
Copy link
Member

@SkyFan2002 SkyFan2002 commented Aug 6, 2025

I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

Summary

This PR introduces a new optimization phase CTEFilterPushdownOptimizer that improves query performance by collecting filters applied to MaterializedCTE references and pushing them down into the CTE definitions.

Example

explain with cte1 as materialized (
    select a, count(*) as cnt
    from t 
    group by a
)
select * from cte1 as c1 where c1.a = 1
union all
select * from cte1 as c2 where c2.a = 2;
----
Sequence
├── MaterializedCTE: cte1
│   └── AggregateFinal
│       ├── output columns: [COUNT(*) (#3), t.a (#0)]
│       ├── group by: [a]
│       ├── aggregate functions: [count()]
│       ├── estimated rows: 2.00
│       └── AggregatePartial
│           ├── group by: [a]
│           ├── aggregate functions: [count()]
│           ├── estimated rows: 2.00
│           └── Filter
│               ├── output columns: [t.a (#0)]
│               ├── filters: [is_true((t.a (#0) = 1 OR t.a (#0) = 2))]
│               ├── estimated rows: 2.78
│               └── TableScan
│                   ├── table: default.default.t
│                   ├── output columns: [a (#0)]
│                   ├── read rows: 5
│                   ├── read size: < 1 KiB
│                   ├── partitions total: 1
│                   ├── partitions scanned: 1
│                   ├── pruning stats: [segments: <range pruning: 1 to 1>, blocks: <range pruning: 1 to 1, bloom pruning: 1 to 1>]
│                   ├── push downs: [filters: [is_true((t.a (#0) = 1 OR t.a (#0) = 2))], limit: NONE]
│                   └── estimated rows: 5.00
└── UnionAll
    ├── output columns: [a (#12), cnt (#13)]
    ├── estimated rows: 2.00
    ├── Filter
    │   ├── output columns: [t.a (#4), COUNT(*) (#7)]
    │   ├── filters: [is_true(c1.a (#4) = 1)]
    │   ├── estimated rows: 1.00
    │   └── MaterializeCTERef
    │       ├── cte_name: cte1
    │       ├── cte_schema: [a (#4), COUNT(*) (#7)]
    │       └── estimated rows: 3.00
    └── Filter
        ├── output columns: [t.a (#8), COUNT(*) (#11)]
        ├── filters: [is_true(c2.a (#8) = 2)]
        ├── estimated rows: 1.00
        └── MaterializeCTERef
            ├── cte_name: cte1
            ├── cte_schema: [a (#8), COUNT(*) (#11)]
            └── estimated rows: 3.00

Benchmark

TPC-DS100, small warehouse

Query v12790 Time (s) pr Time (s) Change (%)
39 3.88 1.67 -56.96%
31 8.489 5.696 -32.90%
75 12.145 9.304 -23.39%

Tests

  • Unit Test
  • Logic Test
  • Benchmark Test
  • No Test - Explain why

Type of change

  • Bug Fix (non-breaking change which fixes an issue)
  • New Feature (non-breaking change which adds functionality)
  • Breaking Change (fix or feature that could cause existing functionality not to work as expected)
  • Documentation Update
  • Refactoring
  • Performance Improvement
  • Other (please describe):

This change is Reviewable

@github-actions github-actions bot added the pr-feature this PR introduces a new feature to the codebase label Aug 6, 2025
@SkyFan2002 SkyFan2002 added the ci-cloud Build docker image for cloud test label Aug 7, 2025
Copy link
Contributor

github-actions bot commented Aug 7, 2025

Docker Image for PR

  • tag: pr-18493-90b7238-1754539870

note: this image tag is only available for internal use.

@SkyFan2002 SkyFan2002 added ci-cloud Build docker image for cloud test and removed ci-cloud Build docker image for cloud test labels Aug 11, 2025
Copy link
Contributor

Docker Image for PR

  • tag: pr-18493-02b4f94-1754899205

note: this image tag is only available for internal use.

@SkyFan2002 SkyFan2002 added ci-cloud Build docker image for cloud test and removed ci-cloud Build docker image for cloud test labels Aug 11, 2025
@SkyFan2002 SkyFan2002 added ci-cloud Build docker image for cloud test and removed ci-cloud Build docker image for cloud test labels Aug 11, 2025
@SkyFan2002 SkyFan2002 added ci-cloud Build docker image for cloud test and removed ci-cloud Build docker image for cloud test labels Aug 11, 2025
Copy link
Contributor

Docker Image for PR

  • tag: pr-18493-2142ac5-1754920113

note: this image tag is only available for internal use.

Copy link
Contributor

Docker Image for PR

  • tag: pr-18493-89371f0-1754921409

note: this image tag is only available for internal use.

Copy link
Contributor

Docker Image for PR

  • tag: pr-18493-4cc886a-1754922545

note: this image tag is only available for internal use.

@SkyFan2002 SkyFan2002 marked this pull request as ready for review August 12, 2025 07:19
@BohuTANG BohuTANG added ci-cloud Build docker image for cloud test and removed ci-cloud Build docker image for cloud test labels Aug 12, 2025
Copy link
Contributor

Docker Image for PR

  • tag: pr-18493-9364201-1754997878

note: this image tag is only available for internal use.

@BohuTANG BohuTANG merged commit f092968 into databendlabs:main Aug 12, 2025
188 of 191 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ci-cloud Build docker image for cloud test pr-feature this PR introduces a new feature to the codebase
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants