Skip to content

feat: Enable custom query condition in snapshot mode (#51)#80

Open
ardakyigit wants to merge 1 commit intomainfrom
feature/enable-custom-query-snapshot
Open

feat: Enable custom query condition in snapshot mode (#51)#80
ardakyigit wants to merge 1 commit intomainfrom
feature/enable-custom-query-snapshot

Conversation

@ardakyigit
Copy link

  • Add QueryCondition to SnapshotConfig and publication.Table
  • Update all snapshot query builders to inject custom conditions
  • Per-table conditions take precedence over global conditions

Fixes #51

- Add QueryCondition to SnapshotConfig and publication.Table
- Update all snapshot query builders to inject custom conditions
- Per-table conditions take precedence over global conditions

Fixes #51

Signed-off-by: Arda Akyigit <arda.akyigit@trendyol.com>
@Abdulsametileri
Copy link
Member

  • Could you fix pipeline errors?
  • Could you write integration test?
  • Could you also put a example for this new feature and add documentation about it [docs/SNAPSHOT_FEATURE.md] and README.md

@Abdulsametileri
Copy link
Member

Abdulsametileri commented Feb 5, 2026

# Global condition - applies to all tables
snapshot:
  enabled: true
  mode: initial
  queryCondition: "is_active = true"  # Only snapshot active records

# Per-table override - takes precedence
publication:
  tables:
    - name: users
      schema: public
      queryCondition: "created_at > '2024-01-01'"  # Recent users only
    - name: orders
      schema: public
      # Uses global condition (is_active = true) 
    - name: archived_data
      schema: public
      queryCondition: "1=1"  # Override: snapshot all rows

it works like that?

HeartbeatInterval time.Duration `json:"heartbeatInterval" yaml:"heartbeatInterval"`
Enabled bool `json:"enabled" yaml:"enabled"`
Resnapshot bool `json:"resnapshot" yaml:"resnapshot"`
QueryCondition string `json:"queryCondition" yaml:"queryCondition"`
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

}

for _, table := range s.tables {
tableSchema := table.Schema
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

getQueryCondition methods takes tableSchema as the first parameter, you also define it here.

We can rename it tblSchema and use it in the for.

 tblSchema := table.Schema  // Farklı isim, karışıklık yok
    if tblSchema == "" {
        tblSchema = "public"
    }

normalizedSchema = "public"
}

for _, table := range s.tables {
Copy link

@sunziping2016 sunziping2016 Feb 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I happened to see this PR and tried it. However, I found per-table QueryCondition not working.

Probably, the s.tables is synthesised from the PGSQL queries and user-provide config. And the synthesizing code is missing in the locations shown below.

go-pq-cdc/config/config.go

Lines 262 to 264 in 7a137ba

if userTable.SnapshotPartitionStrategy != "" {
result[i].SnapshotPartitionStrategy = userTable.SnapshotPartitionStrategy
}

go-pq-cdc/config/config.go

Lines 179 to 181 in 7a137ba

if st.SnapshotPartitionStrategy != "" {
mergedTable.SnapshotPartitionStrategy = st.SnapshotPartitionStrategy
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Enable custom query in snapshot mode

3 participants