v0.37.7

What’s Changed

• bq: rename replication mode to method (#4785) @dtunikov

v0.37.6

What’s Changed

• Reduce catalog contention for RecordMetricsCritical (#4776) @ilidemi
• feat(bigquery): implement watermark column query based replication for bigquery sources (#4739) @dtunikov, @claude
• classify mongo connector KeyNotFound error (#4777) @jgao54
• Update github-actions dependencies (#4779) @renovate[bot]
• Update Node.js to e67514e (#4780) @renovate[bot]
• Exclude unlogged tables from CDC mirror selection (#4772) @heavycrystal, @claude
• feat(bigquery): implement a separate SyncFlow branch for query-based CDC (#4727) @dtunikov, @Amogh-Bharadwaj, @claude, @shockdav
• remove maintenance arg and use cli command intead (#4770) @jgao54
• migrate migration to goose (#4742) @jgao54
• fix(pg-pg): classify pgdump extension missing (#4716) @Amogh-Bharadwaj
• MongoDB CDC: User facing log upon drop and rename collection events (#4771) @pfcoperez
• perf(ssh): 8 MiB channel receive window via x/crypto fork (#4764) @ilidemi
• feat(bigquery): fetched bytes via custom http client (#4738) @dtunikov, @Amogh-Bharadwaj, @claude, @shockdav
• Fix classification for missing tables (#4761) @ilidemi
• Validate ClickHouse table capacity before mirror creation (#4762) @ilidemi
• Symlink skills for codex and other agents (#4752) @ilidemi
• fix correctness and consistency in date and datetime inference (#4765) @jgao54
• Automatic reviews: Use Fable model for PR reviews by default (#4763) @pfcoperez
• fix: bound table validation error details (#4760) @ilidemi
• Update lukemathwalker/cargo-chef Docker tag to latest-rust-1.98.0-alpine (#4756) @renovate[bot]
• Update dependency material-symbols to ^0.47.0 (#4755) @renovate[bot]
• Update golang.org/x/exp digest to e88cd73 (#4758) @renovate[bot]
• Update depot/setup-action digest to 91bc849 (#4753) @renovate[bot]
• Report processing vs receive vs add record time in CDC pull (#4749) @ilidemi
• fix(flow): upgrade to go 1.27 (#4757) @itsbilal
• Don't propagate schema delta for excluded columns in MySQL (#4751) @ilidemi
• fix(alerting): notify user on ClickHouse fatal-shutdown aborts (#4745) @ilidemi
• Update gomod dependencies (#4754) @renovate[bot]
• Lock file maintenance (#4748) @renovate[bot]
• Track CDC event size distribution as a histogram (#4744) @ilidemi
• Preserve large CDC values for non-Snowflake destinations (#4743) @ilidemi
• feat(bigquery): implement appends/changes event based cdc modes (#4707) @dtunikov, @Amogh-Bharadwaj, @claude, @shockdav
• SyncFlow graceful termination handling (#4708) @jgao54
• fix(flow): classify ProxySQL connection timeout as connectivity err (#4735) @itsbilal
• feat: upgrade docker-compose.yml stable image tags (#4736) @github-actions[bot], @jgao54
• Fix PK and Nullable display in mirror columns UI (#4737) @ilidemi

v0.37.5

What's Changed

• report normalize batch id more frequently by @jgao54 in #4689
• optimize hot path by caching contexual attribute by @jgao54 in #4690
• refactor(codegen): convert FlowConnectionConfigs by field number, drop generator by @dtunikov in #4700
• add exclude_snapshot_status to MirrorStatusRequest by @serprex in #4703
• crdb stack/3 cdc changefeeds by @viragtripathi in #4669
• remove feature flag and connector type conditional by @jgao54 in #4686
• CDCFlowWorkFlow refactor by @jgao54 in #4705
• crdb stack/4 peer ui by @viragtripathi in #4670
• feat(bigquery): snapshot export as of BigQuery-clock timestamp T by @dtunikov in #4693
• crdb stack/5 tests ci by @viragtripathi in #4671
• mark bigquery.go as not deprecated by @dtunikov in #4711
• fix(clickhouse): treat CANNOT_ASSIGN_ALTER (517) as retryable by @shockdav in #4702
• feat(pg-pg): classify generated pkey by @Amogh-Bharadwaj in #4713
• fix(cmd): don't panic on CLI app error, exit(1) instead by @dtunikov in #4720
• support legacy 0-byte geometry data by @jgao54 in #4715
• crdb stack/6 resolved buffering by @viragtripathi in #4710
• remove errgroup legacy code by @jgao54 in #4723
• fix(clickhouse): accept cloud_mode_engine modes 3 and 4 in SMT validation by @tpanetti in #4729
• remove dead code in cdcflowworkflow and reduce replay surface area by @jgao54 in #4709
• Fix flaky test checking for continue-as-new by @ilidemi in #4728
• classify too many tables error by @jgao54 in #4731
• fix failure to abort multipart upload due to context cancellation by @jgao54 in #4730
• classify wal segment short error by @jgao54 in #4733
• Update gomod dependencies by @renovate[bot] in #4681
• Update github-actions dependencies by @renovate[bot] in #4684
• Update docker-compose dependencies by @renovate[bot] in #4685
• Update dependency material-symbols to ^0.46.0 by @renovate[bot] in #4704
• Update dockerfile dependencies by @renovate[bot] in #4682
• Lock file maintenance by @renovate[bot] in #4680
• fix(clickhouse): quorum writes on replicated clusters to stop normalize dropping rows by @andreyzhelnin-st in #4676

New Contributors

• @shockdav made their first contribution in #4702

Full Changelog: v0.37.4...v0.37.5

v0.37.4

What’s Changed

• support parallel_view_processing (#4688) @jgao54
• feat(bigquery): support GCP workload identity (#4607) @jkaflik
• crdb stack/2 schema qrep (#4668) @viragtripathi, @jgao54
• Add test cases for detecting incompatible Temporal history (#4677) @jgao54
• fix(clickhouse): validate staging read grants by access method (#4679) @tpanetti, @ilidemi
• crdb stack/1 peer connector (#4667) @viragtripathi, @jgao54
• Fix testcontainers flakes (#4673) @ilidemi
• Update S3 SDK (#4674) @ilidemi
• Lock file maintenance (#4598) @renovate[bot]
• Update gomod dependencies (#4648) @renovate[bot]
• Classify DocumentDB Invalid resume token (#4666) @ilidemi
• fix: classify MySQL error 3032 (server offline mode) as NOTIFY_CONNECTIVITY (#4663) @EvandroLG, @ilidemi
• Fix external contributor flow and Claude review that got it missed (#4665) @ilidemi
• Add CH pkg to Tilt, fix setup peer targets (#4660) @ilidemi
• Notify user when MySQL requires secure transport (#4655) @ilidemi
• Move Mongo validation tests to pkg (#4659) @ilidemi
• feat(postgres): handle postgres DEFAULT in schema changes (#4635) @dtunikov, @claude
• Return connection errors from GetSelectedColumns and other pgx queries missing them (#4657) @ilidemi
• fix(clickhouse): check for S3 grant before validating CH-bound pipe (#4629) @itsbilal
• Classify replication slot state rename failure as recoverable (#4654) @ilidemi, @github-actions[bot]
• Classify GCS HTTP/2 connection loss as recoverable (#4656) @ilidemi
• Classify ALTER PUBLICATION in a read-only transaction as recoverable (#4651) @ilidemi, @github-actions[bot]
• Notify user when the MySQL binlog connection goes stale (#4653) @ilidemi
• Notify user when the replication slot is read on an Aurora reader node (#4652) @ilidemi
• Match all variants of the logical decoding on standby error (#4649) @ilidemi
• Classify missing pg_logical snapshot temp file as recoverable (#4650) @ilidemi
• feat(postgres): emit source replication log space metrics for Postgres (#4640) @dtunikov
• feat(pg-pg): add notification for constraint violation (#4647) @Amogh-Bharadwaj
• fix(mysql): avoid double-counting fetched bytes (#4646) @dtunikov
• feat(mysql): carry MySQL column DEFAULT through to ClickHouse on ADD COLUMN (#4632) @dtunikov
• fix: Add error detail for validation errors in ValidatePeer (#4642) @itsbilal
• PG-PG: Add feature to clean up older raw table data, behind FF (#4624) @Amogh-Bharadwaj
• Refactor staging store validation (#4643) @ilidemi
• Use integration tests in pkg/clickhouse, fix CI for pkg (#4639) @ilidemi
• fix(postgres): correct TLS host resolution for pg_dump schema migration (#4625) @heavycrystal, @claude
• CI: Replace flow service orchestration with Tilt (#4605) @pfcoperez
• CI: Separate agent invocation GH command "@fable" to invoke Anthropic's advanced model separately (#4637) @pfcoperez
• CI Code review: Use latest Opus models (#4636) @pfcoperez
• Retry catalog and Temporal connections at startup (#4623) @dtunikov, @claude
• fix bug in row count estimation (#4619) @jgao54
• Pin dependencies (#4627) @renovate[bot]
• Update docker-compose dependencies (#4628) @renovate[bot]
• Update dockerfile dependencies (#4630) @renovate[bot]
• Update gomod dependencies (#4631) @renovate[bot]
• Classify invalid destination table definitions to notify users (#4626) @pfcoperez, @claude
• CI/Upload test results action: Fail with error when the upload log doesn't contain the summary header (#4622) @pfcoperez
• Optimize ValidateMirrorDestination to only fetchCustomTypeMapping once (#4620) @serprex

v0.37.3

What’s Changed

• Don't panic when changestream recreation fails (#4618) @jgao54
• fix string partitioning edge case to avoid rabbit-holing into single-row partitions (#4617) @jgao54
• fix: estimate row counts for postgres partitioned/inherited tables (#4613) @itsbilal
• support arbitrary string parallel snapshotting (#4522) @jgao54
• ttl_only_drop_parts=1 (#4616) @serprex
• disable aborts missing during db migration (#4615) @jgao54
• CI: Monitor CI stages times with actions-timeline action (#4606) @pfcoperez

v0.37.2

What’s Changed

• Clean up SSH error classification (#4610) @ilidemi
• PEERDB_MYSQL_IGNORE_MISSING_GTID (#4601) @serprex
• Fix zombie snapshot flows after the process restarts; fix a rare conn leak (#4603) @ilidemi
• fix: respect clickhouse max_query_size (#4593) @dtunikov
• maria: classify compressed columns errors + add validation to create mirror (#4559) @dtunikov
• feat: add source/destination/e2e commit lag metrics (#4573) @dtunikov
• Validate PARTITION BY expression with fake virtual table and Dynamic type for all columns (#4599) @pfcoperez
• fix clippy update regression (#4602) @serprex
• bug: CDC Date clamping consistent with initial load limits (#4600) @pfcoperez
• ui: support table removal in edit mirror, show source table in snapshot view (#4594) @serprex
• Include rust builder in renovate, fix script on mac (#4590) @ilidemi
• Heartbeat to Temporal more often (#4591) @ilidemi
• qrep: derive pull columns from source instead of catalog schema (#4583) @dtunikov

v0.37.1

What’s Changed

• fix(pg): set idle_session_timeout=0 for pg connections (#4589) @dtunikov
• fix(pg): classify conn closed explicitly as user notify error (#4587) @dtunikov
• classify error as retryable when pipe closed in deadlineCappableConn (#4567) @jgao54
• support filtering out deletion operation types for mongo (#4585) @jgao54
• fix mongo transient tls error (#4569) @jgao54
• Update github-actions dependencies (#4577) @renovate[bot]
• fix(mysql): gofumpt formatting in cdc.go (#4586) @dtunikov
• Update dockerfile dependencies (#4576) @renovate[bot]
• chore: de noise type change logs (#4581) @dtunikov
• fix: Add test for gh-ost/backup binlog metadata path ADD COLUMN for all types (#4557) @masterashu
• Re-order error classification for MySQLExecuteError (#4555) @jgao54
• Remove checks on pinned versions (#4579) @pfcoperez
• Renovate check and simulation CI workflow (#4578) @pfcoperez
• add filter to force pruning for getTableSchemaForTable query (#4580) @dtunikov
• fetch disk spill metrics for pg 14/15 as well (#4575) @jgao54
• mongodb: fix leaked changestream cursor (#4565) @jgao54
• handle retriable ssh dial error (#4570) @jgao54
• handle clickhouse znode error (#4568) @jgao54
• gcs: add error classification (#4566) @jgao54
• handle incomplete read of msg header transient error (#4571) @jgao54
• classify neon transient error (#4563) @jgao54
• .github: Fix deprecated workflow (#4574) @Jeremyyang920
• DBI-798: Safer check on mirror validation with MySQL sources setting server_id (#4564) @pfcoperez
• fix(ui): fix schema prefixing (#4562) @Amogh-Bharadwaj
• fix(pg-pg): better tls wiring for pgdump command (#4540) @Amogh-Bharadwaj
• feat: upgrade docker-compose.yml stable image tags (#4558) @github-actions[bot]
• feat(api): add endpoint for pg-pg source and destination row count check (#4556) @Amogh-Bharadwaj
• fix(mysql): pull INVISIBLE/GIPK columns during snapshot (#4553) @dtunikov
• fix(mysql): consistent SET column replication between snapshot and cdc (#4551) @dtunikov
• fix(mysql): align JSON representation between snapshot and CDC (#4535) @dtunikov
• Renovate simulation script (#4548) @pfcoperez
• address partition overflow bug (#4550) @jgao54

v0.37.0

What’s Changed

• fix(tests): update peer config to avoid using stale config from previous local run (#4552) @dtunikov
• fix: Add tests for non-pk table cdc works in append-only mode with Merge Tree (#4520) @masterashu
• maria: require resync on partial row events (#4530) @dtunikov
• mongoDB parallel snapshot follow-up (#4531) @jgao54
• fix(mysql): fail fast on partial JSON binlog events (PARTIAL_JSON) (#4538) @dtunikov
• Pin go-mysql digest as well (#4547) @ilidemi
• clickhouse: fix date overflow for initial load (#4534) @dtunikov
• fix(mysql): classify event checksum error (#4537) @dtunikov
• fix(mysql): handle commit for non transactional mysql engines (#4539) @dtunikov
• Update github-actions dependencies (#4542) @renovate[bot]
• Make MySQL binlog staleness threshold configurable (#4453) @ilidemi
• Enable mTLS for PG sources (#4468) @pfcoperez
• Update gomod dependencies (#4545) @renovate[bot]
• Update mysql Docker tag to v9.7.1 (#4543) @renovate[bot]
• Update temporalio/ui Docker tag to v2.51.1 (#4544) @renovate[bot]
• Deprecate non-ClickHouse / non-Postgres destination connectors (#4491) @Jeremyyang920
• feat(ci): Last CI step checking for non-exercised tests (#4533) @pfcoperez
• Lock file maintenance (#4536) @renovate[bot]
• add clickhouse cluster in local tilt (#4527) @masterashu
• fix(mysql): use gtid_binlog_pos for MariaDB CDC start position (#4523) @dtunikov
• feat(mysql): add counter metric for parseSQL errors (#4498) @dtunikov
• Workflow to tag flow-api-client versions (#4532) @pfcoperez
• [DBI-798] MySQL Connector: Allow passing server_id as a peer configuration parameter (#4525) @pfcoperez
• add more mysql/maria column type tests and support uuid/inet types (#4505) @dtunikov
• Update gomod dependencies (#4399) @renovate[bot]
• DBI-609: Add IAM auth support for CDC pipes with AWS Auruora' (#4495) @masterashu
• mongo: partition snapshots for numeric and string _id collections (#4487) @andreyzhelnin-st
• treat string partition ranges as sensitive data (#4501) @jgao54
• fix: PG<15 fallback parameters (#4473) (#4509) @alikonhz
• Fix e2e table schema asserts (#4506) @ilidemi
• Classify CH access denied (#4528) @ilidemi
• Fix claude review on fork PRs (#4502) @ilidemi
• LogFlowError on mysql ConnectionActive failure (#4521) @serprex
• Separate MySQL and MariaDB tests (#4517) @ilidemi
• Disable PG CDC store for CH destinations by default (#4515) @ilidemi
• Update dockerfile dependencies (#4514) @renovate[bot]
• Constrain mysql-debug Renovate updates (#4512) @ilidemi
• Lock file maintenance (#4420) @renovate[bot]
• Update github-actions dependencies (#4511) @renovate[bot]
• Add tag to support propagating pipe name (#4508) @jgao54
• Add a user-facing log when processing large txn (#4500) @jgao54
• error classification: add error code 192 (#4507) @jgao54
• [DBI-829] Extended GTID e2e and UTs (#4493) @pfcoperez
• Optimize encoding maps (#4494) @ilidemi
• add online_schema_migrations metric (#4497) @dtunikov
• Dedup sourcePeerType (#4499) @ilidemi
• Support ANSI_QUOTES in MySQL/MariaDB DDL (#4452) @ilidemi
• fix(mysql): handle transaction payload event (#4433) @dtunikov
• observe mysql/postgres column type changes (#4488) @dtunikov

v0.36.32

What’s Changed

• Disable Postgres CDC store by default for ClickHouse destinations (commit)

v0.36.31

What’s Changed

• support parallel snapshotting for uuid partitioning (#4482) @jgao54
• dbi 604 handle replication slot creation getting stuck (#4439) @masterashu
• Decode BINARY(N) as Bytes on the binlog-metadata add-column path (#4457) @ilidemi
• fix(clickhouse): drop orphaned shard table after mirror resync exchange (#4490) @andreyzhelnin-st
• Support MariaDB VECTOR during CDC (#4454) @ilidemi
• Update docker-compose dependencies (#4469) @renovate[bot]
• Remove SNS and telemetry sender infra (#4492) @ilidemi
• Support null-terminated strings in MySQL DDL (#4459) @ilidemi
• Handle error cases from parsing filepos (#4462) @ilidemi
• Remove incident.io telemetry (#4481) @ilidemi
• fix(flowable): do not gate pg dest logic by pg source (#4483) @Amogh-Bharadwaj
• Tilt environment: Fix ClickHouse docker image selection (#4486) @pfcoperez