Dataflow Templates 2025-11-05-00_RC00

fix(build): update surefire report processing to handle time format (…

Dataflow Templates 2025-10-28-01_RC00

What's Changed

Improvements

• fix(kafka): handle domain-scoped project IDs in bootstrap server URL by @liferoad in #2908
• Yaml: Add yaml syntax UT by @derrickaw in #2872
• [SpannerToSourceDb] Fixing CDC events processing logic for Reverse replication template by @darshan-sj in #2919
• Apply GCP Application Default authentication when using a Managed Service for Apache Kafka cluster. by @fozzie15 in #2925

Bug fixes

• Remove special handling for only PK column tables by @manitgupta in #2929
• [yaml] - move syntax test verification to validate section by @derrickaw in #2936

Package Upgrades

• chore(deps): update Python dependencies by @github-actions[bot] in #2899
• build: update jetty dependencies to 12.0.12 to fix CVE-2024-6763 by @stankiewicz in #2921
• build: update org.apache.commons:commons-configuration to resolve CVE-2024-29131 by @stankiewicz in #2922

New Contributors

• @LucaDai made their first contribution in #2898

Full Changelog: 2025-10-21-00_RC00...2025-10-28-01_RC00

Dataflow Templates 2025-10-28-00_RC00

bump org.apache.commons:commons-configuration to resolve CVE-2024-291…

Dataflow Templates 2025-10-21-00_RC00

What's Changed

Improvements

• KafkaToBigQueryYaml - change hidden to false by @derrickaw in #2879
• Add option to unwrap messages replayed from DLQ to avoid nesting loops. by @claudevdm in #2888

Bug fixes

• fix dependabot labeling workflow name by @derrickaw in #2886
• fix label dependabot workflow by @derrickaw in #2890

Full Changelog: 2025-10-14-00_RC00...2025-10-21-00_RC00

Dataflow Templates 2025-10-14-00_RC00

What's Changed

Improvements

• feat(kafka-to-bigquery): Fix dynamic destination connection churn by @anpag in #2873

Bug fixes

• changed mysql event ordering in datastream to spanner by @bharadwaj-aditya in #2867

New Contributors

• @tianz101 made their first contribution in #2871
• @anpag made their first contribution in #2873

Full Changelog: 2025-10-07-00_RC00...2025-10-14-00_RC00

Dataflow Templates 2025-10-07-00_RC00

What's Changed

Improvements

• Fixed a performance regression in the Spanner change streams to BigQuery template that was introduced in the 2024-09-10-00_RC00.
• Only read schema when the col in mod is not in the stored schema. by @jiangzzhu in #2840
• Skip reading schema if the value capture mode is NEW_ROW_AND_OLD_VALUES. by @jiangzzhu in #2839
• feat: Add support for property graph on view by @jennyjiang98 in #2813
• Remove jinja variables option code by @derrickaw in #2826

New Contributors

• @jiangzzhu made their first contribution in #2840

Full Changelog: 2025-10-01-00_RC00...2025-10-07-00_RC00

Dataflow Templates 2025-10-01-00_RC00

What's Changed

Improvements

• Add Run, Staging and Validation stages for KafkaToBigquery Yaml template by @derrickaw in #2782

Bug fixes

• Fix vector index detection by @fbiville in #2151
• Change default CPU priority of bulk template to HIGH by @manitgupta in #2836

Package Upgrades

• Upgrade to Beam 2.68.0 by @Amar3tto in #2830

Known issues

Users of Spanner change streams to BigQuery template are advised to use the 2024-09-02-00_RC00 version of the template due to a performance regression introduced in the 2024-09-10-00_RC00 release that might affect certain use-cases.

Full Changelog: 2025-09-23-00_RC00...2025-10-01-00_RC00

Dataflow Templates 2025-09-23-00_RC00

What's Changed

Improvements

• Introduce Schemaless graphs to import/export DDL by @smukil in #2749
• build(deps): bump org.awaitility:awaitility from 4.2.0 to 4.3.0 by @dependabot[bot] in #2794
• DataStreamToSpanner - Adding main table read at the beginning of main transaction by @darshan-sj in #2789
• CPU priority in reverse replication jobs by @bharadwaj-aditya in #2795

Known issues

Users of Spanner change streams to BigQuery template are advised to use the 2024-09-02-00_RC00 version of the template due to a performance regression introduced in the 2024-09-10-00_RC00 release that might affect certain use-cases.

Full Changelog: 2025-09-16-00_RC00...2025-09-23-00_RC00

Dataflow Templates 2025-09-16-00_RC00

What's Changed

Improvements

• feat(datastream): add source type override for datastream cdc data by @liferoad in #2715
• Bump org.testcontainers:hivemq from 1.17.5 to 1.21.3 by @dependabot[bot] in #2729
• Fix CVE-2025-58056 by @stankiewicz in #2750

Package Upgrades

• chore(deps): update Python dependencies by @github-actions[bot] in #2778

Known issues

Users of Spanner change streams to BigQuery template are advised to use the 2024-09-02-00_RC00 version of the template due to a performance regression introduced in the 2024-09-10-00_RC00 release that might affect certain use-cases.

Full Changelog: 2025-09-10-00_RC00...2025-09-16-00_RC00

Dataflow Templates 2025-09-10-00_RC00

What's Changed

Improvements

• Bump com.azure:azure-messaging-eventhubs from 5.15.0 to 5.21.0 by @dependabot[bot] in #2711
• Bump org.codehaus.mojo:exec-maven-plugin from 1.6.0 to 3.5.1 by @dependabot[bot] in #2710
• Datastream To SQL - Added Default Casing Parameter / Column Casing Functionality by @rahulkumar0630 in #2687

Bug fixes

• add float32 support to spanner-to-sourcedb by @VardhanThigle in #2722
• fix SourceDbToSpanner desc truncations by @derrickaw in #2734
• update docs and readme about template validation by @derrickaw in #2743

Known issues

Users of Spanner change streams to BigQuery template are advised to use the 2024-09-02-00_RC00 version of the template due to a performance regression introduced in the 2024-09-10-00_RC00 release that might affect certain use-cases.

Full Changelog: 2025-09-02-00_RC00...2025-09-10-00_RC00