Terraform Connector Config: Import all Existing and Future Tables

[jduckwo]

I am doing a POC to load all tables from one Postgres database to another Postgres database. I am configuring the resources using Terraform. I would like to have all existing and future tables to be synced automatically once they are added to the Source PG Publication. I don't want to manually configure each stream in Terraform code manually.

I have the Source, Destination and Connection resources working in general. Where I am running into an issue is getting the existing and future streams to automatically load without needing to be hard coded in the configurations.streams section. If I omit that section altogether, nothing will load at all. When I manually configure one stream it will load that specific stream, but the other existing streams will not load unless I configure them manually or turn them on in the GUI.

When I set up a connection through the GUI, when I select all streams then any new streams get picked up automatically on future job runs. How do I configure this behaviour via Terraform.

Source Config

resource "airbyte_source_postgres" "source__source_1_db" {
  name         = "source_1_db"
  workspace_id = var.workspace_id

  configuration = {
    host     = local.source_host
    port     = local.source_port
    database = local.source_database
    schemas  = local.source_schemas
    username = local.source_username
    password = var.source_password

    ssl_mode      = {disable = {}}
    tunnel_method = {no_tunnel = {}}

    replication_method = {
      read_changes_using_write_ahead_log_cdc = {
        replication_slot                     = "airbyte_slot"
        publication                          = "airbyte_publication"
        invalid_cdc_cursor_position_behavior = "Re-sync data"
      }
    }
  }
}

Destination Config

resource "airbyte_destination_postgres" "destination__central_db" {
  name         = "central_db"
  workspace_id = var.workspace_id

  configuration = {
    host     = local.destination_host
    database = local.destination_database
    port     = local.destination_port
    username = local.destination_username
    password = local.destination_password

    ssl_mode      = {disable = {}}
    tunnel_method = {no_tunnel = {}}
  }
}

Connection Config

resource "airbyte_connection" "connection__source_1_db--central_db" {
  name = "source_1_db -> central_db"

  source_id      = airbyte_source_postgres.source__source_1_db.source_id
  destination_id = airbyte_destination_postgres.destination__central_db.destination_id

  namespace_definition = "custom_format"
  namespace_format     = "${local.code_source_1}__$${SOURCE_NAMESPACE}"
  non_breaking_schema_updates_behavior = "propagate_fully"
  schedule = {
    schedule_type   = "manual"
  }
  status = "active"

  configurations = {
    streams = [
      {
        name      = "table_1"
        sync_mode = "incremental_deduped_history"
      }
    ]
  }
}