It is possible to use the YAML export from a custom connection in terraform?

[anzelpwj-rev]

Topic

terraform builder custom

Relevant information

Hello everyone,

I'm trying to migrate our Airbyte infrastructure (OSS, self-hosted) to terraform, and I'm a bit unclear of how to handle the connections we've created in the Builder. I see there is a airbyte_source_custom option (https://registry.terraform.io/providers/airbytehq/airbyte/latest/docs/resources/source_custom) but I'm very unclear as to if this would work with our custom connection and what I should be putting into the configuration block. Should I just export the YAML from the connection builder, convert it to JSON, and put it in there? If not, how would I go about using airbyte_source_custom?

TIA

[brunopini]

When declaring custom sources, you need to reference its SourceDefinitonId in the definition_id Terraform field. This value can be found using the configuration API and the connector_builder_projects/list endpoint, if it is a low- or no-code source. If you have imported a custom docker source, you can use the source_definitions/list endpoint.

The configuration block in the TF resource is simply a json encoded dictionary-like object with the keys and values needed (mirror what you see in the UI). For example, this source's UI configuration:

Would translate into this configuration block:

locals {
  configuration = jsonencode(merge(
    {
      "advertiser_hashs" : [var.advertiser_hash],
      "username" : data.google_secret_manager_secret_version.username.secret_data,
      "password" : data.google_secret_manager_secret_version.password.secret_data,
      "count_convention" : "ALL_CONVERSIONS"
    },
    var.start_datetime != null ? { "start_date" : var.start_datetime } : {},
    var.end_datetime != null ? { "end_date" : var.end_datetime } : {}
  ))
}

resource "airbyte_source_custom" "my_source" {
  definition_id = var.source_definition_id
  configuration = local.configuration
  name          = "your custom source"
  workspace_id  = var.workspace_id
}

[anzelpwj-rev]

So to clarify, if I used the low-code builder to create a connection, I would just need to use that tool to create the connection in a new Airbyte instance and then make my terraform point to its definition id? Is there a way to automate using the builder to create a new connection and then import the YAML?

[brunopini]

Unfortunately I haven't found any means to avoid this manual step yet. Might be something to look out for in future versions.

[anzelpwj-rev]

Okay, here's another question. Is there a way to coerce whatever custom source I build to have a specific definition id, or do I just have to note down what it is? I'd like to have as much possible already set in Terraform and not have to inject these UUIDs in via environmental variables.

[brunopini]

Unfortunately also no. You won't be able to avoid this manual setup.

[iamliamc]

I've got it to work something like this:

Resource: airbyte_source_custom.privatei.definition_id references
airbyte_declarative_source_definition.privatei.id

resource "airbyte_declarative_source_definition" "privatei" {
  name         = "Private I"
  workspace_id = airbyte_workspace.default.workspace_id
  version      = var.private_i_declarative_source_definition_version
  manifest = jsonencode(yamldecode(templatefile(
    "${path.module}/builder/private_i_source.yml.tftpl",
    {
      PRIVATEI_USERNAME      = var.privatei_connector_username
      PRIVATEI_CLIENT_ID     = var.privatei_connector_client_id
    }
  )))
}


# PrivateI Source Configuration
resource "airbyte_source_custom" "privatei" {
  name          = "Private I"
  workspace_id = airbyte_workspace.default.workspace_id
  definition_id = airbyte_declarative_source_definition.privatei.id

  configuration = jsonencode({
    start_date = "2003-04-01T00:00:00Z"
    client_secret = var.privatei_connector_client_secret
    password = var.privatei_connector_password
  })
}