Skip to content

feat: introduce connection_id for external_table configuration #410

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
+72 −69
Open

feat: introduce connection_id for external_table configuration #410

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
ca54b0a
feat: introduce connection_id for external_table configuration
kdarkhan Aug 5, 2025
b94b0af
chore: make generate_docs
kdarkhan Aug 5, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -196,7 +196,7 @@ This module provisions a dataset and a list of tables with associated JSON schem
| deletion\_protection | Whether or not to allow deletion of tables and external tables defined by this module. Can be overriden by table-level deletion\_protection configuration. | `bool` | `false` | no |
| description | Dataset description. | `string` | `null` | no |
| encryption\_key | Default encryption key to apply to the dataset. Defaults to null (Google-managed). | `string` | `null` | no |
| external\_tables | A list of objects which include table\_id, expiration\_time, external\_data\_configuration, and labels. | <pre>list(object({<br> table_id = string,<br> description = optional(string),<br> autodetect = bool,<br> compression = string,<br> ignore_unknown_values = bool,<br> max_bad_records = number,<br> schema = string,<br> source_format = string,<br> source_uris = list(string),<br> csv_options = object({<br> quote = string,<br> allow_jagged_rows = bool,<br> allow_quoted_newlines = bool,<br> encoding = string,<br> field_delimiter = string,<br> skip_leading_rows = number,<br> }),<br> google_sheets_options = object({<br> range = string,<br> skip_leading_rows = number,<br> }),<br> hive_partitioning_options = object({<br> mode = string,<br> source_uri_prefix = string,<br> }),<br> expiration_time = optional(string, null),<br> max_staleness = optional(string),<br> deletion_protection = optional(bool),<br> labels = optional(map(string), {}),<br> }))</pre> | `[]` | no |
| external\_tables | A list of objects which include table\_id, expiration\_time, external\_data\_configuration, and labels. | <pre>list(object({<br> table_id = string,<br> description = optional(string),<br> autodetect = bool,<br> compression = string,<br> ignore_unknown_values = bool,<br> max_bad_records = number,<br> schema = string,<br> connection_id = optional(string),<br> source_format = string,<br> source_uris = list(string),<br> csv_options = object({<br> quote = string,<br> allow_jagged_rows = bool,<br> allow_quoted_newlines = bool,<br> encoding = string,<br> field_delimiter = string,<br> skip_leading_rows = number,<br> }),<br> google_sheets_options = object({<br> range = string,<br> skip_leading_rows = number,<br> }),<br> hive_partitioning_options = object({<br> mode = string,<br> source_uri_prefix = string,<br> }),<br> expiration_time = optional(string, null),<br> max_staleness = optional(string),<br> deletion_protection = optional(bool),<br> labels = optional(map(string), {}),<br> }))</pre> | `[]` | no |
| location | The location of the dataset. For multi-region, US or EU can be provided. | `string` | `"US"` | no |
| materialized\_views | A list of objects which includes view\_id, view\_query, clustering, time\_partitioning, range\_partitioning, expiration\_time and labels | <pre>list(object({<br> view_id = string,<br> description = optional(string),<br> query = string,<br> enable_refresh = bool,<br> refresh_interval_ms = string,<br> clustering = optional(list(string), []),<br> time_partitioning = optional(object({<br> expiration_ms = string,<br> field = string,<br> type = string,<br> require_partition_filter = bool,<br> }), null),<br> range_partitioning = optional(object({<br> field = string,<br> range = object({<br> start = string,<br> end = string,<br> interval = string,<br> }),<br> }), null),<br> expiration_time = optional(string, null),<br> max_staleness = optional(string),<br> labels = optional(map(string), {}),<br> }))</pre> | `[]` | no |
| max\_time\_travel\_hours | Defines the time travel window in hours | `number` | `null` | no |
Expand Down
1 change: 1 addition & 0 deletions examples/multiple_tables/main.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,7 @@ module "bigquery" {
max_bad_records = 0
source_format = "CSV"
schema = null
connection_id = null
expiration_time = 2524604400000 # 2050/01/01
labels = {
env = "devops"
Expand Down
4 changes: 3 additions & 1 deletion main.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -186,6 +186,7 @@ resource "google_bigquery_table" "external_table" {
table_id = each.key
description = each.value["description"]
labels = each.value["labels"]
schema = each.value["connection_id"] != null ? each.value["schema"] : null
expiration_time = each.value["expiration_time"] != null ? each.value["expiration_time"] : 0
max_staleness = each.value["max_staleness"]
project = var.project_id
Expand All @@ -194,9 +195,10 @@ resource "google_bigquery_table" "external_table" {
external_data_configuration {
autodetect = each.value["autodetect"]
compression = each.value["compression"]
connection_id = each.value["connection_id"]
ignore_unknown_values = each.value["ignore_unknown_values"]
max_bad_records = each.value["max_bad_records"]
schema = each.value["schema"]
schema = each.value["connection_id"] == null ? each.value["schema"] : null
source_format = each.value["source_format"]
source_uris = each.value["source_uris"]

Expand Down
132 changes: 65 additions & 67 deletions metadata.display.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,42 @@ spec:
value: GZIP
- label: NONE
value: NONE
csv_options:
name: csv_options
title: CSV Options
properties:
encoding:
name: encoding
title: Encoding
enumValueLabels:
- label: UTF-8
value: UTF-8
- label: ISO-8859-1
value: ISO-8859-1
skip_leading_rows:
name: skip_leading_rows
title: Skip Leading Rows
google_sheets_options:
name: google_sheets_options
title: Google Sheets Options
properties:
skip_leading_rows:
name: skip_leading_rows
title: Skip Leading Rows
hive_partitioning_options:
name: hive_partitioning_options
title: Hive Partitioning Options
properties:
mode:
name: mode
title: Mode
enumValueLabels:
- label: AUTO
value: AUTO
- label: STRINGS
value: STRINGS
- label: CUSTOM
value: CUSTOM
source_format:
name: source_format
title: Source Format
Expand Down Expand Up @@ -98,44 +134,6 @@ spec:
title: Source Uris
regexValidation: ^gs://([a-z0-9]([a-z0-9-.]*[a-z0-9])?)/([^#?]+)?$
validation: Must be a valid Google Cloud Storage URI.
google_sheets_options:
name: google_sheets_options
title: Google Sheets Options
properties:
skip_leading_rows:
name: skip_leading_rows
title: Skip Leading Rows
min: 0
hive_partitioning_options:
name: hive_partitioning_options
title: Hive Partitioning Options
properties:
mode:
name: mode
title: Mode
enumValueLabels:
- label: AUTO
value: AUTO
- label: STRINGS
value: STRINGS
- label: CUSTOM
value: CUSTOM
csv_options:
name: csv_options
title: CSV Options
properties:
encoding:
name: encoding
title: Encoding
enumValueLabels:
- label: UTF-8
value: UTF-8
- label: ISO-8859-1
value: ISO-8859-1
skip_leading_rows:
name: skip_leading_rows
title: Skip Leading Rows
min: 0
location:
name: location
title: Location
Expand Down Expand Up @@ -174,35 +172,6 @@ spec:
name: routines
title: Routines
properties:
routine_id:
name: routine_id
title: Routine Id
regexValidation: ^[A-Za-z_]{1,256}$
validation: The routine ID must contain only letters, numbers, and underscores, and cannot start with a number.
routine_type:
name: routine_type
title: Routine Type
enumValueLabels:
- label: SCALAR_FUNCTION
value: SCALAR_FUNCTION
- label: PROCEDURE
value: PROCEDURE
- label: TABLE_VALUED_FUNCTION
value: TABLE_VALUED_FUNCTION
language:
name: language
title: Language
enumValueLabels:
- label: SQL
value: SQL
- label: JAVASCRIPT
value: JAVASCRIPT
- label: PYTHON
value: PYTHON
- label: JAVA
value: JAVA
- label: SCALA
value: SCALA
arguments:
name: arguments
title: Arguments
Expand All @@ -225,6 +194,35 @@ spec:
value: OUT
- label: INOUT
value: INOUT
language:
name: language
title: Language
enumValueLabels:
- label: SQL
value: SQL
- label: JAVASCRIPT
value: JAVASCRIPT
- label: PYTHON
value: PYTHON
- label: JAVA
value: JAVA
- label: SCALA
value: SCALA
routine_id:
name: routine_id
title: Routine Id
regexValidation: ^[A-Za-z_]{1,256}$
validation: The routine ID must contain only letters, numbers, and underscores, and cannot start with a number.
routine_type:
name: routine_type
title: Routine Type
enumValueLabels:
- label: SCALAR_FUNCTION
value: SCALAR_FUNCTION
- label: PROCEDURE
value: PROCEDURE
- label: TABLE_VALUED_FUNCTION
value: TABLE_VALUED_FUNCTION
storage_billing_model:
name: storage_billing_model
title: Storage Billing Model
Expand Down
1 change: 1 addition & 0 deletions metadata.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -185,6 +185,7 @@ spec:
ignore_unknown_values = bool,
max_bad_records = number,
schema = string,
connection_id = optional(string),
source_format = string,
source_uris = list(string),
csv_options = object({
Expand Down
1 change: 1 addition & 0 deletions variables.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -199,6 +199,7 @@ variable "external_tables" {
ignore_unknown_values = bool,
max_bad_records = number,
schema = string,
connection_id = optional(string),
source_format = string,
source_uris = list(string),
csv_options = object({
Expand Down