Add upstream mysql and postgres table names

PR change requests

Author: bobbyiliev

compose.yaml

@@ -19,6 +19,7 @@ services:
       - --system-parameter-default=max_clusters=100
       - --system-parameter-default=max_sources=100
       - --system-parameter-default=max_aws_privatelink_connections=10
+      - --system-parameter-default=enable_create_table_from_source=on
       - --all-features
     environment:
       MZ_NO_TELEMETRY: 1
@@ -48,6 +49,7 @@ services:
       - --system-parameter-default=max_sources=100
       - --system-parameter-default=max_aws_privatelink_connections=10
       - --system-parameter-default=transaction_isolation=serializable
+      - --system-parameter-default=enable_create_table_from_source=on
       - --all-features
     environment:
       MZ_NO_TELEMETRY: 1

docs/data-sources/source_reference.md

@@ -0,0 +1,53 @@
+---
+# generated by https://github.com/hashicorp/terraform-plugin-docs
+page_title: "materialize_source_reference Data Source - terraform-provider-materialize"
+subcategory: ""
+description: |-
+  The materialize_source_reference data source retrieves a list of available upstream references for a given Materialize source. These references represent potential tables that can be created based on the source, but they do not necessarily indicate references the source is already ingesting. This allows users to see all upstream data that could be materialized into tables.
+---
+
+# materialize_source_reference (Data Source)
+
+The `materialize_source_reference` data source retrieves a list of *available* upstream references for a given Materialize source. These references represent potential tables that can be created based on the source, but they do not necessarily indicate references the source is already ingesting. This allows users to see all upstream data that could be materialized into tables.
+
+## Example Usage
+
+```terraform
+data "materialize_source_reference" "source_references" {
+  source_id = materialize_source_mysql.test.id
+}
+
+output "source_references" {
+  value = data.materialize_source_reference.my_source_references.references
+}
+```
+
+<!-- schema generated by tfplugindocs -->
+## Schema
+
+### Required
+
+- `source_id` (String) The ID of the source to get references for
+
+### Optional
+
+- `region` (String) The region in which the resource is located.
+
+### Read-Only
+
+- `id` (String) The ID of this resource.
+- `references` (List of Object) The source references (see [below for nested schema](#nestedatt--references))
+
+<a id="nestedatt--references"></a>
+### Nested Schema for `references`
+
+Read-Only:
+
+- `columns` (List of String)
+- `name` (String)
+- `namespace` (String)
+- `source_database_name` (String)
+- `source_name` (String)
+- `source_schema_name` (String)
+- `source_type` (String)
+- `updated_at` (String)

docs/guides/materialize_source_table.md

@@ -1,10 +1,19 @@
-# Source versioning: migrating to `materialize_source_table_{source}` Resource
+---
+# generated by https://github.com/hashicorp/terraform-plugin-docs
+# template file: templates/guides/materialize_source_table.md.tmpl
+page_title: "Source Table Migration Guide"
+subcategory: ""
+description: |-
+  Guide for migrating to the new materialize_source_table_{source_type} resources.
+---
+
+# Source versioning: migrating to `materialize_source_table_{source_type}` Resource
 
 In previous versions of the Materialize Terraform provider, source tables were defined within the source resource itself and were considered subsources of the source rather than separate entities.
 
-This guide will walk you through the process of migrating your existing source table definitions to the new `materialize_source_table_{source}` resource.
+This guide will walk you through the process of migrating your existing source table definitions to the new `materialize_source_table_{source_type}` resource.
 
-For each source type (e.g., MySQL, Postgres, etc.), you will need to create a new `materialize_source_table_{source}` resource for each table that was previously defined within the source resource. This ensures that the tables are preserved during the migration process. For Kafka sources, you will need to create at least one `materialize_source_table_kafka` table to hold data for the kafka topic.
+For each MySQL and Postgres source, you will need to create a new `materialize_source_table_{source_type}` resource for each table that was previously defined within the source resource. This ensures that the tables are preserved during the migration process. For Kafka sources, you will need to create a `materialize_source_table_kafka` table with the same name as the kafka source to contain the data for the kafka topic.
 
 ## Old Approach
 
@@ -54,15 +63,15 @@ resource "materialize_source_kafka" "example_source_kafka_format_text" {
 
 ## New Approach
 
-The new approach separates source definitions and table definitions. You will now create the source without specifying the tables, and then define each table using the `materialize_source_table_{source}` resource.
+The new approach separates source definitions and table definitions. You will now create the source without specifying the tables, and then define each table using the `materialize_source_table_{source_type}` resource.
 
 ## Manual Migration Process
 
-This manual migration process requires users to create new source tables using the new `materialize_source_table_{source}` resource and then remove the old ones. We'll cover examples for both MySQL and Kafka sources.
+This manual migration process requires users to create new source tables using the new `materialize_source_table_{source_type}` resource and then remove the old ones. We'll cover examples for both MySQL and Kafka sources.
 
-### Step 1: Define `materialize_source_table_{source}` Resources
+### Step 1: Define `materialize_source_table_{source_type}` Resources
 
-Before making any changes to your existing source resources, create new `materialize_source_table_{source}` resources for each table that is currently defined within your sources.
+Before making any changes to your existing source resources, create new `materialize_source_table_{source_type}` resources for each table that is currently defined within your sources.
 
 #### MySQL Example:
 
@@ -109,11 +118,11 @@ resource "materialize_source_table_kafka" "kafka_table_from_source" {
 
 ### Step 2: Apply the Changes
 
-Run `terraform plan` and `terraform apply` to create the new `materialize_source_table_{source}` resources.
+Run `terraform plan` and `terraform apply` to create the new `materialize_source_table_{source_type}` resources.
 
 ### Step 3: Remove Table Blocks from Source Resources
 
-Once the new `materialize_source_table_{source}` resources are successfully created, remove all the deprecated and table-specific attributes from your source resources.
+Once the new `materialize_source_table_{source_type}` resources are successfully created, remove all the deprecated and table-specific attributes from your source resources.
 
 #### MySQL Example:
 
@@ -167,9 +176,7 @@ resource "materialize_source_kafka" "kafka_source" {
 }
 ```
 
-In the `lifecycle` block, add the `ignore_changes` meta-argument to prevent Terraform from trying to update these attributes during subsequent applies, that way Terraform won't try to update these values based on incomplete information from the state as they will no longer be defined in the source resource itself but in the new `materialize_source_table_{source}` resources.
-
-> Note: We will make the changes to those attributes a no-op, so the `ignore_changes` block will not be necessary.
+In the `lifecycle` block, add the `ignore_changes` meta-argument to prevent Terraform from trying to update these attributes during subsequent applies, that way Terraform won't try to update these values based on incomplete information from the state as they will no longer be defined in the source resource itself but in the new `materialize_source_table_{source_type}` resources.
 
 ### Step 4: Update Terraform State
 
@@ -179,12 +186,18 @@ After removing the `table` blocks and the table/topic specific attributes from y
 
 After applying the changes, verify that your tables are still correctly set up in Materialize by checking the table definitions using Materialize's SQL commands.
 
+For a more detailed view of a specific table, you can use the `SHOW CREATE TABLE` command:
+
+```sql
+SHOW CREATE TABLE materialize.public.mysql_table1_from_source;
+```
+
 ## Importing Existing Tables
 
 To import existing tables into your Terraform state, use the following command:
 
 ```bash
-terraform import materialize_source_table_{source}.table_name <region>:<table_id>
+terraform import materialize_source_table_{source_type}.table_name <region>:<table_id>
 ```
 
 Replace `{source}` with the appropriate source type (e.g., `mysql`, `kafka`), `<region>` with the actual region, and `<table_id>` with the table ID.
@@ -227,4 +240,4 @@ After importing, you may need to manually update these ignored attributes in you
 
 ## Future Improvements
 
-The Kafka and Webhooks sources are currently being implemented. Once these changes, the migration process will be updated to include them.
+Webhooks sources have not yet been migrated to the new model. Once this changes, the migration process will be updated to include them.

docs/resources/source_kafka.md

@@ -56,19 +56,19 @@ resource "materialize_source_kafka" "example_source_kafka" {
 - `cluster_name` (String) The cluster to maintain this source.
 - `comment` (String) Comment on an object in the database.
 - `database_name` (String) The identifier for the source database in Materialize. Defaults to `MZ_DATABASE` environment variable if set or `materialize` if environment variable is not set.
-- `envelope` (Block List, Max: 1, Deprecated) How Materialize should interpret records (e.g. append-only, upsert). Deprecated: Use the new materialize_source_table_kafka resource instead. (see [below for nested schema](#nestedblock--envelope))
+- `envelope` (Block List, Max: 1, Deprecated) How Materialize should interpret records (e.g. append-only, upsert). Deprecated: Use the new `materialize_source_table_kafka` resource instead. (see [below for nested schema](#nestedblock--envelope))
 - `expose_progress` (Block List, Max: 1) The name of the progress collection for the source. If this is not specified, the collection will be named `<src_name>_progress`. (see [below for nested schema](#nestedblock--expose_progress))
 - `format` (Block List, Max: 1) How to decode raw bytes from different formats into data structures Materialize can understand at runtime. (see [below for nested schema](#nestedblock--format))
-- `include_headers` (Boolean, Deprecated) Include message headers. Deprecated: Use the new materialize_source_table_kafka resource instead.
-- `include_headers_alias` (String, Deprecated) Provide an alias for the headers column. Deprecated: Use the new materialize_source_table_kafka resource instead.
-- `include_key` (Boolean, Deprecated) Include a column containing the Kafka message key. Deprecated: Use the new materialize_source_table_kafka resource instead.
-- `include_key_alias` (String, Deprecated) Provide an alias for the key column. Deprecated: Use the new materialize_source_table_kafka resource instead.
-- `include_offset` (Boolean, Deprecated) Include an offset column containing the Kafka message offset. Deprecated: Use the new materialize_source_table_kafka resource instead.
-- `include_offset_alias` (String, Deprecated) Provide an alias for the offset column. Deprecated: Use the new materialize_source_table_kafka resource instead.
-- `include_partition` (Boolean, Deprecated) Include a partition column containing the Kafka message partition. Deprecated: Use the new materialize_source_table_kafka resource instead.
-- `include_partition_alias` (String, Deprecated) Provide an alias for the partition column. Deprecated: Use the new materialize_source_table_kafka resource instead.
-- `include_timestamp` (Boolean, Deprecated) Include a timestamp column containing the Kafka message timestamp. Deprecated: Use the new materialize_source_table_kafka resource instead.
-- `include_timestamp_alias` (String, Deprecated) Provide an alias for the timestamp column. Deprecated: Use the new materialize_source_table_kafka resource instead.
+- `include_headers` (Boolean, Deprecated) Include message headers. Deprecated: Use the new `materialize_source_table_kafka` resource instead.
+- `include_headers_alias` (String, Deprecated) Provide an alias for the headers column. Deprecated: Use the new `materialize_source_table_kafka` resource instead.
+- `include_key` (Boolean, Deprecated) Include a column containing the Kafka message key. Deprecated: Use the new `materialize_source_table_kafka` resource instead.
+- `include_key_alias` (String, Deprecated) Provide an alias for the key column. Deprecated: Use the new `materialize_source_table_kafka` resource instead.
+- `include_offset` (Boolean, Deprecated) Include an offset column containing the Kafka message offset. Deprecated: Use the new `materialize_source_table_kafka` resource instead.
+- `include_offset_alias` (String, Deprecated) Provide an alias for the offset column. Deprecated: Use the new `materialize_source_table_kafka` resource instead.
+- `include_partition` (Boolean, Deprecated) Include a partition column containing the Kafka message partition. Deprecated: Use the new `materialize_source_table_kafka` resource instead.
+- `include_partition_alias` (String, Deprecated) Provide an alias for the partition column. Deprecated: Use the new `materialize_source_table_kafka` resource instead.
+- `include_timestamp` (Boolean, Deprecated) Include a timestamp column containing the Kafka message timestamp. Deprecated: Use the new `materialize_source_table_kafka` resource instead.
+- `include_timestamp_alias` (String, Deprecated) Provide an alias for the timestamp column. Deprecated: Use the new `materialize_source_table_kafka` resource instead.
 - `key_format` (Block List, Max: 1) Set the key format explicitly. (see [below for nested schema](#nestedblock--key_format))
 - `ownership_role` (String) The owernship role of the object.
 - `region` (String) The region to use for the resource connection. If not set, the default region is used.

docs/resources/source_mysql.md

@@ -53,12 +53,12 @@ resource "materialize_source_mysql" "test" {
 - `comment` (String) Comment on an object in the database.
 - `database_name` (String) The identifier for the source database in Materialize. Defaults to `MZ_DATABASE` environment variable if set or `materialize` if environment variable is not set.
 - `expose_progress` (Block List, Max: 1) The name of the progress collection for the source. If this is not specified, the collection will be named `<src_name>_progress`. (see [below for nested schema](#nestedblock--expose_progress))
-- `ignore_columns` (List of String, Deprecated) Ignore specific columns when reading data from MySQL. Can only be updated in place when also updating a corresponding `table` attribute. Deprecated: Use the new materialize_source_table_mysql resource instead.
+- `ignore_columns` (List of String, Deprecated) Ignore specific columns when reading data from MySQL. Can only be updated in place when also updating a corresponding `table` attribute. Deprecated: Use the new `materialize_source_table_mysql` resource instead.
 - `ownership_role` (String) The owernship role of the object.
 - `region` (String) The region to use for the resource connection. If not set, the default region is used.
 - `schema_name` (String) The identifier for the source schema in Materialize. Defaults to `public`.
-- `table` (Block Set, Deprecated) Specify the tables to be included in the source. Deprecated: Use the new materialize_source_table_mysql resource instead. (see [below for nested schema](#nestedblock--table))
-- `text_columns` (List of String, Deprecated) Decode data as text for specific columns that contain MySQL types that are unsupported in Materialize. Can only be updated in place when also updating a corresponding `table` attribute. Deprecated: Use the new materialize_source_table_mysql resource instead.
+- `table` (Block Set, Deprecated) Specify the tables to be included in the source. Deprecated: Use the new `materialize_source_table_mysql` resource instead. (see [below for nested schema](#nestedblock--table))
+- `text_columns` (List of String, Deprecated) Decode data as text for specific columns that contain MySQL types that are unsupported in Materialize. Can only be updated in place when also updating a corresponding `table` attribute. Deprecated: Use the new `materialize_source_table_mysql` resource instead.
 
 ### Read-Only
 

docs/resources/source_postgres.md

@@ -62,8 +62,8 @@ resource "materialize_source_postgres" "example_source_postgres" {
 - `ownership_role` (String) The owernship role of the object.
 - `region` (String) The region to use for the resource connection. If not set, the default region is used.
 - `schema_name` (String) The identifier for the source schema in Materialize. Defaults to `public`.
-- `table` (Block Set, Deprecated) Creates subsources for specific tables in the Postgres connection. Deprecated: Use the new materialize_source_table_postgres resource instead. (see [below for nested schema](#nestedblock--table))
-- `text_columns` (List of String, Deprecated) Decode data as text for specific columns that contain PostgreSQL types that are unsupported in Materialize. Can only be updated in place when also updating a corresponding `table` attribute. Deprecated: Use the new materialize_source_table_postgres resource instead.
+- `table` (Block Set, Deprecated) Creates subsources for specific tables in the Postgres connection. Deprecated: Use the new `materialize_source_table_postgres` resource instead. (see [below for nested schema](#nestedblock--table))
+- `text_columns` (List of String, Deprecated) Decode data as text for specific columns that contain PostgreSQL types that are unsupported in Materialize. Can only be updated in place when also updating a corresponding `table` attribute. Deprecated: Use the new `materialize_source_table_postgres` resource instead.
 
 ### Read-Only