Skip to content

Commit ee6d0aa

Browse files
authored
docs: clarify alter source add subsource (#34123)
1 parent 225e5b4 commit ee6d0aa

File tree

7 files changed

+53
-4
lines changed

7 files changed

+53
-4
lines changed

doc/user/content/ingest-data/_index.md

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -78,6 +78,11 @@ operation completes. Once the initial snapshot has been ingested, you can start
7878
querying your (sub)sources and Materialize will continue ingesting any new data
7979
as it arrives, in real time.
8080

81+
### Modifying an existing source
82+
83+
{{< include-md file="shared-content/alter-source-snapshot-blocking-behavior.md"
84+
>}}
85+
8186
## Running/steady-state
8287

8388
Once snapshotting completes, Materialize transitions to Running state. During

doc/user/content/ingest-data/troubleshooting.md

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -98,6 +98,18 @@ also be necessary to support increased memory usage during the process. For more
9898
information, see [Use a larger cluster for upsert source
9999
snapshotting](/ingest-data/#use-a-larger-cluster-for-upsert-source-snapshotting).
100100

101+
## Adding a new subsource to an existing source blocks replication. Should I just create a new source instead?
102+
103+
It depends. Materialize provides transactional guarantees for subsource of the
104+
same source, not across different sources. So, if you need transactional
105+
guarantees across the tables between the two sources, you cannot use a new
106+
source. In addition, creating a new source means that you are reading the
107+
replication stream twice.
108+
109+
To use the same source, consider resizing the cluster to speed up the
110+
snapshotting process for the new subsource and once the process finishes, resize
111+
the cluster for steady-state.
112+
101113
## See also
102114

103115
- [Monitoring data ingestion](/ingest-data/monitoring-data-ingestion/)

doc/user/content/sql/alter-source.md

Lines changed: 15 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -31,7 +31,8 @@ menu:
3131
Field | Use
3232
--------|-----
3333
_name_ | The identifier of the source you want to alter.
34-
**ADD SUBSOURCE** ... | Add the identified tables from the upstream database (`table_name`) to the named PostgreSQL or MySQL source, with the option of choosing the name for the subsource in Materialize (`subsrc_name`). Supports [additional options](#add-subsource-with_options).
34+
**ADD SUBSOURCE** ... | Add the identified tables from the upstream database (`table_name`) to the named PostgreSQL/MySQL/SQL Server source, with the option of choosing the name for the subsource in Materialize (`subsrc_name`). Supports [additional options](#add-subsource-with_options). <br><br>{{< include-md file="shared-content/alter-source-snapshot-blocking-behavior.md"
35+
>}}
3536
_retention_period_ | ***Private preview.** This option has known performance or stability issues and is under active development.* Duration for which Materialize retains historical data, which is useful to implement [durable subscriptions](/transform-data/patterns/durable-subscriptions/#history-retention-period). Accepts positive [interval](/sql/types/interval/) values (e.g. `'1hr'`). Default: `1s`.
3637

3738
### **ADD SUBSOURCE** `with_options`
@@ -42,12 +43,17 @@ Field | Value | Description
4243

4344
## Context
4445

45-
### Adding subsources to a PostgreSQL or MySQL source
46+
### Adding subsources to a PostgreSQL/MySQL/SQL Server source
4647

4748
Note that using a combination of dropping and adding subsources lets you change
48-
the schema of the PostgreSQL or MySQL tables that are ingested.
49+
the schema of the PostgreSQL/MySQL/SQL Server tables that are ingested.
4950

50-
### Dropping subsources from a PostgreSQL or MySQL source
51+
{{< important >}}
52+
{{< include-md file="shared-content/alter-source-snapshot-blocking-behavior.md"
53+
>}}
54+
{{< /important >}}
55+
56+
### Dropping subsources from a PostgreSQL/MySQL/SQL Server source
5157

5258
Dropping a subsource prevents Materialize from ingesting any data from it, in
5359
addition to dropping any state that Materialize previously had for the table
@@ -67,6 +73,11 @@ You cannot drop the "progress subsource".
6773
ALTER SOURCE pg_src ADD SUBSOURCE tbl_a, tbl_b AS b WITH (TEXT COLUMNS [tbl_a.col]);
6874
```
6975

76+
{{< important >}}
77+
{{< include-md file="shared-content/alter-source-snapshot-blocking-behavior.md"
78+
>}}
79+
{{< /important >}}
80+
7081
### Dropping subsources
7182

7283
To drop a subsource, use the [`DROP SOURCE`](/sql/drop-source/) command:
Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
When you add a new subsource to an existing source ([`ALTER SOURCE ... ADD
2+
SUBSOURCE ...`](/sql/alter-source/)), Materialize starts the snapshotting
3+
process for the new subsource. During this snapshotting, the data ingestion for
4+
the existing subsources for the same source is temporarily blocked. As such, if
5+
possible, you can resize the cluster to speed up the snapshotting process and
6+
once the process finishes, resize the cluster for steady-state.

doc/user/shared-content/mysql-considerations.md

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,3 +20,8 @@ truncating, you can use an unqualified `DELETE` to remove all rows from the tabl
2020
```mzsql
2121
DELETE FROM t;
2222
```
23+
24+
### Modifying an existing source
25+
26+
{{< include-md file="shared-content/alter-source-snapshot-blocking-behavior.md"
27+
>}}

doc/user/shared-content/postgres-considerations.md

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,3 +10,8 @@ replication slots:
1010

1111
{{< include-md file="shared-content/postgres-remove-unused-replication-slots.md"
1212
>}}
13+
14+
### Modifying an existing source
15+
16+
{{< include-md file="shared-content/alter-source-snapshot-blocking-behavior.md"
17+
>}}

doc/user/shared-content/sql-server-considerations.md

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -54,3 +54,8 @@ system table. For each table, Materialize picks the capture instance with the
5454
most recent `create_date`.
5555

5656
If two capture instances for a table share the same timestamp (unlikely given the millisecond resolution), Materialize selects the `capture_instance` with the lexicographically larger name.
57+
58+
### Modifying an existing source
59+
60+
{{< include-md file="shared-content/alter-source-snapshot-blocking-behavior.md"
61+
>}}

0 commit comments

Comments
 (0)