Single source mount/unmount #874
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -4,4 +4,6 @@ | |
| :page-categories: Management | ||
| :env-linux: true | ||
|
|
||
| For topics with Tiered Storage enabled, you can unmount a topic to safely detach it from a cluster and keep the topic data in the cluster's object storage bucket or container. You can mount the detached topic to either the same origin cluster, or a different one. This allows you to hibernate a topic and free up system resources taken up by the topic, or migrate a topic to a different cluster. | ||
|
Contributor
There was a problem hiding this comment.
Afaik cross-cluster mounting/unmounting is not supported in Cloud and definetely not supported via the cloud api
Contributor
Author
There was a problem hiding this comment. Thanks @weeco I do have a modified version of this intro for the Cloud page. Preview here: https://deploy-preview-874--redpanda-docs-preview.netlify.app/redpanda-cloud/manage/mountable-topics/ |
||
|
|
||
| include::manage:partial$mountable-topics.adoc[] | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,14 +1,21 @@ | ||
| == Prerequisites | ||
|
|
||
| [NOTE] | ||
| ==== | ||
| include::shared:partial$enterprise-license.adoc[] | ||
|
Contributor
Author
There was a problem hiding this comment. @mattschumpert we do not have licensing content for the cloud docs like we do for self-managed. Should I remove/hide the licensing prereq for the Cloud version of this page?
Contributor
Author
There was a problem hiding this comment. cc @towfiqa if you could provide guidance for this as well. |
||
| ==== | ||
|
|
||
| ifndef::env-cloud[] | ||
| . xref:get-started:rpk-install.adoc[Install `rpk`], or ensure that you have access to the Admin API. | ||
| . Enable xref:manage:tiered-storage.adoc[Tiered Storage] for specific topics, or for the entire cluster (all topics). | ||
| endif::[] | ||
| ifdef::env-cloud[] | ||
| . xref:manage:rpk/rpk-install.adoc[Install `rpk`] to run cluster mount and unmount operations on the command line, or xref:manage:api/cloud-api-authentication.adoc[authenticate] to the Cloud API and execute these operations programmatically. | ||
| + | ||
| If using the Cloud API, you'll make requests against Data Plane APIs. Make sure you xref:manage:api/cloud-dataplane-api.adoc#get-data-plane-api-url[retrieve the Data Plane API URL] and use this URL to perform a mount or unmount operation. | ||
| . Enable xref:ROOT:manage:tiered-storage.adoc[Tiered Storage] for the topics you want to unmount or mount. | ||
| endif::[] | ||
|
|
||
|
|
||
| == Unmount a topic from a cluster to object storage | ||
|
|
||
|
|
@@ -32,6 +39,7 @@ In your cluster, run this command to unmount a topic to object storage: | |
| rpk cluster storage unmount <namespace>/<topic-name> | ||
| ``` | ||
| -- | ||
| ifndef::env-cloud[] | ||
| Admin API:: | ||
| + | ||
| -- | ||
|
|
@@ -55,9 +63,27 @@ curl -X POST http://localhost:9644/v1/topics/unmount -d { | |
|
|
||
| You may optionally include the topic namespace (`ns`). Only `kafka` is supported. | ||
| -- | ||
| endif::[] | ||
| ifdef::env-cloud[] | ||
| Cloud API:: | ||
| + | ||
| -- | ||
| To unmount topics from a cluster using the Cloud API, issue a POST request to the `/v1alpha2/cloud-storage/unmount` endpoint. Specify the names of the desired topics in the request body: | ||
|
|
||
| [,bash] | ||
| ---- | ||
| curl -X POST "<dataplane-api-url>/v1alpha2/cloud-storage/topics/unmount" \ | ||
| -H "Authorization: Bearer <token>" \ | ||
| -H "accept: application/json" \ | ||
| -H "content-type: application/json" \ | ||
| -d '{"topics":"<topic-name>"}' | ||
| ---- | ||
|
|
||
| -- | ||
| endif::[] | ||
| ====== | ||
|
|
||
| You can use the ID returned by the command to <<monitor-progress,monitor the progress>> of the unmount operation using `rpk` or the API. | ||
|
|
||
| == Mount a topic to a cluster | ||
|
|
||
|
|
@@ -99,6 +125,7 @@ rpk cluster storage mount <topic-reference> --to <new-topic-name> | |
| + | ||
| You only use the new name for the topic in the target cluster. This name does not persist if you unmount this topic again. Redpanda keeps the original name in object storage if you remount the topic later. | ||
| -- | ||
| ifndef::env-cloud[] | ||
| Admin API:: | ||
| + | ||
| -- | ||
|
|
@@ -110,7 +137,7 @@ curl http://localhost:9644/v1/topics/mountable | |
| + | ||
| The response object contains an array of topics: | ||
| + | ||
| [,bash] | ||
| ---- | ||
| "topics": [ | ||
| { | ||
|
|
@@ -133,28 +160,74 @@ curl -X POST http://localhost:9644/v1/topics/mount -d { | |
| "topics": [ | ||
| { | ||
| "source_topic_reference": {"ns": "kafka", "topic": "<topic-1-name>/<cluster-1-uuid>/<initial-revision>"}, | ||
| "alias": {"topic": "<new-topic-1-name>"} | ||
| }, | ||
| { | ||
| "source_topic_reference": {"ns": "kafka", "topic": "<topic-2-name>"} | ||
| } | ||
| ] | ||
| } | ||
| ``` | ||
| + | ||
| * `ns` is the topic namespace. This field is optional and only `kafka` is supported. | ||
| * You may have multiple topics with the same name that are available to mount from object storage. This can happen if you have unmounted topics using the same name in different clusters. To uniquely identify a source topic, use `<topic-name>/<cluster-uuid>/<initial-revision>` as the topic reference. | ||
| * To rename a topic in the target cluster, use the optional `alias` object in the request body. The following example shows how to specify a new name for topic 1 in the target cluster, while topic 2 retains its original name in the target cluster. | ||
|
|
||
| -- | ||
| endif::[] | ||
| ifdef::env-cloud[] | ||
| Cloud API:: | ||
| + | ||
| -- | ||
| . List the topics that are available to mount from object storage by making a GET request to the `/v1alpha2/cloud-storage/topics/mountable` endpoint. | ||
| + | ||
| ``` | ||
| curl "<dataplane-api-url>/v1alpha2/cloud-storage/topics/mountable" | ||
| ``` | ||
| + | ||
| The response object contains an array of topics: | ||
| + | ||
| [,bash,role=no-placeholders] | ||
| ---- | ||
| "topics": [ | ||
| { | ||
| "name": "topic-1-name", | ||
| "topic_location": "topic-1-name/<cluster-1-uuid>/<initial-revision>" | ||
| }, | ||
| { | ||
| "name": "topic-2-name", | ||
| "topic_location": "topic-2-name/<cluster-1-uuid>/<initial-revision>" | ||
| } | ||
| ] | ||
| ---- | ||
| + | ||
| The `topic_location` is the unique topic location in object storage, in the format `<topic-name>/<cluster-uuid>/<initial-revision>`. Redpanda assigns the number `initial-revision` to a topic upon creation. You can use `topic-location` as the topic reference instead of just the topic name to identify a unique topic to mount in the next step. | ||
|
|
||
| . To mount topics to a target cluster using the Cloud API, make a POST request to the `/cloud-storage/topics/mount` endpoint. Specify the names of the topics in the request body: | ||
| + | ||
| ``` | ||
| curl -X POST "<dataplane-api-url>/v1alpha2/cloud-storage/topics/mount" -d { | ||
| "topics": [ | ||
| { | ||
| "alias": "<new-topic-1-name>", | ||
| "source_topic_reference": "<topic-1-name>/<cluster-1-uuid>/<initial-revision>" | ||
| }, | ||
| { | ||
| "source_topic_reference": "<topic-2-name>" | ||
| } | ||
| ] | ||
| } | ||
| ``` | ||
| + | ||
| * You may have multiple topics with the same name that are available to mount from object storage. This can happen if you have unmounted topics with this name from different clusters. To uniquely identify a source topic, use `<topic-name>/<cluster-uuid>/<initial-revision>` as the topic reference. | ||
| * To rename a topic in the target cluster, use the optional `alias` object in the request body. The following example shows how to specify a new name for topic 1 in the target cluster, while topic 2 retains its original name in the target cluster. | ||
|
|
||
| -- | ||
| endif::[] | ||
|
|
||
| ====== | ||
|
|
||
| You can use the ID returned by the operation to <<monitor-progress,monitor its progress>> using `rpk` or the API. | ||
|
|
||
| When the mount operation is complete, the target cluster handles produce and consume workloads for the topics. | ||
|
|
||
|
|
@@ -172,6 +245,7 @@ rpk cluster storage list-mount | |
| ``` | ||
| -- | ||
|
|
||
| ifndef::env-cloud[] | ||
| Admin API:: | ||
| + | ||
| -- | ||
|
|
@@ -181,6 +255,23 @@ Issue a GET request to the `/migrations` endpoint to view the status of topic mo | |
| curl http://localhost:9644/v1/migrations | ||
| ``` | ||
| -- | ||
| endif::[] | ||
|
|
||
| ifdef::env-cloud[] | ||
| Cloud API:: | ||
| + | ||
| -- | ||
| Issue a GET request to the `/cloud-storage/mount-tasks` endpoint to view the status of topic mount and unmount operations: | ||
|
|
||
| [,bash] | ||
| ---- | ||
| curl "<dataplane-api-url>/v1alpha2/cloud-storage/mount-tasks" \ | ||
| -H "Authorization: Bearer <token>" \ | ||
| -H "accept: application/json" | ||
| ---- | ||
|
|
||
| -- | ||
| endif::[] | ||
| ====== | ||
|
|
||
| You can also retrieve the status of a specific operation by running the command: | ||
|
|
@@ -195,13 +286,28 @@ rpk:: | |
| rpk cluster storage status-mount <migration-id> | ||
| ``` | ||
| -- | ||
| ifndef::env-cloud[] | ||
| Admin API:: | ||
| + | ||
| -- | ||
| ``` | ||
| curl http://localhost:9644/v1/migrations/<migration-id> | ||
| ``` | ||
| -- | ||
| endif::[] | ||
|
|
||
| ifdef::env-cloud[] | ||
| Cloud API:: | ||
| + | ||
| -- | ||
| [,bash] | ||
| ---- | ||
| curl "<dataplane-api-url>/v1alpha2/cloud-storage/mount-tasks/<migration-id>" \ | ||
| -H "Authorization: Bearer <token>" | ||
| ---- | ||
|
|
||
| -- | ||
| endif::[] | ||
| ====== | ||
|
|
||
| `<migration-id>` is the unique identifier of the operation. Redpanda returns this ID when you start a mount or unmount. You can also retrieve the ID by listing <<monitor-progress,existing operations>>. | ||
|
|
@@ -260,18 +366,40 @@ rpk cluster storage cancel-mount <migration-id> | |
| ``` | ||
| -- | ||
|
|
||
| ifndef::env-cloud[] | ||
| Admin API:: | ||
| + | ||
| -- | ||
| ``` | ||
| curl -X POST http://localhost:9644/v1/<migration-id>/?action=cancel | ||
| ``` | ||
| -- | ||
| endif::[] | ||
| ifdef::env-cloud[] | ||
| Cloud API:: | ||
| + | ||
| -- | ||
| [,bash] | ||
| ---- | ||
| curl -X POST "<dataplane-api-url>/v1alpha2/cloud-storage/mount-tasks/<id>" \ | ||
| -H "Authorization: Bearer <token>" \ | ||
| -H "accept: application/json" \ | ||
| -H "content-type: application/json" \ | ||
| -d '{"action":"ACTION_CANCEL"}' | ||
| ---- | ||
|
|
||
| -- | ||
| endif::[] | ||
| ====== | ||
|
|
||
| You cannot cancel mount and unmount operations in the following <<monitor-progress,states>>: | ||
|
|
||
| ifndef::env-cloud[] | ||
| - `planned` (but you may still xref:api:ROOT:admin-api.adoc#delete-/v1/migrations/-id-[delete] a planned mount or unmount) | ||
| endif::[] | ||
| ifdef::env-cloud[] | ||
| - `planned` (but you may still delete a planned mount or unmount) | ||
| endif::[] | ||
| - `cut_over` | ||
| - `finished` | ||
| - `canceling` | ||
|
|
||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@towfiqa This is the overview for the self-managed version of the doc. The cloud version will have this, could you review and let me know if any changes are needed?:
(preview available here)