Skip to content

Commit 6555c4e

Browse files
liljack17liljack17
authored
Merge pull request #205643 from nayakshweta/shwetn-container-copy-actual
Shwetn container copy actual
2 parents 45e21ff + f509dc2 commit 6555c4e

File tree

5 files changed

+269
-10
lines changed

5 files changed

+269
-10
lines changed

‎articles/cosmos-db/.openpublishing.redirection.cosmos-db.json

Lines changed: 0 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -3625,16 +3625,6 @@
36253625
"redirect_url": "/azure/cosmos-db/sql/defender-for-cosmos-db",
36263626
"redirect_document_id": false
36273627
},
3628-
{
3629-
"source_path_from_root": "/articles/cosmos-db/how-to-container-copy.md",
3630-
"redirect_url": "/azure/cosmos-db/",
3631-
"redirect_document_id": false
3632-
},
3633-
{
3634-
"source_path_from_root": "/articles/cosmos-db/intra-account-container-copy.md",
3635-
"redirect_url": "/azure/cosmos-db/",
3636-
"redirect_document_id": false
3637-
},
36383628
{
36393629
"source_path_from_root": "/articles/cosmos-db/sql/create-sql-api-dotnet-v4.md",
36403630
"redirect_url": "/azure/cosmos-db/sql/quickstart-dotnet",

‎articles/cosmos-db/TOC.yml

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -161,6 +161,8 @@
161161
href: data-residency.md
162162
- name: Automatic recommendations
163163
href: automated-recommendations.md
164+
- name: Move data within Azure Cosmos DB
165+
href: intra-account-container-copy.md
164166
- name: SQL/Core API
165167
href: sql/index.yml
166168
- name: Cassandra API
@@ -362,6 +364,9 @@
362364
href: monitor-cosmos-db-reference.md
363365
- name: Third-party monitoring solutions
364366
href: monitoring-solutions.md
367+
- name: Move data within Cosmos DB
368+
displayName: Move data within Cosmos DB
369+
href: how-to-container-copy.md
365370
- name: Migrate data to Cosmos DB
366371
items:
367372
- name: Options to migrate data into Cosmos DB

‎articles/cosmos-db/how-to-container-copy.md

Lines changed: 123 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,123 @@
1+
---
2+
title: Create and manage intra-account container copy jobs in Azure Cosmos DB
3+
description: Learn how to create, monitor, and manage container copy jobs within an Azure Cosmos DB account using CLI commands.
4+
author: nayakshweta
5+
ms.service: cosmos-db
6+
ms.topic: how-to
7+
ms.date: 08/01/2022
8+
ms.author: shwetn
9+
ms.reviewer: sidandrews
10+
---
11+
12+
# Create and manage intra-account container copy jobs in Azure Cosmos DB (Preview)
13+
[!INCLUDE[appliesto-sql-cassandra-api](includes/appliesto-sql-cassandra-api.md)]
14+
15+
[Container copy jobs](intra-account-container-copy.md) help create offline copies of containers within an Azure Cosmos DB account.
16+
17+
This article describes how to create, monitor, and manage intra-account container copy jobs using Azure PowerShell or CLI commands.
18+
19+
## Pre-requisites
20+
21+
* You may use the portal [Cloud Shell](../cloud-shell/quickstart-powershell.md#start-cloud-shell) to run container copy commands. Alternately, you may run the commands locally; make sure you have [Azure CLI](/cli/azure/install-azure-cli) or [Azure PowerShell](/powershell/azure/install-az-ps-msi) downloaded and installed on your machine.
22+
* Currently, container copy is only supported in [these regions](intra-account-container-copy.md#supported-regions). Make sure your account's write region belongs to this list.
23+
24+
25+
## Install the Cosmos DB preview extension
26+
27+
This extension contains the container copy commands.
28+
29+
```azurepowershell-interactive
30+
az extension add --name cosmosdb-preview
31+
```
32+
33+
## Set shell variables
34+
35+
First, set all of the variables that each individual script will use.
36+
37+
```azurepowershell-interactive
38+
$resourceGroup = "<resource-group-name>"
39+
$accountName = "<cosmos-account-name>"
40+
$jobName = ""
41+
$sourceDatabase = ""
42+
$sourceContainer = ""
43+
$destinationDatabase = ""
44+
$destinationContainer = ""
45+
```
46+
47+
## Create an intra-account container copy job for SQL API account
48+
49+
Create a job to copy a container within an Azure Cosmos DB SQL API account:
50+
51+
```azurepowershell-interactive
52+
az cosmosdb dts copy `
53+
--resource-group $resourceGroup `
54+
--account-name $accountName `
55+
--job-name $jobName `
56+
--source-sql-container database=$sourceDatabase container=$sourceContainer `
57+
--dest-sql-container database=$destinationDatabase container=$destinationContainer
58+
```
59+
60+
## Create intra-account container copy job for Cassandra API account
61+
62+
Create a job to copy a container within an Azure Cosmos DB Cassandra API account:
63+
64+
```azurepowershell-interactive
65+
az cosmosdb dts copy `
66+
--resource-group $resourceGroup `
67+
--account-name $accountName `
68+
--job-name $jobName `
69+
--source-cassandra-table keyspace=$sourceKeySpace table=$sourceTable `
70+
--dest-cassandra-table keyspace=$destinationKeySpace table=$destinationTable
71+
```
72+
**Note**: *'--job-name'* should be unique for each job within an account.
73+
74+
## Monitor the progress of a container copy job
75+
76+
View the progress and status of a copy job:
77+
78+
```azurepowershell-interactive
79+
az cosmosdb dts show `
80+
--resource-group $resourceGroup `
81+
--account-name $accountName `
82+
--job-name $jobName
83+
```
84+
85+
## List all the container copy jobs created in an account
86+
87+
To list all the container copy jobs created in an account:
88+
89+
```azurepowershell-interactive
90+
az cosmosdb dts list `
91+
--resource-group $resourceGroup `
92+
--account-name $accountName
93+
```
94+
95+
## Pause a container copy job
96+
97+
In order to pause an ongoing container copy job, you may use the command:
98+
99+
```azurepowershell-interactive
100+
az cosmosdb dts pause `
101+
--resource-group $resourceGroup `
102+
--account-name $accountName `
103+
--job-name $jobName
104+
```
105+
106+
## Resume a container copy job
107+
108+
In order to resume an ongoing container copy job, you may use the command:
109+
110+
```azurepowershell-interactive
111+
az cosmosdb dts resume `
112+
--resource-group $resourceGroup `
113+
--account-name $accountName `
114+
--job-name $jobName
115+
```
116+
117+
## Get support for container copy issues
118+
For issues related to intra-account container copy, please raise a New Support Request from the Azure Portal with the Problem Type as 'Data Migration' and Problem subtype as 'Intra-account container copy'.
119+
120+
121+
## Next steps
122+
123+
- For more information about intra-account container copy jobs, see [Container copy jobs](intra-account-container-copy.md).

‎articles/cosmos-db/includes/appliesto-sql-cassandra-api.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
APPLIES TO: :::image type="icon" source="../media/applies-to/yes.png" border="false":::SQL API :::image type="icon" source="../media/applies-to/yes.png" border="false":::Cassandra API

‎articles/cosmos-db/intra-account-container-copy.md

Lines changed: 140 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,140 @@
1+
---
2+
title: Intra-account container copy jobs in Azure Cosmos DB
3+
description: Learn about container data copy capability within an Azure Cosmos DB account.
4+
author: nayakshweta
5+
ms.service: cosmos-db
6+
ms.topic: conceptual
7+
ms.date: 08/1/2022
8+
ms.author: shwetn
9+
ms.reviewer: sidandrews
10+
ms.custom: references_regions
11+
---
12+
13+
# Intra-account container copy jobs in Azure Cosmos DB (Preview)
14+
[!INCLUDE[appliesto-sql-cassandra-api](includes/appliesto-sql-cassandra-api.md)]
15+
16+
You can perform offline container copy within an Azure Cosmos DB account using container copy jobs.
17+
18+
You may need to copy data within your Azure Cosmos DB account if you want to achieve any of these scenarios:
19+
20+
* Copy all items from one container to another.
21+
* Change the [granularity at which throughput is provisioned - from database to container](set-throughput.md) and vice-versa.
22+
* Change the [partition key](partitioning-overview.md#choose-partitionkey) of a container.
23+
* Update the [unique keys](unique-keys.md) for a container.
24+
* Rename a container/database.
25+
* Adopt new features that are only supported on new containers.
26+
27+
Intra-account container copy jobs can be [created and managed using CLI commands](how-to-container-copy.md).
28+
29+
## Get started
30+
31+
To get started using container copy jobs, register for "Intra-account offline container copy (Cassandra & SQL)" preview from the ['Preview Features'](access-previews.md) list in the Azure portal. Once the registration is complete, the preview will be effective for all Cassandra and SQL API accounts in the subscription.
32+
33+
## Overview of steps needed to do container copy
34+
35+
1. Create the target Cosmos DB container with the desired settings (partition key, throughput granularity, RUs, unique key, etc.).
36+
2. Stop the operations on the source container by pausing the application instances or any clients connecting to it.
37+
3. [Create the container copy job](how-to-container-copy.md).
38+
4. [Monitor the progress of the container copy job](how-to-container-copy.md#monitor-the-progress-of-a-container-copy-job) and wait until it's completed.
39+
5. Resume the operations by appropriately pointing the application or client to the source or target container copy as intended.
40+
41+
## How does intra-account container copy work?
42+
43+
Intra-account container copy jobs perform offline data copy using the source container's incremental change feed log.
44+
45+
* The platform allocates server-side compute instances for the Azure Cosmos DB account.
46+
* These instances are allocated when one or more container copy jobs are created within the account.
47+
* The container copy jobs run on these instances.
48+
* A single job is executed across all instances at any time.
49+
* The instances are shared by all the container copy jobs running within the same account.
50+
* The platform may de-allocate the instances if they're idle for >15 mins.
51+
52+
> [!NOTE]
53+
> We currently only support offline container copy jobs. So, we strongly recommend to stop performing any operations on the source container prior to beginning the container copy.\
54+
> Item deletions and updates done on the source container after beginning the copy job may not be captured. Hence, continuing to perform operations on the source container while the container job is in progress may result in additional or missing data on the target container.
55+
56+
57+
## Factors affecting the rate of a container copy job
58+
59+
The rate of container copy job progress is determined by these factors:
60+
61+
* Source container/database throughput setting.
62+
63+
* Target container/database throughput setting.
64+
65+
* Server-side compute instances allocated to the Azure Cosmos DB account for performing the data transfer.
66+
67+
> [!IMPORTANT]
68+
> The default SKU offers two 4-vCPU 16-GB server-side instances per account.
69+
70+
## FAQs
71+
72+
### Is there an SLA for the container copy jobs?
73+
74+
Container copy jobs are currently supported on best-effort basis. We don't provide any SLA guarantees for the time taken to complete these jobs.
75+
76+
### Can I create multiple container copy jobs within an account?
77+
78+
Yes, you can create multiple jobs within the same account. The jobs will run consecutively. You can [list all the jobs](how-to-container-copy.md#list-all-the-container-copy-jobs-created-in-an-account) created within an account and monitor their progress.
79+
80+
### Can I copy an entire database within the Azure Cosmos DB account?
81+
82+
You'll have to create a job for each container in the database.
83+
84+
### I have an Azure Cosmos DB account with multiple regions. In which region will the container copy job run?
85+
86+
The container copy job will run in the write region. If there are accounts configured with multi-region writes, the job will run in one of the regions from the list.
87+
88+
### What happens to the container copy jobs when the account's write region changes?
89+
90+
The account's write region may change in the rare scenario of a region outage or due to manual failover. In such a scenario, incomplete container copy jobs created within the account would fail. You would need to recreate these failed jobs. Recreated jobs would then run in the new (current) write region.
91+
92+
### Why is a new database *_datatransferstate* created in the account when I run container copy jobs? Am I being charged for this database?
93+
* *_datatransferstate* is a database that is created while running container copy jobs. This database is used by the platform to store the state and progress of the copy job.
94+
* The database uses manual provisioned throughput of 800 RUs. You'll be charged for this database.
95+
* Deleting this database will remove the container copy job history from the account. It can be safely deleted once all the jobs in the account have completed, if you no longer need the job history. The platform will not clean up the *_datatransferstate* database automatically.
96+
97+
## Supported regions
98+
99+
Currently, container copy is supported in the following regions:
100+
101+
| **Americas** | **Europe and Africa** | **Asia Pacific** |
102+
| ------------ | -------- | ----------- |
103+
| Brazil South | France Central | Australia Central |
104+
| Canada Central | France South | Australia Central 2 |
105+
| Canada East | Germany North | Australia East |
106+
| Central US | Germany West Central | Central India |
107+
| Central US EUAP | North Europe | Japan East |
108+
| East US | Norway East | Korea Central |
109+
| East US 2 | Norway West | Southeast Asia |
110+
| East US 2 EUAP | Switzerland North | UAE Central |
111+
| North Central US | Switzerland West | West India |
112+
| South Central US | UK South | |
113+
| West Central US | UK West | |
114+
| West US | West Europe |
115+
| West US 2 | |
116+
117+
## Known/common issues
118+
119+
* Error - Owner resource does not exist
120+
121+
If the job creation fails with the error *"Owner resource does not exist"*, it means that the target container wasn't created or was mis-spelt.
122+
Make sure the target container is created before running the job as specified in the [overview section.](#overview-of-steps-needed-to-do-container-copy)
123+
124+
```
125+
"code": "500",
126+
"message": "Response status code does not indicate success: NotFound (404); Substatus: 1003; ActivityId: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx; Reason: (Message: {\"Errors\":[\"Owner resource does not exist\"]
127+
```
128+
129+
* Error - Shared throughput database creation is not supported for serverless accounts
130+
131+
Job creation on serverless accounts may fail with the error *"Shared throughput database creation is not supported for serverless accounts"*.
132+
As a work-around, create a database called *_datatransferstate* manually within the account and try creating the container copy job again.
133+
134+
```
135+
ERROR: (BadRequest) Response status code does not indicate success: BadRequest (400); Substatus: 0; ActivityId: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx; Reason: (Shared throughput database creation is not supported for serverless accounts.
136+
```
137+
138+
## Next steps
139+
140+
- You can learn [how to create, monitor and manage container copy jobs within Azure Cosmos DB account using CLI commands](how-to-container-copy.md).

0 commit comments

Comments
 (0)