|
| 1 | +--- |
| 2 | +title: Cosmos DB Migration options |
| 3 | +description: This doc describes the various options to migrate your on-premises or cloud data to Azure Cosmos DB |
| 4 | +author: bharathsreenivas |
| 5 | +ms.service: cosmos-db |
| 6 | +ms.topic: conceptual |
| 7 | +ms.date: 10/03/2019 |
| 8 | +ms.author: bharathb |
| 9 | + |
| 10 | +--- |
| 11 | +# Options to migrate your on-premises or cloud data to Azure Cosmos DB |
| 12 | + |
| 13 | +You can load data from various data sources to Azure Cosmos DB. Additionally, since Azure Cosmos DB supports multiple APIs, the targets can be any of the existing APIs. In order to support migration paths from the various sources to the different Azure Cosmos DB APIs, there are multiple solutions that provide specialized handling for each migration path. This document lists the available solutions and describes their advantages and limitations. |
| 14 | + |
| 15 | +## Factors affecting the choice of migration tool |
| 16 | + |
| 17 | +The following factors determine the choice of the migration tool: |
| 18 | +* **Online vs offline migration**: Many migration tools provide a path to do a one-time migration only. This means that the applications accessing the database might experience a period of downtime. Some migration solutions provide a way to do a live migration where there is a replication pipeline set up between the source and the target. |
| 19 | + |
| 20 | +* **Data source**: The existing data can be in various data sources like Oracle DB2, Datastax Cassanda, Azure SQL Server, PostgreSQL, etc. The data can also be in an existing Azure Cosmos DB account and the intent of migration can be to change the data model or repartition the data in a container with a different partition key. |
| 21 | + |
| 22 | +* **Azure Cosmos DB API**: For the SQL API in Azure Cosmos DB, there are a variety of tools developed by the Azure Cosmos DB team which aid in the different migration scenarios. All of the other APIs have their own specialized set of tools developed and maintained by the community. Since Azure Cosmos DB supports these APIs at a wire protocol level, these tools should work as-is while migrating data into Azure Cosmos DB too. However, they might require custom handling for throttles as this concept is specific to Azure Cosmos DB. |
| 23 | + |
| 24 | +* **Size of data**: Most migration tools work very well for smaller datasets. When the data set exceeds a few hundred gigabytes, the choices of migration tools are limited. |
| 25 | + |
| 26 | +* **Expected migration duration**: Migrations can be configured to take place at a slow, incremental pace that consumes less throughput or can consume the entire throughput provisioned on the target Azure Cosmos DB container and complete the migration in less time. |
| 27 | + |
| 28 | +## Azure Cosmos DB SQL API |
| 29 | +|**Migration type**|**Solution**|**Considerations**| |
| 30 | +|---------|---------|---------| |
| 31 | +|Offline|[Data Migration Tool](https://docs.microsoft.com/azure/cosmos-db/import-data)|• Easy to set up and supports multiple sources <br/>• Not suitable for large datasets| |
| 32 | +|Offline|[Azure Data Factory](https://docs.microsoft.com/azure/data-factory/connector-azure-cosmos-db)|• Easy to set up and supports multiple sources <br/>• Makes use of the Azure Cosmos DB bulk executor library <br/>• Suitable for large datasets <br/>• Lack of checkpointing - It means that if an issue occurs during the course of migration, you need to restart the whole migration process<br/>• Lack of a dead letter queue - It means that a few erroneous files can stop the entire migration process.| |
| 33 | +|Offline|[Azure Cosmos DB Spark connector](https://docs.microsoft.com/azure/cosmos-db/spark-connector)|• Makes use of the Azure Cosmos DB bulk executor library <br/>• Suitable for large datasets <br/>• Needs a custom Spark setup <br/>• Spark is sensitive to schema inconsistencies and this can be a problem during migration | |
| 34 | +|Offline|[Custom tool with Cosmos DB bulk executor library](https://docs.microsoft.com/azure/cosmos-db/migrate-cosmosdb-data)|• Provides checkpointing, dead-lettering capabilities which increases migration resiliency <br/>• Suitable for very large datasets (10 TB+) <br/>• Requires custom setup of this tool running as an App Service | |
| 35 | +|Online|[Cosmos DB Functions + ChangeFeed API](https://docs.microsoft.com/azure/cosmos-db/change-feed-functions)|• Easy to set up <br/>• Works only if the source is an Azure Cosmos DB container <br/>• Not suitable for large datasets <br/>• Does not capture deletes from the source container | |
| 36 | +|Online|[Custom Migration Service using ChangeFeed](https://aka.ms/CosmosDBMigrationSample)|• Provides progress tracking <br/>• Works only if the source is an Azure Cosmos DB container <br/>• Works for larger datasets as well <br/>• Requires the user to set up an App Service to host the Change feed processor <br/>• Does not capture deletes from the source container| |
| 37 | +|Online|[Striim](https://docs.microsoft.com/azure/cosmos-db/cosmosdb-sql-api-migrate-data-striim)|• Works with a large variety of sources like Oracle, DB2, SQL Server <br/>• Easy to build ETL pipelines and provides a dashboard for monitoring <br/>• Supports larger datasets <br/>• Since this is a third-party tool, it needs to be purchased from the marketplace and installed in the user's environment| |
| 38 | + |
| 39 | +## Azure Cosmos DB Mongo API |
| 40 | +|**Migration type**|**Solution**|**Considerations**| |
| 41 | +|---------|---------|---------| |
| 42 | +|Offline|[Data Migration Tool](https://docs.microsoft.com/azure/cosmos-db/import-data)|• Easy to set up and supports multiple sources <br/>• Not suitable for large datasets| |
| 43 | +|Offline|[Azure Data Factory](https://docs.microsoft.com/azure/data-factory/connector-azure-cosmos-db)|• Easy to set up and supports multiple sources <br/>• Makes use of the Azure Cosmos DB bulk executor library <br/>• Suitable for large datasets <br/>• Lack of checkpointing means that any issue during the course of migration would require a restart of the whole migration process<br/>• Lack of a dead letter queue would mean that a few erroneous files could stop the entire migration process <br/>• Needs custom code to increase read throughput for certain data sources| |
| 44 | +|Offline|[Existing Mongo Tools (mongodump, mongorestore, Studio3T)](https://azure.microsoft.com/resources/videos/using-mongodb-tools-with-azure-cosmos-db/)|• Easy to set up and integration <br/>• Needs custom handling for throttles| |
| 45 | +|Online|[Azure Database Migration Service](https://docs.microsoft.com/azure/dms/tutorial-mongodb-cosmos-db-online)|• Makes use of the Azure Cosmos DB bulk executor library <br/>• Suitable for large datasets and takes care of replicating live changes <br/>• Works only with other MongoDB sources| |
| 46 | + |
| 47 | +## Azure Cosmos DB Cassandra API |
| 48 | +|**Migration type**|**Solution**|**Considerations**| |
| 49 | +|---------|---------|---------| |
| 50 | +|Offline|[cqlsh COPY command](https://docs.microsoft.com/azure/cosmos-db/cassandra-import-data#migrate-data-using-cqlsh-copy-command)|• Easy to set up <br/>• Not suitable for large datasets <br/>• Works only when the source is a Cassandra table| |
| 51 | +|Offline|[Copy table with Spark](https://docs.microsoft.com/azure/cosmos-db/cassandra-import-data#migrate-data-using-spark) |• Can make use of Spark capabilities to parallelize transformation and ingestion <br/>• Needs configuration with a custom retry policy to handle throttles| |
| 52 | +|Online|[Striim (from Oracle DB/Apache Cassandra)](https://docs.microsoft.com/azure/cosmos-db/cosmosdb-cassandra-api-migrate-data-striim)|• Works with a large variety of sources like Oracle, DB2, SQL Server <br/>• Easy to build ETL pipelines and provides a dashboard for monitoring <br/>• Supports larger datasets <br/>• Since this is a third-party tool, it needs to be purchased from the marketplace and installed in the user's environment| |
| 53 | +|Online|[Blitzz (from Oracle DB/Apache Cassandra)](https://docs.microsoft.com/azure/cosmos-db/oracle-migrate-cosmos-db-blitzz)|<br/>• Supports larger datasets <br/>• Since this is a third-party tool, it needs to be purchased from the marketplace and installed in the user's environment| |
| 54 | + |
| 55 | +## Other APIs |
| 56 | +For APIs other than the SQL API, Mongo API and the Cassandra API, there are various tools supported by each of the API's existing ecosystems. |
| 57 | + |
| 58 | +**Table API** |
| 59 | +* [Data Migration Tool](https://docs.microsoft.com/azure/cosmos-db/table-import#data-migration-tool) |
| 60 | +* [AzCopy](https://docs.microsoft.com/azure/cosmos-db/table-import#migrate-data-by-using-azcopy) |
| 61 | + |
| 62 | +**Gremlin API** |
| 63 | +* [Graph bulk executor library](https://docs.microsoft.com/azure/cosmos-db/bulk-executor-graph-dotnet) |
| 64 | +* [Gremlin Spark](https://github.com/Azure/azure-cosmosdb-spark/blob/2.4/samples/graphframes/main.scala) |
| 65 | + |
| 66 | +## Next steps |
| 67 | + |
| 68 | +* Learn more by trying out the sample applications consuming the bulk executor library in [.NET](bulk-executor-dot-net.md) and [Java](bulk-executor-java.md). |
| 69 | +* The bulk executor library is integrated into the Cosmos DB Spark connector, to learn more, see [Azure Cosmos DB Spark connector](spark-connector.md) article. |
| 70 | +* Contact the Azure Cosmos DB product team by opening a support ticket under the "General Advisory" problem type and "Large (TB+) migrations" problem subtype for additional help with large scale migrations. |
| 71 | +* Use the [Cosmos DB Bootstrap Program](https://azurecosmosdb.github.io/CosmosBootstrap/) to accelerate building or migrating your applications on Azure Cosmos DB. |
| 72 | + |
| 73 | +> [!div class="nextstepaction"] |
| 74 | +> [Cosmos DB Bootstrap Program](https://azurecosmosdb.github.io/CosmosBootstrap/) |
0 commit comments