Merge pull request #210056 from WilliamDAssafMSFT/20220902-storage-dns-limitation

20220902 storage DNS limitation with Synapse COPY

Author: JamesJBarnett

articles/data-factory/connector-troubleshoot-synapse-sql.md

@@ -1,14 +1,17 @@
 ---
 title: Troubleshoot the Azure Synapse Analytics, Azure SQL Database, and SQL Server connectors
 titleSuffix: Azure Data Factory & Azure Synapse
-description: Learn how to troubleshoot issues with the Azure Synapse Analytics, Azure SQL Database, and SQL Server connectors in Azure Data Factory and Azure Synapse Analytics. 
+description: Learn how to troubleshoot issues with the Azure Synapse Analytics, Azure SQL Database, and SQL Server connectors in Azure Data Factory and Azure Synapse Analytics.
 author: jianleishen
+ms.author: jianleishen
+ms.reviewer: joanpo, wiassaf
+ms.date: 09/02/2022
 ms.service: data-factory
 ms.subservice: data-movement
 ms.topic: troubleshooting
-ms.date: 06/29/2022
-ms.author: jianleishen
-ms.custom: has-adal-ref, synapse
+ms.custom:
+  - has-adal-ref
+  - synapse
 ---
 
 # Troubleshoot the Azure Synapse Analytics, Azure SQL Database, and SQL Server connectors in Azure Data Factory and Azure Synapse
@@ -30,7 +33,7 @@ This article provides suggestions to troubleshoot common problems with the Azure
     | If the error message contains the string "SqlException", SQL Database the error indicates that some specific operation failed. | For more information, search by SQL error code in [Database engine errors](/sql/relational-databases/errors-events/database-engine-events-and-errors). For further help, contact Azure SQL support. |
     | If this is a transient issue (for example, an instable network connection), add retry in the activity policy to mitigate. | For more information, see [Pipelines and activities](./concepts-pipelines-activities.md#activity-policy). |
     | If the error message contains the string "Client with IP address '...' is not allowed to access the server", and you're trying to connect to Azure SQL Database, the error is usually caused by an Azure SQL Database firewall issue. | In the Azure SQL Server firewall configuration, enable the **Allow Azure services and resources to access this server** option. For more information, see [Azure SQL Database and Azure Synapse IP firewall rules](/azure/azure-sql/database/firewall-configure). |
-    
+
 ## Error code: SqlOperationFailed
 
 - **Message**: `A database operation failed. Please search error to get more details.`
@@ -44,7 +47,6 @@ This article provides suggestions to troubleshoot common problems with the Azure
     | If the error message contains the string "InvalidOperationException", it's usually caused by invalid input data. | To identify which row has encountered the problem, enable the fault tolerance feature on the copy activity, which can redirect problematic rows to the storage for further investigation. For more information, see [Fault tolerance of copy activity](./copy-activity-fault-tolerance.md). |
     | If the error message contains "Execution Timeout Expired", it's usually caused by query timeout. | Configure **Query timeout** in the source and **Write batch timeout** in the sink to increase timeout. |
 
-
 ## Error code: SqlUnauthorizedAccess
 
 - **Message**: `Cannot connect to '%connectorName;'. Detail Message: '%message;'`
@@ -53,7 +55,6 @@ This article provides suggestions to troubleshoot common problems with the Azure
 
 - **Recommendation**:  Check to ensure that the login account has sufficient permissions to access the SQL database.
 
-
 ## Error code: SqlOpenConnectionTimeout
 
 - **Message**: `Open connection to database timeout after '%timeoutValue;' seconds.`
@@ -62,7 +63,6 @@ This article provides suggestions to troubleshoot common problems with the Azure
 
 - **Recommendation**:  Retry the operation to update the linked service connection string with a larger connection timeout value.
 
-
 ## Error code: SqlAutoCreateTableTypeMapFailed
 
 - **Message**: `Type '%dataType;' in source side cannot be mapped to a type that supported by sink side(column name:'%columnName;') in autocreate table.`
@@ -71,7 +71,6 @@ This article provides suggestions to troubleshoot common problems with the Azure
 
 - **Recommendation**:  Update the column type in *mappings*, or manually create the sink table in the target server.
 
-
 ## Error code: SqlDataTypeNotSupported
 
 - **Message**: `A database operation failed. Check the SQL errors.`
@@ -84,7 +83,6 @@ This article provides suggestions to troubleshoot common problems with the Azure
 
 - **Recommendation**:  Update the corresponding column type to the *datetime2* type in the sink table.
 
-
 ## Error code: SqlInvalidDbStoredProcedure
 
 - **Message**: `The specified Stored Procedure is not valid. It could be caused by that the stored procedure doesn't return any data. Invalid Stored Procedure script: '%scriptName;'.`
@@ -104,7 +102,6 @@ This article provides suggestions to troubleshoot common problems with the Azure
 
 - **Recommendation**:  Validate the SQL query by using SQL Tools. Make sure that the query can return data.
 
-
 ## Error code: SqlInvalidColumnName
 
 - **Message**: `Column '%column;' does not exist in the table '%tableName;', ServerName: '%serverName;', DatabaseName: '%dbName;'.`
@@ -113,7 +110,6 @@ This article provides suggestions to troubleshoot common problems with the Azure
 
 - **Recommendation**:  Verify the column in the query, *structure* in the dataset, and *mappings* in the activity.
 
-
 ## Error code: SqlBatchWriteTimeout
 
 - **Message**: `Timeouts in SQL write operation.`
@@ -122,7 +118,6 @@ This article provides suggestions to troubleshoot common problems with the Azure
 
 - **Recommendation**:  Retry the operation. If the problem persists, contact Azure SQL support.
 
-
 ## Error code: SqlBatchWriteTransactionFailed
 
 - **Message**: `SQL transaction commits failed.`
@@ -135,7 +130,6 @@ This article provides suggestions to troubleshoot common problems with the Azure
 
 - **Recommendation**:  Retry the activity and review the SQL database side metrics.
 
-
 ## Error code: SqlBulkCopyInvalidColumnLength
 
 - **Message**: `SQL Bulk Copy failed due to receive an invalid column length from the bcp client.`
@@ -144,7 +138,6 @@ This article provides suggestions to troubleshoot common problems with the Azure
 
 - **Recommendation**:  To identify which row has encountered the problem, enable the fault tolerance feature on the copy activity. This can redirect problematic rows to the storage for further investigation. For more information, see [Fault tolerance of copy activity](./copy-activity-fault-tolerance.md).
 
-
 ## Error code: SqlConnectionIsClosed
 
 - **Message**: `The connection is closed by SQL Database.`
@@ -159,15 +152,15 @@ This article provides suggestions to troubleshoot common problems with the Azure
 
 - **Cause**: The linked service was not configured properly.
 
-- **Recommendation**: Validate and fix the SQL server linked service. 
+- **Recommendation**: Validate and fix the SQL server linked service.
 
 ## Error code: SqlParallelFailedToDetectPartitionColumn
 
 - **Message**: `Failed to detect the partition column with command '%command;', %message;.`
 
 - **Cause**: There is no primary key or unique key in the table.
 
-- **Recommendation**: Check the table to make sure that a primary key or a unique index is created. 
+- **Recommendation**: Check the table to make sure that a primary key or a unique index is created.
 
 ## Error code: SqlParallelFailedToDetectPhysicalPartitions
 
@@ -189,37 +182,35 @@ This article provides suggestions to troubleshoot common problems with the Azure
 
 - **Symptoms**: When you copy data from a tabular data source (such as SQL Server) into Azure Synapse Analytics using staged copy and PolyBase, you receive the following error:
 
-   `ErrorCode=FailedDbOperation,Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException, 
-    Message=Error happened when loading data into Azure Synapse Analytics., 
-    Source=Microsoft.DataTransfer.ClientLibrary,Type=System.Data.SqlClient.SqlException, 
+   `ErrorCode=FailedDbOperation,Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,  
+    Message=Error happened when loading data into Azure Synapse Analytics.,  
+    Source=Microsoft.DataTransfer.ClientLibrary,Type=System.Data.SqlClient.SqlException,  
     Message=Conversion failed when converting from a character string to uniqueidentifier...`
 
 - **Cause**: Azure Synapse Analytics PolyBase can't convert an empty string to a GUID.
 
 - **Resolution**: In the copy activity sink, under PolyBase settings, set the **use type default** option to *false*.
 
-
 ## Error message: Expected data type: DECIMAL(x,x), Offending value
 
 - **Symptoms**: When you copy data from a tabular data source (such as SQL Server) into Azure Synapse Analytics by using staged copy and PolyBase, you receive the following error:
 
-    `ErrorCode=FailedDbOperation,Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException, 
-    Message=Error happened when loading data into Azure Synapse Analytics., 
-    Source=Microsoft.DataTransfer.ClientLibrary,Type=System.Data.SqlClient.SqlException, 
+    `ErrorCode=FailedDbOperation,Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,  
+    Message=Error happened when loading data into Azure Synapse Analytics.,  
+    Source=Microsoft.DataTransfer.ClientLibrary,Type=System.Data.SqlClient.SqlException,  
     Message=Query aborted-- the maximum reject threshold (0 rows) was reached while reading from an external source: 1 rows rejected out of total 415 rows processed. (/file_name.txt)  
     Column ordinal: 18, Expected data type: DECIMAL(x,x), Offending value:..`
 
 - **Cause**: Azure Synapse Analytics PolyBase can't insert an empty string (null value) into a decimal column.
 
 - **Resolution**: In the copy activity sink, under PolyBase settings, set the **use type default** option to false.
 
-
 ## Error message: Java exception message: HdfsBridge::CreateRecordReader
 
 - **Symptoms**: You copy data into Azure Synapse Analytics by using PolyBase and receive the following error:
 
     `Message=110802;An internal DMS error occurred that caused this operation to fail.  
-    Details: Exception: Microsoft.SqlServer.DataWarehouse.DataMovement.Common.ExternalAccess.HdfsAccessException, 
+    Details: Exception: Microsoft.SqlServer.DataWarehouse.DataMovement.Common.ExternalAccess.HdfsAccessException,  
     Message: Java exception raised on call to HdfsBridge_CreateRecordReader.  
     Java exception message:HdfsBridge::CreateRecordReader - Unexpected error encountered creating the record reader.: Error [HdfsBridge::CreateRecordReader - Unexpected error encountered creating the record reader.] occurred while accessing external file.....`
 
@@ -241,11 +232,10 @@ This article provides suggestions to troubleshoot common problems with the Azure
     - Time = 12 bytes
     - Tinyint = 1 byte
 
-- **Resolution**: 
+- **Resolution**:  
     - Reduce column width to less than 1 MB.
     - Or use a bulk insert approach by disabling PolyBase.
 
-
 ## Error message: The condition specified using HTTP conditional header(s) is not met
 
 - **Symptoms**: You use SQL query to pull data from Azure Synapse Analytics and receive the following error:
@@ -256,48 +246,51 @@ This article provides suggestions to troubleshoot common problems with the Azure
 
 - **Resolution**: Run the same query in SQL Server Management Studio (SSMS) and check to see whether you get the same result. If you do, open a support ticket to Azure Synapse Analytics and provide your Azure Synapse Analytics server and database name.
 
-
 ## Performance tier is low and leads to copy failure
 
 - **Symptoms**: You copy data into Azure SQL Database and receive the following error: `Database operation failed. Error message from database execution : ExecuteNonQuery requires an open and available Connection. The connection's current state is closed.`
 
 - **Cause**: Azure SQL Database s1 has hit input/output (I/O) limits.
 
-- **Resolution**: Upgrade the Azure SQL Database performance tier to fix the issue. 
-
+- **Resolution**: Upgrade the Azure SQL Database performance tier to fix the issue.
 
-## SQL table can't be found 
+## SQL table can't be found
 
 - **Symptoms**: You copy data from hybrid into an on-premises SQL Server table and receive the following error：`Cannot find the object "dbo.Contoso" because it does not exist or you do not have permissions.`
 
 - **Cause**: The current SQL account doesn't have sufficient permissions to execute requests issued by .NET SqlBulkCopy.WriteToServer.
 
 - **Resolution**: Switch to a more privileged SQL account.
 
-
 ## Error message: String or binary data is truncated
 
-- **Symptoms**: An error occurs when you copy data into an on-premises Azure SQL Server table. 
+- **Symptoms**: An error occurs when you copy data into an on-premises Azure SQL Server table.
 
-- **Cause**: The Cx SQL table schema definition has one or more columns with less length than expected.
+- **Cause**: The SQL table schema definition has one or more columns with less length than expected.
 
 - **Resolution**: To resolve the issue, try the following:
 
-    1. To troubleshoot which rows have the issue, apply SQL sink [fault tolerance](./copy-activity-fault-tolerance.md), especially "redirectIncompatibleRowSettings."
+    1. To troubleshoot which rows have the issue, apply SQL sink [fault tolerance](./copy-activity-fault-tolerance.md), especially `redirectIncompatibleRowSettings`.
 
-    	> [!NOTE]
-    	> Fault tolerance might require additional execution time, which could lead to higher costs.
+        > [!NOTE]  
+        > Fault tolerance might require additional execution time, which could lead to higher costs.
 
-    2. Double-check the redirected data against the SQL table schema column length to see which columns need to be updated.
+    1. Double-check the redirected data against the SQL table schema column length to see which columns need to be updated.
 
-    3. Update the table schema accordingly.
+    1. Update the table schema accordingly.
 
 ## Error code: FailedDbOperation
 
 - **Message**: `User does not have permission to perform this action.`
 
 - **Recommendation**: Make sure the user configured in the Azure Synapse Analytics connector must have 'CONTROL' permission on the target database while using PolyBase to load data. For more detailed information, refer to this [document](./connector-azure-sql-data-warehouse.md#required-database-permission).
 
+## Error code: Msg 105208
+
+- **Symptoms**: Error code: `Error code: Msg 105208, Level 16, State 1, Line 1 COPY statement failed with the following error when validating value of option 'FROM': '105200;COPY statement failed because the value for option 'FROM' is invalid.'`
+- **Cause**: Currently, ingesting data using the COPY command into an Azure Storage account that is using the new DNS partitioning feature results in an error. DNS partition feature enables customers to create up to 5000 storage accounts per subscription.  
+- **Resolutions**: Provision a storage account in a subscription that does not use the new [Azure Storage DNS partition feature](https://techcommunity.microsoft.com/t5/azure-storage-blog/public-preview-create-additional-5000-azure-storage-accounts/ba-p/3465466) (currently in Public Preview).
+
 ## Next steps
 
 For more troubleshooting help, try these resources:

articles/synapse-analytics/sql-data-warehouse/sql-data-warehouse-load-from-azure-data-lake-store.md

@@ -1,14 +1,13 @@
 ---
-title: 'Tutorial load data from Azure Data Lake Storage'
+title: "Tutorial load data from Azure Data Lake Storage"
 description: Use the COPY statement to load data from Azure Data Lake Storage for dedicated SQL pools.
 author: WilliamDAssafMSFT
-manager: craigg
+ms.author: wiassaf
+ms.reviewer: joanpo 
+ms.date: 09/02/2022
 ms.service: synapse-analytics
+ms.subservice: sql-dw
 ms.topic: conceptual
-ms.subservice: sql-dw 
-ms.date: 11/20/2020
-ms.author: wiassaf
-ms.reviewer: wiassaf
 ms.custom: azure-synapse
 ---
 
@@ -33,11 +32,12 @@ Before you begin this tutorial, download and install the newest version of [SQL
 To run this tutorial, you need:
 
 * A dedicated SQL pool. See [Create a dedicated SQL pool and query data](create-data-warehouse-portal.md).
-* A Data Lake Storage account. See [Get started with Azure Data Lake Storage](../../data-lake-store/data-lake-store-get-started-portal.md?toc=/azure/synapse-analytics/sql-data-warehouse/toc.json&bc=/azure/synapse-analytics/sql-data-warehouse/breadcrumb/toc.json). For this storage account, you will need to configure or specify one of the following credentials to load: A storage account key, shared access signature (SAS) key, an Azure Directory Application user, or an AAD user which has the appropriate Azure role to the storage account.
+* A Data Lake Storage account. See [Get started with Azure Data Lake Storage](../../data-lake-store/data-lake-store-get-started-portal.md?toc=/azure/synapse-analytics/sql-data-warehouse/toc.json&bc=/azure/synapse-analytics/sql-data-warehouse/breadcrumb/toc.json). For this storage account, you will need to configure or specify one of the following credentials to load: A storage account key, shared access signature (SAS) key, an Azure Directory Application user, or an Azure AD user that has the appropriate Azure role to the storage account.
+* Currently, ingesting data using the COPY command into an Azure Storage account that is using the new [Azure Storage DNS partition feature](https://techcommunity.microsoft.com/t5/azure-storage-blog/public-preview-create-additional-5000-azure-storage-accounts/ba-p/3465466) results in an error. Provision a storage account in a subscription that does not use DNS partitioning for this tutorial.
 
 ## Create the target table
 
-Connect to your dedicated SQL pool and create the target table you will to load to. In this example, we are creating a product dimension table.
+Connect to your dedicated SQL pool and create the target table you will load to. In this example, we are creating a product dimension table.
 
 ```sql
 -- A: Create the target table
@@ -56,26 +56,25 @@ WITH
 );
 ```
 
-
 ## Create the COPY statement
 
 Connect to your SQL dedicated pool and run the COPY statement. For a complete list of examples, visit the following documentation: [Securely load data using dedicated SQL pools](./quickstart-bulk-load-copy-tsql-examples.md).
 
 ```sql
 -- B: Create and execute the COPY statement
 
-COPY INTO [dbo].[DimProduct] 
---The column list allows you map, omit, or reorder input file columns to target table columns. 
+COPY INTO [dbo].[DimProduct]  
+--The column list allows you map, omit, or reorder input file columns to target table columns.  
 --You can also specify the default value when there is a NULL value in the file.
 --When the column list is not specified, columns will be mapped based on source and target ordinality
 (
-	ProductKey default -1 1,
-	ProductLabel default 'myStringDefaultWhenNull' 2,
-	ProductName default 'myStringDefaultWhenNull' 3
+    ProductKey default -1 1,
+    ProductLabel default 'myStringDefaultWhenNull' 2,
+    ProductName default 'myStringDefaultWhenNull' 3
 )
 --The storage account location where you data is staged
 FROM 'https://storageaccount.blob.core.windows.net/container/directory/'
-WITH 
+WITH  
 (
    --CREDENTIAL: Specifies the authentication method and credential access your storage account
    CREDENTIAL = (IDENTITY = '', SECRET = ''),
@@ -121,6 +120,7 @@ The following example is a good starting point for creating statistics. It creat
 You have successfully loaded data into your data warehouse. Great job!
 
 ## Next steps
+
 Loading data is the first step to developing a data warehouse solution using Azure Synapse Analytics. Check out our development resources.
 
 > [!div class="nextstepaction"]
@@ -129,4 +129,4 @@ Loading data is the first step to developing a data warehouse solution using Azu
 For more loading examples and references, view the following documentation:
 - [COPY statement reference documentation](/sql/t-sql/statements/copy-into-transact-sql?view=azure-sqldw-latest&preserve-view=true#syntax)
 - [COPY examples for each authentication method](./quickstart-bulk-load-copy-tsql-examples.md)
-- [COPY quickstart for a single table](./quickstart-bulk-load-copy-tsql.md)
+- [COPY quickstart for a single table](./quickstart-bulk-load-copy-tsql.md)