Merge pull request #174248 from jonburchel/2021-10-01-breaks-up-long-connector-troubleshooting-article

Initial commit just to test TOC behavior before full commit

Author: ShannonLeavitt

articles/data-factory/TOC.yml

@@ -941,8 +941,44 @@ items:
       href: data-factory-troubleshoot-guide.md
       displayName: timeout
     - name: Connectors
-      href: connector-troubleshoot-guide.md
-      displayName: polybase, timeout
+      items:
+      - name: Overview and general copy activity errors
+        href: connector-troubleshoot-guide.md
+        displayName: polybase, timeout
+      - name: Azure Blob Storage
+        href: connector-troubleshoot-azure-blob-storage.md
+      - name: Azure Cosmos DB (including SQL API)
+        href: connector-troubleshoot-azure-cosmos-db.md
+      - name: Azure Data Lake (Gen1 and Gen2)
+        href: connector-troubleshoot-azure-data-lake.md
+      - name: Azure database for PostgreSQL
+        href: connector-troubleshoot-postgresql.md
+      - name: Azure files
+        href: connector-troubleshoot-azure-files.md
+      - name: Azure Synapse Analytics, Azure SQL Database, and SQL Server
+        href: connector-troubleshoot-synapse-sql.md
+      - name: Azure Table Storage
+        href: connector-troubleshoot-azure-table-storage.md
+      - name: DB2
+        href: connector-troubleshoot-db2.md
+      - name: Delimited text format
+        href: connector-troubleshoot-delimited-text.md
+      - name: Dynamics 365, Dataverse (Common Data Service), and Dynamics CRM
+        href: connector-troubleshoot-dynamics-dataverse.md
+      - name: FTP, SFTP and HTTP
+        href: connector-troubleshoot-ftp-sftp-http.md
+      - name: Oracle
+        href: connector-troubleshoot-oracle.md
+      - name: ORC format
+        href: connector-troubleshoot-orc.md
+      - name: Parquet format
+        href: connector-troubleshoot-parquet.md
+      - name: REST
+        href: connector-troubleshoot-rest.md
+      - name: SharePoint Online list
+        href: connector-troubleshoot-sharepoint-online-list.md
+      - name: XML format
+        href: connector-troubleshoot-xml.md
     - name: Pipeline Triggers 
       href: pipeline-trigger-troubleshoot-guide.md 
     - name: Data Flows

articles/data-factory/connector-troubleshoot-azure-blob-storage.md

@@ -0,0 +1,73 @@
+---
+title: Troubleshoot the Azure Blob Storage connector
+titleSuffix: Azure Data Factory & Azure Synapse
+description: Learn how to troubleshoot issues with the Azure Blob Storage connector in Azure Data Factory and Azure Synapse Analytics. 
+author: jianleishen
+ms.service: data-factory
+ms.subservice: data-movement
+ms.topic: troubleshooting
+ms.date: 10/01/2021
+ms.author: jianleishen
+ms.custom: has-adal-ref, synapse
+---
+
+# Troubleshoot the Azure Blob Storage connector in Azure Data Factory and Azure Synapse
+
+[!INCLUDE[appliesto-adf-asa-md](includes/appliesto-adf-asa-md.md)]
+
+This article provides suggestions to troubleshoot common problems with the Azure Blob Storage connector in Azure Data Factory and Azure Synapse.
+
+## Error code: AzureBlobOperationFailed
+
+- **Message**: "Blob operation Failed. ContainerName: %containerName;, path: %path;."
+
+- **Cause**: A problem with the Blob Storage operation.
+
+- **Recommendation**:  To check the error details, see [Blob Storage error codes](/rest/api/storageservices/blob-service-error-codes). For further help, contact the Blob Storage team.
+
+
+## Invalid property during copy activity
+
+- **Message**: `Copy activity \<Activity Name> has an invalid "source" property. The source type is not compatible with the dataset \<Dataset Name> and its linked service \<Linked Service Name>. Please verify your input against.`
+
+- **Cause**: The type defined in the dataset is inconsistent with the source or sink type that's defined in the copy activity.
+
+- **Resolution**: Edit the dataset or pipeline JSON definition to make the types consistent, and then rerun the deployment.
+
+## Error code: FIPSModeIsNotSupport
+
+- **Message**: `Fail to read data form Azure Blob Storage for Azure Blob connector needs MD5 algorithm which can't co-work with FIPS mode. Please change diawp.exe.config in self-hosted integration runtime install directory to disable FIPS policy following https://docs.microsoft.com/en-us/dotnet/framework/configure-apps/file-schema/runtime/enforcefipspolicy-element.`
+
+- **Cause**: Then FIPS policy is enabled on the VM where the self-hosted integration runtime was installed.
+
+- **Recommendation**: Disable the FIPS mode on the VM where the self-hosted integration runtime was installed. Windows doesn't recommend the FIPS mode.
+
+## Error code: AzureBlobInvalidBlockSize
+
+- **Message**: `Block size should between %minSize; MB and 100 MB.`
+
+- **Cause**: The block size is over the blob limitation.
+
+## Error code: AzureStorageOperationFailedConcurrentWrite
+
+- **Message**: `Error occurred when trying to upload a file. It's possible because you have multiple concurrent copy activities runs writing to the same file '%name;'. Check your ADF configuration.`
+
+- **Cause**: You have multiple concurrent copy activity runs or applications writing to the same file.
+
+## Error code: AzureAppendBlobConcurrentWriteConflict
+
+- **Message**: `Detected concurrent write to the same append blob file, it's possible because you have multiple concurrent copy activities runs or applications writing to the same file '%name;'. Please check your ADF configuration and retry.`
+
+- **Cause**: Multiple concurrent writing requests occur, which causes conflicts on file content.
+
+## Next steps
+
+For more troubleshooting help, try these resources:
+
+- [Connector troubleshooting guide](connector-troubleshoot-guide.md)
+- [Data Factory blog](https://azure.microsoft.com/blog/tag/azure-data-factory/)
+- [Data Factory feature requests](/answers/topics/azure-data-factory.html)
+- [Azure videos](https://azure.microsoft.com/resources/videos/index/?sort=newest&services=data-factory)
+- [Microsoft Q&A page](/answers/topics/azure-data-factory.html)
+- [Stack Overflow forum for Data Factory](https://stackoverflow.com/questions/tagged/azure-data-factory)
+- [Twitter information about Data Factory](https://twitter.com/hashtag/DataFactory)

articles/data-factory/connector-troubleshoot-azure-cosmos-db.md

@@ -0,0 +1,102 @@
+---
+title: Troubleshoot the Azure Cosmos DB connector
+titleSuffix: Azure Data Factory & Azure Synapse
+description: Learn how to troubleshoot issues with the Azure Cosmos DB and Azure Cosmos DB (SQL API) connectors in Azure Data Factory and Azure Synapse Analytics. 
+author: jianleishen
+ms.service: data-factory
+ms.subservice: data-movement
+ms.topic: troubleshooting
+ms.date: 10/01/2021
+ms.author: jianleishen
+ms.custom: has-adal-ref, synapse
+---
+
+# Troubleshoot the Azure Cosmos DB connector in Azure Data Factory and Azure Synapse
+
+[!INCLUDE[appliesto-adf-asa-md](includes/appliesto-adf-asa-md.md)]
+
+This article provides suggestions to troubleshoot common problems with the Azure Cosmos DB and Azure Cosmos DB (SQL API) connectors in Azure Data Factory and Azure Synapse.
+
+## Error message: Request size is too large
+
+- **Symptoms**: When you copy data into Azure Cosmos DB with a default write batch size, you receive the following error: `Request size is too large.`
+
+- **Cause**: Azure Cosmos DB limits the size of a single request to 2 MB. The formula is *request size = single document size \* write batch size*. If your document size is large, the default behavior will result in a request size that's too large. You can tune the write batch size.
+
+- **Resolution**: In the copy activity sink, reduce the *write batch size* value (the default value is 10000).
+
+## Error message: Unique index constraint violation
+
+- **Symptoms**: When you copy data into Azure Cosmos DB, you receive the following error:
+
+    `Message=Partition range id 0 | Failed to import mini-batch. 
+    Exception was Message: {"Errors":["Encountered exception while executing function. Exception = Error: {\"Errors\":[\"Unique index constraint violation.\"]}...`
+
+- **Cause**: There are two possible causes:
+
+    - Cause 1: If you use **Insert** as the write behavior, this error means that your source data has rows or objects with same ID.
+    - Cause 2: If you use **Upsert** as the write behavior and you set another unique key to the container, this error means that your source data has rows or objects with different IDs but the same value for the defined unique key.
+
+- **Resolution**: 
+
+    - For cause 1, set **Upsert** as the write behavior.
+    - For cause 2, make sure that each document has a different value for the defined unique key.
+
+## Error message: Request rate is large
+
+- **Symptoms**: When you copy data into Azure Cosmos DB, you receive the following error:
+
+    `Type=Microsoft.Azure.Documents.DocumentClientException,
+    Message=Message: {"Errors":["Request rate is large"]}`
+
+- **Cause**: The number of used request units (RUs) is greater than the available RUs configured in Azure Cosmos DB. To learn how
+Azure Cosmos DB calculates RUs, see [Request units in Azure Cosmos DB](../cosmos-db/request-units.md#request-unit-considerations).
+
+- **Resolution**: Try either of the following two solutions:
+
+    - Increase the *container RUs* number to a greater value in Azure Cosmos DB. This solution will improve the copy activity performance, but it will incur more cost in Azure Cosmos DB. 
+    - Decrease *writeBatchSize* to a lesser value, such as 1000, and decrease *parallelCopies* to a lesser value, such as 1. This solution will reduce copy run performance, but it won't incur more cost in Azure Cosmos DB.
+
+## Columns missing in column mapping
+
+- **Symptoms**: When you import a schema for Azure Cosmos DB for column mapping, some columns are missing. 
+
+- **Cause**: Azure Data Factory and Synapse pipelines infer the schema from the first 10 Azure Cosmos DB documents. If some document columns or properties don't contain values, the schema isn't detected and consequently isn't displayed.
+
+- **Resolution**: You can tune the query as shown in the following code to force the column values to be displayed in the result set with empty values. Assume that the *impossible* column is missing in the first 10 documents). Alternatively, you can manually add the column for mapping.
+
+    ```sql
+    select c.company, c.category, c.comments, (c.impossible??'') as impossible from c
+    ```
+
+## Error message: The GuidRepresentation for the reader is CSharpLegacy
+
+- **Symptoms**: When you copy data from Azure Cosmos DB MongoAPI or MongoDB with the universally unique identifier (UUID) field, you receive the following error:
+
+    `Failed to read data via MongoDB client., 
+    Source=Microsoft.DataTransfer.Runtime.MongoDbV2Connector,Type=System.FormatException, 
+    Message=The GuidRepresentation for the reader is CSharpLegacy which requires the binary sub type to be UuidLegacy not UuidStandard.,Source=MongoDB.Bson,’“,`
+
+- **Cause**: There are two ways to represent the UUID in Binary JSON (BSON): UuidStardard and UuidLegacy. By default, UuidLegacy is used to read data. You will receive an error if your UUID data in MongoDB is UuidStandard.
+
+- **Resolution**: In the MongoDB connection string, add the *uuidRepresentation=standard* option. For more information, see [MongoDB connection string](connector-mongodb.md#linked-service-properties).
+
+## Error code: CosmosDbSqlApiOperationFailed
+
+- **Message**: `CosmosDbSqlApi operation Failed. ErrorMessage: %msg;.`
+
+- **Cause**: A problem with the CosmosDbSqlApi operation.  This applies to the Cosmos DB (SQL API) connector specifically.
+
+- **Recommendation**:  To check the error details, see [Azure Cosmos DB help document](../cosmos-db/troubleshoot-dot-net-sdk.md). For further help, contact the Azure Cosmos DB team.
+
+## Next steps
+
+For more troubleshooting help, try these resources:
+
+- [Connector troubleshooting guide](connector-troubleshoot-guide.md)
+- [Data Factory blog](https://azure.microsoft.com/blog/tag/azure-data-factory/)
+- [Data Factory feature requests](/answers/topics/azure-data-factory.html)
+- [Azure videos](https://azure.microsoft.com/resources/videos/index/?sort=newest&services=data-factory)
+- [Microsoft Q&A page](/answers/topics/azure-data-factory.html)
+- [Stack Overflow forum for Data Factory](https://stackoverflow.com/questions/tagged/azure-data-factory)
+- [Twitter information about Data Factory](https://twitter.com/hashtag/DataFactory)

articles/data-factory/connector-troubleshoot-azure-data-lake.md

@@ -0,0 +1,109 @@
+---
+title: Troubleshoot the Azure Data Lake Storage connectors
+titleSuffix: Azure Data Factory & Azure Synapse
+description: Learn how to troubleshoot issues with the Azure Data Lake Storage Gen1 and Gen2 connectors in Azure Data Factory and Azure Synapse Analytics. 
+author: jianleishen
+ms.service: data-factory
+ms.subservice: data-movement
+ms.topic: troubleshooting
+ms.date: 10/01/2021
+ms.author: jianleishen
+ms.custom: has-adal-ref, synapse
+---
+
+# Troubleshoot the Azure Data Lake Storage connectors in Azure Data Factory and Azure Synapse
+
+[!INCLUDE[appliesto-adf-asa-md](includes/appliesto-adf-asa-md.md)]
+
+This article provides suggestions to troubleshoot common problems with the Azure Data Lake Storage Gen1 and Gen2 connectors in Azure Data Factory and Azure Synapse.
+
+## Azure Data Lake Storage Gen1
+
+### Error message: The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel.
+
+- **Symptoms**: Copy activity fails with the following error: 
+
+    `Message: ErrorCode = UserErrorFailedFileOperation, Error Message = The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel.`
+
+- **Cause**: The certificate validation failed during the TLS handshake.
+
+- **Resolution**: As a workaround, use the staged copy to skip the Transport Layer Security (TLS) validation for Azure Data Lake Storage Gen1. You need to reproduce this issue and gather the network monitor (netmon) trace, and then engage your network team to check the local network configuration.
+
+    :::image type="content" source="./media/connector-troubleshoot-guide/adls-troubleshoot.png" alt-text="Diagram of Azure Data Lake Storage Gen1 connections for troubleshooting issues.":::
+
+
+### Error message: The remote server returned an error: (403) Forbidden
+
+- **Symptoms**: Copy activity fail with the following error: 
+
+   `Message: The remote server returned an error: (403) Forbidden.   
+    Response details: {"RemoteException":{"exception":"AccessControlException""message":"CREATE failed with error 0x83090aa2 (Forbidden. ACL verification failed. Either the resource does not exist or the user is not authorized to perform the requested operation.)....`
+
+- **Cause**: One possible cause is that the service principal or managed identity you use doesn't have permission to access certain folders or files.
+
+- **Resolution**: Grant appropriate permissions to all the folders and subfolders you need to copy. For more information, see [Copy data to or from Azure Data Lake Storage Gen1](connector-azure-data-lake-store.md#linked-service-properties).
+
+### Error message: Failed to get access token by using service principal. ADAL Error: service_unavailable
+
+- **Symptoms**: Copy activity fails with the following error:
+
+    `Failed to get access token by using service principal.  
+    ADAL Error: service_unavailable, The remote server returned an error: (503) Server Unavailable.`
+
+- **Cause**: When the Service Token Server (STS) that's owned by Azure Active Directory is not available, that means it's too busy to handle requests, and it returns HTTP error 503. 
+
+- **Resolution**: Rerun the copy activity after several minutes.
+
+## Azure Data Lake Storage Gen2
+
+### Error code: ADLSGen2OperationFailed
+
+- **Message**: `ADLS Gen2 operation failed for: %adlsGen2Message;.%exceptionData;.`
+
+- **Causes and recommendations**: Different causes may lead to this error. Check below list for possible cause analysis and related recommendation.
+
+  | Cause analysis                                               | Recommendation                                               |
+  | :----------------------------------------------------------- | :----------------------------------------------------------- |
+  | If Azure Data Lake Storage Gen2 throws error indicating some operation failed.| Check the detailed error message thrown by Azure Data Lake Storage Gen2. If the error is a transient failure, retry the operation. For further help, contact Azure Storage support, and provide the request ID in error message. |
+  | If the error message contains the string "Forbidden", the service principal or managed identity you use might not have sufficient permission to access Azure Data Lake Storage Gen2. | To troubleshoot this error, see [Copy and transform data in Azure Data Lake Storage Gen2](./connector-azure-data-lake-storage.md#service-principal-authentication). |
+  | If the error message contains the string "InternalServerError", the error is returned by Azure Data Lake Storage Gen2. | The error might be caused by a transient failure. If so, retry the operation. If the issue persists, contact Azure Storage support and provide the request ID from the error message. |
+
+### Request to Azure Data Lake Storage Gen2 account caused a timeout error
+
+- **Message**: 
+  * Error Code = `UserErrorFailedBlobFSOperation`
+  * Error Message = `BlobFS operation failed for: A task was canceled.`
+
+- **Cause**: The issue is caused by the Azure Data Lake Storage Gen2 sink timeout error, which usually occurs on the Self-hosted Integration Runtime (IR) machine.
+
+- **Recommendation**: 
+
+    - Place your Self-hosted IR machine and target Azure Data Lake Storage Gen2 account in the same region, if possible. This can help avoid a random timeout error and produce better performance.
+
+    - Check whether there's a special network setting, such as ExpressRoute, and ensure that the network has enough bandwidth. We suggest that you lower the Self-hosted IR concurrent jobs setting when the overall bandwidth is low. Doing so can help avoid network resource competition across multiple concurrent jobs.
+
+    - If the file size is moderate or small, use a smaller block size for nonbinary copy to mitigate such a timeout error. For more information, see [Blob Storage Put Block](/rest/api/storageservices/put-block).
+
+       To specify the custom block size, edit the property in your JSON file editor as shown here:
+    
+        ```
+        "sink": {
+            "type": "DelimitedTextSink",
+            "storeSettings": {
+                "type": "AzureBlobFSWriteSettings",
+                "blockSizeInMB": 8
+            }
+        }
+        ```
+
+## Next steps
+
+For more troubleshooting help, try these resources:
+
+- [Connector troubleshooting guide](connector-troubleshoot-guide.md)
+- [Data Factory blog](https://azure.microsoft.com/blog/tag/azure-data-factory/)
+- [Data Factory feature requests](/answers/topics/azure-data-factory.html)
+- [Azure videos](https://azure.microsoft.com/resources/videos/index/?sort=newest&services=data-factory)
+- [Microsoft Q&A page](/answers/topics/azure-data-factory.html)
+- [Stack Overflow forum for Data Factory](https://stackoverflow.com/questions/tagged/azure-data-factory)
+- [Twitter information about Data Factory](https://twitter.com/hashtag/DataFactory)