Merge pull request #279015 from Clare-Zheng82/0621-Add_Snowflake_integration_doc

prmerger-automator[bot] · web-flow · commit 5b2d0e8c3359 · 2024-07-10T01:51:46.000Z
[New feature] Add Snowflake storage integration
diff --git a/articles/data-factory/connector-snowflake-legacy.md b/articles/data-factory/connector-snowflake-legacy.md
@@ -8,7 +8,7 @@ ms.service: data-factory
 ms.subservice: data-movement
 ms.topic: conceptual
 ms.custom: synapse
-ms.date: 05/22/2024
+ms.date: 07/02/2024
 ---
 
 # Copy and transform data in Snowflake using Azure Data Factory or Azure Synapse Analytics (legacy)
@@ -253,7 +253,7 @@ If your sink data store and format meet the criteria described in this section,
         "typeProperties": {
             "source": {
                 "type": "SnowflakeSource",
-                "sqlReaderQuery": "SELECT * FROM MYTABLE",
+                "query": "SELECT * FROM MYTABLE",
                 "exportSettings": {
                     "type": "SnowflakeExportCopyCommand",
                     "additionalCopyOptions": {
@@ -304,7 +304,7 @@ To use this feature, create an [Azure Blob storage linked service](connector-azu
         "typeProperties": {
             "source": {
                 "type": "SnowflakeSource",               
-                "sqlReaderQuery": "SELECT * FROM MyTable",
+                "query": "SELECT * FROM MyTable",
                 "exportSettings": {
                     "type": "SnowflakeExportCopyCommand"
                 }
diff --git a/articles/data-factory/connector-snowflake.md b/articles/data-factory/connector-snowflake.md
@@ -8,7 +8,7 @@ ms.service: data-factory
 ms.subservice: data-movement
 ms.topic: conceptual
 ms.custom: synapse
-ms.date: 05/22/2024
+ms.date: 06/24/2024
 ---
 
 # Copy and transform data in Snowflake using Azure Data Factory or Azure Synapse Analytics
@@ -267,6 +267,7 @@ To copy data from Snowflake, the following properties are supported in the Copy
 | exportSettings | Advanced settings used to retrieve data from Snowflake. You can configure the ones supported by the COPY into command that the service will pass through when you invoke the statement. | Yes       |
 | ***Under `exportSettings`:*** |  |  |
 | type | The type of export command, set to **SnowflakeExportCopyCommand**. | Yes |
+| storageIntegration | Specify the name of your storage integration that you created in the Snowflake. For the prerequisite steps of using the storage integration, see [Configuring a Snowflake storage integration](https://docs.snowflake.com/en/user-guide/data-load-azure-config#option-1-configuring-a-snowflake-storage-integration). | No |
 | additionalCopyOptions | Additional copy options, provided as a dictionary of key-value pairs. Examples: MAX_FILE_SIZE, OVERWRITE. For more information, see [Snowflake Copy Options](https://docs.snowflake.com/en/sql-reference/sql/copy-into-location.html#copy-options-copyoptions). | No |
 | additionalFormatOptions | Additional file format options that are provided to COPY command as a dictionary of key-value pairs. Examples: DATE_FORMAT, TIME_FORMAT, TIMESTAMP_FORMAT. For more information, see [Snowflake Format Type Options](https://docs.snowflake.com/en/sql-reference/sql/copy-into-location.html#format-type-options-formattypeoptions). | No |
 
@@ -279,7 +280,17 @@ To copy data from Snowflake, the following properties are supported in the Copy
 
 If your sink data store and format meet the criteria described in this section, you can use the Copy activity to directly copy from Snowflake to sink. The service checks the settings and fails the Copy activity run if the following criteria isn't met:
 
-- The **sink linked service** is [**Azure Blob storage**](connector-azure-blob-storage.md) with **shared access signature** authentication. If you want to directly copy data to Azure Data Lake Storage Gen2 in the following supported format, you can create an Azure Blob linked service with SAS authentication against your ADLS Gen2 account, to avoid using [staged copy from Snowflake](#staged-copy-from-snowflake).
+- When you specify `storageIntegration` in the source:
+
+  The sink data store is the Azure Blob Storage that you referred in the external stage in Snowflake. You need to complete the following steps before copying data:
+
+    1. Create an [**Azure Blob Storage**](connector-azure-blob-storage.md) linked service for the sink Azure Blob Storage with any supported authentication types.
+    
+    2. Grant at least **Storage Blob Data Contributor** role to the Snowflake service principal in the sink Azure Blob Storage **Access Control (IAM)**.
+
+- When you don't specify `storageIntegration` in the source:
+  
+  The **sink linked service** is [**Azure Blob storage**](connector-azure-blob-storage.md) with **shared access signature** authentication. If you want to directly copy data to Azure Data Lake Storage Gen2 in the following supported format, you can create an Azure Blob Storage linked service with SAS authentication against your Azure Data Lake Storage Gen2 account, to avoid using [staged copy from Snowflake](#staged-copy-from-snowflake).
 
 - The **sink data format** is of **Parquet**, **delimited text**, or **JSON** with the following configurations:
 
@@ -318,7 +329,7 @@ If your sink data store and format meet the criteria described in this section,
         "typeProperties": {
             "source": {
                 "type": "SnowflakeV2Source",
-                "sqlReaderQuery": "SELECT * FROM MYTABLE",
+                "query": "SELECT * FROM MYTABLE",
                 "exportSettings": {
                     "type": "SnowflakeExportCopyCommand",
                     "additionalCopyOptions": {
@@ -327,7 +338,8 @@ If your sink data store and format meet the criteria described in this section,
                     },
                     "additionalFormatOptions": {
                         "DATE_FORMAT": "'MM/DD/YYYY'"
-                    }
+                    },
+                    "storageIntegration": "< Snowflake storage integration name >"
                 }
             },
             "sink": {
@@ -344,8 +356,9 @@ When your sink data store or format isn't natively compatible with the Snowflake
 
 To use this feature, create an [Azure Blob storage linked service](connector-azure-blob-storage.md#linked-service-properties) that refers to the Azure storage account as the interim staging. Then specify the `enableStaging` and `stagingSettings` properties in the Copy activity.
 
-> [!NOTE]
-> The staging Azure Blob storage linked service must use shared access signature authentication, as required by the Snowflake COPY command. Make sure you grant proper access permission to Snowflake in the staging Azure Blob storage. To learn more about this, see this [article](https://docs.snowflake.com/en/user-guide/data-load-azure-config.html#option-2-generating-a-sas-token). 
+- When you specify `storageIntegration` in the source, the interim staging Azure Blob Storage should be the one that you referred in the external stage in Snowflake. Ensure that you create an [Azure Blob Storage](connector-azure-blob-storage.md) linked service for it with any supported authentication, and grant at least **Storage Blob Data Contributor** role to the Snowflake service principal in the staging Azure Blob Storage **Access Control (IAM)**. 
+
+- When you don't specify `storageIntegration` in the source, the staging Azure Blob Storage linked service must use shared access signature authentication, as required by the Snowflake COPY command. Make sure you grant proper access permission to Snowflake in the staging Azure Blob Storage. To learn more about this, see this [article](https://docs.snowflake.com/en/user-guide/data-load-azure-config.html#option-2-generating-a-sas-token). 
 
 **Example:**
 
@@ -369,9 +382,10 @@ To use this feature, create an [Azure Blob storage linked service](connector-azu
         "typeProperties": {
             "source": {
                 "type": "SnowflakeV2Source",               
-                "sqlReaderQuery": "SELECT * FROM MyTable",
+                "query": "SELECT * FROM MyTable",
                 "exportSettings": {
-                    "type": "SnowflakeExportCopyCommand"
+                    "type": "SnowflakeExportCopyCommand",
+                    "storageIntegration": "< Snowflake storage integration name >"                    
                 }
             },
             "sink": {
@@ -405,6 +419,7 @@ To copy data to Snowflake, the following properties are supported in the Copy ac
 | importSettings | Advanced settings used to write data into Snowflake. You can configure the ones supported by the COPY into command that the service will pass through when you invoke the statement. | Yes |
 | ***Under `importSettings`:*** |                                                              |  |
 | type | The type of import command, set to **SnowflakeImportCopyCommand**. | Yes |
+| storageIntegration | Specify the name of your storage integration that you created in the Snowflake. For the prerequisite steps of using the storage integration, see [Configuring a Snowflake storage integration](https://docs.snowflake.com/en/user-guide/data-load-azure-config#option-1-configuring-a-snowflake-storage-integration). | No |
 | additionalCopyOptions | Additional copy options, provided as a dictionary of key-value pairs. Examples: ON_ERROR, FORCE, LOAD_UNCERTAIN_FILES. For more information, see [Snowflake Copy Options](https://docs.snowflake.com/en/sql-reference/sql/copy-into-table.html#copy-options-copyoptions). | No |
 | additionalFormatOptions | Additional file format options provided to the COPY command, provided as a dictionary of key-value pairs. Examples: DATE_FORMAT, TIME_FORMAT, TIMESTAMP_FORMAT. For more information, see [Snowflake Format Type Options](https://docs.snowflake.com/en/sql-reference/sql/copy-into-table.html#format-type-options-formattypeoptions). | No |
 
@@ -421,7 +436,17 @@ To copy data to Snowflake, the following properties are supported in the Copy ac
 
 If your source data store and format meet the criteria described in this section, you can use the Copy activity to directly copy from source to Snowflake. The service checks the settings and fails the Copy activity run if the following criteria isn't met:
 
-- The **source linked service** is [**Azure Blob storage**](connector-azure-blob-storage.md) with **shared access signature** authentication. If you want to directly copy data from Azure Data Lake Storage Gen2 in the following supported format, you can create an Azure Blob linked service with SAS authentication against your ADLS Gen2 account, to avoid using  [staged copy to Snowflake](#staged-copy-to-snowflake).
+- When you specify `storageIntegration` in the sink:
+
+  The source data store is the Azure Blob Storage that you referred in the external stage in Snowflake. You need to complete the following steps before copying data:
+
+    1. Create an [**Azure Blob Storage**](connector-azure-blob-storage.md) linked service for the source Azure Blob Storage with any supported authentication types. 
+
+    2. Grant at least **Storage Blob Data Reader** role to the Snowflake service principal in the source Azure Blob Storage **Access Control (IAM)**.
+
+- When you don't specify `storageIntegration` in the sink: 
+
+  The **source linked service** is [**Azure Blob storage**](connector-azure-blob-storage.md) with **shared access signature** authentication. If you want to directly copy data from Azure Data Lake Storage Gen2 in the following supported format, you can create an Azure Blob Storage linked service with SAS authentication against your Azure Data Lake Storage Gen2 account, to avoid using [staged copy to Snowflake](#staged-copy-to-snowflake).
 
 - The **source data format** is **Parquet**, **Delimited text**, or **JSON** with the following configurations:
 
@@ -476,7 +501,8 @@ If your source data store and format meet the criteria described in this section
                     },
                     "fileFormatOptions": {
                         "DATE_FORMAT": "YYYY-MM-DD"
-                    }
+                    },
+                    "storageIntegration": "< Snowflake storage integration name >"
                 }
             }
         }
@@ -490,8 +516,9 @@ When your source data store or format isn't natively compatible with the Snowfla
 
 To use this feature, create an [Azure Blob storage linked service](connector-azure-blob-storage.md#linked-service-properties) that refers to the Azure storage account as the interim staging. Then specify the `enableStaging` and `stagingSettings` properties in the Copy activity.
 
-> [!NOTE]
-> The staging Azure Blob storage linked service need to use shared access signature authentication as required by the Snowflake COPY command.
+- When you specify `storageIntegration` in the sink, the interim staging Azure Blob Storage should be the one that you referred in the external stage in Snowflake. Ensure that you create an [Azure Blob Storage](connector-azure-blob-storage.md) linked service for it with any supported authentication, and grant at least **Storage Blob Data Reader** role to the Snowflake service principal in the staging Azure Blob Storage **Access Control (IAM)**. 
+
+- When you don't specify `storageIntegration` in the sink, the staging Azure Blob Storage linked service need to use shared access signature authentication as required by the Snowflake COPY command.
 
 **Example:**
 
@@ -519,7 +546,8 @@ To use this feature, create an [Azure Blob storage linked service](connector-azu
             "sink": {
                 "type": "SnowflakeV2Sink",
                 "importSettings": {
-                    "type": "SnowflakeImportCopyCommand"
+                    "type": "SnowflakeImportCopyCommand",
+                    "storageIntegration": "< Snowflake storage integration name >"
                 }
             },
             "enableStaging": true,
@@ -645,7 +673,6 @@ The Snowflake connector offers new functionalities and is compatible with most f
 | :----------- | :------- |
 | Support Basic and Key pair authentication. | Support Basic authentication. | 
 | Script parameters are not supported in Script activity currently. As an alternative, utilize dynamic expressions for script parameters. For more information, see [Expressions and functions in Azure Data Factory and Azure Synapse Analytics](control-flow-expression-language-functions.md). | Support script parameters in Script activity. | 
-| Multiple SQL statements execution in Script activity is not supported currently. To execute multiple SQL statements, divide the query into several script blocks. | Support multiple SQL statements execution in Script activity. | 
 | Support BigDecimal in Lookup activity. The NUMBER type, as defined in Snowflake, will be displayed as a string in Lookup activity. | BigDecimal is not supported in Lookup activity.  | 
 
 ## Related content
diff --git a/articles/data-factory/copy-activity-performance-features.md b/articles/data-factory/copy-activity-performance-features.md
@@ -8,7 +8,7 @@ ms.service: data-factory
 ms.subservice: data-movement
 ms.topic: conceptual
 ms.custom: synapse
-ms.date: 06/17/2024
+ms.date: 06/24/2024
 
 ---
 
@@ -153,7 +153,7 @@ Configure the **enableStaging** setting in the copy activity to specify whether
 | --- | --- | --- | --- |
 | enableStaging |Specify whether you want to copy data via an interim staging store. |False |No |
 | linkedServiceName |Specify the name of an [Azure Blob storage](connector-azure-blob-storage.md#linked-service-properties) or [Azure Data Lake Storage Gen2](connector-azure-data-lake-storage.md#linked-service-properties) linked service, which refers to the instance of Storage that you use as an interim staging store. |N/A |Yes, when **enableStaging** is set to TRUE |
-| path |Specify the path that you want to contain the staged data. If you don't provide a path, the service creates a container to store temporary data. |N/A |No |
+| path |Specify the path that you want to contain the staged data. If you don't provide a path, the service creates a container to store temporary data. |N/A |No (Yes when `storageIntegration` in Snowflake connector is specified) |
 | enableCompression |Specifies whether data should be compressed before it's copied to the destination. This setting reduces the volume of data being transferred. |False |No |
 
 >[!NOTE]
