Skip to content

Commit 83d59fa

Browse files
ttorblettorble
authored
Merge pull request #297106 from Clare-Zheng82/0325-Spark_v2
[New feature] Add Spark v2
2 parents 9de7ed8 + bee218d commit 83d59fa

File tree

2 files changed

+102
-11
lines changed

2 files changed

+102
-11
lines changed

‎articles/data-factory/connector-spark.md

Lines changed: 102 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ author: jianleishen
77
ms.subservice: data-movement
88
ms.topic: conceptual
99
ms.custom: synapse
10-
ms.date: 10/20/2023
10+
ms.date: 03/26/2025
1111
---
1212

1313
# Copy data from Spark using Azure Data Factory or Synapse Analytics
@@ -67,19 +67,67 @@ The following sections provide details about properties that are used to define
6767

6868
## Linked service properties
6969

70-
The following properties are supported for Spark linked service:
70+
The Spark connector now supports version 2.0 (Preview). Refer to this [section](#upgrade-the-spark-connector) to upgrade your Spark connector version from version 1.0. For the property details, see the corresponding sections.
71+
72+
- [Version 2.0 (Preview)](#version-20-preview)
73+
- [Version 1.0](#version-10)
74+
75+
### Version 2.0 (Preview)
76+
77+
The following properties are supported for Spark linked service version 2.0 (Preview):
7178

7279
| Property | Description | Required |
7380
|:--- |:--- |:--- |
7481
| type | The type property must be set to: **Spark** | Yes |
82+
| version | The version that you specify. The value is `2.0`. | Yes |
7583
| host | IP address or host name of the Spark server | Yes |
7684
| port | The TCP port that the Spark server uses to listen for client connections. If you connect to Azure HDInsights, specify port as 443. | Yes |
77-
| serverType | The type of Spark server. <br/>Allowed values are: **SharkServer**, **SharkServer2**, **SparkThriftServer** | No |
78-
| thriftTransportProtocol | The transport protocol to use in the Thrift layer. <br/>Allowed values are: **Binary**, **SASL**, **HTTP** | No |
79-
| authenticationType | The authentication method used to access the Spark server. <br/>Allowed values are: **Anonymous**, **Username**, **UsernameAndPassword**, **WindowsAzureHDInsightService** | Yes |
85+
| serverType | The type of Spark server. <br/>The allowed value is: **SparkThriftServer** | No |
86+
| thriftTransportProtocol | The transport protocol to use in the Thrift layer. <br/>The allowed value is: **HTTP** | No |
87+
| authenticationType | The authentication method used to access the Spark server. <br/>Allowed values are: **Anonymous**, **UsernameAndPassword**, **WindowsAzureHDInsightService** | Yes |
8088
| username | The user name that you use to access Spark Server. | No |
8189
| password | The password corresponding to the user. Mark this field as a SecureString to store it securely, or [reference a secret stored in Azure Key Vault](store-credentials-in-key-vault.md). | No |
8290
| httpPath | The partial URL corresponding to the Spark server. | No |
91+
| enableSsl | Specifies whether the connections to the server are encrypted using TLS. The default value is true. | No |
92+
| connectVia | The [Integration Runtime](concepts-integration-runtime.md) to be used to connect to the data store. Learn more from [Prerequisites](#prerequisites) section. If not specified, it uses the default Azure Integration Runtime. |No |
93+
94+
**Example:**
95+
96+
```json
97+
{
98+
"name": "SparkLinkedService",
99+
"properties": {
100+
"type": "Spark",
101+
"version": "2.0",
102+
"typeProperties": {
103+
"host": "<cluster>.azurehdinsight.net",
104+
"port": "<port>",
105+
"authenticationType": "WindowsAzureHDInsightService",
106+
"username": "<username>",
107+
"password": {
108+
"type": "SecureString",
109+
"value": "<password>"
110+
}
111+
}
112+
}
113+
}
114+
```
115+
116+
### Version 1.0
117+
118+
The following properties are supported for Spark linked service version 1.0:
119+
120+
| Property | Description | Required |
121+
|:--- |:--- |:--- |
122+
| type | The type property must be set to: **Spark** | Yes |
123+
| host | IP address or host name of the Spark server | Yes |
124+
| port | The TCP port that the Spark server uses to listen for client connections. If you connect to Azure HDInsights, specify port as 443. | Yes |
125+
| serverType | The type of Spark server. <br/>Allowed values are: **SharkServer**, **SharkServer2**, **SparkThriftServer** | No |
126+
| thriftTransportProtocol | The transport protocol to use in the Thrift layer. <br/>Allowed values are: **Binary**, **SASL**, **HTTP** | No |
127+
| authenticationType | The authentication method used to access the Spark server. <br/>Allowed values are: **Anonymous**, **Username**, **UsernameAndPassword**, **WindowsAzureHDInsightService** | Yes |
128+
| username | The user name that you use to access Spark Server.| No |
129+
| password | The password corresponding to the user. Mark this field as a SecureString to store it securely, or [reference a secret stored in Azure Key Vault](store-credentials-in-key-vault.md).| No |
130+
| httpPath | The partial URL corresponding to the Spark server.| No |
83131
| enableSsl | Specifies whether the connections to the server are encrypted using TLS. The default value is false. | No |
84132
| trustedCertPath | The full path of the .pem file containing trusted CA certificates for verifying the server when connecting over TLS. This property can only be set when using TLS on self-hosted IR. The default value is the cacerts.pem file installed with the IR. | No |
85133
| useSystemTrustStore | Specifies whether to use a CA certificate from the system trust store or from a specified PEM file. The default value is false. | No |
@@ -95,13 +143,13 @@ The following properties are supported for Spark linked service:
95143
"properties": {
96144
"type": "Spark",
97145
"typeProperties": {
98-
"host" : "<cluster>.azurehdinsight.net",
99-
"port" : "<port>",
100-
"authenticationType" : "WindowsAzureHDInsightService",
101-
"username" : "<username>",
146+
"host": "<cluster>.azurehdinsight.net",
147+
"port": "<port>",
148+
"authenticationType": "WindowsAzureHDInsightService",
149+
"username": "<username>",
102150
"password": {
103-
"type": "SecureString",
104-
"value": "<password>"
151+
"type": "SecureString",
152+
"value": "<password>"
105153
}
106154
}
107155
}
@@ -183,9 +231,52 @@ To copy data from Spark, set the source type in the copy activity to **SparkSour
183231
]
184232
```
185233

234+
## Data type mapping for Spark
235+
236+
When you copy data from and to Spark, the following interim data type mappings are used within the service. To learn about how the copy activity maps the source schema and data type to the sink, see [Schema and data type mappings](copy-activity-schema-and-type-mapping.md).
237+
238+
| Spark data type | Interim service data type (for version 2.0 (Preview)) | Interim service data type (for version 1.0) |
239+
|:--- |:--- |:--- |
240+
| BooleanType  | Boolean  | Boolean  |
241+
| ByteType  | Sbyte  | Int16  |
242+
| ShortType  | Int16  | Int16  |
243+
| IntegerType  | Int32  | Int32  |
244+
| LongType  | Int64  | Int64  |
245+
| FloatType  | Single  | Single  |
246+
| DoubleType  | Double  | Double  |
247+
| DateType  | DateTime  | DateTime  |
248+
| TimestampType  | DateTimeOffset  | DateTime  |
249+
| StringType  | String  | String  |
250+
| BinaryType  | Byte[]  | Byte[]  |
251+
| DecimalType  | Decimal  | Decimal  |
252+
| ArrayType  | String  | String  |
253+
| StructType  | String  | String  |
254+
| MapType  | String  | String  |
255+
| TimestampNTZType  | DateTime  | DateTime  |
256+
| YearMonthIntervalType  | String  | Not supported.  |
257+
| DayTimeIntervalType  | String  | Not supported. |
258+
186259
## Lookup activity properties
187260

188261
To learn details about the properties, check [Lookup activity](control-flow-lookup-activity.md).
189262

263+
## Upgrade the Spark connector
264+
265+
1. In **Edit linked service** page, select 2.0 for version and configure the linked service by referring to [Linked service properties version 2.0 (Preview)](#version-20-preview).
266+
267+
1. The data type mapping for the Spark linked service version 2.0 (Preview) is different from that for the version 1.0. To learn the latest data type mapping, see [Data type mapping for Spark](#data-type-mapping-for-spark).
268+
269+
## Differences between Spark version 2.0 (Preview) and version 1.0
270+
271+
The Spark connector version 2.0 (Preview) offers new functionalities and is compatible with most features of version 1.0. The following table shows the feature differences between version 2.0 (Preview) and version 1.0.
272+
273+
| Version 2.0 (Preview)  | Version 1.0  |
274+
|:--- |:--- |
275+
| SharkServer and SharkServer2 are not supported for `serverType`. | Support SharkServer and SharkServer2 for `serverType`. |
276+
| Binary and SASL are not supported for `thriftTransportProtocl`. | Support Binary and SASL for `thriftTransportProtocl`. |
277+
| Username authentication type is not supported. | Support Username authentication type. |
278+
| The default value of `enableSSL` is true. `trustedCertPath`, `useSystemTrustStore`, `allowHostNameCNMismatch` and `allowSelfSignedServerCert` are not supported.| The default value of `enableSSL` is false. Additionally, support `trustedCertPath`, `useSystemTrustStore`, `allowHostNameCNMismatch` and `allowSelfSignedServerCert`. |
279+
| The following mappings are used from Spark data types to interim service data types used by the service internally.<br><br>TimestampType -> DateTimeOffset <br>YearMonthIntervalType -> String<br>DayTimeIntervalType -> String | The following mappings are used from Spark data types to interim service data types used by the service internally.<br><br>TimestampType -> DateTime<br>Other mappings supported by version 2.0 (Preview) listed left are not supported by version 1.0. |
280+
190281
## Related content
191282
For a list of data stores supported as sources and sinks by the copy activity, see [supported data stores](copy-activity-overview.md#supported-data-stores-and-formats).

‎articles/data-factory/media/connector-spark/configure-spark-linked-service.png

477 Bytes
Loading

0 commit comments

Comments
 (0)