Skip to content

Commit 0e6a36d

Browse files
committed
add GBQ from v1 to v2 section
1 parent 648af77 commit 0e6a36d

File tree

2 files changed

+87
-4
lines changed

2 files changed

+87
-4
lines changed

articles/data-factory/connector-google-bigquery-legacy.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ author: jianleishen
77
ms.subservice: data-movement
88
ms.topic: conceptual
99
ms.custom: synapse
10-
ms.date: 06/13/2025
10+
ms.date: 04/14/2025
1111
---
1212

1313
# Copy data from Google BigQuery V1 using Azure Data Factory or Synapse Analytics
@@ -16,7 +16,7 @@ ms.date: 06/13/2025
1616
This article outlines how to use Copy Activity in Azure Data Factory and Synapse Analytics pipelines to copy data from Google BigQuery. It builds on the [Copy Activity overview](copy-activity-overview.md) article that presents a general overview of the copy activity.
1717

1818
> [!IMPORTANT]
19-
> The [Google BigQuery V2 connector](connector-google-bigquery.md) provides improved native Google BigQuery support. If you are using the [Google BigQuery V1 connector](connector-google-bigquery-legacy.md) in your solution, you are recommended to [upgrade your Google BigQuery connector](connector-google-bigquery.md#upgrade-the-google-bigquery-v2-connector-from-version-10-to-version-11) before **September 30, 2025**. Refer to this [section](connector-google-bigquery.md#differences-between-google-bigquery-and-google-bigquery-legacy) for details on the difference between V2 and V1.
19+
> The [Google BigQuery V2 connector](connector-google-bigquery.md) provides improved native Google BigQuery support. If you are using the [Google BigQuery V1 connector](connector-google-bigquery-legacy.md) in your solution, please [upgrade your Google BigQuery connector](connector-google-bigquery.md#upgrade-the-google-bigquery-linked-service) as V1 is at [End of Support stage](connector-deprecation-plan.md). Your pipeline will fail after **September 30, 2025** if not upgraded. Refer to this [section](connector-google-bigquery.md#differences-between-google-bigquery-and-google-bigquery-legacy) for details on the difference between V2 and V1.
2020
2121
## Supported capabilities
2222

articles/data-factory/connector-google-bigquery.md

Lines changed: 85 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@ ms.date: 06/09/2025
1717
This article outlines how to use Copy Activity in Azure Data Factory and Synapse Analytics pipelines to copy data from Google BigQuery. It builds on the [Copy Activity overview](copy-activity-overview.md) article that presents a general overview of the copy activity.
1818

1919
> [!IMPORTANT]
20-
> The [Google BigQuery V2 connector](connector-google-bigquery.md) provides improved native Google BigQuery support. If you are using the [Google BigQuery V1 connector](connector-google-bigquery-legacy.md) in your solution, you are recommended to [upgrade your Google BigQuery connector](#upgrade-the-google-bigquery-v2-connector-from-version-10-to-version-11) before **September 30, 2025**. Refer to this [section](#differences-between-google-bigquery-and-google-bigquery-legacy) for details on the difference between V2 and V1.
20+
> The [Google BigQuery V2 connector](connector-google-bigquery.md) provides improved native Google BigQuery support. If you are using the [Google BigQuery V1 connector](connector-google-bigquery-legacy.md) in your solution, please [upgrade your Google BigQuery connector](#upgrade-the-google-bigquery-linked-service) as V1 is at [End of Support stage](connector-deprecation-plan.md). Your pipeline will fail after **September 30, 2025** if not upgraded. Refer to this [section](#differences-between-google-bigquery-and-google-bigquery-legacy) for details on the difference between V2 and V1.
2121
2222
## Supported capabilities
2323

@@ -223,7 +223,6 @@ To copy data from Google BigQuery, set the source type in the copy activity to *
223223

224224
To learn details about the properties, check [Lookup activity](control-flow-lookup-activity.md).
225225

226-
227226
## <a name="differences-between-google-bigquery-and-google-bigquery-legacy"></a> Google BigQuery connector lifecycle and upgrade
228227

229228
The following table shows the release stage and change logs for different versions of the Google BigQuery connector:
@@ -234,6 +233,90 @@ The following table shows the release stage and change logs for different versio
234233
| Google BigQuery V2 (version 1.0) | GA version available | • Service authentication is supported by the Azure integration runtime and the self-hosted integration runtime. <br>The properties `trustedCertPath`, `useSystemTrustStore`, `email` and `keyFilePath` are not supported as they are available on the self-hosted integration runtime only. <br><br> • `requestGoogleDriveScope` is not supported. You need additionally apply the permission in Google BigQuery service by referring to [Choose Google Drive API scopes](https://developers.google.com/drive/api/guides/api-specific-auth) and [Query Drive data](https://cloud.google.com/bigquery/docs/query-drive-data). <br><br> • `additionalProjects` is not supported. As an alternative, [query a public dataset with the Google Cloud console](https://cloud.google.com/bigquery/docs/quickstarts/query-public-dataset-console).<br><br> • NUMBER is read as Decimal data type. <br><br> • Timestamp and Datetime are read as DateTimeOffset data type.|
235234
| Google BigQuery V2 (version 1.1) | GA version available | • When executing multiple statements, the `query` returns the complete set of results instead of only the result of the first statement. |
236235

236+
### <a name="upgrade-the-google-bigquery-linked-service"></a> Upgrade the Google BigQuery connector from V1 to V2
237+
238+
To upgrade the Google BigQuery connector from V1 to V2, you can do a side-by-side upgrade, or an in-place upgrade.
239+
240+
#### Side-by-side upgrade
241+
242+
To perform a side-by-side upgrade, complete the following steps:
243+
244+
1. Create a new Google BigQuery linked service and configure it by referring to the V2 linked service properties.
245+
1. Create a dataset based on the newly created Google BigQuery linked service.
246+
1. Replace the new linked service and dataset with the existing ones in the pipelines that targets the V1 objects.
247+
248+
#### In-place upgrade
249+
250+
To perform an in-place upgrade, you need to edit the existing linked service payload and update dataset to use the new linked service.
251+
252+
1. Update the type from **GoogleBigQuery** to **GoogleBigQueryV2**.
253+
1. Modify the linked service payload from its V1 format to V2. You can either fill in each field from the user interface after changing the type mentioned above, or update the payload directly through the JSON Editor. Refer to the [Linked service properties](#linked-service-properties) section in this article for the supported connection properties. The following examples show the differences in payload for the V1 and V2 Google BigQuery linked services:
254+
255+
**Google BigQuery V1 linked service JSON payload:**
256+
```json
257+
{
258+
"name": "GoogleBigQuery1",
259+
"type": "Microsoft.DataFactory/factories/linkedservices",
260+
"properties": {
261+
"annotations": [],
262+
"type": "GoogleBigQuery",
263+
"typeProperties": {
264+
"project" : "<project ID>",
265+
"additionalProjects" : "<additional project IDs>",
266+
"requestGoogleDriveScope" : true,
267+
"authenticationType" : "UserAuthentication",
268+
"clientId": "<id of the application used to generate the refresh token>",
269+
"clientSecret": {
270+
"type": "SecureString",
271+
"value":"<secret of the application used to generate the refresh token>"
272+
},
273+
"refreshToken": {
274+
"type": "SecureString",
275+
"value": "<refresh token>"
276+
}
277+
},
278+
"connectVia": {
279+
"referenceName": "AzureIntegrationRuntime",
280+
"type": "IntegrationRuntimeReference"
281+
}
282+
}
283+
}
284+
```
285+
286+
**Google BigQuery V2 linked service JSON payload:**
287+
```json
288+
{
289+
"name": "GoogleBigQuery2",
290+
"type": "Microsoft.DataFactory/factories/linkedservices",
291+
"properties": {
292+
"annotations": [],
293+
"type": "GoogleBigQueryV2",
294+
"typeProperties": {
295+
"projectId" : "<project ID>",
296+
"authenticationType" : "UserAuthentication",
297+
"clientId": "<client ID>",
298+
"clientSecret": {
299+
"type": "SecureString",
300+
"value":"<client secret>"
301+
},
302+
"refreshToken": {
303+
"type": "SecureString",
304+
"value": "<refresh token>"
305+
}
306+
},
307+
"connectVia": {
308+
"referenceName": "AutoResolveIntegrationRuntime",
309+
"type": "IntegrationRuntimeReference"
310+
}
311+
}
312+
}
313+
```
314+
315+
1. Update dataset to use the new linked service. You can either create a new dataset based on the newly created linked service, or update an existing dataset's type property from **GoogleBigQueryObject** to **GoogleBigQueryV2Object**.
316+
317+
>[!NOTE]
318+
>When transitioning linked services, the override template parameter section might only display database properties. You can resolve this by manually editing the parameters. After that the **Override template parameters** section will show the connection strings.
319+
237320
### Upgrade the Google BigQuery V2 connector from version 1.0 to version 1.1
238321

239322
In **Edit linked service** page, select 1.1 for version. For more information, see [Linked service properties](#linked-service-properties).

0 commit comments

Comments
 (0)