Skip to content

Commit 0829a2e

Browse files
kokoro-teamkokoro-team
committed
Release 0.34.1.
1 parent b7349b0 commit 0829a2e

File tree

2 files changed

+24
-24
lines changed

2 files changed

+24
-24
lines changed

‎CHANGES.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# Release Notes
22

3-
## Next
3+
## 0.34.1 - 2025-03-23
44

55
* PR #1212: Support OpenLineage in spark-3.x-bigquery connectors
66

‎README.md‎

Lines changed: 23 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -57,13 +57,13 @@ The latest version of the connector is publicly available in the following links
5757

5858
| version | Link |
5959
|------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
60-
| Spark 3.4 | `gs://spark-lib/bigquery/spark-3.4-bigquery-0.34.0.jar`([HTTP link](https://storage.googleapis.com/spark-lib/bigquery/spark-3.4-bigquery-0.34.0.jar)) |
61-
| Spark 3.3 | `gs://spark-lib/bigquery/spark-3.3-bigquery-0.34.0.jar`([HTTP link](https://storage.googleapis.com/spark-lib/bigquery/spark-3.3-bigquery-0.34.0.jar)) |
62-
| Spark 3.2 | `gs://spark-lib/bigquery/spark-3.2-bigquery-0.34.0.jar`([HTTP link](https://storage.googleapis.com/spark-lib/bigquery/spark-3.2-bigquery-0.34.0.jar)) |
63-
| Spark 3.1 | `gs://spark-lib/bigquery/spark-3.1-bigquery-0.34.0.jar`([HTTP link](https://storage.googleapis.com/spark-lib/bigquery/spark-3.1-bigquery-0.34.0.jar)) |
64-
| Spark 2.4 | `gs://spark-lib/bigquery/spark-2.4-bigquery-0.34.0.jar`([HTTP link](https://storage.googleapis.com/spark-lib/bigquery/spark-2.4-bigquery-0.34.0.jar)) |
65-
| Scala 2.13 | `gs://spark-lib/bigquery/spark-bigquery-with-dependencies_2.13-0.34.0.jar` ([HTTP link](https://storage.googleapis.com/spark-lib/bigquery/spark-bigquery-with-dependencies_2.13-0.34.0.jar)) |
66-
| Scala 2.12 | `gs://spark-lib/bigquery/spark-bigquery-with-dependencies_2.12-0.34.0.jar` ([HTTP link](https://storage.googleapis.com/spark-lib/bigquery/spark-bigquery-with-dependencies_2.12-0.34.0.jar)) |
60+
| Spark 3.4 | `gs://spark-lib/bigquery/spark-3.4-bigquery-0.34.1.jar`([HTTP link](https://storage.googleapis.com/spark-lib/bigquery/spark-3.4-bigquery-0.34.1.jar)) |
61+
| Spark 3.3 | `gs://spark-lib/bigquery/spark-3.3-bigquery-0.34.1.jar`([HTTP link](https://storage.googleapis.com/spark-lib/bigquery/spark-3.3-bigquery-0.34.1.jar)) |
62+
| Spark 3.2 | `gs://spark-lib/bigquery/spark-3.2-bigquery-0.34.1.jar`([HTTP link](https://storage.googleapis.com/spark-lib/bigquery/spark-3.2-bigquery-0.34.1.jar)) |
63+
| Spark 3.1 | `gs://spark-lib/bigquery/spark-3.1-bigquery-0.34.1.jar`([HTTP link](https://storage.googleapis.com/spark-lib/bigquery/spark-3.1-bigquery-0.34.1.jar)) |
64+
| Spark 2.4 | `gs://spark-lib/bigquery/spark-2.4-bigquery-0.34.1.jar`([HTTP link](https://storage.googleapis.com/spark-lib/bigquery/spark-2.4-bigquery-0.34.1.jar)) |
65+
| Scala 2.13 | `gs://spark-lib/bigquery/spark-bigquery-with-dependencies_2.13-0.34.1.jar` ([HTTP link](https://storage.googleapis.com/spark-lib/bigquery/spark-bigquery-with-dependencies_2.13-0.34.1.jar)) |
66+
| Scala 2.12 | `gs://spark-lib/bigquery/spark-bigquery-with-dependencies_2.12-0.34.1.jar` ([HTTP link](https://storage.googleapis.com/spark-lib/bigquery/spark-bigquery-with-dependencies_2.12-0.34.1.jar)) |
6767
| Scala 2.11 | `gs://spark-lib/bigquery/spark-bigquery-with-dependencies_2.11-0.29.0.jar` ([HTTP link](https://storage.googleapis.com/spark-lib/bigquery/spark-bigquery-with-dependencies_2.11-0.29.0.jar)) |
6868

6969
The first four versions are Java based connectors targeting Spark 2.4/3.1/3.2/3.3 of all Scala versions built on the new
@@ -104,13 +104,13 @@ repository. It can be used using the `--packages` option or the
104104

105105
| version | Connector Artifact |
106106
|------------|------------------------------------------------------------------------------------|
107-
| Spark 3.4 | `com.google.cloud.spark:spark-3.4-bigquery:0.34.0` |
108-
| Spark 3.3 | `com.google.cloud.spark:spark-3.3-bigquery:0.34.0` |
109-
| Spark 3.2 | `com.google.cloud.spark:spark-3.2-bigquery:0.34.0` |
110-
| Spark 3.1 | `com.google.cloud.spark:spark-3.1-bigquery:0.34.0` |
111-
| Spark 2.4 | `com.google.cloud.spark:spark-2.4-bigquery:0.34.0` |
112-
| Scala 2.13 | `com.google.cloud.spark:spark-bigquery-with-dependencies_2.13:0.34.0` |
113-
| Scala 2.12 | `com.google.cloud.spark:spark-bigquery-with-dependencies_2.12:0.34.0` |
107+
| Spark 3.4 | `com.google.cloud.spark:spark-3.4-bigquery:0.34.1` |
108+
| Spark 3.3 | `com.google.cloud.spark:spark-3.3-bigquery:0.34.1` |
109+
| Spark 3.2 | `com.google.cloud.spark:spark-3.2-bigquery:0.34.1` |
110+
| Spark 3.1 | `com.google.cloud.spark:spark-3.1-bigquery:0.34.1` |
111+
| Spark 2.4 | `com.google.cloud.spark:spark-2.4-bigquery:0.34.1` |
112+
| Scala 2.13 | `com.google.cloud.spark:spark-bigquery-with-dependencies_2.13:0.34.1` |
113+
| Scala 2.12 | `com.google.cloud.spark:spark-bigquery-with-dependencies_2.12:0.34.1` |
114114
| Scala 2.11 | `com.google.cloud.spark:spark-bigquery-with-dependencies_2.11:0.29.0` |
115115

116116
### Specifying the Spark BigQuery connector version in a Dataproc cluster
@@ -120,8 +120,8 @@ Using the standard `--jars` or `--packages` (or alternatively, the `spark.jars`/
120120

121121
To use another version than the built-in one, please do one of the following:
122122

123-
* For Dataproc clusters, using image 2.1 and above, add the following flag on cluster creation to upgrade the version `--metadata SPARK_BQ_CONNECTOR_VERSION=0.34.0`, or `--metadata SPARK_BQ_CONNECTOR_URL=gs://spark-lib/bigquery/spark-3.3-bigquery-0.34.0.jar` to create the cluster with a different jar. The URL can point to any valid connector JAR for the cluster's Spark version.
124-
* For Dataproc serverless batches, add the following property on batch creation to upgrade the version: `--properties dataproc.sparkBqConnector.version=0.34.0`, or `--properties dataproc.sparkBqConnector.uri=gs://spark-lib/bigquery/spark-3.3-bigquery-0.34.0.jar` to create the batch with a different jar. The URL can point to any valid connector JAR for the runtime's Spark version.
123+
* For Dataproc clusters, using image 2.1 and above, add the following flag on cluster creation to upgrade the version `--metadata SPARK_BQ_CONNECTOR_VERSION=0.34.1`, or `--metadata SPARK_BQ_CONNECTOR_URL=gs://spark-lib/bigquery/spark-3.3-bigquery-0.34.1.jar` to create the cluster with a different jar. The URL can point to any valid connector JAR for the cluster's Spark version.
124+
* For Dataproc serverless batches, add the following property on batch creation to upgrade the version: `--properties dataproc.sparkBqConnector.version=0.34.1`, or `--properties dataproc.sparkBqConnector.uri=gs://spark-lib/bigquery/spark-3.3-bigquery-0.34.1.jar` to create the batch with a different jar. The URL can point to any valid connector JAR for the runtime's Spark version.
125125

126126
## Hello World Example
127127

@@ -131,7 +131,7 @@ You can run a simple PySpark wordcount against the API without compilation by ru
131131

132132
```
133133
gcloud dataproc jobs submit pyspark --cluster "$MY_CLUSTER" \
134-
--jars gs://spark-lib/bigquery/spark-bigquery-with-dependencies_2.12-0.34.0.jar \
134+
--jars gs://spark-lib/bigquery/spark-bigquery-with-dependencies_2.12-0.34.1.jar \
135135
examples/python/shakespeare.py
136136
```
137137

@@ -1148,7 +1148,7 @@ using the following code:
11481148
```python
11491149
from pyspark.sql import SparkSession
11501150
spark = SparkSession.builder \
1151-
.config("spark.jars.packages", "com.google.cloud.spark:spark-bigquery-with-dependencies_2.12:0.34.0") \
1151+
.config("spark.jars.packages", "com.google.cloud.spark:spark-bigquery-with-dependencies_2.12:0.34.1") \
11521152
.getOrCreate()
11531153
df = spark.read.format("bigquery") \
11541154
.load("dataset.table")
@@ -1157,15 +1157,15 @@ df = spark.read.format("bigquery") \
11571157
**Scala:**
11581158
```scala
11591159
val spark = SparkSession.builder
1160-
.config("spark.jars.packages", "com.google.cloud.spark:spark-bigquery-with-dependencies_2.12:0.34.0")
1160+
.config("spark.jars.packages", "com.google.cloud.spark:spark-bigquery-with-dependencies_2.12:0.34.1")
11611161
.getOrCreate()
11621162
val df = spark.read.format("bigquery")
11631163
.load("dataset.table")
11641164
```
11651165

11661166
In case Spark cluster is using Scala 2.12 (it's optional for Spark 2.4.x,
11671167
mandatory in 3.0.x), then the relevant package is
1168-
com.google.cloud.spark:spark-bigquery-with-dependencies_**2.12**:0.34.0. In
1168+
com.google.cloud.spark:spark-bigquery-with-dependencies_**2.12**:0.34.1. In
11691169
order to know which Scala version is used, please run the following code:
11701170

11711171
**Python:**
@@ -1189,14 +1189,14 @@ To include the connector in your project:
11891189
<dependency>
11901190
<groupId>com.google.cloud.spark</groupId>
11911191
<artifactId>spark-bigquery-with-dependencies_${scala.version}</artifactId>
1192-
<version>0.34.0</version>
1192+
<version>0.34.1</version>
11931193
</dependency>
11941194
```
11951195

11961196
### SBT
11971197

11981198
```sbt
1199-
libraryDependencies += "com.google.cloud.spark" %% "spark-bigquery-with-dependencies" % "0.34.0"
1199+
libraryDependencies += "com.google.cloud.spark" %% "spark-bigquery-with-dependencies" % "0.34.1"
12001200
```
12011201

12021202
### Connector metrics and how to view them
@@ -1241,7 +1241,7 @@ word-break:break-word
12411241
</table>
12421242

12431243

1244-
**Note:** To use the metrics in the Spark UI page, you need to make sure the `spark-bigquery-metrics-0.34.0.jar` is the class path before starting the history-server and the connector version is `spark-3.2` or above.
1244+
**Note:** To use the metrics in the Spark UI page, you need to make sure the `spark-bigquery-metrics-0.34.1.jar` is the class path before starting the history-server and the connector version is `spark-3.2` or above.
12451245

12461246
## FAQ
12471247

0 commit comments

Comments
 (0)