|
| 1 | +# Google BigQuery JDBC Client for Java |
| 2 | + |
| 3 | +Java idiomatic client for [BigQuery JDBC][product-docs]. |
| 4 | + |
| 5 | +[![Maven][maven-version-image]][maven-version-link] |
| 6 | +![Stability][stability-image] |
| 7 | + |
| 8 | +- [Product Documentation][product-docs] |
| 9 | +- [Client Library Documentation][javadocs] |
| 10 | + |
| 11 | + |
| 12 | +## Quickstart |
| 13 | + |
| 14 | + |
| 15 | +If you are using Maven, add this to your pom.xml file: |
| 16 | + |
| 17 | +<!-- {x-version-update-start:google-cloud-bigquery-jdbc:released} --> |
| 18 | + |
| 19 | +```xml |
| 20 | +<dependency> |
| 21 | + <groupId>com.google.cloud</groupId> |
| 22 | + <artifactId>google-cloud-bigquery-jdbc</artifactId> |
| 23 | + <version>0.1.0</version> |
| 24 | +</dependency> |
| 25 | +``` |
| 26 | + |
| 27 | +If you are using Gradle without BOM, add this to your dependencies: |
| 28 | + |
| 29 | +```Groovy |
| 30 | +implementation 'com.google.cloud:google-cloud-bigquery-jdbc:0.1.0' |
| 31 | +``` |
| 32 | + |
| 33 | +If you are using SBT, add this to your dependencies: |
| 34 | + |
| 35 | +```Scala |
| 36 | +libraryDependencies += "com.google.cloud" % "google-cloud-bigquery-jdbc" % "0.1.0" |
| 37 | +``` |
| 38 | +<!-- {x-version-update-end} --> |
| 39 | + |
| 40 | +## Authentication |
| 41 | + |
| 42 | +See the [Authentication][authentication] section in the base directory's README. |
| 43 | + |
| 44 | +## Authorization |
| 45 | + |
| 46 | +The client application making API calls must be granted [authorization scopes][auth-scopes] required for the desired BigQuery JDBC APIs, and the authenticated principal must have the [IAM role(s)][predefined-iam-roles] required to access GCP resources using the BigQuery JDBC API calls. |
| 47 | + |
| 48 | +## Developer Guide |
| 49 | + |
| 50 | +### Prerequisites |
| 51 | + |
| 52 | +You need to have either Java with Maven installed or Docker. You might want to install [`Make`](https://www.gnu.org/software/make/) to simplify running commands, otherwise please look into Makefile to check for specific configurations. |
| 53 | + |
| 54 | +### Setup |
| 55 | + |
| 56 | +`make install` primarily relies on `mvn install` command. All following commands are primarily applicable for the `google-cloud-bigquery-jdbc` project. |
| 57 | + |
| 58 | +`bigquery-external-jdbc-tests` at this moment is indepdnent and has its own configuration. |
| 59 | + |
| 60 | +### Running tests |
| 61 | + |
| 62 | +#### Unittests |
| 63 | + |
| 64 | +Run all unittests |
| 65 | + |
| 66 | +`make unittest` |
| 67 | + |
| 68 | +Run specific unittests |
| 69 | + |
| 70 | +`make unittest test=<filter>` |
| 71 | + |
| 72 | +Please reference [Maven documentation](https://maven.apache.org/surefire/maven-surefire-plugin/examples/single-test.html) for details about `<filter>` |
| 73 | + |
| 74 | +Example: `make unittest test=BigQueryArrowStructTest` |
| 75 | + |
| 76 | +#### Integration tests |
| 77 | + |
| 78 | +IMPORTANT: Running integration tests will skip unit tests. Primary focus of this command is to run specific set of tests without a lot of overhead. |
| 79 | + |
| 80 | +Majority of Integration tests rely on certain environment variables: |
| 81 | + |
| 82 | +``` |
| 83 | +# Default gcloud auth setup |
| 84 | +export GOOGLE_APPLICATION_CREDENTIALS=<path-to-service-account-json-config-file> |
| 85 | +
|
| 86 | +# Test specfic envs |
| 87 | +export SA_EMAIL=email@email.com |
| 88 | +export SA_SECRET=<path-to-service-account-json-config-file> |
| 89 | +# Alternatively it can be json content: |
| 90 | +export SA_SECRET=`cat <path-to-service-account-json-config-file>` |
| 91 | +``` |
| 92 | + |
| 93 | +Run all integration tests (currently takes 15+ minutes, so this is discouraged). |
| 94 | + |
| 95 | +`make integration-test` |
| 96 | + |
| 97 | +Run specific unitintegration test |
| 98 | + |
| 99 | +`make integration-test test=<filter>` |
| 100 | + |
| 101 | +Please reference [Maven documentation](https://maven.apache.org/surefire/maven-surefire-plugin/examples/single-test.html) for details about `<filter>` |
| 102 | + |
| 103 | +Example: `make unittest test=ITBigQueryJDBCTest#testValidServiceAccountAuthenticationOAuthPvtKey` |
| 104 | + |
| 105 | +### Dockerized environment |
| 106 | + |
| 107 | +If you don't have Java or Maven, or if you want to test changes with a different Java version, you can leverage dockerized environment. |
| 108 | + |
| 109 | +One-time run commands are similar to local development make commands: |
| 110 | + |
| 111 | +``` |
| 112 | +make docker-build |
| 113 | +make docker-unittest |
| 114 | +make docker-integration-test |
| 115 | +``` |
| 116 | + |
| 117 | +Please note that running unit of integration tests within docker doesn't leverage maven cache because it is not persisted. |
| 118 | +If you want to run multiple commands, you can start a session and treat this shell session as your local environment. |
| 119 | + |
| 120 | +``` |
| 121 | +make docker-session |
| 122 | +``` |
| 123 | + |
| 124 | +All Docker commands rely only on `GOOGLE_APPLICATION_CREDENTIALS` env being present, it will create rest of env vars as needed. |
| 125 | + |
| 126 | +### Packaging |
| 127 | + |
| 128 | +There are two ways to package Google JDBC Driver: thin jar and jar with all dependencies. |
| 129 | + |
| 130 | +#### Thin jar |
| 131 | + |
| 132 | +Thin jar can be generated by running `make docker-package`. It generates jar and list of dependencies in format compatible with copying into `pom.xml` file. |
| 133 | + |
| 134 | +#### All dependencies |
| 135 | + |
| 136 | +Jar with all dependencies can be generated by running `make docker-package-all-dependencies`. Produced jar includes all depenencies and can be used as a standalon jar with tools like R-Studio. |
| 137 | + |
| 138 | +### Custom query tests |
| 139 | + |
| 140 | +Thin client using JDBC driver can be found in 'bigquery-jdbc-test-client' folder. It allows you to run custom queries with custom connection string ot generate logs or measure performance. |
| 141 | + |
| 142 | +It can be built for Google JDBC drivers or any other JDBC drivers. For other JDBC drivers, you need to specify a version and it will download it. |
| 143 | + |
| 144 | +#### Google test |
| 145 | + |
| 146 | +For Google, you need to build an uber jar following previous steps. When it is ready, you can use these |
| 147 | +sample commands to run it: |
| 148 | + |
| 149 | +``` |
| 150 | +cd bigquery-jdbc-test-client |
| 151 | +make docker-package-for-validation JDBC_JAR_PATH=<path-to-uber-jar> |
| 152 | +make run QUERY=small |
| 153 | +``` |
| 154 | + |
| 155 | +### Night builds |
| 156 | + |
| 157 | +Nightly build is running full set of Integration tests and extended Nightly tests that includes some long-running tests (takes 20+ minutes to complete). |
| 158 | + |
| 159 | +Nightly Integration Tests include a step to build a full and thin jars and upload them to [Google Storage](https://pantheon.corp.google.com/storage/browser/bq_devtools_release_private/drivers/jdbc;tab=objects?inv=1&invt=Ab4E6g&project=bigquery-devtools-drivers&prefix=&forceOnObjectsSortingFiltering=false). |
| 160 | + |
| 161 | +They can be retrieved via following commands: |
| 162 | + |
| 163 | +``` |
| 164 | +gsutil cp gs://bq_devtools_release_private/drivers/jdbc/google-cloud-bigquery-jdbc-latest-full.jar . |
| 165 | +gsutil cp gs://bq_devtools_release_private/drivers/jdbc/google-cloud-bigquery-jdbc-latest.zip . |
| 166 | +``` |
| 167 | + |
| 168 | +#### Performance tests |
| 169 | + |
| 170 | +[Cloud Build Pipeline](https://pantheon.corp.google.com/cloud-build/triggers;region=us-east1/edit/125214da-8603-449d-8336-b6dba7196bb4?project=bigquery-devtools-drivers) |
| 171 | +is uploading latest full jar to the internal location for perf tests once a week, at the moment scheduled to do this Sunday night. |
| 172 | + |
| 173 | +### Code Coverage |
| 174 | + |
| 175 | +We're using [JaCoCo](https://www.eclemma.org/jacoco/) to track Code Coverage. `Makefile` has 2 separate set of commands for unittests and integration tests reports. |
| 176 | + |
| 177 | +You can run `make docker-coverage` and it will produce both results, you can find `jacoco-(unittest|full).zip` file in the root with results. |
| 178 | + |
| 179 | +## Getting Started |
| 180 | + |
| 181 | +### Prerequisites |
| 182 | + |
| 183 | +You will need a [Google Cloud Platform Console][developer-console] project with the BigQuery JDBC [API enabled][enable-api]. |
| 184 | + |
| 185 | +[Follow these instructions][create-project] to get your project set up. You will also need to set up the local development environment by |
| 186 | +[installing the Google Cloud Command Line Interface][cloud-cli] and running the following commands in command line: |
| 187 | +`gcloud auth login` and `gcloud config set project [YOUR PROJECT ID]`. |
| 188 | + |
| 189 | +### Installation and setup |
| 190 | + |
| 191 | +You'll need to obtain the `google-cloud-bigquery-jdbc` library. See the [Quickstart](#quickstart) section |
| 192 | +to add `google-cloud-bigquery-jdbc` as a dependency in your code. |
| 193 | + |
| 194 | +## About BigQuery JDBC |
| 195 | + |
| 196 | + |
| 197 | +[BigQuery JDBC][product-docs] |
| 198 | + |
| 199 | +See the [BigQuery JDBC client library docs][javadocs] to learn how to |
| 200 | +use this BigQuery JDBC Client Library. |
| 201 | + |
| 202 | + |
| 203 | + |
| 204 | + |
| 205 | + |
| 206 | + |
| 207 | +## Troubleshooting |
| 208 | + |
| 209 | +To get help, follow the instructions in the [shared Troubleshooting document][troubleshooting]. |
| 210 | + |
| 211 | +## Supported Java Versions |
| 212 | + |
| 213 | +Java 8 or above is required for using this client. |
| 214 | + |
| 215 | +Google's Java client libraries, |
| 216 | +[Google Cloud Client Libraries][cloudlibs] |
| 217 | +and |
| 218 | +[Google Cloud API Libraries][apilibs], |
| 219 | +follow the |
| 220 | +[Oracle Java SE support roadmap][oracle] |
| 221 | +(see the Oracle Java SE Product Releases section). |
| 222 | + |
| 223 | +### For new development |
| 224 | + |
| 225 | +In general, new feature development occurs with support for the lowest Java |
| 226 | +LTS version covered by Oracle's Premier Support (which typically lasts 5 years |
| 227 | +from initial General Availability). If the minimum required JVM for a given |
| 228 | +library is changed, it is accompanied by a [semver][semver] major release. |
| 229 | + |
| 230 | +Java 11 and (in September 2021) Java 17 are the best choices for new |
| 231 | +development. |
| 232 | + |
| 233 | +### Keeping production systems current |
| 234 | + |
| 235 | +Google tests its client libraries with all current LTS versions covered by |
| 236 | +Oracle's Extended Support (which typically lasts 8 years from initial |
| 237 | +General Availability). |
| 238 | + |
| 239 | +#### Legacy support |
| 240 | + |
| 241 | +Google's client libraries support legacy versions of Java runtimes with long |
| 242 | +term stable libraries that don't receive feature updates on a best efforts basis |
| 243 | +as it may not be possible to backport all patches. |
| 244 | + |
| 245 | +Google provides updates on a best efforts basis to apps that continue to use |
| 246 | +Java 7, though apps might need to upgrade to current versions of the library |
| 247 | +that supports their JVM. |
| 248 | + |
| 249 | +#### Where to find specific information |
| 250 | + |
| 251 | +The latest versions and the supported Java versions are identified on |
| 252 | +the individual GitHub repository `github.com/GoogleAPIs/java-SERVICENAME` |
| 253 | +and on [google-cloud-java][g-c-j]. |
| 254 | + |
| 255 | +## Versioning |
| 256 | + |
| 257 | + |
| 258 | +This library follows [Semantic Versioning](http://semver.org/). |
| 259 | + |
| 260 | + |
| 261 | + |
| 262 | +## Contributing |
| 263 | + |
| 264 | + |
| 265 | +Contributions to this library are always welcome and highly encouraged. |
| 266 | + |
| 267 | +See [CONTRIBUTING][contributing] for more information how to get started. |
| 268 | + |
| 269 | +Please note that this project is released with a Contributor Code of Conduct. By participating in |
| 270 | +this project you agree to abide by its terms. See [Code of Conduct][code-of-conduct] for more |
| 271 | +information. |
| 272 | + |
| 273 | + |
| 274 | +## License |
| 275 | + |
| 276 | +Apache 2.0 - See [LICENSE][license] for more information. |
| 277 | + |
| 278 | +## CI Status |
| 279 | + |
| 280 | +Java Version | Status |
| 281 | +------------ | ------ |
| 282 | +Java 8 | [![Kokoro CI][kokoro-badge-image-2]][kokoro-badge-link-2] |
| 283 | +Java 8 OSX | [![Kokoro CI][kokoro-badge-image-3]][kokoro-badge-link-3] |
| 284 | +Java 8 Windows | [![Kokoro CI][kokoro-badge-image-4]][kokoro-badge-link-4] |
| 285 | +Java 11 | [![Kokoro CI][kokoro-badge-image-5]][kokoro-badge-link-5] |
| 286 | + |
| 287 | +Java is a registered trademark of Oracle and/or its affiliates. |
| 288 | + |
| 289 | +[product-docs]: https://cloud.google.com/bigquery |
| 290 | +[javadocs]: https://cloud.google.com/java/docs/reference/google-cloud-bigquery/latest/history |
| 291 | +[kokoro-badge-image-1]: http://storage.googleapis.com/cloud-devrel-public/java/badges/java-bigquery-jdbc/java7.svg |
| 292 | +[kokoro-badge-link-1]: http://storage.googleapis.com/cloud-devrel-public/java/badges/java-bigquery-jdbc/java7.html |
| 293 | +[kokoro-badge-image-2]: http://storage.googleapis.com/cloud-devrel-public/java/badges/java-bigquery-jdbc/java8.svg |
| 294 | +[kokoro-badge-link-2]: http://storage.googleapis.com/cloud-devrel-public/java/badges/java-bigquery-jdbc/java8.html |
| 295 | +[kokoro-badge-image-3]: http://storage.googleapis.com/cloud-devrel-public/java/badges/java-bigquery-jdbc/java8-osx.svg |
| 296 | +[kokoro-badge-link-3]: http://storage.googleapis.com/cloud-devrel-public/java/badges/java-bigquery-jdbc/java8-osx.html |
| 297 | +[kokoro-badge-image-4]: http://storage.googleapis.com/cloud-devrel-public/java/badges/java-bigquery-jdbc/java8-win.svg |
| 298 | +[kokoro-badge-link-4]: http://storage.googleapis.com/cloud-devrel-public/java/badges/java-bigquery-jdbc/java8-win.html |
| 299 | +[kokoro-badge-image-5]: http://storage.googleapis.com/cloud-devrel-public/java/badges/java-bigquery-jdbc/java11.svg |
| 300 | +[kokoro-badge-link-5]: http://storage.googleapis.com/cloud-devrel-public/java/badges/java-bigquery-jdbc/java11.html |
| 301 | +[stability-image]: https://img.shields.io/badge/stability-unknown-red |
| 302 | +[maven-version-image]: https://img.shields.io/maven-central/v/com.google.cloud/google-cloud-bigquery-jdbc.svg |
| 303 | +[maven-version-link]: https://central.sonatype.com/artifact/com.google.cloud/google-cloud-bigquery-jdbc/0.0.0 |
| 304 | +[authentication]: https://github.com/googleapis/google-cloud-java#authentication |
| 305 | +[auth-scopes]: https://developers.google.com/identity/protocols/oauth2/scopes |
| 306 | +[predefined-iam-roles]: https://cloud.google.com/iam/docs/understanding-roles#predefined_roles |
| 307 | +[iam-policy]: https://cloud.google.com/iam/docs/overview#cloud-iam-policy |
| 308 | +[developer-console]: https://console.developers.google.com/ |
| 309 | +[create-project]: https://cloud.google.com/resource-manager/docs/creating-managing-projects |
| 310 | +[cloud-cli]: https://cloud.google.com/cli |
| 311 | +[troubleshooting]: https://github.com/googleapis/google-cloud-java/blob/main/TROUBLESHOOTING.md |
| 312 | +[contributing]: https://github.com/googleapis/java-bigquery/blob/main/CONTRIBUTING.md |
| 313 | +[code-of-conduct]: https://github.com/googleapis/java-bigquery/blob/main/CODE_OF_CONDUCT.md |
| 314 | +[license]: https://github.com/googleapis/java-bigquery/blob/main/LICENSE |
| 315 | + |
| 316 | + |
| 317 | +[libraries-bom]: https://github.com/GoogleCloudPlatform/cloud-opensource-java/wiki/The-Google-Cloud-Platform-Libraries-BOM |
| 318 | +[shell_img]: https://gstatic.com/cloudssh/images/open-btn.png |
| 319 | + |
| 320 | +[semver]: https://semver.org/ |
| 321 | +[cloudlibs]: https://cloud.google.com/apis/docs/client-libraries-explained |
| 322 | +[apilibs]: https://cloud.google.com/apis/docs/client-libraries-explained#google_api_client_libraries |
| 323 | +[oracle]: https://www.oracle.com/java/technologies/java-se-support-roadmap.html |
| 324 | +[g-c-j]: http://github.com/googleapis/google-cloud-java |
0 commit comments