Ccdb 5226 rework big query pr to fix data loss issue

.github/CODEOWNERS

@@ -0,0 +1,2 @@
+# See go/codeowners - automatically generated for confluentinc/kafka-connect-bigquery:
+*	@confluentinc/connect

.gitignore

@@ -1,8 +1,10 @@
 bin/
 build/
+target/
 *.class
 *.jar
 *.tar
+*.zip
 
 .gradle
 
@@ -22,6 +24,6 @@ build/
 key.json
 
 test.conf
-kcbq-connector/src/integration-test/resources/test.properties
+kcbq-connector/src/test/resources/test.properties
 kcbq-connector/test/docker/connect/properties/
 kcbq-connector/out/

Jenkinsfile

@@ -0,0 +1,30 @@
+#!/usr/bin/env groovy
+/*
+ * Copyright 2020 Confluent, Inc.
+ *
+ * This software contains code derived from the WePay BigQuery Kafka Connector, Copyright WePay, Inc.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+common {
+  slackChannel = '#connect-warn'
+  nodeLabel = 'docker-oraclejdk8'
+  publish = false
+  downStreamValidate = false
+  secret_file_list = [
+          ['gcp/kcbq', 'creds',   '/tmp/creds.json', 'KCBQ_TEST_KEYFILE'],
+          ['gcp/kcbq', 'creds',   '/tmp/creds.json', 'GOOGLE_APPLICATION_CREDENTIALS']
+  ]
+  timeoutHours = 2
+}

README.md

@@ -6,6 +6,14 @@
 This is an implementation of a sink connector from [Apache Kafka] to [Google BigQuery], built on top 
 of [Apache Kafka Connect]. For a comprehensive list of configuration options, see the [Connector Configuration Wiki].
 
+## History
+
+This connector was [originally developed by WePay](https://github.com/wepay/kafka-connect-bigquery).
+In late 2020 the project moved to [Confluent](https://github.com/confluentinc/kafka-connect-bigquery),
+with both companies taking on maintenance duties. All new activity such as filing issues and opening
+pull requests should now target the [Confluent](https://github.com/confluentinc/kafka-connect-bigquery)
+fork of the project.
+
 ## Download
 
 The latest releases are available in the GitHub release tab, or via [tarballs in Maven central](http://search.maven.org/#search%7Cga%7C1%7Ca%3A%22kcbq-connector%22).
@@ -40,25 +48,36 @@ save the properties file.
 
 Once you get more familiar with the connector, you might want to revisit the `connector.properties`
 file and experiment with tweaking its settings.
-
-### Building and Extracting a Tarball
+
+#### Migrating to 2.x.x
+In accordance with the introduction of schema unionization in version 2.0.0, the following changes
+to configs have been introduced and should be made when migrating:
+1. `autoUpdateSchemas` has been removed
+2. `allowNewBigQueryFields` and `allowBigQueryRequiredFieldRelaxation` have been introduced
+3. `allowSchemaUnionization` has been introduced
+
+Setting `allowNewBigQueryFields` and `allowBigQueryRequiredFieldRelaxation` to `true` while
+setting `allowSchemaUnionization` to false results in the same behavior that setting `autoUpdateSchemas`
+to `true` used to.
+
+### Building and Extracting a Confluent Hub archive
 
 If you haven't already, move into the repository's top-level directory:
 
 ```bash
 $ cd /path/to/kafka-connect-bigquery/
 ```
 
-Begin by creating a tarball of the connector with the Confluent Schema Retriever included:
+Begin by creating Confluent Hub archive of the connector with the Confluent Schema Retriever included:
 
 ```bash
-$ ./gradlew clean distTar
+$ mvn clean package -DskipTests
 ```
 
 And then extract its contents:
 
 ```bash
-$ mkdir -p bin/jar/ && tar -C bin/jar/ -xf kcbq-confluent/build/distributions/kcbq-confluent-*.tar
+$ mkdir -p bin/jar/ && cp kcbq-connector/target/components/packages/wepay-kafka-connect-bigquery-*/wepay-kafka-connect-bigquery-*/lib/*.jar bin/jar/
 ```
 
 ### Setting-Up Background Processes
@@ -131,22 +150,54 @@ adjusting flags given to the Avro Console Producer and tweaking the config setti
 
 ## Integration Testing the Connector
 
-> **NOTE**: You must have [Docker] installed and running on your machine in order to run integration
-tests for the connector.
+### Configuring the tests
+
+You must supply the following environment variables in order to run the tests:
+
+- `$KCBQ_TEST_PROJECT`: The name of the BigQuery project to use for the test
+- `$KCBQ_TEST_DATASET`: The name of the BigQuery dataset to use for the test
+- `$KCBQ_TEST_KEYFILE`: The key file used to authenticate with BigQuery during the test
+- `$KCBQ_TEST_BUCKET`: The name of the GCS bucket to use (for testing the GCS batch loading feature)
+
+The `$KCBQ_TEST_FOLDER` variable can be supplied to specify which subfolder of the GCS bucket should
+be used when testing the GCS batch loading feature; if not supplied, the top-level folder will be
+used.
+
+### Adding new GCP Credentials & BigQuery DataSet
+This section is optional in case one wants to use a different GCP project and generate new creds for that
+- **Create a GCP Service Account:** Follow instructions from https://cloud.google.com/iam/docs/creating-managing-service-accounts e.g.
+```
+gcloud iam service-accounts create kcbq-test --description="service account key for bigquery sink integration test" --display-name="kcbq-test"
+```
+- **Create Service Account Keys:** Follow instructions from https://cloud.google.com/iam/docs/creating-managing-service-account-keys e.g.
+```
+gcloud iam service-accounts keys create /tmp/creds.json --iam-account=kcbq-test@<GCP_PROJECT_NAME>.iam.gserviceaccount.com
+```
+- **Give BigQuery & Storage Admin Permissions to Service Account:**  
+  - Open https://console.cloud.google.com/iam-admin/iam?project=<GCP_PROJECT_NAME>
+  - Click on Add and enter New Principal as created above e.g. `kcbq-test@<GCP_PROJECT_NAME>.iam.gserviceaccount.com`
+  - Add following 2 roles from "Select a role" drop down menu:
+    - BigQuery -> BigQuery Admin
+    - Cloud Storage -> Storage Admin
+- **Add a BigQuery DataSet into the Project:**
+  - Open https://console.cloud.google.com/bigquery?project=<GCP_PROJECT_NAME>
+  - Click on the 3 vertical dots against the project name and click on "Create dataset" and follow the steps there.
+
+### Running the Integration Tests
 
-This all takes place in the `kcbq-connector` directory.
+```bash
+# (Re)builds the project and runs the integration tests, skipping unit tests to save a bit of time
+mvn clean package integration-test -Dskip.unit.tests=true
+```
 
 ### How Integration Testing Works
 
-Integration tests run by creating [Docker] instances for [Zookeeper], [Kafka], [Schema Registry], 
+Integration tests run by creating embedded instances for [Zookeeper], [Kafka], [Schema Registry],
 and the BigQuery Connector itself, then verifying the results using a [JUnit] test.
 
-They use schemas and data that can be found in the `test/docker/populate/test_schemas/` directory, 
-and rely on a user-provided JSON key file (like in the `quickstart` example) to access BigQuery.
-
-The project and dataset they write to, as well as the specific JSON key file they use, can be
-specified by command-line flag, environment variable, or configuration file — the exact details of
-each can be found by running the integration test script with the `-?` flag.
+They use schemas and data that can be found in the
+`kcbq-connector/src/test/resources/integration_test_cases/` directory, and rely on a user-provided
+JSON key file (like in the `quickstart` example) to access BigQuery.
 
 ### Data Corruption Concerns
 
@@ -156,70 +207,49 @@ if you have any tables in your dataset whose names begin with `kcbq_test_` and m
 name of any of the `test_schema` subdirectories. If that is the case, you should probably consider
 writing to a different project/dataset.
 
-Because Kafka and Schema Registry are run in Docker, there is no risk that running integration 
-tests will corrupt any existing data that is already on your machine, and there is also no need to 
-free up any of your ports that might currently be in use by real instances of the programs that are 
-faked in the process of testing.
-
-### Running the Integration Tests
-
-Running the series of integration tests is easy:
-
-```bash
-$ test/integrationtest.sh
-```
-
-This assumes that the project, dataset, and key file have been specified by variable or 
-configuration file. For more information on how to specify these, run the test script with
-the `--help` flag.
-
-> **NOTE:** You must have a recent version of [boot2docker], [Docker Machine], [Docker], etc.
-installed. Older versions will hang when cleaning containers, and linking doesn't work properly.
+Kafka, Schema Registry, Zookeeper, and Kafka Connect are all run as temporary embedded instances, so
+there is no risk that running integration tests will corrupt any existing data that is already on
+your machine, and there is also no need to free up any of your ports that might currently be in use
+by instances of the services that are brought up in the process of testing.
 
 ### Adding New Integration Tests
 
 Adding an integration test is a little more involved, and consists of two major steps: specifying
-Avro data to be sent to Kafka, and specifying via JUnit test how to verify that such data made 
+Avro data to be sent to Kafka, and specifying via JUnit test how to verify that such data made
 it to BigQuery as expected.
 
-To specify input data, you must create a new directory in the `test/resources/test_schemas/`
-directory with whatever name you want the Kafka topic of your test to be named, and whatever 
-string you want the name of your test's BigQuery table to be derived from. Then, create two files 
-in that directory:
+To specify input data, you must create a new directory in the
+`kcbq-connector/src/test/resources/integration_test_cases/` directory with whatever name you want
+the Kafka topic of your test to be named, and whatever string you want the name of your test's
+BigQuery table to be derived from. Then, create two files in that directory:
 
 * `schema.json` will contain the Avro schema of the type of data the new test will send
 through the connector.
 
-* `data.json` will contain a series of JSON objects, each of which should represent an [Avro] record 
-that matches the specified schema. **Each JSON object must occupy its own line, and each object 
-cannot occupy more than one line** (this inconvenience is due to limitations in the Avro 
+* `data.json` will contain a series of JSON objects, each of which should represent an [Avro] record
+that matches the specified schema. **Each JSON object must occupy its own line, and each object
+cannot occupy more than one line** (this inconvenience is due to limitations in the Avro
 Console Producer, and may be addressed in future commits).
 
-To specify data verification, add a new JUnit test to the file 
-`src/integration-test/java/com/wepay/kafka/connect/bigquery/it/BigQueryConnectorIntegrationTest.java`.
-Rows that are retrieved from BigQuery in the test are only returned as _Lists_ of _Objects_. The 
-names of their columns are not tracked. Construct a _List_ of the _Objects_ that you expect to be 
-stored in the test's BigQuery table, retrieve the actual _List_ of _Objects_ stored via a call to 
-`readAllRows()`, and then compare the two via a call to `testRows()`.
+To specify data verification, add to the test cases present in the
+`kcbq-connector/src/test/java/com/wepay/kafka/connect/bigquery/integration/BigQuerySinkConnectorIT.java`
 
 > **NOTE**: Because the order of rows is not guaranteed when reading test results from BigQuery, 
-you must include a row number as the first field of any of your test schemas, and every row of test 
-data must have a unique value for its row number (row numbers are one-indexed).
+you must include a numeric column named "row" number in all of your test schemas, and every row of
+test data must have a unique value for its row number. When data is read back from BigQuery to
+verify its accuracy, it will be returned in ascending order based on that "row" column.
 
   [Apache Avro]: https://avro.apache.org
-  [Apache Kafka Connect]: http://docs.confluent.io/3.0.0/connect/
+  [Apache Kafka Connect]: http://docs.confluent.io/current/connect/
   [Apache Kafka]: http://kafka.apache.org
   [Apache Maven]: https://maven.apache.org
   [Avro]: https://avro.apache.org
   [BigQuery]: https://cloud.google.com/bigquery/
-  [boot2docker]: http://boot2docker.io
-  [Confluent Platform]: http://docs.confluent.io/3.0.0/installation.html
+  [Confluent Platform]: http://docs.confluent.io/current/installation.html
   [Connector Configuration Wiki]: https://github.com/wepay/kafka-connect-bigquery/wiki/Connector-Configuration
-  [Docker Machine]: https://docs.docker.com/machine/
-  [Docker]: https://www.docker.com
   [Google BigQuery]: https://cloud.google.com/bigquery/
   [JUnit]: http://junit.org
-  [Kafka Connect]: http://docs.confluent.io/3.0.0/connect/
+  [Kafka Connect]: http://docs.confluent.io/current/connect/
   [Kafka]: http://kafka.apache.org
   [Maven]: https://maven.apache.org
   [Schema Registry]: https://github.com/confluentinc/schema-registry