feat: timestamp check for columns (#85)

Raystack rename
chore: modify error message on stopped error bq sink
feat: BQ streaming API
fix: Remove bytefield
fix: bytes to base64 in old api
Bug fix
fix: recreate writer if closed
fix: bq timestamp filter constraints
chore: getFieldByName with lowercase
Camel case fix
Timestamp check for columns

Author: ravisuhag

README.md

@@ -7,24 +7,24 @@ GRPC)
 
 ## Key Features
 
-* Instrumentation support with statsd
-* Log Sink
-* Bigquery Sink
+- Instrumentation support with statsd
+- Log Sink
+- Bigquery Sink
 
 Depot is a sink connector, which acts as a bridge between data processing systems and real sink. The APIs in this
 library can be used to push data to various sinks. Common sinks implementations will be added in this repo.
 
 ## Requirements
 
-* java8 or higher
-* gradle
+- java8 or higher
+- gradle
 
 ## How to use
 
 Explore the following resources to get started
 
-* [Reference](docs/reference) contains details about configurations of metrics and various sinks
-* [Contribute](docs/contribute/contribution.md) contains resources for anyone who wants to contribute.
+- [Reference](docs/reference) contains details about configurations of metrics and various sinks
+- [Contribute](docs/contribute/contribution.md) contains resources for anyone who wants to contribute.
 
 ### Build and run tests
 
@@ -47,21 +47,21 @@ $ ./gradlew clean
 ```xml
 
 <dependency>
-    <groupId>io.odpf</groupId>
+    <groupId>org.raystack</groupId>
     <artifactId>depot</artifactId>
     <version>version</version>
 </dependency>
 ```
 
 ```sh
-implementation group: 'io.odpf', name: 'depot', version: 'version'
+implementation group: 'org.raystack', name: 'depot', version: 'version'
 ```
 
 ### Usage example:
 
 ```java
-public interface OdpfSink extends Closeable {
-    OdpfSinkResponse pushToSink(List<OdpfMessage> messages) throws OdpfSinkException;
+public interface Sink extends Closeable {
+    SinkResponse pushToSink(List<Message> messages) throws SinkException;
 }
 ```
 
@@ -87,17 +87,18 @@ class MyClass {
 
 ### Data types
 
-Currently, sink connector library is supporting protobuf and Json format. We can set the datatype of `OdpfMessage` by
+Currently, sink connector library is supporting protobuf and Json format. We can set the datatype of `Message` by
 setting `SINK_CONNECTOR_SCHEMA_DATA_TYPE`. Each datatype has parsers which takes care of deserialization.
 
 ### Adding a new Sink
 
-Each sink will have to implement `OdpfSink` interface. The pushToSink take a batch of messages and return a response
+Each sink will have to implement `Sink` interface. The pushToSink take a batch of messages and return a response
 with error list.
 
 ### Configurations
 
 Please check the docs folder for details.
 
 ## License
+
 Firehose is [Apache 2.0](LICENSE) licensed.

build.gradle

@@ -21,8 +21,8 @@ plugins {
     id "io.github.gradle-nexus.publish-plugin" version "1.1.0"
 }
 
-group 'io.odpf'
-version '0.3.8'
+group 'org.raystack'
+version '0.4.0'
 
 repositories {
     mavenCentral()
@@ -33,13 +33,14 @@ dependencies {
     implementation group: 'com.google.protobuf', name: 'protobuf-java', version: '3.1.0'
     implementation group: 'com.datadoghq', name: 'java-dogstatsd-client', version: '2.13.0'
     implementation group: 'com.google.protobuf', name: 'protobuf-java-util', version: '3.1.0'
-    implementation group: 'io.odpf', name: 'stencil', version: '0.2.1' exclude group: 'org.slf4j'
+    implementation group: 'org.raystack', name: 'stencil', version: '0.4.0' exclude group: 'org.slf4j'
     implementation group: 'org.aeonbits.owner', name: 'owner', version: '1.0.9'
-    implementation 'com.google.cloud:google-cloud-bigquery:1.115.0'
+    implementation 'com.google.cloud:google-cloud-bigquerystorage:2.34.2'
+    implementation 'com.google.cloud:google-cloud-bigquery:2.23.2'
     implementation (group: 'com.google.cloud', name:'google-cloud-bigtable', version:'2.11.2') {
         exclude group: "io.grpc"
     }
-    implementation "io.grpc:grpc-all:1.38.0"
+    implementation 'io.grpc:grpc-all:1.53.0'
     implementation group: 'org.slf4j', name: 'jul-to-slf4j', version: '1.7.35'
     implementation group: 'redis.clients', name: 'jedis', version: '3.0.1'
     implementation group: 'org.apache.commons', name: 'commons-lang3', version: '3.5'
@@ -93,6 +94,8 @@ idea {
         testSourceDirs += file("$projectDir/src/generated/test/java")
     }
 }
+sourceCompatibility = JavaVersion.VERSION_1_8
+targetCompatibility = JavaVersion.VERSION_1_8
 
 protobuf {
     generatedFilesBaseDir = "$projectDir/src/generated"
@@ -142,10 +145,10 @@ publishing {
                 version = project.version
                 name = 'Depot'
                 description = 'A sink connector library containing multiple sink implementations'
-                url = 'https://github.com/odpf/depot'
+                url = 'https://github.com/raystack/depot'
 
                 scm {
-                    url = 'https://github.com/odpf/depot.git'
+                    url = 'https://github.com/raystack/depot.git'
                 }
 
                 licenses {

docs/README.md

@@ -7,26 +7,26 @@ GRPC)
 
 ## Key Features
 
-* Instrumentation support with statsd
-* Log Sink
-* Bigquery Sink
-* Redis Sink
-* Bigtable Sink
+- Instrumentation support with statsd
+- Log Sink
+- Bigquery Sink
+- Redis Sink
+- Bigtable Sink
 
 Depot is a sink connector, which acts as a bridge between data processing systems and real sink. The APIs in this
 library can be used to push data to various sinks. Common sinks implementations will be added in this repo.
 
 ## Requirements
 
-* java8 or higher
-* gradle
+- java8 or higher
+- gradle
 
 ## How to use
 
 Explore the following resources to get started
 
-* [Reference](docs/reference) contains details about configurations of metrics and various sinks
-* [Contribute](docs/contribute/contribution.md) contains resources for anyone who wants to contribute.
+- [Reference](docs/reference) contains details about configurations of metrics and various sinks
+- [Contribute](docs/contribute/contribution.md) contains resources for anyone who wants to contribute.
 
 ### Build and run tests
 
@@ -49,21 +49,21 @@ $ ./gradlew clean
 ```xml
 
 <dependency>
-    <groupId>io.odpf</groupId>
+    <groupId>org.raystack</groupId>
     <artifactId>depot</artifactId>
     <version>version</version>
 </dependency>
 ```
 
 ```sh
-implementation group: 'io.odpf', name: 'depot', version: 'version'
+implementation group: 'org.raystack', name: 'depot', version: 'version'
 ```
 
 ### Usage example:
 
 ```java
-public interface OdpfSink extends Closeable {
-    OdpfSinkResponse pushToSink(List<OdpfMessage> messages) throws OdpfSinkException;
+public interface Sink extends Closeable {
+    SinkResponse pushToSink(List<Message> messages) throws SinkException;
 }
 ```
 
@@ -89,12 +89,12 @@ class MyClass {
 
 ### Data types
 
-Currently, sink connector library is supporting protobuf and Json format. We can set the datatype of `OdpfMessage` by
+Currently, sink connector library is supporting protobuf and Json format. We can set the datatype of `Message` by
 setting `SINK_CONNECTOR_SCHEMA_DATA_TYPE`. Each datatype has parsers which takes care of deserialization.
 
 ### Adding a new Sink
 
-Each sink will have to implement `OdpfSink` interface. The pushToSink take a batch of messages and return a response
+Each sink will have to implement `Sink` interface. The pushToSink take a batch of messages and return a response
 with error list.
 
 ### Configurations

docs/contribute/contribution.md

@@ -2,7 +2,7 @@
 
 The following is a set of guidelines for contributing to Depot. These are mostly guidelines, not rules. Use your best judgment, and feel free to propose changes to this document in a pull request. Here are some important resources:
 
-* Github [issues](https://github.com/odpf/depot/issues) track the ongoing and reported issues.
+- Github [issues](https://github.com/raystack/depot/issues) track the ongoing and reported issues.
 
 Development of Depot happens in the open on GitHub, and we are grateful to the community for contributing bug fixes and improvements. Read below to learn how you can take part in improving Depot.
 
@@ -14,44 +14,42 @@ You don’t have to be a developer to make a contribution. We also need technica
 
 The following parts are open for contribution:
 
-* Adding a new functionality
-* Improve an existing functionality
-* Adding a new sink
-* Improve an existing sink
-* Provide suggestions to make the user experience better
-* Provide suggestions to Improve the documentation
-
+- Adding a new functionality
+- Improve an existing functionality
+- Adding a new sink
+- Improve an existing sink
+- Provide suggestions to make the user experience better
+- Provide suggestions to Improve the documentation
 
 ## How can I contribute?
 
 We use RFCs and GitHub issues to communicate ideas.
 
-* You can report a bug or suggest a feature enhancement or can just ask questions. Reach out on Github discussions for this purpose.
-* You are also welcome to add a new sink, improve monitoring and logging and improve code quality.
-* You can help with documenting new features or improve existing documentation.
-* You can also review and accept other contributions if you are a maintainer.
+- You can report a bug or suggest a feature enhancement or can just ask questions. Reach out on Github discussions for this purpose.
+- You are also welcome to add a new sink, improve monitoring and logging and improve code quality.
+- You can help with documenting new features or improve existing documentation.
+- You can also review and accept other contributions if you are a maintainer.
 
-Please submit a PR to the main branch of the `depot` repository once you are ready to submit your contribution. 
+Please submit a PR to the main branch of the `depot` repository once you are ready to submit your contribution.
 Code submission \(including a submission from project maintainers\) requires review and approval from maintainers or code owners.
-PRs that are submitted by the general public need to pass the build. 
+PRs that are submitted by the general public need to pass the build.
 Once the build is passed community members will help to review the pull request.
 
 ## Becoming a maintainer
 
 We are always interested in adding new maintainers. What we look for is a series of contributions, good taste, and an ongoing interest in the project.
 
-* maintainers will have write-access to the `depot` repositories.
-* There is no strict protocol for becoming a maintainer or PMC member. Candidates for new maintainers are typically people that are active contributors and community members.
-* Candidates for new maintainers can also be suggested by current maintainers or PMC members.
-* If you would like to become a maintainer, you should start contributing in any of the ways mentioned. You might also want to talk to other maintainers and ask for their advice and guidance.
+- maintainers will have write-access to the `depot` repositories.
+- There is no strict protocol for becoming a maintainer or PMC member. Candidates for new maintainers are typically people that are active contributors and community members.
+- Candidates for new maintainers can also be suggested by current maintainers or PMC members.
+- If you would like to become a maintainer, you should start contributing in any of the ways mentioned. You might also want to talk to other maintainers and ask for their advice and guidance.
 
 ## Guidelines
 
 Please follow these practices for your change to get merged fast and smoothly:
 
-* Contributions can only be accepted if they contain appropriate testing \(Unit and Integration Tests\).
-* If you are introducing a completely new feature or making any major changes to an existing one, we recommend starting with an RFC and get consensus on the basic design first.
-* Make sure your local build is running with all the tests and checkstyle passing.
-* If your change is related to user-facing protocols/configurations, you need to make the corresponding change in the documentation as well.
-* Docs live in the code repo under [`docs`](https://github.com/odpf/depot/tree/main/docs/README.md) so that changes to that can be done in the same PR as changes to the code.
-
+- Contributions can only be accepted if they contain appropriate testing \(Unit and Integration Tests\).
+- If you are introducing a completely new feature or making any major changes to an existing one, we recommend starting with an RFC and get consensus on the basic design first.
+- Make sure your local build is running with all the tests and checkstyle passing.
+- If your change is related to user-facing protocols/configurations, you need to make the corresponding change in the documentation as well.
+- Docs live in the code repo under [`docs`](https://github.com/raystack/depot/tree/main/docs/README.md) so that changes to that can be done in the same PR as changes to the code.