Merge pull request #214 from RADAR-base/release-2.0.0

Release 2.0.0

Author: blootsvoets

README.md

@@ -1,12 +1,12 @@
 # Restructure Kafka connector output files
 
-[![Build Status](https://travis-ci.org/RADAR-base/Restructure-HDFS-topic.svg?branch=master)](https://travis-ci.org/RADAR-base/Restructure-HDFS-topic)
-
 Data streamed by a Kafka Connector will be converted to a RADAR-base oriented output directory, by organizing it by project, user and collection date.
-It supports data written by [RADAR HDFS sink connector](https://github.com/RADAR-base/RADAR-HDFS-Sink-Connector) is streamed to files based on topic name only. This package transforms that output to a local directory structure as follows: `projectId/userId/topic/date_hour.csv`. The date and hour are extracted from the `time` field of each record, and is formatted in UTC time. This package is included in the [RADAR-Docker](https://github.com/RADAR-base/RADAR-Docker) repository, in the `dcompose/radar-cp-hadoop-stack/bin/hdfs-restructure` script.
+It supports data written by [RADAR S3 sink connector](https://github.com/RADAR-base/RADAR-S3-Connector) is streamed to files based on topic name only. This package transforms that output to a local directory structure as follows: `projectId/userId/topic/date_hour.csv`. The date and hour are extracted from the `time` field of each record, and is formatted in UTC time. This package is included in the [RADAR-Docker](https://github.com/RADAR-base/RADAR-Docker) repository, in the `dcompose/radar-cp-hadoop-stack/bin/hdfs-restructure` script.
 
 ## Upgrade instructions
 
+Since version 2.0.0, HDFS is no longer supported, only AWS S3 or Azure Blob Storage, and local file system compatible. If HDFS is still needed, please implement a HDFS source storage factory with constructor `org.radarbase.output.source.HdfsSourceStorageFactory(resourceConfig: ResourceConfig, tempPath: Path)` with method `createSourceStorage(): SourceStorage`. This implementation may be added as a separate JAR in the `lib/radar-output-plugins/` directory of where the distribution is installed.
+
 When upgrading to version 1.2.0, please follow the following instructions:
 
   - When using local target storage, ensure that:
@@ -70,21 +70,11 @@ When upgrading to version 0.6.0 from version 0.5.x or earlier, please follow the
 This package is available as docker image [`radarbase/radar-output-restructure`](https://hub.docker.com/r/radarbase/radar-output-restructure). The entrypoint of the image is the current application. So in all the commands listed in usage, replace `radar-output-restructure` with for example:
 
 ```shell
-docker run --rm -t --network hadoop -v "$PWD/output:/output" radarbase/radar-output-restructure:1.2.1 -n hdfs-namenode -o /output /myTopic
+docker run --rm -t --network s3 -v "$PWD/output:/output" radarbase/radar-output-restructure:2.0.0 -o /output /myTopic
 ```
 
 ## Command line usage
 
-When the application is installed, it can be used as follows:
-
-```shell
-radar-output-restructure --nameservice <hdfs_node> --output-directory <output_folder> <input_path_1> [<input_path_2> ...]
-```
-or you can use the short form as well:
-```shell
-radar-output-restructure -n <hdfs_node> -o <output_folder> <input_path_1> [<input_path_2> ...]
-```
-
 To display the usage and all available options you can use the help option as follows:
 ```shell
 radar-output-restructure --help
@@ -97,7 +87,7 @@ Each argument, as well as much more, can be supplied in a config file. The defau
 
 By default, this will output the data in CSV format. If JSON format is preferred, use the following instead:
 ```shell
-radar-output-restructure --format json --nameservice <hdfs_node> --output-directory <output_folder>  <input_path_1> [<input_path_2> ...]
+radar-output-restructure --format json --output-directory <output_folder>  <input_path_1> [<input_path_2> ...]
 ```
 
 By default, files records are not deduplicated after writing. To enable this behaviour, specify the option `--deduplicate` or `-d`. This set to false by default because of an issue with Biovotion data. Please see - [issue #16](https://github.com/RADAR-base/Restructure-HDFS-topic/issues/16) before enabling it. Deduplication can also be enabled or disabled per topic using the config file. If lines should be deduplicated using a subset of fields, e.g. only `sourceId` and `time` define a unique record and only the last record with duplicate values should be kept, then specify `topics: <topicName>: deduplication: distinctFields: [key.sourceId, value.time]`.
@@ -106,7 +96,7 @@ By default, files records are not deduplicated after writing. To enable this beh
 
 Another option is to output the data in compressed form. All files will get the `gz` suffix, and can be decompressed with a GZIP decoder. Note that for a very small number of records, this may actually increase the file size. Zip compression is also available.
 ```
-radar-output-restructure --compression gzip  --nameservice <hdfs_node> --output-directory <output_folder> <input_path_1> [<input_path_2> ...]
+radar-output-restructure --compression gzip --output-directory <output_folder> <input_path_1> [<input_path_2> ...]
 ```
 
 ### Redis
@@ -115,26 +105,26 @@ This package assumes a Redis service running. See the example `restructure.yml`
 
 ### Source and target
 
-The `source` and `target` properties contain resource descriptions. The source can have two types, `hdfs` and `s3`:
+The `source` and `target` properties contain resource descriptions. The source can have two types, `azure` and `s3`:
 
 ```yaml
 source:
-  type: s3  # hdfs or s3
+  type: s3  # azure or s3
   s3:
     endpoint: http://localhost:9000  # using AWS S3 endpoint is also possible.
     bucket: radar
     accessToken: minioadmin
     secretKey: minioadmin
   # only actually needed if source type is hdfs
-  hdfs:
-    nameNodes: [hdfs-namenode-1, hdfs-namenode-2]
+  azure:
+    # azure options
 ```
 
-The target is similar, but it does not support HDFS, but the local file system (`local`) or `s3`.
+The target is similar, and in addition supports the local file system (`local`).
 
 ```yaml
 target:
-  type: s3  # s3 or local
+  type: s3  # s3, local or azure
   s3:
     endpoint: http://localhost:9000
     bucket: out
@@ -179,7 +169,7 @@ The cleaner can also be enabled with the `--cleaner` command-line flag. To run t
 
 ### Service
 
-To run the output generator as a service that will regularly poll the HDFS directory, add the `--service` flag and optionally the `--interval` flag to adjust the polling interval or use the corresponding configuration file parameters.
+To run the output generator as a service that will regularly poll the source directory, add the `--service` flag and optionally the `--interval` flag to adjust the polling interval or use the corresponding configuration file parameters.
 
 ## Local build
 
@@ -192,7 +182,7 @@ This package requires at least Java JDK 8. Build the distribution with
 and install the package into `/usr/local` with for example
 ```shell
 sudo mkdir -p /usr/local
-sudo tar -xzf build/distributions/radar-output-restructure-1.2.1.tar.gz -C /usr/local --strip-components=1
+sudo tar -xzf build/distributions/radar-output-restructure-2.0.0.tar.gz -C /usr/local --strip-components=1
 ```
 
 Now the `radar-output-restructure` command should be available.

build.gradle.kts

@@ -1,6 +1,5 @@
 import com.github.benmanes.gradle.versions.updates.DependencyUpdatesTask
 import org.gradle.api.tasks.testing.logging.TestExceptionFormat.FULL
-import org.jetbrains.dokka.gradle.DokkaTask
 import org.jetbrains.kotlin.gradle.tasks.KotlinCompile
 import java.time.Duration
 
@@ -16,22 +15,18 @@ plugins {
 }
 
 group = "org.radarbase"
-version = "1.2.1"
+version = "2.0.0"
+
+repositories {
+    mavenCentral()
+}
 
 description = "RADAR-base output restructuring"
 val website = "https://radar-base.org"
 val githubRepoName = "RADAR-base/radar-output-restructure"
 val githubUrl = "https://github.com/${githubRepoName}"
 val issueUrl = "${githubUrl}/issues"
 
-application {
-    mainClass.set("org.radarbase.output.Application")
-}
-
-repositories {
-    mavenCentral()
-}
-
 sourceSets {
     create("integrationTest") {
         compileClasspath += sourceSets.main.get().output
@@ -51,13 +46,18 @@ configurations["integrationTestRuntimeOnly"].extendsFrom(
 dependencies {
     val avroVersion: String by project
     api("org.apache.avro:avro:$avroVersion")
+    val snappyVersion: String by project
+    runtimeOnly("org.xerial.snappy:snappy-java:$snappyVersion")
+
+    implementation(kotlin("reflect"))
 
     val jacksonVersion: String by project
-    implementation("com.fasterxml.jackson.core:jackson-databind:$jacksonVersion")
-    implementation("com.fasterxml.jackson.dataformat:jackson-dataformat-yaml:$jacksonVersion")
-    implementation("com.fasterxml.jackson.dataformat:jackson-dataformat-csv:$jacksonVersion")
-    implementation("com.fasterxml.jackson.module:jackson-module-kotlin:$jacksonVersion")
-    implementation("com.fasterxml.jackson.datatype:jackson-datatype-jsr310:$jacksonVersion")
+    implementation("com.fasterxml.jackson:jackson-bom:$jacksonVersion")
+    implementation("com.fasterxml.jackson.core:jackson-databind")
+    implementation("com.fasterxml.jackson.dataformat:jackson-dataformat-yaml")
+    implementation("com.fasterxml.jackson.dataformat:jackson-dataformat-csv")
+    implementation("com.fasterxml.jackson.module:jackson-module-kotlin")
+    implementation("com.fasterxml.jackson.datatype:jackson-datatype-jsr310")
 
     val jedisVersion: String by project
     implementation("redis.clients:jedis:$jedisVersion")
@@ -75,16 +75,11 @@ dependencies {
     implementation("com.azure:azure-storage-blob:$azureStorageVersion")
     implementation("com.opencsv:opencsv:5.4")
 
-    implementation("org.apache.avro:avro-mapred:$avroVersion")
-
-    val hadoopVersion: String by project
-    implementation("org.apache.hadoop:hadoop-common:$hadoopVersion")
-
     val slf4jVersion: String by project
     implementation("org.slf4j:slf4j-api:$slf4jVersion")
 
-    runtimeOnly("org.slf4j:slf4j-log4j12:$slf4jVersion")
-    runtimeOnly("org.apache.hadoop:hadoop-hdfs-client:$hadoopVersion")
+    val log4jVersion: String by project
+    runtimeOnly("org.apache.logging.log4j:log4j-slf4j-impl:$log4jVersion")
 
     val radarSchemasVersion: String by project
     testImplementation("org.radarbase:radar-schemas-commons:$radarSchemasVersion")
@@ -100,12 +95,8 @@ dependencies {
     dokkaHtmlPlugin("org.jetbrains.dokka:kotlin-as-java-plugin:$dokkaVersion")
 }
 
-tasks.withType<KotlinCompile> {
-    kotlinOptions {
-        jvmTarget = "11"
-        apiVersion = "1.4"
-        languageVersion = "1.4"
-    }
+application {
+    mainClass.set("org.radarbase.output.Application")
 }
 
 distributions {
@@ -118,63 +109,12 @@ distributions {
     }
 }
 
-tasks.startScripts {
-    classpath = classpath?.let { it + files("lib/PlaceHolderForPluginPath") }
-
-    doLast {
-        val windowsScriptFile = file(getWindowsScript())
-        val unixScriptFile    = file(getUnixScript())
-        windowsScriptFile.writeText(windowsScriptFile.readText().replace("PlaceHolderForPluginPath", "radar-output-plugins\\*"))
-        unixScriptFile.writeText(unixScriptFile.readText().replace("PlaceHolderForPluginPath", "radar-output-plugins/*"))
-    }
-}
-
-val integrationTest by tasks.registering(Test::class) {
-    description = "Runs integration tests."
-    group = "verification"
-
-    testClassesDirs = sourceSets["integrationTest"].output.classesDirs
-    classpath = sourceSets["integrationTest"].runtimeClasspath
-    outputs.upToDateWhen { false }
-    shouldRunAfter("test")
-}
-
-tasks.withType<Test> {
-    useJUnitPlatform()
-    testLogging {
-        events("passed", "skipped", "failed")
-        showStandardStreams = true
-        setExceptionFormat(FULL)
-    }
-}
-
-dockerCompose {
-    waitForTcpPortsTimeout = Duration.ofSeconds(30)
-    environment["SERVICES_HOST"] = "localhost"
-    captureContainersOutputToFiles = project.file("build/container-logs")
-    isRequiredBy(integrationTest)
-}
-
-val check by tasks
-check.dependsOn(integrationTest)
-
-tasks.withType<Tar> {
-    compression = Compression.GZIP
-    archiveExtension.set("tar.gz")
-}
-
-tasks.register("downloadDependencies") {
-    doLast {
-        description = "Pre-downloads dependencies"
-        configurations.compileClasspath.get().files
-        configurations.runtimeClasspath.get().files
+tasks.withType<KotlinCompile> {
+    kotlinOptions {
+        jvmTarget = "11"
+        apiVersion = "1.4"
+        languageVersion = "1.4"
     }
-    outputs.upToDateWhen { false }
-}
-
-tasks.register<Copy>("copyDependencies") {
-    from(configurations.runtimeClasspath.get().files)
-    into("$buildDir/third-party/")
 }
 
 // custom tasks for creating source/javadoc jars
@@ -190,6 +130,11 @@ val dokkaJar by tasks.registering(Jar::class) {
     dependsOn(tasks.dokkaJavadoc)
 }
 
+tasks.withType<Tar> {
+    compression = Compression.GZIP
+    archiveExtension.set("tar.gz")
+}
+
 tasks.withType<Jar> {
     manifest {
         attributes(
@@ -199,6 +144,15 @@ tasks.withType<Jar> {
     }
 }
 
+tasks.startScripts {
+    classpath = classpath?.let { it + files("lib/PlaceHolderForPluginPath") }
+
+    doLast {
+        windowsScript.writeText(windowsScript.readText().replace("PlaceHolderForPluginPath", "radar-output-plugins\\*"))
+        unixScript.writeText(unixScript.readText().replace("PlaceHolderForPluginPath", "radar-output-plugins/*"))
+    }
+}
+
 publishing {
     publications {
         create<MavenPublication>("mavenJar") {
@@ -213,7 +167,7 @@ publishing {
                 licenses {
                     license {
                         name.set("The Apache Software License, Version 2.0")
-                        url.set("http://www.apache.org/licenses/LICENSE-2.0.txt")
+                        url.set("https://www.apache.org/licenses/LICENSE-2.0.txt")
                         distribution.set("repo")
                     }
                 }
@@ -271,6 +225,49 @@ nexusPublishing {
     }
 }
 
+val integrationTest by tasks.registering(Test::class) {
+    description = "Runs integration tests."
+    group = "verification"
+
+    testClassesDirs = sourceSets["integrationTest"].output.classesDirs
+    classpath = sourceSets["integrationTest"].runtimeClasspath
+    outputs.upToDateWhen { false }
+    shouldRunAfter("test")
+}
+
+dockerCompose {
+    waitForTcpPortsTimeout = Duration.ofSeconds(30)
+    environment["SERVICES_HOST"] = "localhost"
+    captureContainersOutputToFiles = project.file("build/container-logs")
+    isRequiredBy(integrationTest)
+}
+
+val check by tasks
+check.dependsOn(integrationTest)
+
+tasks.withType<Test> {
+    useJUnitPlatform()
+    testLogging {
+        events("passed", "skipped", "failed")
+        showStandardStreams = true
+        exceptionFormat = FULL
+    }
+}
+
+tasks.register("downloadDependencies") {
+    doLast {
+        description = "Pre-downloads dependencies"
+        configurations.compileClasspath.get().files
+        configurations.runtimeClasspath.get().files
+    }
+    outputs.upToDateWhen { false }
+}
+
+tasks.register<Copy>("copyDependencies") {
+    from(configurations.runtimeClasspath.get().files)
+    into("$buildDir/third-party/")
+}
+
 fun isNonStable(version: String): Boolean {
     val stableKeyword = listOf("RELEASE", "FINAL", "GA").any { version.toUpperCase().contains(it) }
     val regex = "^[0-9,.v-]+(-r)?$".toRegex()
@@ -285,5 +282,5 @@ tasks.named<DependencyUpdatesTask>("dependencyUpdates").configure {
 }
 
 tasks.wrapper {
-    gradleVersion = "7.0"
+    gradleVersion = "7.0.2"
 }

gradle.properties

@@ -1,16 +1,18 @@
-kotlinVersion=1.4.32
+kotlinVersion=1.5.0
 kotlin.code.style=official
 
 avroVersion=1.10.2
+snappyVersion=1.1.8.4
 jacksonVersion=2.12.3
 hadoopVersion=3.3.0
 jCommanderVersion=1.81
 almworksVersion=1.1.2
-junitVersion=5.7.1
+junitVersion=5.7.2
 minioVersion=8.2.1
 jedisVersion=3.6.0
 slf4jVersion=1.7.30
-azureStorageVersion=12.11.0
+log4jVersion=2.14.1
+azureStorageVersion=12.11.1
 radarSchemasVersion=0.6.0
 mockitoVersion=3.8.0
 dokkaVersion=1.4.30

gradle/wrapper/gradle-wrapper.properties

@@ -1,5 +1,5 @@
 distributionBase=GRADLE_USER_HOME
 distributionPath=wrapper/dists
-distributionUrl=https\://services.gradle.org/distributions/gradle-7.0-bin.zip
+distributionUrl=https\://services.gradle.org/distributions/gradle-7.0.2-bin.zip
 zipStoreBase=GRADLE_USER_HOME
 zipStorePath=wrapper/dists