apache#34009 avro generic record to beam row conversion added support for a… (apache#34024)

* apache#34009 avro generic record to beam row conversion added support for all logical types and conversions

* using string comparison to avoid class not found issues with earlier versions of avro

* using string comparison to avoid class not found issues with earlier versions of avro

* com.github.davidmc24.gradle.plugin:gradle-avro-plugin:1.9.1

* using string comparison to avoid class not found issues with earlier versions of avro

* com.github.davidmc24.gradle.plugin:gradle-avro-plugin:1.9.1

* Add `types.Unalias` to types assertions and types switches to get an underlying type instead of types.Alias (apache#33868)

* Revert huggingface transformers to 4.30.0 (apache#34025)

* add endpoint type to WorkerMetadataResponse proto (apache#33953)

* add endpoint type to WorkerMetadataResponse proto

* add default value to endpoint_type

* add hashcode/equals to WaitTest helper classes to avoid log error (apache#34006)

* Add enable_lineage experiment to Dataflow tests (apache#34027)

* Add UUID support in SpannerSchema (apache#34034)

* Add UUID support in Spanner Schema

* Add test

* fix dashboard link (apache#34023)

* [Go SDK] Add missing type inspection case for Alias types. (apache#34039)

* removed unneeded license header

* remove unneeded license header

* Added tests for specific records generated with avro 1.8.2 and 1.9.2, and to add custom conversions

* Supporting different UUID representations in different avro versions

* Spotless fixes

* fix dependency typo

---------

Co-authored-by: Alfredo Scaccialepre <alfredo.scaccialepre@edreamsodigeo.com>
Co-authored-by: synenka <97878236+synenka@users.noreply.github.com>
Co-authored-by: Vitaly Terentyev <vitaly.terentyev@akvelon.com>
Co-authored-by: martin trieu <martinkt@google.com>
Co-authored-by: scwhittle <scwhittle@users.noreply.github.com>
Co-authored-by: Yi Hu <yathu@google.com>
Co-authored-by: Luv Agarwal <luvagarwal.k@gmail.com>
Co-authored-by: Ahmed Abualsaud <65791736+ahmedabu98@users.noreply.github.com>
Co-authored-by: Robert Burke <lostluck@users.noreply.github.com>

Author: 10 people

build.gradle.kts

@@ -44,6 +44,7 @@ tasks.rat {
 
     "**/package-list",
     "**/test.avsc",
+    "**/logical-types.avsc",
     "**/user.avsc",
     "**/test/resources/**/*.txt",
     "**/test/resources/**/*.csv",

buildSrc/build.gradle.kts

@@ -45,7 +45,7 @@ dependencies {
   implementation("com.github.spotbugs.snom:spotbugs-gradle-plugin:5.0.14")
 
   runtimeOnly("com.google.protobuf:protobuf-gradle-plugin:0.8.13")                                         // Enable proto code generation
-  runtimeOnly("com.github.davidmc24.gradle.plugin:gradle-avro-plugin:1.9.1")                               // Enable Avro code generation
+  runtimeOnly("com.github.davidmc24.gradle.plugin:gradle-avro-plugin:1.9.1")                               // Enable Avro code generation. Version 1.1.0 is the last supporting avro 1.10.2
   runtimeOnly("com.diffplug.spotless:spotless-plugin-gradle:5.6.1")                                        // Enable a code formatting plugin
   runtimeOnly("gradle.plugin.com.dorongold.plugins:task-tree:1.5")                                         // Adds a 'taskTree' task to print task dependency tree
   runtimeOnly("gradle.plugin.com.github.johnrengelman:shadow:7.1.1")                                       // Enable shading Java dependencies

sdks/java/extensions/avro/build.gradle

@@ -67,6 +67,7 @@ dependencies {
   implementation library.java.error_prone_annotations
   implementation library.java.avro
   implementation library.java.joda_time
+  implementation library.java.commons_lang3
   testImplementation(project(path: ":sdks:java:core", configuration: "shadowTest")) {
     // Exclude Avro dependencies from "core" since Avro support moved to this extension
     exclude group: "org.apache.avro", module: "avro"
@@ -143,9 +144,11 @@ avroVersions.each { k, v ->
     main = "org.apache.avro.tool.Main"
     args = [
         "compile",
+        "-bigDecimal", // Use BigDecimal for logical type decimal, similarly to what gradle-avro-plugin does
         "schema",
         "src/test/avro/org/apache/beam/sdk/extensions/avro/io/user.avsc",
         "src/test/avro/org/apache/beam/sdk/extensions/avro/schemas/test.avsc",
+        "src/test/avro/org/apache/beam/sdk/extensions/avro/schemas/logicaltypes/logical-types.avsc",
         "build/generated/sources/avro$k/test/java"
     ]
   }
@@ -161,4 +164,4 @@ static def createTaskNames(Map<String, String> prefixMap, String suffix) {
   return prefixMap.keySet().stream()
       .map { version -> "avroVersion${version}${suffix}" }
       .collect(Collectors.toList())
-}
+}