CLI-3553: Avro JSON Encoding Union Type Fails to Decode

[dangoslen]

When attempting to write records to Confluent Cloud using the CLI, we sometimes get errors related JSON Encoded Avro records that deal with union types.

According to the JSON Encoding spec from Avro, when using a union type

• if its type is null, then it is encoded as a JSON null;
• otherwise it is encoded as a JSON object with one name/value pair whose name is the type’s name and whose value is the recursively encoded value. For Avro’s named types (record, fixed or enum) the user-specified name is used, for other types the type name is used.

We can do this locally using the included /usr/bin/kafka-avro-console-producer inside of the CP Schema Registry docker container, but not using the CLI. Instead, we get

Error: cannot decode textual record "<record-namespace>.<record-name>": cannot decode textual union: cannot decode textual map: cannot determine codec: "<type>" for key: "<field-name>"

Below is the full schema (with some sensitive bits removed) and the sample record we are using

{
  "type" : "record",
  "name" : "Updated",
  "namespace" : "namespace",
  "doc" : "Updated Event",
  "fields" : [ {
    "name" : "subjectivity_id",
    "type" : {
      "type" : "string",
      "logicalType" : "string"
    },
    "doc" : "Identifier"
  }, {
    "name" : "topic",
    "type" : "string",
    "doc" : "Topic name"
  }, {
    "name" : "created_at",
    "type" : [ "null", {
      "type" : "long",
      "logicalType" : "timestamp-millis"
    } ],
    "doc" : "Date indicating when the record was created",
    "default" : null
  }, {
    "name" : "updated_at",
    "type" : [ "null", {
      "type" : "long",
      "logicalType" : "timestamp-millis"
    } ],
    "doc" : "Date indicating when the record was updated",
    "default" : null
  }]
}

{ "subjectivity_id": "subj-1234","topic": "subjectivity_updated","created_at": null,"updated_at": { "long": 1696613296 }}

Let me know what other data we can provide!