KAFKA-19634: Document the encoding of nullable struct #20614
Conversation
| BYTES, COMPACT_BYTES, NULLABLE_BYTES, COMPACT_NULLABLE_BYTES, | ||
| RECORDS, COMPACT_RECORDS, new ArrayOf(STRING), new CompactArrayOf(COMPACT_STRING)}; | ||
| RECORDS, COMPACT_RECORDS, new ArrayOf(STRING), new CompactArrayOf(COMPACT_STRING), | ||
| new Schema()}; |
There was a problem hiding this comment.
Can you generate the website with this change?
|
|
||
| @Override | ||
| public String documentation() { | ||
| return "Represents a composite object or null. " + |
There was a problem hiding this comment.
We should first say sth like "A struct is named by a string with a capitalized first letter and consists of one or more fields".
| public final class Schema extends Type { | ||
| public final class Schema extends DocumentedType { | ||
|
|
||
| private static final String STRUCT_TYPE_NAME = "NULLABLE_STRUCT"; |
There was a problem hiding this comment.
The current Schema class is written for a struct that's not nullable since the write() method just writes all fields without the null indicator. We should name this STRUCT and create a new NullableStruct version.
There was a problem hiding this comment.
without the null indicator
The documentation for other nullable types also needs to be updated to mention the null indicator.
| STRING, COMPACT_STRING, NULLABLE_STRING, COMPACT_NULLABLE_STRING, | ||
| BYTES, COMPACT_BYTES, NULLABLE_BYTES, COMPACT_NULLABLE_BYTES, | ||
| RECORDS, COMPACT_RECORDS, new ArrayOf(STRING), new CompactArrayOf(COMPACT_STRING)}; | ||
| RECORDS, COMPACT_RECORDS, new ArrayOf(STRING), new CompactArrayOf(COMPACT_STRING), |
There was a problem hiding this comment.
There are multiple of existing issues with the generated schemas in https://kafka.apache.org/protocol#protocol_api_keys.
-
The current RECORDS class is written as NULLABLE_RECORDS. So, we should rename it NULLABLE_RECORDS. We also need to create a new RECORDS class that's not nullable. In SchemaGenerator, we should distinguish between RECORDS and NULLABLE_RECORDS. Currently, the generated schema only has RECORDS for both nullable and non-nullable fields, which is misleading. Ditto for COMPACT_RECORDS.
-
For ArrayOf, it takes nullable as the input, but its name is always ARRAY. We need to add the nullable name. Ditto for CompactArrayOf. Currently, the generator schema only uses [] to represent an array. We need a way to denote whether it's nullable and compact.
-
When generating the schema for a struct, we just fold in the fields in the struct like the following without indicating whether that struct is nullable or not. We need a way to make that clear.
assignment => [topic_partitions] _tagged_fields
topic_partitions => topic_id [partitions] _tagged_fields
topic_id => UUID
partitions => INT32
| */ | ||
| package org.apache.kafka.common.protocol.types; | ||
|
|
||
| import org.apache.kafka.common.protocol.types.Type.DocumentedType; |
There was a problem hiding this comment.
There are a few things that we need to fix in clients/src/main/resources/common/message/README.md.
- We need to add type struct in Field Types.
- In Nullable Fields, it says uuid is nullable. This is incorrect (the generator throws an exception if a uuid filed is nullable) and needs to be removed.
- We need to add struct as a nullable type.
This patch adds documentation for nullable struct encoding in the protocol specification.
For nullable structs, the encoding is defined as follows: