Protobuf field presence tracking is broken for editions 2023 and 2024.

[ubiquitousbyte]

After migrating to Protobuf 4, the schema registry has managed to preserve correct field presence tracking behavior for files following the proto2 and proto3 syntax.

However, the current implementation does not take edition semantics into consideration for primitive or repeated fields. For example, It is now possible to track field presence per file descriptor and omit the optional keyword completely:

edition = "2023";
package events;

import "google/protobuf/timestamp.proto";

option features.field_presence = EXPLICIT;

message MyEvent {
    string id = 1 [features.field_presence = IMPLICIT];
    google.protobuf.Timestamp ts = 2;

    string cluster = 3;
    string host = 4;
    string release = 5;
}

In the example above, the ts, cluster, host and release fields are all optional, even though the optional keyword is missing. This is controlled by the top-level file option. The id field is not optional, as defined by its field-level option.

The current schema mapping implementation checks if the optional keyword is present in the field declaration and if it is not, it assumes it should use the default value. See here. That is incorrect behavior in editions 2023 and 2024.

The fix is to check hasPresence instead of getProto3Optional.

[rayokota]

The underlying wire parser doesn't yet support parsing Protobuf files with the edition syntax. Once that is available, we'll probably need to change the method to the following:

 private boolean isOptional(FieldDescriptor fieldDescriptor) {
    if (getEdition(fieldDescriptor.getFile()) == Edition.EDITION_PROTO2) {
      return supportOptionalForProto2 && hasOptionalKeyword(fieldDescriptor);
    } else {
      return fieldDescriptor.hasPresence();
    }
  }

[ubiquitousbyte]

Robert Yokota (@rayokota) is the ProtobufData class the underlying wire parser you're referring to? From what I can tell you guys are already on Protobuf 4, i.e parsing edition syntax is already possible.

[rayokota]

Nas Denkov (@ubiquitousbyte) , we're waiting for square/wire#2546