feat: support unknown fields#471
Conversation
Signed-off-by: Anthony Petrov <anthony@swirldslabs.com>
Signed-off-by: Anthony Petrov <anthony@swirldslabs.com>
pbj-core/pbj-compiler/src/main/java/com/hedera/pbj/compiler/impl/generators/ModelGenerator.java
Outdated
Show resolved
Hide resolved
pbj-core/pbj-runtime/src/main/java/com/hedera/pbj/runtime/io/ReadableSequentialData.java
Outdated
Show resolved
Hide resolved
pbj-core/pbj-runtime/src/main/java/com/hedera/pbj/runtime/UnknownField.java
Show resolved
Hide resolved
...rc/main/java/com/hedera/pbj/compiler/impl/generators/protobuf/CodecParseMethodGenerator.java
Outdated
Show resolved
Hide resolved
pbj-core/pbj-compiler/src/main/java/com/hedera/pbj/compiler/impl/generators/ModelGenerator.java
Outdated
Show resolved
Hide resolved
pbj-core/pbj-compiler/src/main/java/com/hedera/pbj/compiler/impl/generators/ModelGenerator.java
Outdated
Show resolved
Hide resolved
...rc/main/java/com/hedera/pbj/compiler/impl/generators/protobuf/CodecParseMethodGenerator.java
Outdated
Show resolved
Hide resolved
pbj-core/pbj-compiler/src/main/java/com/hedera/pbj/compiler/impl/generators/ModelGenerator.java
Outdated
Show resolved
Hide resolved
...rc/main/java/com/hedera/pbj/compiler/impl/generators/protobuf/CodecParseMethodGenerator.java
Show resolved
Hide resolved
pbj-core/pbj-runtime/src/main/java/com/hedera/pbj/runtime/UnknownField.java
Outdated
Show resolved
Hide resolved
pbj-integration-tests/src/test/java/com/hedera/pbj/integration/test/UnknownFieldsTest.java
Show resolved
Hide resolved
Signed-off-by: Anthony Petrov <anthony@swirldslabs.com>
Signed-off-by: Anthony Petrov <anthony@swirldslabs.com>
Signed-off-by: Anthony Petrov <anthony@swirldslabs.com>
jasperpotts
left a comment
There was a problem hiding this comment.
Also model generator, toString, equals, compareTo and hashCode should all include unknown fields.
Also do we test unknown oneOf values?
...rc/main/java/com/hedera/pbj/compiler/impl/generators/protobuf/CodecWriteMethodGenerator.java
Show resolved
Hide resolved
pbj-integration-tests/src/test/java/com/hedera/pbj/integration/test/UnknownFieldsTest.java
Show resolved
Hide resolved
...rc/main/java/com/hedera/pbj/compiler/impl/generators/protobuf/CodecParseMethodGenerator.java
Outdated
Show resolved
Hide resolved
pbj-core/pbj-compiler/src/main/java/com/hedera/pbj/compiler/impl/generators/ModelGenerator.java
Show resolved
Hide resolved
Signed-off-by: Anthony Petrov <anthony@swirldslabs.com>
Signed-off-by: Anthony Petrov <anthony@swirldslabs.com>
Done in latest commits.
The "oneOf" is a model-level concept. On the wire, oneOf values are no different from regular fields, and the wire format doesn't tell if it's a standalone field or a part of a oneOf. One has to know the latest model in order to interpret the value as a oneOf. In case of unknown fields, we don't know the latest model by definition, and therefore, we cannot treat oneOfs specially. They are just added to the unknown fields list as regular fields. |
jasperpotts
left a comment
There was a problem hiding this comment.
I think all good, just please check the handling of unknown fields with default values and think through stable hashcode and equals needed for use of PBJ model objects as keys in DB. While needing to be able to add fields to those objects over time and the keys still match. Maybe a unit test to make sure that is true and stays true would be good.
pbj-core/pbj-compiler/src/main/java/com/hedera/pbj/compiler/impl/generators/ModelGenerator.java
Show resolved
Hide resolved
...rc/main/java/com/hedera/pbj/compiler/impl/generators/protobuf/CodecParseMethodGenerator.java
Show resolved
Hide resolved
pbj-core/pbj-runtime/src/main/java/com/hedera/pbj/runtime/UnknownField.java
Show resolved
Hide resolved
|
It would be great to have Artem and Joseph review this PR as well because there are so many edge details to remember like stable hashcode. |
pbj-core/pbj-compiler/src/main/java/com/hedera/pbj/compiler/impl/generators/ModelGenerator.java
Show resolved
Hide resolved
pbj-core/pbj-compiler/src/main/java/com/hedera/pbj/compiler/impl/generators/ModelGenerator.java
Show resolved
Hide resolved
pbj-core/pbj-runtime/src/main/java/com/hedera/pbj/runtime/io/buffer/Bytes.java
Show resolved
Hide resolved
Signed-off-by: Anthony Petrov <anthony@swirldslabs.com>
Description:
Introducing support for unknown fields. The default behavior is unchanged and PBJ skips unknown fields by default. To activate the new behavior, the
parseUnknownFieldsargument of theCodec.parse()method needs to be set totrue. This enables the parser to collect all the known fields in a map in the model class. Later on, the codec can use this map to serialize all the unknown fields if they're present. Also, applications are able to examine the unknown fields by callingmodel.getUnknownFields()to get an unmodifiable map with the fields.Revision 2:
protobufSize()to account for unknown fields, and incopyBuilder()to account for them as well.Revision 3:
Also note: in Google Protobuf, the JSON format officially doesn't support unknown fields. So PBJ won't support them in its JSONCodec as well. See https://protobuf.dev/programming-guides/proto3/#retaining
Revision 4:
UnknownFieldrecord.Related issue(s):
Fixes #456
Notes for reviewer:
A new integ test is added.
Checklist