Illegal Wire Type 4 (EndGroup) appended to generated gRPC response messages

[fabiocarneiro]

I am encountering a protocol violation where the server appends an unexpected tag to the end of a gRPC response. Specifically, after the expected fields of the response message are serialized, the stream contains an additional Field Number 12 with Wire Type 4 (EndGroup).

This causes standard Protobuf parsers and network inspectors to fail with an "Unknown or unsupported wiretype" error.

Wire Type 4 seems to be deprecated in proto3. Its presence here as a standalone "footer" violates the Proto3 wire specification and breaks compatibility with non-Kotlin gRPC clients. I suspect this started with the addition of the custom codecs, because in the latest releases (0.10.0-grpc-127 or 0.10.1-grpc-155), the same model was working fine.

Reproduction Structure

Sample .proto Definition

syntax = "proto3";

package com.example.generic;

message ChildItem {
    int64 timestamp = 1;
    double value = 2;
}

message EmptyRequest {}

message RootResponse {
    repeated ChildItem items = 1;
    string metadata = 2;
}

service ExampleService {
    rpc GetData (EmptyRequest) returns (RootResponse);
}

Evidence (Network Analysis)

Below is a breakdown of the response stream. The parser correctly iterates through the generated message fields, but fails immediately after the last valid field of the root message.

Binary Breakdown of the Malformed Footer: [Hex: 64] -> [Binary: 01100 100] -> [Field Number: 12, Wire Type: 4 (End Group)]

Inspector Log (Wireshark)

Protocol Buffers: /com.example.generic.ExampleService/GetData, response
  Message: com.example.generic.RootResponse
    Field(1): items (message)
      Value Length: 280
      items: (280 bytes)
        Message: com.example.generic.ChildItem
          Field(1): timestamp = 1738368000000 (int64)
          Field(2): value = 309883250.0 (double)
    Field(2): metadata = "sample_meta" (string)
    
    Field(12): <UNKNOWN>
      .110 0... = Field Number: 12
      .... .100 = Wire Type: End group (deprecated) (4)
      [Expert Info (Warning/Protocol): Unknown or unsupported wiretype]

Debugging and Maintenance Barriers

Investigating this issue is currently difficult due to the project's infrastructure and release workflow:

• Lack of Snapshot/Nightly Releases: There are no automated builds for grpc-common changes. I am forced to build the project locally and publish to mavenLocal to debug. This makes it nearly impossible to provide a shareable, isolated reproduction project, as the required dependencies are not available in any public repository.
• Build System Portability: The build process for kotlinx-rpc appears highly optimized for internal JetBrains environments. Several release tools trigger warnings regarding missing internal developer tokens, making it cumbersome for community members to contribute or verify fixes.

Proposal: Automated Releases

To improve the feedback loop for wire-protocol issues, I beg that an automated Nightly or Snapshot release mechanism for gRPC-related modules be implemented.

Moving away from manual pushes to specific grpc-release branches and providing public snapshots would:

1. Enable users to verify fixes in real-world applications without complex local build setups.
2. Reduce maintainer effort to release intermediate release candidate versions
3. Allow for shareable reproductions that point to public experimental artifacts.

[Mr3zee]

Hi @fabiocarneiro ! First of all, I'm impressed that somebody already plays with the branch and find issues. Thank you for reporting it, we'll fix it soon.

Secondly, let me address the concerns:

Lack of Snapshot/Nightly Releases: There are no automated builds for grpc-common changes. I am forced to build the project locally and publish to mavenLocal to debug. This makes it nearly impossible to provide a shareable, isolated reproduction project, as the required dependencies are not available in any public repository.

Well, that's because it is not released yet 😅
Though I really appreciate the enthusiasm, it would take us extra time to create processes that would not be needed in some months anyway because we'll move to a regular release scheme (to which we may include nightlies). I hope this clarifies the current state.

Build System Portability: The build process for kotlinx-rpc appears highly optimized for internal JetBrains environments. Several release tools trigger warnings regarding missing internal developer tokens, making it cumbersome for community members to contribute or verify fixes.

Please add specifics.

[Mr3zee]

I suspect this started with the addition of the custom codecs, because in the latest releases (0.10.0-grpc-127 or 0.10.1-grpc-155), the same model was working fine.

@Jozott00 any idea what that may be?

[fabiocarneiro]

It would take us extra time to create processes that would not be needed in some months

@Mr3zee I was not asking for a one-time/throw-away solution. My proposal is to have a permanent release-candidate release process. Once this gets stable and you are working on 1.x or 1.0.x, it will still be useful to have a 1.1-127, 1.1-155, etc. It's something you set up once and will always be there; the earlier you get it done, the more time you'll save on future manual releases. I'd be happy to help on that myself, but you guys are not using GitHub Actions ;)

Please add specifics.

│ * What went wrong:                                                                                                                                                                                                                                                                                                                                                                                                                                                                    │
│ ANDROID_HOME is not set 

Publishing build scan...                                                                                                                                                                                                                                                                                                                                                                                                                                                              │
│                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       │
│ The Develocity server (ge.jetbrains.com) rejected the request due to authentication being required.                                                                                                                                                                                                                                                                                                                                                                                   │
│ If you are able to sign in to this server in a web browser, you can authenticate your environment by running the following task:                                                                                                                                                                                                                                                                                                                                                      │
│                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       │
│ :provisionDevelocityAccessKey    

[Mr3zee]

Android home is a one time thing, last one can be ignored.

The processes I referred to was about feature branches like grpc-common and grpc-release. Having nightlies for main would indeed benefit, but right now we don't have time to set it up, and we probably will after KotlinConf. Github Actions are a bit too primitive for our processes, so moving to them is not a option

[Jozott00]

This is quite a weird bug, as we never encode GROUP_END tags because they are deprecated, which you correctly pointed out. Also, the field number (12) appears to be arbitrary. I suspect that some bytes were unintentionally included in the encoding and interpreted as GROUP_END with field number 12.

However, I was not yet able to reproduce this. Given the data from your Wireshark logs, I used the official protoc to encode

items {
  timestamp: 1738368000000
  value: 3.0988325E8
}
metadata: "sample_meta"

with

protoc --encode=com.example.generic.RootResponse test.proto < input.txt | xxd -p

which gives me

0a100880c0baf6cb3211000000727178b241120b73616d706c655f6d657461

This is the same output as what our protobuf encoder produces for the above message.

I will further investigate if our gRPC server implementation appends some invalid bytes to the message. A first test with a python gRPC client works fine.

@fabiocarneiro just to be sure, you did use the current state of the grpc-common branch right?

[Jozott00]

I tested the issue using Wireshark, but unfortunately, I was unable to reproduce it.

Protocol Buffers: /com.example.generic.ExampleService/GetData,response
    Message: com.example.generic.RootResponse
        [Message Name: com.example.generic.RootResponse]
        Field(1): items  (message)
            Value Length: 16
            items: (16 bytes)
                Message: com.example.generic.ChildItem
                    [Message Name: com.example.generic.ChildItem]
                    Field(1): timestamp = 1738368000000 (int64)
                    Field(2): value = 309883250.000000 (double)
        Field(2): metadata = My Metadata (string)