Gossipsub max raw data size

[kalabukdima]

Hello!
We're using gossipsub to broadcast batches of events, so I would like to know the exact limit of the message length that can be sent. There is a config option max_transmit_size, but it is applied to the wrapper struct, not the raw data. How can I find the maximum number of bytes that can fit in the raw data vec?

[dariusc93]

I dont believe there is an explicit limit that is implied to the raw data itself. At least skimming over gossipsub spec i dont see, so you would be free to increase the max_transmit_size to your use-case. Do keep in mind that increasing it also means that you would be accepting the message size from other peers up to that set limit.

[kalabukdima]

I dont believe there is an explicit limit that is implied to the raw data itself. At least skimming over gossipsub spec i dont see, so you would be free to increase the max_transmit_size to your use-case. Do keep in mind that increasing it also means that you would be accepting the message size from other peers up to that set limit.

I mean, there's no particular data size I'm trying to fit in the message, but I want to know how much data I can pass to the publish function without getting the PublishError::MessageTooLarge error. I could use something dumb like config.max_transmit_size - 5000, but why lose precious bytes if the limit may be calculated accurately?

I could try deriving it from the RawMessage struct size, but is it guaranteed not to change?

[jxs]

would you like to submit a PR addressing it?

[kalabukdima]

I don't really understand what the receiver is supposed to do in this case.

Basically, there are two kinds of message sizes that can be limited.

• user_message_size — the size of the message passed into the publish function. That's the one I'd like to know the limit of.
• wire_message_size — the size of the actual message that gets transmitted through the network.

As I understand, there is a single limit on the wire_message_size on both sender and receiver ends. And we probably want the receiver to calculate the incoming message size before parsing, so it can only apply the limit on the wire_message_size, right?
It brings us back to the original problem — if I know the maximum possible user_message_size, how do I calculate the maximum possible wire_message_size (or vice versa)?

[jxs]

you can't currently the maximum possible wire_message_size so what I am suggesting is that you submit a PR with this:

diff --git a/protocols/gossipsub/src/behaviour.rs b/protocols/gossipsub/src/behaviour.rs
index 16adb555a..a9b9c1942 100644
--- a/protocols/gossipsub/src/behaviour.rs
+++ b/protocols/gossipsub/src/behaviour.rs
@@ -583,6 +583,11 @@ where
             .data_transform
             .outbound_transform(&topic, data.clone())?;
 
+        // check that the size doesn't exceed the max transmission size
+        if transformed_data.len() > self.config.max_transmit_size() {
+            return Err(PublishError::MessageTooLarge);
+        }
+
         let raw_message = self.build_raw_message(topic, transformed_data)?;
 
         // calculate the message id from the un-transformed data
@@ -593,11 +598,6 @@ where
             topic: raw_message.topic.clone(),
         });
 
-        // check that the size doesn't exceed the max transmission size
-        if raw_message.raw_protobuf_len() > self.config.max_transmit_size() {
-            return Err(PublishError::MessageTooLarge);
-        }
-
         // Check the if the message has been published before
         if self.duplicate_cache.contains(&msg_id) {
             // This message has already been seen. We don't re-publish messages that have already

that would solve your problem right?

[kalabukdima]

I didn't realize the limit is not enforced on receiving or retransmitting messages. Made a PR: #5642

[kalabukdima]

By the way, isn't it a vulnerability that any peer is allowed to push you an unlimited amount of data in the gossipsub messages?

[jxs]

Solved with #5642