From 8644e8546c2dfccb6ef7527ec4a6db2597f28fdc Mon Sep 17 00:00:00 2001 From: Maxim Semenov Date: Mon, 17 Mar 2025 15:08:23 -0700 Subject: [PATCH 01/23] Add ADR for large message chunking in MQTT protocol --- doc/dev/adr/0020-large-message-chunking.md | 57 ++++++++++++++++++++++ 1 file changed, 57 insertions(+) create mode 100644 doc/dev/adr/0020-large-message-chunking.md diff --git a/doc/dev/adr/0020-large-message-chunking.md b/doc/dev/adr/0020-large-message-chunking.md new file mode 100644 index 0000000000..9bd0c27f04 --- /dev/null +++ b/doc/dev/adr/0020-large-message-chunking.md @@ -0,0 +1,57 @@ +# ADR 20: Large Message Chunking in MQTT Protocol + +## Status +Proposed + +## Context +The MQTT protocol has inherent message size limitations imposed by brokers and network constraints. Azure IoT Operations scenarios often require transmitting payloads that exceed these limits (e.g., firmware updates, large telemetry batches, complex configurations). Without a standardized chunking mechanism, applications must implement their own fragmentation strategies, leading to inconsistent implementations and interoperability issues. + +## Decision +We will implement protocol-level message chunking as part of the MQTT protocol layer by using MQTT user properties to carry chunk metadata. This approach will make the chunking mechanism explicit in the protocol rather than hiding it in higher or lower layers. + +The chunking mechanism will: +1. Be applied only to MQTT PUBLISH packets +2. Use standardized user properties for chunk metadata: + - `__ci`: Chunk index in the sequence (0-based) + - `__tc`: Total number of chunks in the complete message + - `__mid`: Original message identifier for reassembly + - `__cs`: Optional hash/checksum of complete payload + +### Protocol Flow +**Sending Process:** +- When a payload exceeds the maximum packet size, the client intercepts it before transmission +- The message is split into fixed-size chunks (with potentially smaller last chunk) +- Each chunk is sent as a separate MQTT message with the same topic but with chunk metadata. +- Any user properties and additional metadata not mandated by the MQTT protocol to appear in every message, originally set on the initial PUBLISH packet, will be included only in the first chunk. +- QoS settings are maintained across all chunks. + +**Receiving Process:** + - The MQTT client receives messages and identifies chunked messages by the presence of chunk metadata. + - Chunks are stored in a temporary buffer, indexed by message ID (`__mid`) and chunk index (`__ci`). + - When all chunks for a message ID are received, they are reassembled in order. + - The reconstructed message is then processed as a single message by the application callback. + +## Consequences + +### Benefits +- **Standards-Based:** Uses existing MQTT features rather than custom transport mechanisms +- **Protocol Transparent:** Makes chunking behavior explicit in the protocol +- **Property Preservation:** Maintains topic, QoS, and other message properties consistently +- **Network Optimized:** Allows efficient transmission of large payloads over constrained networks + +### Implementation Considerations +- **Error Handling:** + - Chunk timeout mechanisms + - Missing chunk detection + - Error propagation to application code +- **Performance Optimization:** + - Dynamic chunk sizing based on broker limitations + - Concurrent chunk transmission + - Efficient memory usage during reassembly +- **Security:** + - Validate message integrity across chunks + - Prevent chunk injection attacks + +## Open Questions +1. How do we determine the optimal chunk size? Should it be based on the broker's max size, network conditions, or configurable by the application? +2. Do we create a new API method (`PublishLargeAsync()`) or use the existing `PublishAsync()` API with transparent chunking for oversized payloads? From 6de0c8ef130415dd2e3033f81271284f271e0e5c Mon Sep 17 00:00:00 2001 From: Maxim Semenov <87454347+maximsemenov80@users.noreply.github.com> Date: Tue, 18 Mar 2025 16:13:20 -0700 Subject: [PATCH 02/23] accept suggestion Co-authored-by: Valerie Avva Lim <54871851+vaavva@users.noreply.github.com> --- doc/dev/adr/0020-large-message-chunking.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/dev/adr/0020-large-message-chunking.md b/doc/dev/adr/0020-large-message-chunking.md index 9bd0c27f04..3d33a9aeb7 100644 --- a/doc/dev/adr/0020-large-message-chunking.md +++ b/doc/dev/adr/0020-large-message-chunking.md @@ -19,7 +19,7 @@ The chunking mechanism will: ### Protocol Flow **Sending Process:** -- When a payload exceeds the maximum packet size, the client intercepts it before transmission +- When a payload exceeds the maximum packet size, the MQTT client intercepts it before transmission - The message is split into fixed-size chunks (with potentially smaller last chunk) - Each chunk is sent as a separate MQTT message with the same topic but with chunk metadata. - Any user properties and additional metadata not mandated by the MQTT protocol to appear in every message, originally set on the initial PUBLISH packet, will be included only in the first chunk. From 75a767cc1ea3dedf0ef0b70100ebf6845d634ce9 Mon Sep 17 00:00:00 2001 From: Maxim Semenov <87454347+maximsemenov80@users.noreply.github.com> Date: Tue, 18 Mar 2025 16:19:56 -0700 Subject: [PATCH 03/23] accept suggestion Co-authored-by: Valerie Avva Lim <54871851+vaavva@users.noreply.github.com> --- doc/dev/adr/0020-large-message-chunking.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/dev/adr/0020-large-message-chunking.md b/doc/dev/adr/0020-large-message-chunking.md index 3d33a9aeb7..c09f4b9c4d 100644 --- a/doc/dev/adr/0020-large-message-chunking.md +++ b/doc/dev/adr/0020-large-message-chunking.md @@ -7,7 +7,7 @@ Proposed The MQTT protocol has inherent message size limitations imposed by brokers and network constraints. Azure IoT Operations scenarios often require transmitting payloads that exceed these limits (e.g., firmware updates, large telemetry batches, complex configurations). Without a standardized chunking mechanism, applications must implement their own fragmentation strategies, leading to inconsistent implementations and interoperability issues. ## Decision -We will implement protocol-level message chunking as part of the MQTT protocol layer by using MQTT user properties to carry chunk metadata. This approach will make the chunking mechanism explicit in the protocol rather than hiding it in higher or lower layers. +We will implement sdk-level message chunking as part of the MQTT layer by using MQTT user properties to carry chunk metadata. This approach will make the chunking mechanism explicit in the SDK rather than hiding it in higher or lower layers. The chunking mechanism will: 1. Be applied only to MQTT PUBLISH packets From e28408328bbb9c5d91d564a41ca3ff7e7c5b9c20 Mon Sep 17 00:00:00 2001 From: Maxim Semenov Date: Wed, 19 Mar 2025 13:32:24 -0700 Subject: [PATCH 04/23] Address review comments --- doc/dev/adr/0020-large-message-chunking.md | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/doc/dev/adr/0020-large-message-chunking.md b/doc/dev/adr/0020-large-message-chunking.md index c09f4b9c4d..5a5a413a20 100644 --- a/doc/dev/adr/0020-large-message-chunking.md +++ b/doc/dev/adr/0020-large-message-chunking.md @@ -12,10 +12,7 @@ We will implement sdk-level message chunking as part of the MQTT layer by using The chunking mechanism will: 1. Be applied only to MQTT PUBLISH packets 2. Use standardized user properties for chunk metadata: - - `__ci`: Chunk index in the sequence (0-based) - - `__tc`: Total number of chunks in the complete message - - `__mid`: Original message identifier for reassembly - - `__cs`: Optional hash/checksum of complete payload + - `__chunk`: `mid:;ci:;tc:;cs:`; `mid,ci` - present for every chunk; `tc,cs` - present only for the first chunk. ### Protocol Flow **Sending Process:** From 8c29c44c0c2018fd4ccaa4697fa6674c29cd1f19 Mon Sep 17 00:00:00 2001 From: Maxim Semenov Date: Mon, 24 Mar 2025 10:04:35 -0700 Subject: [PATCH 05/23] progress --- doc/dev/adr/0020-large-message-chunking.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/doc/dev/adr/0020-large-message-chunking.md b/doc/dev/adr/0020-large-message-chunking.md index 5a5a413a20..34d7c37d53 100644 --- a/doc/dev/adr/0020-large-message-chunking.md +++ b/doc/dev/adr/0020-large-message-chunking.md @@ -12,7 +12,7 @@ We will implement sdk-level message chunking as part of the MQTT layer by using The chunking mechanism will: 1. Be applied only to MQTT PUBLISH packets 2. Use standardized user properties for chunk metadata: - - `__chunk`: `mid:;ci:;tc:;cs:`; `mid,ci` - present for every chunk; `tc,cs` - present only for the first chunk. + - `__chunk`: `;;;`; `,` - present for every chunk; `,` - present only for the first chunk. ### Protocol Flow **Sending Process:** @@ -24,22 +24,21 @@ The chunking mechanism will: **Receiving Process:** - The MQTT client receives messages and identifies chunked messages by the presence of chunk metadata. - - Chunks are stored in a temporary buffer, indexed by message ID (`__mid`) and chunk index (`__ci`). - - When all chunks for a message ID are received, they are reassembled in order. + - Chunks are stored in a temporary buffer, indexed by message ID and chunk index. + - When all chunks for a message ID are received, they are reassembled in order and message checksum verified. - The reconstructed message is then processed as a single message by the application callback. ## Consequences ### Benefits - **Standards-Based:** Uses existing MQTT features rather than custom transport mechanisms -- **Protocol Transparent:** Makes chunking behavior explicit in the protocol +- **Protocol Transparent:** Makes chunking behavior explicit in the MQTT protocol - **Property Preservation:** Maintains topic, QoS, and other message properties consistently - **Network Optimized:** Allows efficient transmission of large payloads over constrained networks ### Implementation Considerations - **Error Handling:** - Chunk timeout mechanisms - - Missing chunk detection - Error propagation to application code - **Performance Optimization:** - Dynamic chunk sizing based on broker limitations @@ -52,3 +51,4 @@ The chunking mechanism will: ## Open Questions 1. How do we determine the optimal chunk size? Should it be based on the broker's max size, network conditions, or configurable by the application? 2. Do we create a new API method (`PublishLargeAsync()`) or use the existing `PublishAsync()` API with transparent chunking for oversized payloads? +3. Chunking and shared subscriptions: How do we handle chunked messages across multiple subscribers? From 2251ff3f98ebe527c5dbe9dbe65c9ff5defa6fc6 Mon Sep 17 00:00:00 2001 From: Maxim Semenov Date: Mon, 24 Mar 2025 11:19:04 -0700 Subject: [PATCH 06/23] progress --- doc/dev/adr/0020-large-message-chunking.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/doc/dev/adr/0020-large-message-chunking.md b/doc/dev/adr/0020-large-message-chunking.md index 34d7c37d53..767a924016 100644 --- a/doc/dev/adr/0020-large-message-chunking.md +++ b/doc/dev/adr/0020-large-message-chunking.md @@ -28,6 +28,9 @@ The chunking mechanism will: - When all chunks for a message ID are received, they are reassembled in order and message checksum verified. - The reconstructed message is then processed as a single message by the application callback. +**Note:** +Chunk size is a three party agreement: *publisher-broker-subscriber*. Publisher can set the chunk size, but it is capped by the subscriber's or broker's maximum packet size which ever is smaller. + ## Consequences ### Benefits @@ -41,7 +44,6 @@ The chunking mechanism will: - Chunk timeout mechanisms - Error propagation to application code - **Performance Optimization:** - - Dynamic chunk sizing based on broker limitations - Concurrent chunk transmission - Efficient memory usage during reassembly - **Security:** @@ -49,6 +51,5 @@ The chunking mechanism will: - Prevent chunk injection attacks ## Open Questions -1. How do we determine the optimal chunk size? Should it be based on the broker's max size, network conditions, or configurable by the application? -2. Do we create a new API method (`PublishLargeAsync()`) or use the existing `PublishAsync()` API with transparent chunking for oversized payloads? -3. Chunking and shared subscriptions: How do we handle chunked messages across multiple subscribers? +1. Do we create a new API method (`PublishLargeAsync()`) or use the existing `PublishAsync()` API with transparent chunking for oversized payloads? +2. Chunking and shared subscriptions: How do we handle chunked messages across multiple subscribers? From a8a9cc1325fed5c37478273bc205660c8040048d Mon Sep 17 00:00:00 2001 From: Maxim Semenov Date: Tue, 13 May 2025 09:38:09 -0700 Subject: [PATCH 07/23] Address coments to large message chunking implementation details --- doc/dev/adr/0020-large-message-chunking.md | 83 ++++++++++++++-------- 1 file changed, 54 insertions(+), 29 deletions(-) diff --git a/doc/dev/adr/0020-large-message-chunking.md b/doc/dev/adr/0020-large-message-chunking.md index 767a924016..59f05567a6 100644 --- a/doc/dev/adr/0020-large-message-chunking.md +++ b/doc/dev/adr/0020-large-message-chunking.md @@ -7,49 +7,74 @@ Proposed The MQTT protocol has inherent message size limitations imposed by brokers and network constraints. Azure IoT Operations scenarios often require transmitting payloads that exceed these limits (e.g., firmware updates, large telemetry batches, complex configurations). Without a standardized chunking mechanism, applications must implement their own fragmentation strategies, leading to inconsistent implementations and interoperability issues. ## Decision -We will implement sdk-level message chunking as part of the MQTT layer by using MQTT user properties to carry chunk metadata. This approach will make the chunking mechanism explicit in the SDK rather than hiding it in higher or lower layers. +We will implement sdk-level message chunking as part of the Protocol layer to transparently handle messages exceeding the MQTT broker's maximum packet size. -The chunking mechanism will: -1. Be applied only to MQTT PUBLISH packets -2. Use standardized user properties for chunk metadata: - - `__chunk`: `;;;`; `,` - present for every chunk; `,` - present only for the first chunk. +**The chunking mechanism will**: +- Be enabled/disabled by a configuration setting. +- Use standardized user properties for chunk metadata: + - The `__chunk` user property will contain a JSON object with chunking metadata. + - The JSON structure will include: + ```json + { + "messageId": "unique-id-for-chunked-message", + "chunkIndex": 0, + "timeout" : "00:00:10", + "totalChunks": 5, + "checksum": "optional-message-hash" + } + ``` + - `messageId, chunkIndex, timeout` - present for every chunk; `totalChunks, checksum` - present only for the first chunk. -### Protocol Flow -**Sending Process:** -- When a payload exceeds the maximum packet size, the MQTT client intercepts it before transmission -- The message is split into fixed-size chunks (with potentially smaller last chunk) -- Each chunk is sent as a separate MQTT message with the same topic but with chunk metadata. -- Any user properties and additional metadata not mandated by the MQTT protocol to appear in every message, originally set on the initial PUBLISH packet, will be included only in the first chunk. -- QoS settings are maintained across all chunks. +**Chunk size calculation**: +- Maximum chunk size will be derived from the MQTT CONNECT packet's Maximum Packet Size. +- A static overhead value will be subtracted from the Maximum Packet Size to account for MQTT packet headers, topic name, user properties, and other metadata. +- The overhead size will be configurable, large enough to simplify calculations while ensuring we stay under the broker's limit. -**Receiving Process:** - - The MQTT client receives messages and identifies chunked messages by the presence of chunk metadata. - - Chunks are stored in a temporary buffer, indexed by message ID and chunk index. - - When all chunks for a message ID are received, they are reassembled in order and message checksum verified. - - The reconstructed message is then processed as a single message by the application callback. +**Implementation layer**: +- Chunking will be implemented as middleware in the Protocol layer between serialization and MQTT client. + ``` + Application → Protocol Layer (Serialization) → Chunking Middleware → MQTT Client → Broker + ``` +- This makes chunking transparent to application code and compatible with all serialization formats. -**Note:** -Chunk size is a three party agreement: *publisher-broker-subscriber*. Publisher can set the chunk size, but it is capped by the subscriber's or broker's maximum packet size which ever is smaller. - -## Consequences +- Sending Process: + - When a payload exceeds the maximum packet size, the midlware intercepts it before transmission + - The message is split into fixed-size chunks (with potentially smaller last chunk) + - Each chunk is sent as a separate MQTT message with the same topic but with chunk metadata. + - Effort should be made to minimize user properties copied over to every chunk: first chunk will have full set of original user properties and the rest only thoses that are neccessary to reassamble original message (ex.: ```$partition``` property to support shared subscriptions:). + - QoS settings are maintained across all chunks. +- Receiving Process: + - The Chunking aware client receives messages and identifies chunked messages by the presence of chunk metadata. + - Chunks are stored in a temporary buffer, indexed by message ID and chunk index. + - When all chunks for a message ID are received, they are reassembled in order and message checksum verified. + - The reconstructed message is then processed as a single message by the application. ### Benefits -- **Standards-Based:** Uses existing MQTT features rather than custom transport mechanisms -- **Protocol Transparent:** Makes chunking behavior explicit in the MQTT protocol - **Property Preservation:** Maintains topic, QoS, and other message properties consistently - **Network Optimized:** Allows efficient transmission of large payloads over constrained networks ### Implementation Considerations - **Error Handling:** - - Chunk timeout mechanisms + - Chunk timeout mechanisms, fixed or sliding timeout window approaches can be used (see Chunk Timeout Mechanism Options in Appendix) - Error propagation to application code - **Performance Optimization:** - Concurrent chunk transmission - Efficient memory usage during reassembly - **Security:** - - Validate message integrity across chunks - - Prevent chunk injection attacks + - Validate message integrity across chunks and prevent chunk injection attacks (covered if checksumm implemented) + +# Appendix + +## Chunk Timeout Mechanism Options + +1. Fixed Timeout Window + - Set a single timeout period after receiving the first chunk + - If all chunks aren't received within this window, the message is considered failed + - **Pros**: Simple implementation, predictable behavior + - **Cons**: Not adaptive to message size or network conditions -## Open Questions -1. Do we create a new API method (`PublishLargeAsync()`) or use the existing `PublishAsync()` API with transparent chunking for oversized payloads? -2. Chunking and shared subscriptions: How do we handle chunked messages across multiple subscribers? +2. Sliding Timeout Window + - Reset the timeout each time a new chunk arrives + - Only expire the chunked message if there's a long gap between chunks + - **Pros**: Tolerates varying network conditions and delivery rates + - **Cons**: Could keep resources allocated for extended periods \ No newline at end of file From 496715078d536646cd55e68540d6cbc87b861038 Mon Sep 17 00:00:00 2001 From: Maxim Semenov Date: Tue, 13 May 2025 09:52:16 -0700 Subject: [PATCH 08/23] progress --- doc/dev/adr/0020-large-message-chunking.md | 28 ++++++++++++++++++---- 1 file changed, 24 insertions(+), 4 deletions(-) diff --git a/doc/dev/adr/0020-large-message-chunking.md b/doc/dev/adr/0020-large-message-chunking.md index 59f05567a6..c6435d65c7 100644 --- a/doc/dev/adr/0020-large-message-chunking.md +++ b/doc/dev/adr/0020-large-message-chunking.md @@ -20,7 +20,7 @@ We will implement sdk-level message chunking as part of the Protocol layer to tr "chunkIndex": 0, "timeout" : "00:00:10", "totalChunks": 5, - "checksum": "optional-message-hash" + "checksum": "message-hash" } ``` - `messageId, chunkIndex, timeout` - present for every chunk; `totalChunks, checksum` - present only for the first chunk. @@ -55,13 +55,13 @@ We will implement sdk-level message chunking as part of the Protocol layer to tr ### Implementation Considerations - **Error Handling:** - - Chunk timeout mechanisms, fixed or sliding timeout window approaches can be used (see Chunk Timeout Mechanism Options in Appendix) + - Chunk timeout mechanisms (see Chunk Timeout Mechanism Options in the Appendix) - Error propagation to application code - **Performance Optimization:** - Concurrent chunk transmission - Efficient memory usage during reassembly - **Security:** - - Validate message integrity across chunks and prevent chunk injection attacks (covered if checksumm implemented) + - Validate message integrity across chunks and prevent chunk injection attacks (see Checksum Algorithm Options for MQTT Message Chunking in the Appendix) # Appendix @@ -77,4 +77,24 @@ We will implement sdk-level message chunking as part of the Protocol layer to tr - Reset the timeout each time a new chunk arrives - Only expire the chunked message if there's a long gap between chunks - **Pros**: Tolerates varying network conditions and delivery rates - - **Cons**: Could keep resources allocated for extended periods \ No newline at end of file + - **Cons**: Could keep resources allocated for extended periods + +## Checksum Algorithm Options for MQTT Message Chunking + +1. MD5 + - **Description**: 128-bit hash function + - **Pros**: Good performance, reasonable size (16 bytes), widely implemented + - **Cons**: No longer considered cryptographically secure + - **Best for**: Basic integrity verification without security requirements + +2. SHA-256 + - **Description**: Secure hash algorithm producing 256-bit output + - **Pros**: Cryptographically secure, widely supported in all target languages + - **Cons**: Larger output size (32 bytes), more computation required + - **Best for**: Applications requiring message security and tamper protection + +3. BLAKE2b + - **Description**: Modern cryptographic hash function + - **Pros**: Faster than MD5 but with security comparable to SHA-3 + - **Cons**: May not be as universally available in standard libraries + - **Best for**: Performance-critical applications that still need security \ No newline at end of file From 0af2742526d7b6ff78796725e408aa8ceaf6c486 Mon Sep 17 00:00:00 2001 From: Maxim Semenov <87454347+maximsemenov80@users.noreply.github.com> Date: Wed, 14 May 2025 11:09:37 -0700 Subject: [PATCH 09/23] Update doc/dev/adr/0020-large-message-chunking.md Co-authored-by: Tim Taylor --- doc/dev/adr/0020-large-message-chunking.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/dev/adr/0020-large-message-chunking.md b/doc/dev/adr/0020-large-message-chunking.md index c6435d65c7..4b239f843d 100644 --- a/doc/dev/adr/0020-large-message-chunking.md +++ b/doc/dev/adr/0020-large-message-chunking.md @@ -38,7 +38,7 @@ We will implement sdk-level message chunking as part of the Protocol layer to tr - This makes chunking transparent to application code and compatible with all serialization formats. - Sending Process: - - When a payload exceeds the maximum packet size, the midlware intercepts it before transmission + - When a payload exceeds the maximum packet size, the middleware intercepts it before transmission - The message is split into fixed-size chunks (with potentially smaller last chunk) - Each chunk is sent as a separate MQTT message with the same topic but with chunk metadata. - Effort should be made to minimize user properties copied over to every chunk: first chunk will have full set of original user properties and the rest only thoses that are neccessary to reassamble original message (ex.: ```$partition``` property to support shared subscriptions:). From cc724bb11fcb3f6655c4e8926627985ca8e0442c Mon Sep 17 00:00:00 2001 From: Maxim Semenov Date: Thu, 5 Jun 2025 15:09:00 -0700 Subject: [PATCH 10/23] Address review comments --- doc/dev/adr/0020-large-message-chunking.md | 87 ++++++++-------------- 1 file changed, 32 insertions(+), 55 deletions(-) diff --git a/doc/dev/adr/0020-large-message-chunking.md b/doc/dev/adr/0020-large-message-chunking.md index 4b239f843d..b9645a23fe 100644 --- a/doc/dev/adr/0020-large-message-chunking.md +++ b/doc/dev/adr/0020-large-message-chunking.md @@ -1,47 +1,58 @@ # ADR 20: Large Message Chunking in MQTT Protocol ## Status + Proposed ## Context + The MQTT protocol has inherent message size limitations imposed by brokers and network constraints. Azure IoT Operations scenarios often require transmitting payloads that exceed these limits (e.g., firmware updates, large telemetry batches, complex configurations). Without a standardized chunking mechanism, applications must implement their own fragmentation strategies, leading to inconsistent implementations and interoperability issues. ## Decision + We will implement sdk-level message chunking as part of the Protocol layer to transparently handle messages exceeding the MQTT broker's maximum packet size. **The chunking mechanism will**: + - Be enabled/disabled by a configuration setting. - Use standardized user properties for chunk metadata: - - The `__chunk` user property will contain a JSON object with chunking metadata. - - The JSON structure will include: - ```json - { + + - The `__chunk` user property will contain a JSON object with chunking metadata. + - The JSON structure will include: + + ```json + { "messageId": "unique-id-for-chunked-message", "chunkIndex": 0, - "timeout" : "00:00:10", + "timeout" : "10000", "totalChunks": 5, "checksum": "message-hash" } ``` - - `messageId, chunkIndex, timeout` - present for every chunk; `totalChunks, checksum` - present only for the first chunk. + + - `messageId, chunkIndex, timeout` - present for every chunk; `totalChunks, checksum` - present only for the first chunk. `messageId` is UUID, `timeout` is milliseconds. **Chunk size calculation**: + - Maximum chunk size will be derived from the MQTT CONNECT packet's Maximum Packet Size. - A static overhead value will be subtracted from the Maximum Packet Size to account for MQTT packet headers, topic name, user properties, and other metadata. - The overhead size will be configurable, large enough to simplify calculations while ensuring we stay under the broker's limit. -**Implementation layer**: -- Chunking will be implemented as middleware in the Protocol layer between serialization and MQTT client. - ``` - Application → Protocol Layer (Serialization) → Chunking Middleware → MQTT Client → Broker - ``` -- This makes chunking transparent to application code and compatible with all serialization formats. +**Chunk Timeout Mechanism:** + +- Set a single timeout period after receiving the first chunk. +- If all chunks aren't received within this window, the message is considered failed. + +**Checksum Algorithm Options for MQTT Message Chunking** + +SDK will provide user with options to inject their algorithm of choice or use SDK's default SHA-256. + +**Implementation layer:** - Sending Process: - - When a payload exceeds the maximum packet size, the middleware intercepts it before transmission - - The message is split into fixed-size chunks (with potentially smaller last chunk) + - When a payload exceeds the maximum packet size, the message is split into fixed-size chunks (with potentially smaller last chunk) - Each chunk is sent as a separate MQTT message with the same topic but with chunk metadata. - - Effort should be made to minimize user properties copied over to every chunk: first chunk will have full set of original user properties and the rest only thoses that are neccessary to reassamble original message (ex.: ```$partition``` property to support shared subscriptions:). + - Effort should be made to minimize user properties copied over to every chunk: first chunk will have full set of original user properties and the rest only those that are necessary to reassemble original message (ex.: ```$partition``` property to support shared subscriptions:). - QoS settings are maintained across all chunks. - Receiving Process: - The Chunking aware client receives messages and identifies chunked messages by the presence of chunk metadata. @@ -50,51 +61,17 @@ We will implement sdk-level message chunking as part of the Protocol layer to tr - The reconstructed message is then processed as a single message by the application. ### Benefits -- **Property Preservation:** Maintains topic, QoS, and other message properties consistently -- **Network Optimized:** Allows efficient transmission of large payloads over constrained networks + +- **Property Preservation:** maintains topic, QoS, and other message properties consistently. +- **Network Optimized:** allows efficient transmission of large payloads over constrained networks. ### Implementation Considerations + - **Error Handling:** - - Chunk timeout mechanisms (see Chunk Timeout Mechanism Options in the Appendix) + - Chunk timeout mechanisms (see Chunk Timeout Mechanism) - Error propagation to application code - **Performance Optimization:** - Concurrent chunk transmission - Efficient memory usage during reassembly - **Security:** - - Validate message integrity across chunks and prevent chunk injection attacks (see Checksum Algorithm Options for MQTT Message Chunking in the Appendix) - -# Appendix - -## Chunk Timeout Mechanism Options - -1. Fixed Timeout Window - - Set a single timeout period after receiving the first chunk - - If all chunks aren't received within this window, the message is considered failed - - **Pros**: Simple implementation, predictable behavior - - **Cons**: Not adaptive to message size or network conditions - -2. Sliding Timeout Window - - Reset the timeout each time a new chunk arrives - - Only expire the chunked message if there's a long gap between chunks - - **Pros**: Tolerates varying network conditions and delivery rates - - **Cons**: Could keep resources allocated for extended periods - -## Checksum Algorithm Options for MQTT Message Chunking - -1. MD5 - - **Description**: 128-bit hash function - - **Pros**: Good performance, reasonable size (16 bytes), widely implemented - - **Cons**: No longer considered cryptographically secure - - **Best for**: Basic integrity verification without security requirements - -2. SHA-256 - - **Description**: Secure hash algorithm producing 256-bit output - - **Pros**: Cryptographically secure, widely supported in all target languages - - **Cons**: Larger output size (32 bytes), more computation required - - **Best for**: Applications requiring message security and tamper protection - -3. BLAKE2b - - **Description**: Modern cryptographic hash function - - **Pros**: Faster than MD5 but with security comparable to SHA-3 - - **Cons**: May not be as universally available in standard libraries - - **Best for**: Performance-critical applications that still need security \ No newline at end of file + - Validate message integrity across chunks and prevent chunk injection attacks (see Checksum Algorithm Options for MQTT Message Chunking) \ No newline at end of file From 11de62b43ff82c94bd40d2290c3157394c211906 Mon Sep 17 00:00:00 2001 From: Maxim Semenov Date: Fri, 6 Jun 2025 14:29:32 -0700 Subject: [PATCH 11/23] progress --- doc/dev/adr/0020-large-message-chunking.md | 57 ++++++++++++++++++---- 1 file changed, 48 insertions(+), 9 deletions(-) diff --git a/doc/dev/adr/0020-large-message-chunking.md b/doc/dev/adr/0020-large-message-chunking.md index b9645a23fe..c860f39575 100644 --- a/doc/dev/adr/0020-large-message-chunking.md +++ b/doc/dev/adr/0020-large-message-chunking.md @@ -24,7 +24,7 @@ We will implement sdk-level message chunking as part of the Protocol layer to tr { "messageId": "unique-id-for-chunked-message", "chunkIndex": 0, - "timeout" : "10000", + "timeout" : 10000, "totalChunks": 5, "checksum": "message-hash" } @@ -57,14 +57,9 @@ SDK will provide user with options to inject their algorithm of choice or use SD - Receiving Process: - The Chunking aware client receives messages and identifies chunked messages by the presence of chunk metadata. - Chunks are stored in a temporary buffer, indexed by message ID and chunk index. - - When all chunks for a message ID are received, they are reassembled in order and message checksum verified. + - When all chunks for a message ID are received, they are reassembled in order and message checksum verified (see Checksum Algorithm Options for MQTT Message Chunking). - The reconstructed message is then processed as a single message by the application. -### Benefits - -- **Property Preservation:** maintains topic, QoS, and other message properties consistently. -- **Network Optimized:** allows efficient transmission of large payloads over constrained networks. - ### Implementation Considerations - **Error Handling:** @@ -73,5 +68,49 @@ SDK will provide user with options to inject their algorithm of choice or use SD - **Performance Optimization:** - Concurrent chunk transmission - Efficient memory usage during reassembly -- **Security:** - - Validate message integrity across chunks and prevent chunk injection attacks (see Checksum Algorithm Options for MQTT Message Chunking) \ No newline at end of file + - Maximum reassembly buffer size limits (configurable) + +### Benefits + +- **Property Preservation:** maintains topic, QoS, and other message properties consistently. +- **Network Optimized:** allows efficient transmission of large payloads over constrained networks. + +### Compatibility + +- Non-chunking-aware clients will receive individual chunks as separate messages. + +## Appendix + +### Message Flow Diagram + +```mermaid +sequenceDiagram + participant Sender as Sending Client + participant Broker as MQTT Broker + participant Receiver as Receiving Client + + Note over Sender: Large message (>max size) + Sender->>Sender: Split into chunks + Note right of Sender: Calculate chunk size:
MaxPacketSize - Overhead + + loop For each chunk + Sender->>Broker: MQTT PUBLISH with __chunk metadata + Note over Broker: No special handling
required by broker + Broker->>Receiver: Forward chunk + Note over Receiver: First chucnk starts timeout count down + Receiver->>Receiver: Store in buffer + Note left of Receiver: Index by:
messageId + chunkIndex + end + + Note over Receiver: All chunks received + Receiver->>Receiver: Verify checksum + Note right of Receiver: SHA-256 or
custom algorithm + Receiver->>Receiver: Reassemble message + Note over Receiver: Process complete message + + rect rgba(255, 0, 0, 0.1) + Note over Sender,Receiver: Error Path + Receiver-->>Receiver: Timeout occurred + Receiver-->>Receiver: Cleanup buffers + end +``` From 0bb4de6be65c2e2f235938245670e5d25809463b Mon Sep 17 00:00:00 2001 From: Maxim Semenov Date: Fri, 6 Jun 2025 16:51:24 -0700 Subject: [PATCH 12/23] progress --- doc/dev/adr/0020-large-message-chunking.md | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/doc/dev/adr/0020-large-message-chunking.md b/doc/dev/adr/0020-large-message-chunking.md index c860f39575..490519e08e 100644 --- a/doc/dev/adr/0020-large-message-chunking.md +++ b/doc/dev/adr/0020-large-message-chunking.md @@ -24,24 +24,25 @@ We will implement sdk-level message chunking as part of the Protocol layer to tr { "messageId": "unique-id-for-chunked-message", "chunkIndex": 0, - "timeout" : 10000, "totalChunks": 5, "checksum": "message-hash" } ``` - - `messageId, chunkIndex, timeout` - present for every chunk; `totalChunks, checksum` - present only for the first chunk. `messageId` is UUID, `timeout` is milliseconds. + - `messageId, chunkIndex` - present for every chunk; `totalChunks, checksum` - present only for the first chunk. `messageId` is UUID. -**Chunk size calculation**: +**Chunk size calculation:** - Maximum chunk size will be derived from the MQTT CONNECT packet's Maximum Packet Size. - A static overhead value will be subtracted from the Maximum Packet Size to account for MQTT packet headers, topic name, user properties, and other metadata. - The overhead size will be configurable, large enough to simplify calculations while ensuring we stay under the broker's limit. -**Chunk Timeout Mechanism:** +**Chunk Timeout Mechanism** -- Set a single timeout period after receiving the first chunk. -- If all chunks aren't received within this window, the message is considered failed. +> [MQTT-3.3.2-6] | The PUBLISH packet sent to a Client by the Server MUST contain a Message Expiry Interval set to the received value minus the time that the message has been waiting in the Server. + +The receiving client uses the Message Expiry Interval from the first chunk as the timeout period for collecting all remaining chunks of the message. +**Edge case:** Since the Message Expiry Interval is specified in seconds, chunked messages may behave differently than single messages when the expiry interval is very short (e.g., 1 second remaining). For a single large message, the QoS flow would complete even if the expiry interval expires during transmission. However, with chunking, if the remaining expiry interval is too short to receive all chunks, the message reassembly will fail due to timeout. **Checksum Algorithm Options for MQTT Message Chunking** From 18b36fe0ac3365f0fa142201cba44c4f8ad26cd2 Mon Sep 17 00:00:00 2001 From: Maxim Semenov Date: Mon, 9 Jun 2025 08:41:37 -0700 Subject: [PATCH 13/23] Enhance large message chunking documentation with failure handling and configuration settings --- doc/dev/adr/0020-large-message-chunking.md | 46 ++++++++++++++++------ 1 file changed, 34 insertions(+), 12 deletions(-) diff --git a/doc/dev/adr/0020-large-message-chunking.md b/doc/dev/adr/0020-large-message-chunking.md index 490519e08e..e87c4564c9 100644 --- a/doc/dev/adr/0020-large-message-chunking.md +++ b/doc/dev/adr/0020-large-message-chunking.md @@ -53,13 +53,22 @@ SDK will provide user with options to inject their algorithm of choice or use SD - Sending Process: - When a payload exceeds the maximum packet size, the message is split into fixed-size chunks (with potentially smaller last chunk) - Each chunk is sent as a separate MQTT message with the same topic but with chunk metadata. - - Effort should be made to minimize user properties copied over to every chunk: first chunk will have full set of original user properties and the rest only those that are necessary to reassemble original message (ex.: ```$partition``` property to support shared subscriptions:). + - Effort should be made to minimize user properties copied over to every chunk: first chunk will have full set of original user properties and the rest only those that are necessary to reassemble original message (ex.: ```$partition``` property to support shared subscriptions). - QoS settings are maintained across all chunks. - Receiving Process: - The Chunking aware client receives messages and identifies chunked messages by the presence of chunk metadata. - Chunks are stored in a temporary buffer, indexed by message ID and chunk index. - When all chunks for a message ID are received, they are reassembled in order and message checksum verified (see Checksum Algorithm Options for MQTT Message Chunking). - The reconstructed message is then processed as a single message by the application. +- Receiving Failures: + - Message timeout interval ended before all chunks received. + - Reassembly buffer size limit reached before all chunks received. + - Calculated checksum does not match checksumm from chunk metadata. + +**Configuration settings:** +- Enable/Disable +- Overhead size +- Reassembly buffer size limit ### Implementation Considerations @@ -98,20 +107,33 @@ sequenceDiagram Sender->>Broker: MQTT PUBLISH with __chunk metadata Note over Broker: No special handling
required by broker Broker->>Receiver: Forward chunk - Note over Receiver: First chucnk starts timeout count down + Note over Receiver: First chunk starts timeout countdown Receiver->>Receiver: Store in buffer Note left of Receiver: Index by:
messageId + chunkIndex end - Note over Receiver: All chunks received - Receiver->>Receiver: Verify checksum - Note right of Receiver: SHA-256 or
custom algorithm - Receiver->>Receiver: Reassemble message - Note over Receiver: Process complete message - - rect rgba(255, 0, 0, 0.1) - Note over Sender,Receiver: Error Path - Receiver-->>Receiver: Timeout occurred - Receiver-->>Receiver: Cleanup buffers + alt Success Path + Note over Receiver: All chunks received + Receiver->>Receiver: Verify checksum + Note right of Receiver: SHA-256 or
custom algorithm + Receiver->>Receiver: Reassemble message + Note over Receiver: Process complete message + else Failure: Timeout + Note over Receiver: Message Expiry Interval exceeded + Receiver->>Receiver: Timeout occurred + Receiver->>Receiver: Cleanup buffers + Note over Receiver: Notify application:
ChunkTimeoutError + else Failure: Buffer Limit + Note over Receiver: Reassembly buffer full + Receiver->>Receiver: Reject new chunks + Receiver->>Receiver: Cleanup oldest incomplete messages + Note over Receiver: Notify application:
BufferLimitExceededError + else Failure: Checksum Mismatch + Note over Receiver: All chunks received + Receiver->>Receiver: Calculate checksum + Note over Receiver: Checksum verification failed + Receiver->>Receiver: Discard reassembled message + Receiver->>Receiver: Cleanup buffers + Note over Receiver: Notify application:
ChecksumMismatchError end ``` From 833774f434171eeb89d819853dc1761d1b40a8df Mon Sep 17 00:00:00 2001 From: Maxim Semenov Date: Mon, 9 Jun 2025 10:04:47 -0700 Subject: [PATCH 14/23] Clarify handling of message chunks for non-chunking-aware clients in documentation --- doc/dev/adr/0020-large-message-chunking.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/dev/adr/0020-large-message-chunking.md b/doc/dev/adr/0020-large-message-chunking.md index e87c4564c9..ba58af2c2c 100644 --- a/doc/dev/adr/0020-large-message-chunking.md +++ b/doc/dev/adr/0020-large-message-chunking.md @@ -87,7 +87,7 @@ SDK will provide user with options to inject their algorithm of choice or use SD ### Compatibility -- Non-chunking-aware clients will receive individual chunks as separate messages. +- Non-chunking-aware clients will receive individual chunks as separate messages. Chunks reassembly could be implemented on the application side, given chunking implementation is known. ## Appendix From 0199e109ebe0945c10991bfe807b6e0ecd9ebde7 Mon Sep 17 00:00:00 2001 From: Maxim Semenov <87454347+maximsemenov80@users.noreply.github.com> Date: Mon, 9 Jun 2025 14:40:26 -0700 Subject: [PATCH 15/23] Update 0020-large-message-chunking.md --- doc/dev/adr/0020-large-message-chunking.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/doc/dev/adr/0020-large-message-chunking.md b/doc/dev/adr/0020-large-message-chunking.md index ba58af2c2c..2ad0c15f3d 100644 --- a/doc/dev/adr/0020-large-message-chunking.md +++ b/doc/dev/adr/0020-large-message-chunking.md @@ -42,7 +42,10 @@ We will implement sdk-level message chunking as part of the Protocol layer to tr > [MQTT-3.3.2-6] | The PUBLISH packet sent to a Client by the Server MUST contain a Message Expiry Interval set to the received value minus the time that the message has been waiting in the Server. The receiving client uses the Message Expiry Interval from the first chunk as the timeout period for collecting all remaining chunks of the message. -**Edge case:** Since the Message Expiry Interval is specified in seconds, chunked messages may behave differently than single messages when the expiry interval is very short (e.g., 1 second remaining). For a single large message, the QoS flow would complete even if the expiry interval expires during transmission. However, with chunking, if the remaining expiry interval is too short to receive all chunks, the message reassembly will fail due to timeout. + +Edge case: +- Since the Message Expiry Interval is specified in seconds, chunked messages may behave differently than single messages when the expiry interval is very short (e.g., 1 second remaining). For a single large message, the QoS flow would complete even if the expiry interval expires during transmission. However, with chunking, if the remaining expiry interval is too short to receive all chunks, the message reassembly will fail due to timeout. +- In case of QoS 0 and no Message Expiry Interval (forever message) if any of the chunks lost during transmission client will never cleanup assembler buffer for this message, thus use of the chunking should be restricted to QoS 1/2. **Checksum Algorithm Options for MQTT Message Chunking** From 448f8e2a6fc9577e2ccff6946002c4b7cf5b0102 Mon Sep 17 00:00:00 2001 From: Maxim Semenov <87454347+maximsemenov80@users.noreply.github.com> Date: Tue, 10 Jun 2025 12:04:27 -0700 Subject: [PATCH 16/23] Update 0020-large-message-chunking.md --- doc/dev/adr/0020-large-message-chunking.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/dev/adr/0020-large-message-chunking.md b/doc/dev/adr/0020-large-message-chunking.md index 2ad0c15f3d..c8899a6819 100644 --- a/doc/dev/adr/0020-large-message-chunking.md +++ b/doc/dev/adr/0020-large-message-chunking.md @@ -112,7 +112,7 @@ sequenceDiagram Broker->>Receiver: Forward chunk Note over Receiver: First chunk starts timeout countdown Receiver->>Receiver: Store in buffer - Note left of Receiver: Index by:
messageId + chunkIndex + Note over Receiver: Index by:
messageId + chunkIndex end alt Success Path From aa8e946053636ebec2a53c6df001bc0daee342f1 Mon Sep 17 00:00:00 2001 From: Maxim Semenov <87454347+maximsemenov80@users.noreply.github.com> Date: Tue, 10 Jun 2025 16:40:44 -0700 Subject: [PATCH 17/23] Update 0020-large-message-chunking.md --- doc/dev/adr/0020-large-message-chunking.md | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/doc/dev/adr/0020-large-message-chunking.md b/doc/dev/adr/0020-large-message-chunking.md index c8899a6819..e7c3c29e3b 100644 --- a/doc/dev/adr/0020-large-message-chunking.md +++ b/doc/dev/adr/0020-large-message-chunking.md @@ -41,15 +41,15 @@ We will implement sdk-level message chunking as part of the Protocol layer to tr > [MQTT-3.3.2-6] | The PUBLISH packet sent to a Client by the Server MUST contain a Message Expiry Interval set to the received value minus the time that the message has been waiting in the Server. -The receiving client uses the Message Expiry Interval from the first chunk as the timeout period for collecting all remaining chunks of the message. +The receiving client uses the Message Expiry Interval from the first chunk as the timeout period for collecting all remaining chunks of the message. Chunking mechanism is relaying on the Protocol level requirement of having Message Expiry Interval to be set for every message to avoid "forever message" edge case (see below). Edge case: - Since the Message Expiry Interval is specified in seconds, chunked messages may behave differently than single messages when the expiry interval is very short (e.g., 1 second remaining). For a single large message, the QoS flow would complete even if the expiry interval expires during transmission. However, with chunking, if the remaining expiry interval is too short to receive all chunks, the message reassembly will fail due to timeout. -- In case of QoS 0 and no Message Expiry Interval (forever message) if any of the chunks lost during transmission client will never cleanup assembler buffer for this message, thus use of the chunking should be restricted to QoS 1/2. +- In case of QoS 0 and no Message Expiry Interval (forever message) if any of the chunks lost during transmission client will never cleanup assembler buffer for this message. -**Checksum Algorithm Options for MQTT Message Chunking** +**Checksum Algorithm for MQTT Message Chunking** -SDK will provide user with options to inject their algorithm of choice or use SDK's default SHA-256. +Chunking will use SHA-256 for checksum calculation. **Implementation layer:** @@ -65,13 +65,11 @@ SDK will provide user with options to inject their algorithm of choice or use SD - The reconstructed message is then processed as a single message by the application. - Receiving Failures: - Message timeout interval ended before all chunks received. - - Reassembly buffer size limit reached before all chunks received. - Calculated checksum does not match checksumm from chunk metadata. **Configuration settings:** - Enable/Disable - Overhead size -- Reassembly buffer size limit ### Implementation Considerations From 8e9c76019124eab5435ca194dd0e7849a9a13461 Mon Sep 17 00:00:00 2001 From: Maxim Semenov <87454347+maximsemenov80@users.noreply.github.com> Date: Wed, 11 Jun 2025 11:37:09 -0700 Subject: [PATCH 18/23] Update 0020-large-message-chunking.md --- doc/dev/adr/0020-large-message-chunking.md | 25 +++++----------------- 1 file changed, 5 insertions(+), 20 deletions(-) diff --git a/doc/dev/adr/0020-large-message-chunking.md b/doc/dev/adr/0020-large-message-chunking.md index e7c3c29e3b..b48f3324a2 100644 --- a/doc/dev/adr/0020-large-message-chunking.md +++ b/doc/dev/adr/0020-large-message-chunking.md @@ -15,21 +15,11 @@ We will implement sdk-level message chunking as part of the Protocol layer to tr **The chunking mechanism will**: - Be enabled/disabled by a configuration setting. -- Use standardized user properties for chunk metadata: - - - The `__chunk` user property will contain a JSON object with chunking metadata. - - The JSON structure will include: - - ```json - { - "messageId": "unique-id-for-chunked-message", - "chunkIndex": 0, - "totalChunks": 5, - "checksum": "message-hash" - } - ``` - - - `messageId, chunkIndex` - present for every chunk; `totalChunks, checksum` - present only for the first chunk. `messageId` is UUID. +- Use standardized user properties for chunk metadata. The `__chunk` user property will contain a colon-separated string with chunking metadata: ```:::```. The string will include: + - `messageId` - UUID string in the 8-4-4-4-12 format, present for every chunk. + - `chunkIndex` - unsigned 32 bit integer in decimal format, present for every chunk; + - `totalChunks` - unsigned 32 bit integer in decimal format, present only for the first chunk. + - `checksum` - SHA-256 hash in hexadecimal format (64 characters long), present only for the first chunk. **Chunk size calculation:** @@ -124,11 +114,6 @@ sequenceDiagram Receiver->>Receiver: Timeout occurred Receiver->>Receiver: Cleanup buffers Note over Receiver: Notify application:
ChunkTimeoutError - else Failure: Buffer Limit - Note over Receiver: Reassembly buffer full - Receiver->>Receiver: Reject new chunks - Receiver->>Receiver: Cleanup oldest incomplete messages - Note over Receiver: Notify application:
BufferLimitExceededError else Failure: Checksum Mismatch Note over Receiver: All chunks received Receiver->>Receiver: Calculate checksum From e32378051cf7e3eefa8f21f84c9597f190ece279 Mon Sep 17 00:00:00 2001 From: Maxim Semenov Date: Wed, 9 Jul 2025 13:57:26 -0700 Subject: [PATCH 19/23] rename file --- ...0-large-message-chunking.md => 0023-large-message-chunking.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename doc/dev/adr/{0020-large-message-chunking.md => 0023-large-message-chunking.md} (100%) diff --git a/doc/dev/adr/0020-large-message-chunking.md b/doc/dev/adr/0023-large-message-chunking.md similarity index 100% rename from doc/dev/adr/0020-large-message-chunking.md rename to doc/dev/adr/0023-large-message-chunking.md From f6894f8c6867c96cfdfdd39e7f12d691074bf6aa Mon Sep 17 00:00:00 2001 From: Maxim Semenov <87454347+maximsemenov80@users.noreply.github.com> Date: Wed, 9 Jul 2025 14:49:38 -0700 Subject: [PATCH 20/23] Update 0023-large-message-chunking.md --- doc/dev/adr/0023-large-message-chunking.md | 21 ++++++++------------- 1 file changed, 8 insertions(+), 13 deletions(-) diff --git a/doc/dev/adr/0023-large-message-chunking.md b/doc/dev/adr/0023-large-message-chunking.md index b48f3324a2..285b3ce138 100644 --- a/doc/dev/adr/0023-large-message-chunking.md +++ b/doc/dev/adr/0023-large-message-chunking.md @@ -24,14 +24,13 @@ We will implement sdk-level message chunking as part of the Protocol layer to tr **Chunk size calculation:** - Maximum chunk size will be derived from the MQTT CONNECT packet's Maximum Packet Size. -- A static overhead value will be subtracted from the Maximum Packet Size to account for MQTT packet headers, topic name, user properties, and other metadata. -- The overhead size will be configurable, large enough to simplify calculations while ensuring we stay under the broker's limit. +- Message overhead (MQTT packet headers, topic name, user properties, and other metadata) value will be subtracted from the Maximum Packet Size. **Chunk Timeout Mechanism** > [MQTT-3.3.2-6] | The PUBLISH packet sent to a Client by the Server MUST contain a Message Expiry Interval set to the received value minus the time that the message has been waiting in the Server. -The receiving client uses the Message Expiry Interval from the first chunk as the timeout period for collecting all remaining chunks of the message. Chunking mechanism is relaying on the Protocol level requirement of having Message Expiry Interval to be set for every message to avoid "forever message" edge case (see below). +The receiving client uses the Message Expiry Interval from the first chunk as the timeout period for collecting all remaining chunks of the message. Chunking mechanism is relaying on the existing Protocol level requirement of having Message Expiry Interval to be set for every message to avoid "forever message" edge case (see below). Edge case: - Since the Message Expiry Interval is specified in seconds, chunked messages may behave differently than single messages when the expiry interval is very short (e.g., 1 second remaining). For a single large message, the QoS flow would complete even if the expiry interval expires during transmission. However, with chunking, if the remaining expiry interval is too short to receive all chunks, the message reassembly will fail due to timeout. @@ -45,7 +44,7 @@ Chunking will use SHA-256 for checksum calculation. - Sending Process: - When a payload exceeds the maximum packet size, the message is split into fixed-size chunks (with potentially smaller last chunk) - - Each chunk is sent as a separate MQTT message with the same topic but with chunk metadata. + - Each chunk is sent as a separate MQTT message with the same topic and with chunk metadata added. - Effort should be made to minimize user properties copied over to every chunk: first chunk will have full set of original user properties and the rest only those that are necessary to reassemble original message (ex.: ```$partition``` property to support shared subscriptions). - QoS settings are maintained across all chunks. - Receiving Process: @@ -59,7 +58,6 @@ Chunking will use SHA-256 for checksum calculation. **Configuration settings:** - Enable/Disable -- Overhead size ### Implementation Considerations @@ -69,7 +67,6 @@ Chunking will use SHA-256 for checksum calculation. - **Performance Optimization:** - Concurrent chunk transmission - Efficient memory usage during reassembly - - Maximum reassembly buffer size limits (configurable) ### Benefits @@ -90,9 +87,8 @@ sequenceDiagram participant Broker as MQTT Broker participant Receiver as Receiving Client - Note over Sender: Large message (>max size) + Note over Sender: Large message (>max size).
Calculate chunk size. Sender->>Sender: Split into chunks - Note right of Sender: Calculate chunk size:
MaxPacketSize - Overhead loop For each chunk Sender->>Broker: MQTT PUBLISH with __chunk metadata @@ -105,20 +101,19 @@ sequenceDiagram alt Success Path Note over Receiver: All chunks received - Receiver->>Receiver: Verify checksum - Note right of Receiver: SHA-256 or
custom algorithm Receiver->>Receiver: Reassemble message + Receiver->>Receiver: Verify checksum + Note right of Receiver: SHA-256 Note over Receiver: Process complete message else Failure: Timeout Note over Receiver: Message Expiry Interval exceeded - Receiver->>Receiver: Timeout occurred Receiver->>Receiver: Cleanup buffers Note over Receiver: Notify application:
ChunkTimeoutError else Failure: Checksum Mismatch Note over Receiver: All chunks received - Receiver->>Receiver: Calculate checksum + Receiver->>Receiver: Reassemble message + Receiver->>Receiver: Verify checksum Note over Receiver: Checksum verification failed - Receiver->>Receiver: Discard reassembled message Receiver->>Receiver: Cleanup buffers Note over Receiver: Notify application:
ChecksumMismatchError end From 98baf9bbfd3f6d1dafeedc1979d227b664d9f01a Mon Sep 17 00:00:00 2001 From: Maxim Semenov <87454347+maximsemenov80@users.noreply.github.com> Date: Wed, 9 Jul 2025 14:49:56 -0700 Subject: [PATCH 21/23] Update 0023-large-message-chunking.md --- doc/dev/adr/0023-large-message-chunking.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/dev/adr/0023-large-message-chunking.md b/doc/dev/adr/0023-large-message-chunking.md index 285b3ce138..ba08607a9c 100644 --- a/doc/dev/adr/0023-large-message-chunking.md +++ b/doc/dev/adr/0023-large-message-chunking.md @@ -1,4 +1,4 @@ -# ADR 20: Large Message Chunking in MQTT Protocol +# ADR 23: Large Message Chunking in MQTT Protocol ## Status From f95fe5c0a68fee479d4d2d51355bfac7b39c78ec Mon Sep 17 00:00:00 2001 From: Maxim Semenov <87454347+maximsemenov80@users.noreply.github.com> Date: Wed, 9 Jul 2025 14:53:28 -0700 Subject: [PATCH 22/23] Update 0023-large-message-chunking.md --- doc/dev/adr/0023-large-message-chunking.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/dev/adr/0023-large-message-chunking.md b/doc/dev/adr/0023-large-message-chunking.md index ba08607a9c..1730ed75aa 100644 --- a/doc/dev/adr/0023-large-message-chunking.md +++ b/doc/dev/adr/0023-large-message-chunking.md @@ -75,7 +75,7 @@ Chunking will use SHA-256 for checksum calculation. ### Compatibility -- Non-chunking-aware clients will receive individual chunks as separate messages. Chunks reassembly could be implemented on the application side, given chunking implementation is known. +- Non-chunking-aware clients will receive individual chunks as separate messages. Chunks reassembly could be implemented on the application side, given described above chunking implementation is known to the application author. ## Appendix From 788dc3d1cec4f6035beb333647538cf14568759f Mon Sep 17 00:00:00 2001 From: Maxim Semenov <87454347+maximsemenov80@users.noreply.github.com> Date: Tue, 5 Aug 2025 13:18:15 -0700 Subject: [PATCH 23/23] Update 0023-large-message-chunking.md --- doc/dev/adr/0023-large-message-chunking.md | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/doc/dev/adr/0023-large-message-chunking.md b/doc/dev/adr/0023-large-message-chunking.md index 1730ed75aa..7446000466 100644 --- a/doc/dev/adr/0023-large-message-chunking.md +++ b/doc/dev/adr/0023-large-message-chunking.md @@ -14,7 +14,7 @@ We will implement sdk-level message chunking as part of the Protocol layer to tr **The chunking mechanism will**: -- Be enabled/disabled by a configuration setting. +- Be part of the protocol starting some version. [^1] - Use standardized user properties for chunk metadata. The `__chunk` user property will contain a colon-separated string with chunking metadata: ```:::```. The string will include: - `messageId` - UUID string in the 8-4-4-4-12 format, present for every chunk. - `chunkIndex` - unsigned 32 bit integer in decimal format, present for every chunk; @@ -77,6 +77,12 @@ Chunking will use SHA-256 for checksum calculation. - Non-chunking-aware clients will receive individual chunks as separate messages. Chunks reassembly could be implemented on the application side, given described above chunking implementation is known to the application author. +[^1]: The group decided that supporting a specific protocol version should imply mandatory support for chunking, with no option to opt out, to simplify compatibility and avoid the complexity of feature flags or negotiation mechanisms. +This approach ensures that all clients and servers claiming compliance with a given protocol version can reliably handle chunked messages, reducing ambiguity and potential errors. +It was noted that making chunking always available (though not always used) streamlines the design and avoids the need for additional feature negotiation logic. +The team acknowledged that while this is a stricter stance, it keeps the protocol simpler, especially for constrained devices, and avoids layering new features on top of feature negotiation. +There was recognition that this approach means any implementation supporting the new protocol version must handle chunking, and rejecting chunked messages would make it non-compliant. + ## Appendix ### Message Flow Diagram