feat: message chunking

[ldrozdz93]

Problem

Message chunking logic was duplicated across worker-cli and orb-discovery, using a heuristic algorithm. It turned out to not be enough for a customer's dataset.

Solution

Centralized chunking logic in the SDK with a configurable chunk size parameter (default: 3.0 MB). The implementation uses greedy bin-packing that accumulates entities until adding the next entity would exceed the size limit, then starts a new chunk.

Changes

• Added netboxlabs/diode/sdk/chunking.py with create_message_chunks() and estimate_message_size()
• Added comprehensive test suite (tests/test_chunking.py) with 10 test cases covering edge cases

[github-actions]

Coverage Report

File	Stmts	Miss	Cover	Missing
/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/netboxlabs/diode/sdk
chunking.py	33	1	97%	112
client.py	509	40	92%	165–166, 185–187, 190–193, 482, 548, 553, 557, 643–648, 693–695, 700, 705, 710, 715, 725, 729, 733, 755, 776, 778, 852, 883, 891, 933, 969, 984–985, 992–993
exceptions.py	44	3	93%	69, 82–83
TOTAL	601	44	93%

Tests	Skipped	Failures	Errors	Time
185	0 💤	0 ❌	0 🔥	1.589s ⏱️

[Copilot]

Pull request overview

This pull request centralizes message chunking logic into the SDK to eliminate duplication across worker-cli and orb-discovery. The implementation provides a configurable chunk size parameter with a default of 3.0 MB and uses greedy bin-packing to split entities into appropriately-sized chunks for gRPC ingestion.

Changes:

• Added chunking module with create_message_chunks() and estimate_message_size() functions
• Exported chunking functions from the SDK's public API
• Added comprehensive test suite covering edge cases including empty lists, single/multiple chunks, custom chunk sizes, order preservation, and large entities

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File	Description
netboxlabs/diode/sdk/chunking.py	New module implementing greedy bin-packing chunking algorithm with size estimation
netboxlabs/diode/sdk/init.py	Exports chunking functions to SDK's public API
tests/test_chunking.py	Comprehensive test suite with 10 test cases covering various chunking scenarios

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[mfiedorowicz]

Thanks @ldrozdz93 , generally it's advisable to create GitHub issue first so we can track, triage and prioritise issues properly. Nevertheless I see it valuable, the main ask here is to document chunking in the README. Additionally, we would like to have functional parity in Diode Go SDK too, hence could you please add feature request issue there?

[mfiedorowicz]

Mind failing linting issues:

Error: netboxlabs/diode/sdk/chunking.py:3:1: D213 Multi-line docstring summary should start at the second line
Error: netboxlabs/diode/sdk/chunking.py:10:1: UP035 Import from `collections.abc` instead: `Iterable`
Error: netboxlabs/diode/sdk/chunking.py:18:5: D213 Multi-line docstring summary should start at the second line
Error: netboxlabs/diode/sdk/chunking.py:35:5: D413 Missing blank line after last section ("Examples")
Error: netboxlabs/diode/sdk/chunking.py:88:5: D213 Multi-line docstring summary should start at the second line
Error: netboxlabs/diode/sdk/chunking.py:99:5: D413 Missing blank line after last section ("Examples")
Error: tests/test_chunking.py:182:5: D213 Multi-line docstring summary should start at the second line

[ldrozdz93]

Thanks @ldrozdz93 , generally it's advisable to create GitHub issue first so we can track, triage and prioritise issues properly. Nevertheless I see it valuable, the main ask here is to document chunking in the README. Additionally, we would like to have functional parity in Diode Go SDK too, hence could you please add feature request issue there?

Thanks for the feedback @mfiedorowicz. I'm aware this is kind of a shortcut. A customer is actively blocked by this and I wanted to make it quick.

I'll create a FR for Go SDK and add docs.

[ldrozdz93]

@mfiedorowicz ready for review

[ldrozdz93]

@mfiedorowicz I've created a feature request for diode-sdk-go. Let me know if that's not what you meant.

[github-actions]

🎉 This PR is included in version 1.9.0 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀