Fix long message handling with dynamic buffer growth #158
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The socket transport plugin used a fixed 64KB buffer size which caused messages surpassing that size to be truncated For UDP/Unix datagram sockets, this resulted in parsing errors like "unexpected end of input" This change allows the buffer to grow (up to a limit depending on the protocol) to accommodate larger messages. Closes: OSPRH-23826
Contributor
Author
|
Still working on unit tests |
Use sendTCPSocketMessage name instead
vyzigold
reviewed
Jan 8, 2026
Contributor
vyzigold
left a comment
vyzigold
left a comment
There was a problem hiding this comment.
I see one thing in the tests (see my comment) which needs a change / discussion. But otherwise this looks quite good. Thank you for the cleanup of the tests!
| assert.Equal(t, true, len(receivedMsg) > 0) | ||
|
|
||
| // If we received a complete message, verify the content | ||
| if len(receivedMsg) == largeBuffSize+len(addition) { |
Contributor
There was a problem hiding this comment.
Hmm. Does it actually get inside this if? If I understand it correctly. You're sending a 128kb message, but the socket buffer hasn't expanded yet (it's still set to 64kb), so a truncated string is received and sent further (which means only the first half of the sent message gets assigned to receivedMsg right?). So this condition would be 64kb == 128kb + 19. So even though the test passes, I don't think the code inside the condition ever executes.
What I think should happen in this testcase:
- Send the large message +
addition - Optionally check that receivedMsg only has the first half of the message
- Send the same message again
- Check that we now have 128kb of data with the last character being '$'
- Send the same message again
- Check that we now have 128kb + 19 bytes of data and the end is "wubba lubba dub dub"
Contributor
Author
There was a problem hiding this comment.
Good catch! Indeed, we should extend this test to send a large message, catch the log message and verify we only received the base buffer size and then retry sending a long message and see it succeeds. I will fix this.
Contributor
Author
|
Thanks for the review! |
This test verifies the dynamic buffer growth by sending three messages In each iteration the buffer grows from the initial size of 65535 bytes to 3 times the initial size. Also verifies the content of the received message
vkmc
commented
Jan 8, 2026
| assert.Equal(t, true, len(receivedMsg) > 0) | ||
|
|
||
| // If we received a complete message, verify the content | ||
| if len(receivedMsg) == largeBuffSize+len(addition) { |
Contributor
Author
There was a problem hiding this comment.
Good catch! Indeed, we should extend this test to send a large message, catch the log message and verify we only received the base buffer size and then retry sending a long message and see it succeeds. I will fix this.
vyzigold
approved these changes
Jan 13, 2026
vkmc
added a commit
that referenced
this pull request
Jan 23, 2026
* Pin Ceilometer to stable/2024.2 (#154) Even though we are pinning to the stable/2024.2 when cloning Devstack, we are pulling Ceilometer requirements from the master branch and this is generating dependencies conflict. Explicitily set stable/2024.2 branch when cloning Ceilometer to fix this. Closes-Bug: OSPRH-19908 * Bump gopkg.in/yaml.v3 from 3.0.0 to 3.0.1 (#153) --- updated-dependencies: - dependency-name: gopkg.in/yaml.v3 dependency-version: 3.0.1 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Victoria Martinez de la Cruz <victoria@redhat.com> * Protect the expiryProc list with a mutex (#155) The container/list data structure used in expiry.go is not safe for concurrent use. See golang/go#25105 Serialize the access to the shared list to avoid having check() on empty lists. Closes-Bug: OSPRH-18979 * Fix long message handling with dynamic buffer growth (#158) The socket transport plugin used a fixed 64KB buffer size which caused messages surpassing that size to be truncated For UDP/Unix datagram sockets, this resulted in parsing errors like "unexpected end of input" This change allows the buffer to grow (up to a limit depending on the protocol) to accommodate larger messages. * Create helper functions for TCP connection handling * Add helper sendUDPSocketMessage * Update TestUnixSocketTransport large message test This test verifies the dynamic buffer growth by sending three messages In each iteration the buffer grows from the initial size of 65535 bytes to 3 times the initial size. Also verifies the content of the received message Closes: OSPRH-23826 * Update to golang 1.24.11 (#159) * Fix non-constant format string in call Go 1.24.11 has stricter checking for format strings in printf-style functions to prevent format string vulnerabilities. * Bump golangci-lint version to 1.64.2 * Replace exportloopref for copyloopvar The linter 'exportloopref' is deprecated (since v1.60.2) * Update coveralls-badge workflow to be non-blocking (#160) Added "continue-on-error: true" to prevent badge failures from blocking PRs. Explicitly defined pull-requests: write permissions for GITHUB_TOKEN. Added try-catch safety logic and async/await to the script. * Improve test coverage for expiry.go (#164) Add tests for check() edge cases, run() function, and concurrent access. Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com> * Improve test coverage for prometheus application plugin (#166) Add tests for New, Config, PromCollector methods, metricExpiry, collectorExpiry, ReceiveMetric, UpdateMetrics, Describe, and Collect. Omit logging and HTTP server tests. * Fix golangci-lint empty-block error in prometheus tests Replace empty for-range loop with explicit assignment to satisfy revive linter. * Add timestamp verification assertions to prometheus collector tests Enhance test coverage by adding assertions that verify the presence or absence of timestamps in collected metrics. The "collect with timestamp" test now verifies that metrics include the expected timestamp value, while the "collect with zero timestamp" test confirms that metrics with zero timestamps are sent without timestamp metadata. Also fixes default port assertion and cleans up empty-block lint issue. * Fix golangci-lint empty-block error in channel drain loop Replace empty for-range loop with explicit variable usage to resolve the revive empty-block linting error. Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com> * Improve test coverage for socket transport plugin (#165) Add tests for Config, New, Listen, init errors, dump messages, and TCP parsing edge cases. * Fix golangci-lint errors in socket transport tests - Check error returns from binary.Write calls - Reduce file permissions from 0644 to 0600 Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * Add comprehensive test coverage for parser libraries (#168) Ceilometer: - ParseInputJSON with various message formats - ParseInputMsgPack for msgpack parsing - sanitize function with escaped quotes and payload formatting - Error handling for invalid JSON and malformed data - Edge cases: empty payloads, multiple metrics, user metadata Collectd: - ParseInputByte for all metric variations - Multi-dimensional metrics with multiple values - Optional fields (plugin_instance, type_instance) - Error handling for invalid JSON and non-array data - Edge cases: empty arrays, zero values, negative values, large values - Real-world virt plugin data formats * Add comprehensive test coverage for sensu parser library Create test file for sensubility-metrics sensu parser library with complete coverage of all validation and error building functions. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * Add test coverage for manager package (#167) Create manager_test.go test cases covering setter functions, plugin initialization error paths, and edge cases. Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The socket transport plugin used a fixed 64KB buffer size which caused messages surpassing that size to be truncated
For UDP/Unix datagram sockets, this resulted in parsing errors like "unexpected end of input"
This change allows the buffer to grow (up to a limit depending on the protocol) to accommodate larger messages.
Closes: OSPRH-23826