fix: Preserve types when decoding records [sc-143811] #114
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ThomasBrady
changed the title
stoksc
reviewed
Jul 30, 2025
| } | ||
|
|
||
| return out, fmt.Errorf("msgpack unmarshal event time: %w", err) | ||
| } |
Contributor
There was a problem hiding this comment.
unrelated to you but i dont understand the code above this at all. in the event of an error, it looks like we do a lot of stuff w/o any side effects and then return an error in all paths.
Author
There was a problem hiding this comment.
Hmm yeah that seems like a bug. It tries to decode the time, then has a couple fallback ways of decoding it but returns an error regardless of if the fallbacks succeed.
stoksc
reviewed
Jul 30, 2025
Comment on lines
+523
to
+530
| // Create a decoder that preserves number types (int64 vs float64) | ||
| // instead of converting all numbers to float64 like the default msgpack.Unmarshal | ||
| recordDecoder := msgpack.NewDecoder(bytes.NewReader(entry[1])) | ||
| // Configure decoder to preserve exact number types instead of converting to float64 | ||
| recordDecoder.UseLooseInterfaceDecoding(false) | ||
|
|
||
| var record map[string]any | ||
| if err := msgpack.Unmarshal(entry[1], &record); err != nil { | ||
| if err := recordDecoder.Decode(&record); err != nil { |
Contributor
There was a problem hiding this comment.
isn't this only for Go output plugins?
Author
There was a problem hiding this comment.
Hmmm yeah upon further investigation this isn't the cause of the behavior I'm seeing when testing the json processing as http_sink doesn't use this. May as well fix this here tho.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Preserves the types of fields in records by using a custom decoder with
UseLooseInterfaceDecoding(false)set.https://app.shortcut.com/chronosphere/story/143811/number-type-loss-in-calyptia-plugin-messagepack-decoding