Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
dc131bd
refactor: Replace unsigned int32 binary append func
RafaelCenzano Dec 20, 2025
56af566
refactor: Replace signed int32 binary append func
RafaelCenzano Dec 20, 2025
f241f11
feat: Create function to replace int64 binary appending
RafaelCenzano Dec 20, 2025
1801ea0
refactor: Replace signed int64 binary append func
RafaelCenzano Dec 20, 2025
3f4a212
refactor: Replace unsigned int64 binary append func
RafaelCenzano Dec 20, 2025
78477dd
feat: Create read int32 function
RafaelCenzano Dec 22, 2025
a71e8ad
refactor: Use wrappers for generic function
RafaelCenzano Dec 22, 2025
6e6903d
refactor: Remove readu32 from bson package
RafaelCenzano Dec 22, 2025
8f3d1c5
refactor: Replace use of readi32 function
RafaelCenzano Dec 22, 2025
d2b6ff7
feat: Create read uint64 and int64 func
RafaelCenzano Dec 22, 2025
6e8c2c0
refactor: Replace use of readu64 func
RafaelCenzano Dec 22, 2025
000b91b
refactor: Replace use of readi64 func
RafaelCenzano Dec 22, 2025
cd7bea2
feat: Create ReadCString funcs
RafaelCenzano Dec 22, 2025
672e8b7
refactor: Replace readcstringbytes fun
RafaelCenzano Dec 22, 2025
815508e
refactor: Replace readcstring func
RafaelCenzano Dec 22, 2025
ceadabd
Merge branch 'master' into refactor/godriver-3707-deduplicate-binary-…
RafaelCenzano Dec 22, 2025
58a4d9a
refactor: remove bytes read generic function
RafaelCenzano Jan 2, 2026
93fcc89
test: new tests and benchmarks for binaryutil functions
RafaelCenzano Jan 2, 2026
8d747d3
chore: add comments to explain decisions for binaryutil functions
RafaelCenzano Jan 2, 2026
ddb6a09
refactor: use standard library in unsigned int read
RafaelCenzano Jan 5, 2026
8cc3586
perf: use code from standard library
RafaelCenzano Jan 5, 2026
0ccd9c1
chore: remove uneeded int32 converions
RafaelCenzano Jan 5, 2026
299cb36
chore: update comment to reflect update function
RafaelCenzano Jan 5, 2026
b2ae3df
Merge branch 'master' into refactor/godriver-3707-deduplicate-binary-…
RafaelCenzano Jan 5, 2026
36f3eae
test: use bytes.Equal to simplify binaryutil test code
RafaelCenzano Jan 7, 2026
3ce86a1
Merge branch 'master' into refactor/godriver-3707-deduplicate-binary-…
RafaelCenzano Jan 9, 2026
18366cc
Merge branch 'master' into refactor/godriver-3707-deduplicate-binary-…
RafaelCenzano Jan 9, 2026
1e6f7fe
refactor: repalce readi32unsafe with ReadI32 from binaryutils package
RafaelCenzano Jan 10, 2026
f238fb7
docs: create doc.go to explain benchmark differences in Read functions
RafaelCenzano Jan 10, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
135 changes: 135 additions & 0 deletions internal/binaryutil/binaryutil.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,135 @@
// Copyright (C) MongoDB, Inc. 2025-present.
//
// Licensed under the Apache License, Version 2.0 (the "License"); you may
// not use this file except in compliance with the License. You may obtain
// a copy of the License at http://www.apache.org/licenses/LICENSE-2.0

// Package binaryutil provides utility functions for working with binary data.
package binaryutil

import (
"bytes"
"encoding/binary"
)

// Append32 appends a uint32 or int32 value to dst in little-endian byte order.
// Byte shifting is done directly to prevent overflow security errors, in
// compliance with gosec G115.
//
// See: https://cs.opensource.google/go/go/+/refs/tags/go1.19:src/encoding/binary/binary.go;l=92
func Append32[T ~uint32 | ~int32](dst []byte, v T) []byte {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[blocking] Each function should have a description. In the description we should note why we do byte shifting directly, instead of using the binary package. We should also note that the order is little-endian:

Append32 appends a uint32 or int32 value to dst in little-endian byte order. Byte shifting is done directly to prevent overflow security errors, in compliance with gosec 115.

See here: https://cs.opensource.google/go/go/+/refs/tags/go1.19:src/encoding/binary/binary.go;l=84

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comments added

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[blocking] There is a significant performance cost of this function compared to the standard library (0.5335 ns/op v 0.1760 ns/op):

❯ go test -bench=BenchmarkAppend32
goos: darwin
goarch: arm64
pkg: go.mongodb.org/mongo-driver/v2/internal/binaryutil
cpu: Apple M1 Pro
BenchmarkAppend32-10            1000000000               0.5335 ns/op   7498.15 MB/s
PASS
ok      go.mongodb.org/mongo-driver/v2/internal/binaryutil      0.907s
❯ go test -bench=BenchmarkStdlibAppendUint32
goos: darwin
goarch: arm64
pkg: go.mongodb.org/mongo-driver/v2/internal/binaryutil
cpu: Apple M1 Pro
BenchmarkStdlibAppendUint32-10          1000000000               0.1760 ns/op   22726.89 MB/s

Suggest copying the standard library directly:

func Append32[T ~uint32 | ~int32](dst []byte, v T) []byte {
	return append(dst,
		byte(v),
		byte(v>>8),
		byte(v>>16),
		byte(v>>24),
	)
}

This problem is also present in append64.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct me if this is an incorrect way to do this I got the commands from AI, but when I run the benchmark my results are very close between the new implementation and standard library. Here is my output for averages over 100 runs from the append 32 functions:

go test ./internal/binaryutil/... -bench=BenchmarkAppend32 -benchmem -count=100 > bench1.out
go test ./internal/binaryutil/... -bench=BenchmarkStdlibAppendUint32 -benchmem -count=100 > bench2.out
awk '/BenchmarkAppend32-/{sum+=$3; n++} END {print "avg ns/op =", sum/n}' bench1.out
avg ns/op = 0.023329
awk '/BenchmarkStdlibAppendUint32-/{sum+=$3; n++} END {print "avg ns/op =", sum/n}' bench2.out
avg ns/op = 0.0234163

For append 32 those numbers are nearly the same over 100 runs.

For append 64 I do see a very drastic difference:

go test ./internal/binaryutil/... -bench=BenchmarkAppend64 -benchmem -count=100 > bench3.out
go test ./internal/binaryutil/... -bench=BenchmarkStdlibAppendUint64 -benchmem -count=100 > bench4.out
awk '/BenchmarkAppend64-/{sum+=$3; n++} END {print "avg ns/op =", sum/n}' bench3.out
avg ns/op = 0.128843
awk '/BenchmarkStdlibAppendUint64-/{sum+=$3; n++} END {print "avg ns/op =", sum/n}' bench4.out
avg ns/op = 0.0231089

To keep things uniform I will apply your suggestion to both functions, thanks for catching that

Copy link
Member

@prestonvasquez prestonvasquez Jan 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm seeing something similar using benchstat. Regardless, it would be ideal to keep the code as close to the source as possible.

return append(dst,
byte(v),
byte(v>>8),
byte(v>>16),
byte(v>>24),
)
}

// Append64 appends a uint64 or int64 value to dst in little-endian byte order.
// Byte shifting is done directly to prevent overflow security errors, in
// compliance with gosec G115.
//
// See: https://cs.opensource.google/go/go/+/refs/tags/go1.19:src/encoding/binary/binary.go;l=119
func Append64[T ~uint64 | ~int64](dst []byte, v T) []byte {
return append(dst,
byte(v),
byte(v>>8),
byte(v>>16),
byte(v>>24),
byte(v>>32),
byte(v>>40),
byte(v>>48),
byte(v>>56),
)
}

// ReadU32 reads a uint32 from src in little-endian byte order. ReadU32 and
// ReadI32 are separate functions to avoid unsafe casting between unsigned and
// signed integers.
func ReadU32(src []byte) (uint32, []byte, bool) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[blocking] This function should wrap the standard library function since doing so doesn't require a type conversion:

func ReadU32(src []byte) (uint32, []byte, bool) {
	if len(src) < 4 {
		return 0, src, false
	}

	return binary.LittleEndian.Uint32(src), src[4:], true
}

if len(src) < 4 {
return 0, src, false
}

return binary.LittleEndian.Uint32(src), src[4:], true
}

// ReadI32 reads an int32 from src in little-endian byte order.
// Byte shifting is done directly to prevent overflow security errors, in
// compliance with gosec G115. ReadU32 and ReadI32 are separate functions to
// avoid unsafe casting between unsigned and signed integers.
//
// See: https://cs.opensource.google/go/go/+/refs/tags/go1.19:src/encoding/binary/binary.go;l=79
func ReadI32(src []byte) (int32, []byte, bool) {
Copy link
Member

@prestonvasquez prestonvasquez Jan 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@RafaelCenzano Like we discussed in office hours, I can't tell why this isn't benchmarking the same as ReadU32 (more specifically, binary.LittleEndian.Uint32(src)) . But it appears to be the same locally: https://github.com/prestonvasquez/go-playground/blob/main/benchmark/binary_inline_test.go

Just noting here since it sounds like we should accept this solution but are seeing weirdness from the benchmarks.

CC: @matthewdale

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gave ai a stab at the question, it's answer stated it may be related to the inlining. This may be wrong but it stated because the code in ReadU is inline already things are easy to compile and run while ReadI it inlines the standard library call, and then it may have issues properly optimizing the checks/hints like _ = src[3] // bounds check hint to compiler. I'm not familiar with the innerworkings of the benchmarking and testing framework in go. But could this be the reason for the difference?

Also as a note ReadU64 and ReadI64 have the same difference in benchmark

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@RafaelCenzano The Go compiler would inline binary.LittleEndian.Uint32, which is evident in the results from the go-playground example.

Also as a note ReadU64 and ReadI64 have the same difference in benchmark

Could you elaborate on what you mean by this?

Copy link
Contributor Author

@RafaelCenzano RafaelCenzano Jan 8, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When I ran the benchmark on those two functions I saw the same issue with performance differences.

go test ./internal/binaryutil/... -bench=BenchmarkReadI64 -benchmem -count=100 > bench1.out
go test ./internal/binaryutil/... -bench=BenchmarkReadU64 -benchmem -count=100 > bench2.out
awk '/BenchmarkReadI64-/{sum+=$3; n++} END {print "avg ns/op =", sum/n}' bench1.out
avg ns/op = 0.0235705
awk '/BenchmarkReadU64-/{sum+=$3; n++} END {print "avg ns/op =", sum/n}' bench2.out
avg ns/op = 0.0415546

We discussed in our meeting today. I'm going to spend a few hours looking into this before we merge to try to find a root cause if possible

Copy link
Member

@prestonvasquez prestonvasquez Jan 8, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@RafaelCenzano Somehow the problem is this block:

if len(src) < 4 {
    return 0, src, false
}

The assembly ends up being [effectively] the same for either case, though. This is bizarre. I think we should just do type casting, despite the gosec issues. We should not introduce a regression.

  func ReadI32(src []byte) (int32, []byte, bool) {
      if len(src) < 4 {
          return 0, src, false
      }
      //nolint:gosec // G115: Safe bit-pattern reinterpretation
      return int32(binary.LittleEndian.Uint32(src)), src[4:], true
  }

if len(src) < 4 {
return 0, src, false
}

_ = src[3] // bounds check hint to compiler

value := int32(src[0]) |
int32(src[1])<<8 |
int32(src[2])<<16 |
int32(src[3])<<24

return value, src[4:], true
}

// ReadU64 reads a uint64 from src in little-endian byte order. ReadU64 and
// ReadI64 are separate functions to avoid unsafe casting between unsigned and
// signed integers.
func ReadU64(src []byte) (uint64, []byte, bool) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[blocking] This function should wrap the standard library function since doing so doesn't require a type conversion.

if len(src) < 8 {
return 0, src, false
}

return binary.LittleEndian.Uint64(src), src[8:], true
}

// ReadI64 reads an int64 from src in little-endian byte order.
// Byte shifting is done directly to prevent overflow security errors, in
// compliance with gosec G115. ReadU64 and ReadI64 are separate functions to
// avoid unsafe casting between unsigned and signed integers.
//
// See: https://cs.opensource.google/go/go/+/refs/tags/go1.19:src/encoding/binary/binary.go;l=101
func ReadI64(src []byte) (int64, []byte, bool) {
if len(src) < 8 {
return 0, src, false
}

_ = src[7] // bounds check hint to compiler

value := int64(src[0]) |
int64(src[1])<<8 |
int64(src[2])<<16 |
int64(src[3])<<24 |
int64(src[4])<<32 |
int64(src[5])<<40 |
int64(src[6])<<48 |
int64(src[7])<<56

return value, src[8:], true
}

// ReadCStringBytes reads a null-terminated C string from src as a byte slice.
// This is the base implementation used by ReadCString to ensure a single source
// of truth for C string parsing logic.
func ReadCStringBytes(src []byte) ([]byte, []byte, bool) {
idx := bytes.IndexByte(src, 0x00)
if idx < 0 {
return nil, src, false
}
return src[:idx], src[idx+1:], true
}

// ReadCString reads a null-terminated C string from src as a string.
// It delegates to ReadCStringBytes to maintain a single source of truth for
// C string parsing logic.
func ReadCString(src []byte) (string, []byte, bool) {
cstr, rem, ok := ReadCStringBytes(src)
if !ok {
return "", src, false
}
return string(cstr), rem, true
}
Loading
Loading