Skip to content

Replace VarInt.getLength with Protobuf's branchless implementation #36959

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
sjvanrossum wants to merge 4 commits into
base: master
Choose a base branch
from
Open

Replace VarInt.getLength with Protobuf's branchless implementation #36959

Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
8a82c18
Replace VarInt.getLength with Protobuf's branchless implementation
sjvanrossum Dec 2, 2025
07b1ee1
Use CodedOutputStream's computeUInt32SizeNoTag and computeUInt64SizeN…
sjvanrossum Dec 2, 2025
f859813
Run spotless
sjvanrossum Dec 3, 2025
76fc879
Merge remote-tracking branch 'origin/master' into varint-get-length
sjvanrossum Dec 3, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 10 additions & 7 deletions sdks/java/core/src/main/java/org/apache/beam/sdk/util/VarInt.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -136,16 +136,19 @@ public static long decodeLong(InputStream stream) throws IOException {

/** Returns the length of the encoding of the given value (in bytes). */
public static int getLength(int v) {
return getLength(convertIntToLongNoSignExtend(v));
// log2(v) / 7 + 1 rewritten as multiplication by 9/64 instead of a division by 7.
// Log2 is performed using a bit counting instruction.
// Multiplication by 9 is performed using a 3-bit left shift and add.
// Division by 64 is performed using a 6-bit right shift.
return ((Integer.SIZE * 9 + (1 << 6)) - (Integer.numberOfLeadingZeros(v) * 9)) >>> 6;
Copy link
Contributor

@scwhittle scwhittle Dec 2, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could we instead call the public static methods on CodedOutputStream?
ie CodedOutputStream.computeUInt32SizeNoTag here and 64 below?

Then if further improvements are made we get them for free.

Copy link
Contributor Author

@sjvanrossum sjvanrossum Dec 2, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't realize core depends on protobuf, I thought that everything had moved to extensions/protobuf.
I've imported CodedOutputStream from the unvendored com.google.protobuf package, but would it make more sense to pull it in from org.apache.beam.vendor.grpc.v1p69p0.com.google.protobuf?

Copy link
Contributor

@scwhittle scwhittle Dec 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since we're not leaking an object just calling the function, I think the unvendored package is fine. But looping in @Abacn to confirm.

}

/** Returns the length of the encoding of the given value (in bytes). */
public static int getLength(long v) {
int result = 0;
do {
result++;
v >>>= 7;
} while (v != 0);
return result;
// log2(v) / 7 + 1 rewritten as multiplication by 9/64 instead of a division by 7.
// Log2 is performed using a bit counting instruction.
// Multiplication by 9 is performed using a 3-bit left shift and add.
// Division by 64 is performed using a 6-bit right shift.
return ((Long.SIZE * 9 + (1 << 6)) - (Long.numberOfLeadingZeros(v) * 9)) >>> 6;
}
}
Loading