[mypyc] feat: new primitive for `int.to_bytes` #19674

BobTheBuidler · 2025-08-16T15:51:42Z

This PR adds a new primitive for all arg combinations of int.to_bytes

for more information, see https://pre-commit.ci

… to-bytes

ilevkivskyi

Nice! I will keep this open for a day or two in case @JukkaL has some comments.

mypyc/lib-rt/int_ops.c

mypyc/lib-rt/CPy.h

mypyc/lib-rt/int_ops.c

mypyc/test-data/run-integers.test

ilevkivskyi

On the second look I think I have some more questions.

ilevkivskyi · 2025-08-21T09:56:17Z

mypyc/lib-rt/int_ops.c

+// int.to_bytes(length, byteorder, signed=False)
+PyObject *CPyTagged_ToBytes(CPyTagged self, Py_ssize_t length, PyObject *byteorder, int signed_flag) {
+    PyObject *pyint = CPyTagged_StealAsObject(self);
+    if (!PyLong_Check(pyint)) {


On the second thought, all these type checks look unnecessary, normally Python wrappers should do them. You can probably verify this by adding some run tests with Anys in them.

what, like this?

def f(x: Any) -> bytes: return int.to_bytes(x)

based on @JukkaL response to a similar question on #19673 I think we can safely remove this check since CPyTagged_StealAsObject guarantees the type

I think CPyTagged_StealAsObject is not correct there, since it will transfer the ownership of the parameter, and this can cause a double free. CPyTagged_AsObject will return a new reference which you can decref at the end of the function.

@BobTheBuidler

what, like this?

First, not just the self, second I think you should try more something like this

def to_bytes(n: int, length: int, byteorder: str = "little", signed: bool = False) -> bytes: return n.to_bytes(length, byteorder, signed=signed) x: Any = "no" bad: Any = "way" to_bytes(x, bad)

and check that a TypeError will be given even before getting to your specialized code.

I could implement this test, but wouldn't we then just be testing the standard python-wrapper type validation functionality, as opposed to some specific functionality related to this PR?

I can still add the tests accordingly, I just want to make sure we have the same understanding of things before I proceed.

I could implement this test, but wouldn't we then just be testing the standard python-wrapper type validation functionality, as opposed to some specific functionality related to this PR?

That's exactly my point. This whole thread started from me saying "On the second thought, all these type checks look unnecessary, normally Python wrappers should do them. You can probably verify this by adding some run tests with Anys in them."

To be clear, you don't need to add the tests to the PR, but you can (if you want) to check that I am right by running such a test locally.

ilevkivskyi · 2025-08-21T09:59:24Z

mypyc/test-data/run-integers.test

+    assert to_bytes(255, 2, "big") == b'\x00\xff'
+    assert to_bytes(255, 2, "little") == b'\xff\x00'
+    assert to_bytes(-1, 2, "big", True) == b'\xff\xff'
+    assert to_bytes(0, 1, "big") == b'\x00'


Maybe also test calling to_bytes() function from interpreted code.

@ilevkivskyi how would I implement that? Is there a good example I can look from?

mypyc/lib-rt/int_ops.c

for more information, see https://pre-commit.ci

… to-bytes

for more information, see https://pre-commit.ci

… to-bytes

for more information, see https://pre-commit.ci

… to-bytes

BobTheBuidler · 2025-09-10T09:47:11Z

1.18 vs this branch

There was a ~5-8% increase in speed for various permutations of the int_to_big_endian benchmark which includes not only a call to to_bytes but a decent bit of other stuff as well

Okay, with the specializer this becomes a 9% improvement for all cases

… to-bytes

for more information, see https://pre-commit.ci

ilevkivskyi · 2025-10-10T12:39:33Z

@BobTheBuidler just in case you didn't notice, some tests are failing now.

ilevkivskyi · 2025-10-15T16:19:31Z

@BobTheBuidler please ping me when this is ready for review/merge.

BobTheBuidler · 2025-10-15T16:25:14Z

That's odd. I fixed the kw-only arg in to_bytes stub, but doing so breaks the IR. Is there a way for me to cover this case with a method_op?

@ilevkivskyi this open question is my only remaining blocker, unsure how to navigate this situation

ilevkivskyi · 2025-10-15T17:07:06Z

I don't know, maybe add some prints to specialize_int_to_bytes. For starter I think self is not included in CallExpr.args for methods, but maybe this is something else.

BobTheBuidler and others added 22 commits August 16, 2025 15:50

[mypyc] feat: new primitive for int.to_bytes

2058f81

[pre-commit.ci] auto fixes from pre-commit.com hooks

f62bfd2

for more information, see https://pre-commit.ci

add headers

813c510

Merge branch 'to-bytes' of https://github.com/BobTheBuidler/mypy into…

4be77d6

… to-bytes

cover all arg combos

cb93329

CPyLong_ToBytes header

6ff1c9b

;

166b6f4

fix name

be2d2de

fix _PyLong_AsByteArray compile err

9ff3a7c

Update ir.py

8399619

define header

db7b483

Merge branch 'to-bytes' of https://github.com/BobTheBuidler/mypy into…

ed05c00

… to-bytes

fix ir

a0c147b

Update ir.py

5bab265

fix ir

b346bbf

Merge branch 'to-bytes' of https://github.com/BobTheBuidler/mypy into…

8d523a0

… to-bytes

fix ir

85652a2

add ir test

8e97165

fix py 3.10

3660a01

use _PyLong_AsByteArray on all pythons

6a8a83c

fix: py3.13 and 3.14

ec95008

optimize if check

95fa8b0

BobTheBuidler marked this pull request as ready for review August 17, 2025 16:39

ilevkivskyi approved these changes Aug 20, 2025

View reviewed changes

Merge branch 'master' into to-bytes

78f4ca1

JukkaL reviewed Aug 21, 2025

View reviewed changes

mypyc/lib-rt/int_ops.c Outdated Show resolved Hide resolved

mypyc/lib-rt/CPy.h Outdated Show resolved Hide resolved

mypyc/lib-rt/int_ops.c Outdated Show resolved Hide resolved

mypyc/test-data/run-integers.test Outdated Show resolved Hide resolved

ilevkivskyi reviewed Aug 21, 2025

View reviewed changes

BobTheBuidler added 3 commits August 21, 2025 09:46

Update CPy.h

326484b

Update int_ops.c

245a122

Update int_ops.c

d2994e1

BobTheBuidler and others added 20 commits September 10, 2025 07:27

feat: specialize

bdd320e

[pre-commit.ci] auto fixes from pre-commit.com hooks

4518aea

for more information, see https://pre-commit.ci

missing }

97873da

Merge branch 'to-bytes' of https://github.com/BobTheBuidler/mypy into…

d7b0023

… to-bytes

fix: specialize

26cd5c6

[pre-commit.ci] auto fixes from pre-commit.com hooks

30c4a30

for more information, see https://pre-commit.ci

fix

84acab3

[pre-commit.ci] auto fixes from pre-commit.com hooks

b5a6c06

for more information, see https://pre-commit.ci

fix

ef5b1d7

Merge branch 'to-bytes' of https://github.com/BobTheBuidler/mypy into…

9b629a5

… to-bytes

[pre-commit.ci] auto fixes from pre-commit.com hooks

9b69d58

for more information, see https://pre-commit.ci

fix

6a5de07

Merge branch 'to-bytes' of https://github.com/BobTheBuidler/mypy into…

251986e

… to-bytes

fix

f5f9d30

Update irbuild-int.test

9a27e5d

Update run-integers.test

6c82357

Update irbuild-int.test

c4d9673

maybe fix

3ad1e50

Update int_ops.py

77535d3

Update irbuild-int.test

d291fb1

BobTheBuidler and others added 4 commits September 10, 2025 13:22

Merge branch 'to-bytes' of https://github.com/BobTheBuidler/mypy into…

66bb191

… to-bytes

Merge branch 'master' into to-bytes

420b8e2

Merge branch 'master' into to-bytes

2fc6433

[pre-commit.ci] auto fixes from pre-commit.com hooks

11a6aea

for more information, see https://pre-commit.ci

Merge branch 'master' into to-bytes

2d69419

Uh oh!

[mypyc] feat: new primitive for int.to_bytes #19674

Are you sure you want to change the base?

[mypyc] feat: new primitive for int.to_bytes #19674

Uh oh!

Conversation

BobTheBuidler commented Aug 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ilevkivskyi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ilevkivskyi left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

BobTheBuidler commented Sep 10, 2025

Uh oh!

ilevkivskyi commented Oct 10, 2025

Uh oh!

ilevkivskyi commented Oct 15, 2025

Uh oh!

BobTheBuidler commented Oct 15, 2025

Uh oh!

ilevkivskyi commented Oct 15, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

[mypyc] feat: new primitive for `int.to_bytes` #19674

[mypyc] feat: new primitive for `int.to_bytes` #19674

BobTheBuidler commented Aug 16, 2025 •

edited

Loading