pickletools optimize performance degraded with factor 3-4 in python 3.14

[dirkroorda]

Bug report

Bug description:

The pickletools.optimize code suddenly performs bad starting with python 3.14, with or without JIT.
The degradation is a factor 3 to 4.

It is easy to test.

First use this script to make a sizeable dict and pickle it and write it to file.

import pickle

x = {}

for i in range(1, 1000000):
    x[i] = f"ii{i:>07}"

with open("pickled", "wb") as fh:
    pickle.dump(x, fh, protocol=4)

Then write a script, let's call it optpickle.py to read the pickled dict from file and optimize it and write the result to another file.

# optpickle.py

import pickletools

with open("pickled", "rb") as fh:
    p = fh.read()

s = pickletools.optimize(p)

with open("pickledopt", "wb") as fh:
    fh.write(s)

Now execute this in three separate terminal windows, one geared to python 3.13, one to python 3.14.0 without JIT, and one to python 3.14.0 with JIT.

My results on a mac with M1 processor were

version	time elapsed	cpu time
3.13	1.67	1.773
3.14.0 no JIT	6.52	6.664
3.14.0 JIT	6.52	6.642

That is a dramatic drop in performance!

The filesizes are 16.9 MB for the unoptimized file and 15.9 for the optimized one.

I decided to drop the optimization in an app where I used it heavily

CPython versions tested on:

3.14

Operating systems tested on:

macOS

[StanFromIreland]

Are your results an average, or just one off tests?

[tim-one]

Just curious about how things might change if you added:

import gc
gc.disable()

Another recent report (sorry, can't find it now) pointed out slowdowns when creating lots of little tuples, due to changes in the cyclic garbage collector (which isn't really needed here). optimize() does create lots of little tuples.

[sergey-miryanov]

@tim-one #139951

[tim-one]

FYI, on my box (Win10 Pro with the released 3.14.0 from pyhon.org), the time for your second script drops from about 12 seconds to about 4 seconds when disabling gc, and enclosing the body of your script in a loop to run both ways repeatedly (to factor out possible "warmup" distortions).

[dirkroorda]

My app Text-Fabric used the optimize function when loading a dataset consisting of approx hundred largish dicts with integer keys. That's where I noticed the time lag.

I have played with the gc in the past, when I tried to squeeze out every performance gain I could get. I'll have another look at that.

At the same time, it seems that the gain of optimizing is very little. The optimized file size is not much less than the unoptimized one, and I did not notice slower load times.

So my app is better off without the optimize.

[dirkroorda]

Here are updated results, now showing the influence of the garbage collector:

Python version	JIT enabled	GC enabled	Real (s)	CPU (s)
13	no	yes	1.722	1.719
13	no	no	1.660	1.659
14	no	yes	6.545	6.543
14	no	no	1.444	1.444
14	yes	yes	6.558	6.558
14	yes	no	1.473	1.473

Here is the code

[dirkroorda]

In the past, I had switched off the gc in this part of the app, but I did not notice a dramatic difference, so I switched it on again. If I had left it off since then, I would not have noticed the drop in performance going from 3.13 to 3.14, and this issue would not have been created!

[dirkroorda]

I multiplied the dict size by 10, now there are 10 million items, and the pickled size is 169,9 MB.

In Python 3.13, the time spent is indeed linear: almost exactly 10 times as much.

But in Python 3.14 things are worse, when the GC is on. I'm afraid it is worse than quadratic. A 77-fold increase.

Python version	JIT enabled	GC enabled	Real (s)	CPU (s)
13	no	yes	17.250	17.205
13	no	no	16.567	16.561
14	no	yes	506.329	505.930
14	no	no	14.906	14.892
14	yes	yes	509.202	508.373
14	yes	no	14.755	14.748

[dirkroorda]

@sergey-miryanov Thanks for pointing to the issue on the garbage collector and its behaviour with small tuples. Although for me GC is largely a black box, it is enlightening to see where the performance drop comes from.

For my app, dicts are absolutely crucial as an economical (fast and not too memory hungry) solution to dealing with sparse arrays. My dicts typically have integer keys, so I'm not bitten by the small tuple problem.

[tim-one]

@dirkroorda, thank you for the very thorough and helpful follow-ups! You've done more than your share here 😄

pickletools.optimize() rarely saves much. What it's aiming at: pickling is a 1-pass process, writing out "opcodes" as it goes along. When an object is pickled, there's no way to know whether that object may appear again as a sub-object in the entire object graph being pickled. So, "just in case", a PUT opcode is pumped out after a single object's pickle has been produced. If the same object is encountered again, a GET opcode is emitted referencing the index of that object saved away by PUT.

It's often the case that a given PUT is never referenced again. All optimize() aims at is removing the PUT opcodes that are never used. But a PUT opcode occupies few bytes, and there can't be more of them than there are individual sub-objects pickled. Bytes produced for the payload (actual data bytes) typically account for the vast bulk of a pickle stream. optimize() can't do anything to shrink those.

So optimize() is rarely worth the bother. I wrote optimize(), so that's not just sour grapes 😉. Nevertheless, it can be useful in some niche cases. I used to be a "gc expert" too, but that's changed a lot since my days on it, and I no longer have actual insight into how it currently works. So I can't do more here than confirm that this really has almost nothing to do with optimize(), apart from that it creates object graphs of kinds that provoke 3.14's version of gc into bad performance.

I have no doubt that the current gc experts have already thought of several ways that could be repaired.