Don't quicken in-place.

[markshannon]

Now that we are quickening in-place, I wonder if we shouldn't.

An alternative scheme would be to add back the co_firstinstr pointer, and original co_code bytes object.
For functions, co_firstinstr would initially point to a QUICKEN bytecode, to quicken without an additional test on each call.
For classes and modules, co_firstinstr would point to the start of the code in the co_code bytes object.

There are a number of advantages:

• It would potentially save quite a lot of memory for applications that have many functions that are never called.
• No need to unpack the bytecode during unmarshaling.
• It reduces the changes needed to marshal, as QUICKEN does the work of expanding the bytecode.
• It means that the compiler (and templating engines that generate bytecode) don't need to worry about quickening or caches.

Although there are also some disadvantages:

• Extra memory use for quickened code: one pointer plus the size of the compact code form.
• A very small slowdown of calls: getting the instruction pointer would require a memory read
• We would need extra BINARY_OP_UNSPECIALIZED and BINARY_SUBSCR_UNSPECIALIZED bytecodes for quickened code when we want an unspecialized form of those operations, as the base forms would not skip the following cache.

Cost analysis:

• Code that is never run: Less memory used, faster unmarshaling.
• Run once code: Provided that we know that it is run once statically, slightly faster due to lower unmarshaling cost. Execution speed should be the same.
• Code that is run at least once: Increased memory use. No difference in performance: We don't need to unpack during unmarshaling, as the QUICKEN bytecode does the job for us.

The QUICKEN bytecode

This would expand out the compact form into the form with caches, doing quickening at the same time.
It would then update the co_firstinstr and internal variables, and resume execution.

[markshannon]

It is important that quickening, if it happens at all, must occur on the first call.
That way we don't lose one of the advantages of quickening in-place, that we could store pointers to instructions, not just offsets.

[brandtbucher]

I'm a bit worried that this will negatively impact startup time, due to needing to allocate and expand the bytecode and all debugging tables for every function's first call.

It seems that the main issue here is memory bloat due to the debugging tables growing to accommodate more CACHE/EXTENDED_ARG instructions (the related marshal and compiler changes seem well-contained and not-too-complicated). Shouldn't that be adequately addressed by #324?

What's the estimated break-even point (% of code objects ever actually executed once) where this would result in memory savings over the current scheme?

[markshannon]

I'm a bit worried that this will negatively impact startup time, due to needing to allocate and expand the bytecode and all debugging tables for every function's first call.

We currently need to get the expanded bytecode during unmarshaling, so this shouldn't be worse; less data to fetch from disk, but an additional allocation and some copying.

Shouldn't that be adequately addressed by #324?

Yes, #324 should cover it. But we need to implement that as well.

What's the estimated break-even point (% of code objects ever actually executed once) where this would result in memory savings over the current scheme?

The break-even is about 65%, given that the bytecode with caches is about 3 times the compact form.
If half code objects are executed, then this scheme costs about 4.5 bytes per instruction, versus the current scheme's cost of about 6 bytes per instruction.

[gvanrossum]

Can we collect stats about unused code objects? Not from benchmarks. Maybe Cinder knows this for the Instagram code?

I imagine CLI apps might have lots of code that’s never run in any particular invocation.