Expose called function on frame object

[ericsnowcurrently]

BPO	12857
Nosy	@jcea, @ncoghlan, @benjaminp, @njsmith, @meadori, @durban, @markshannon, @ericsnowcurrently, @1st1, @jstasiak, @DimitrisJim
Files	called_function.diff called_function_2.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/ericsnowcurrently'
closed_at = None
created_at = <Date 2011-08-30.05:54:27.258>
labels = ['interpreter-core', 'type-feature']
title = 'Expose called function on frame object'
updated_at = <Date 2019-06-14.19:51:36.791>
user = 'https://github.com/ericsnowcurrently'

bugs.python.org fields:

activity = <Date 2019-06-14.19:51:36.791>
actor = 'zach.ware'
assignee = 'eric.snow'
closed = False
closed_date = None
closer = None
components = ['Interpreter Core']
creation = <Date 2011-08-30.05:54:27.258>
creator = 'eric.snow'
dependencies = []
files = ['23071', '23149']
hgrepos = []
issue_num = 12857
keywords = ['patch']
message_count = 19.0
messages = ['143201', '143248', '143249', '144008', '144064', '150280', '150282', '156961', '156962', '222158', '279846', '294295', '294412', '294430', '294431', '294432', '294433', '294436', '294475']
nosy_count = 14.0
nosy_names = ['jcea', 'ncoghlan', 'benjamin.peterson', 'Arfrever', 'njs', 'cvrebert', 'meador.inge', 'daniel.urban', 'Mark.Shannon', 'eric.snow', 'yselivanov', 'jstasiak', 'Jim Fasarakis-Hilliard', 'kevlevrone']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue12857'
versions = ['Python 3.5']

[ericsnowcurrently]

This patch adds f_func to PyFrameObject and sets it for functions that get called (in PyFrame_New). For classes and modules it is set to None. The difference in performance was not noticable, as far as I could tell. However, I am willing to do more than just time 'make test' a few times if there is any concern.

A couple weeks ago a thread on python-ideas centered on the subject matter of PEP-3130[1]. The discussion started with, and mainly involved, the idea of adding __function__ to the frame locals during execution of the function. __function__ would point to the function that was called, which had resulted in the frame.

I spent quite a bit of time getting this to work using a closure, but the result was overkill. It also made it too easy to use __function__ for recursion, which Guido did not like (and I agree).

At this point it dawned on me that it would be much simpler to just add the called function to the frame object. This patch is the result. In the end it is much more efficient than the locals approach I had been taking.

[1] http://mail.python.org/pipermail/python-ideas/2011-August/011062.html

[ericsnowcurrently]

Thanks for the review, Nick. I'll be uploading a new patch in a couple hours with your recommended fixes.

Regarding the comments on python-ideas, would it be better to use a weakref proxy around the function, to help with the reference cycles? That's what I was doing with my closure-based solution. I didn't do it here just to see what would happen and I didn't see any problems in my very limited testing (basically just 'make test'). I don't mind using weakrefs and, if it matters, I could pre-allocate the weakref proxy in PyFunction_New to save a little overhead at each call.

For the moment I left in the code to limit f_func to only functions. I'll respond to that on python-ideas.

Finally, how does this patch relate to the ABI? I'm not too familiar with it (read through PEP-384) and want to make sure I'm okay here.

[ericsnowcurrently]

On second thought, I probably won't be able to get an updated patch tonight. I need to mull over the PyEval_EvalFunction implementation and the interaction with fast_function.

[ericsnowcurrently]

Finally had a chance to get back to this. Here's a new patch in response to Nick's review.

My only concern is the new _PyEval_EvalFunctionCode function. It is basically the existing PyEval_EvalCodeEx function with an extra parameter. I've turned PyEval_EvalCodeEx() into a very light wrapper around _PyEval_EvalFunctionCode(). This is how I tackled the backwards incompatibility of my previous patch. However, I am nervous about messing around with something like PyEval_EvalCodeEx().

I was toying with the idea of doing a more comprehensive refactor involving call_function, fast_function, and do_call, which seems like it would be a better fix. However, I'm even more reticent to make that scale of changes, especially on something so critical, even if it seems right to me currently.

I figured the safe thing for now would be to name the new function with a leading underscore to mark it as private.

Here are the results from my cursory check before and after my latest patch:

BEFORE
3 tests failed:
test_distutils test_packaging test_socket
[293234 refs]
real 14m40.578s
user 11m43.547s
sys 0m52.096s

AFTER
3 tests failed:
test_distutils test_packaging test_socket
[293171 refs]
real 14m33.785s
user 11m55.437s
sys 0m53.850s

[ericsnowcurrently]

Nick, does thing look better?

[benjaminp]

What would you use the functionality provided by this patch for?

[ericsnowcurrently]

My response to a similar query:

What it enables is the ability to introspect the *actual* function object belonging to the currently executing code, whether in the same execution frame or in a later one on the stack. We don't have that now. The best we can do is guess, and I'm trying to refuse the temptation by building in an alternative that doesn't get in the way (and doesn't provide an attractive nuisance).

My motivation for this patch was to be able to find out, in one of my functions, what function called it and to interact with that function object. The name or file represented in the code object on the stack didn't cut it; and decorators and function factories make the whole thing a mess. Guido sounded rather positive to the idea, which is why I pursued it. From Guido's post to which I linked in the issue[1]:

> For me the use case involves determining what function called my
> function. Currently you can tell in which execution frame a function
> was called, and thereby which code object, but reliably matching that
> to a function is not so simple. I recognize that my case is likely
> not a general one.

But it is a nice one. It solves some issues that pdb currently solves
by just using a file/line reference.

Not that it necessarily matters, but I could dig up the bit of code that motivated the whole thing. I recall it was something related to writing an enum library (a la flufl.enum). Anyway, I'd love to see this move forward.

[1] http://mail.python.org/pipermail/python-ideas/2011-August/011062.html

[ericsnowcurrently]

It's been a while and I probably need to clean up that last patch a little. I also need to address the questions of:

1. f_func for class/module execution
2. thread-safety

see:
http://mail.python.org/pipermail/python-ideas/2011-August/011321.html
and
http://mail.python.org/pipermail/python-ideas/2011-August/011064.html

[ericsnowcurrently]

for reference, a related issue:

http://bugs.python.org/issue13672

[BreamoreBoy]

Note that bpo-13672 referred to in msg156962 refers to bpo-13855.

[ncoghlan]

This topic came up in a discussion I was having with Yury, and thanks to generators and coroutines, I don't think "f_func" would be the right name for an attribute like this, with something more neutral like "f_origin" potentially being preferable

The specific runtime debugging use case I have in mind is this:

1. Given a frame reference, navigate to the generator-iterator or coroutine that spawned that frame
2. Given gi_back and cr_back doubly-linked counterparts to the current gi_yieldfrom and cr_await introspection attributes on generators and coroutines, expose the relevant generator and/or coroutine stack in addition to the frame stack

These secondary stacks could then potentially be incorporated into traceback outputs by default

[njsmith]

I'd also like to make use of this in trio, as a way to get safer and less complicated control-C handling without having to implement things in C. (Exhaustive discussion: https://vorpus.org/blog/control-c-handling-in-python-and-trio/)

@nick: I understand your proposal is to add a field on the frame that for regular function calls points to the function object, and for generator/coroutines points to the generator/coroutine object. Is that right? Two possible concerns there: (a) this creates a reference cycle, because generator/coroutine objects own a reference to the frame object, (b) AFAICT you also currently can't get from a generator/coroutine object back to the function object, so this would break some (all?) of the use cases for this. I guess the solution to (a) is to make it a weak reference, and for (b) either keep f_func as always pointing to the function and make f_generator a separate field, or else make it f_origin and also add gi_func/co_func fields to generator/coroutine objects. (Which might be useful in any case.)

Also, I'm not quite following the proposed use case. When a generator/coroutine is resumed, then send() steps through the whole yield stack and reconstructs the frame stack so it matches the generator/coroutine stack. (throw() currently doesn't do this, but that's a bug that breaks profilers/debuggers/etc - bpo-29590 - so we should fix it anyway, and I think Yury is planning to do that soon.) So if you get a frame from sys._getframe() or similar, then the stack is correct. And if a coroutine/generator isn't running, then the only way to get to the frame is by starting with the coroutine/generator object, so we don't really need a way to get back.

Tracebacks are a different case again because they continue to hold frame references after the stack is unwound, but generally the traceback already has the correct stack because the exception propagates up the coroutine/generator stack, and also aren't the {gi,co}_frame references cleared as the exception propagates anyway?