Include `Pdb._exec_in_closure()` as code-path in `exec()` itself, or make it a public utility otherwise?

[krassowski]

Feature or enhancement

Proposal:

When IPython is embedded into a non-IPython interpreter, it evaluates code using frame locals (FrameLocalsProxy) in the same way as pdb does in Python stdlib.

This does not work with objects which require access to the closure scope:

• comprehensions in Python <3.13 (fixed for 3.13+ by inlining comprehensions introduced in PEP 709)
• generators

This bug affected pdb module too (#65360) but it was fixed/worked around by:

• gh-83151: Make closure work on pdb #111094

For example, the following does not work:

import sys
call_frame = sys._getframe(0).f_back
local_ns = call_frame.f_locals
exec('x = 1; sum(x * i for i in range(5))', locals=local_ns)

Produces:

Traceback (most recent call last):
  File "<python-input-0>", line 4, in <module>
    exec('x = 1; sum(x * i for i in range(5))', locals=local_ns)
    ~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<string>", line 1, in <module>
  File "<string>", line 1, in <genexpr>
NameError: name 'x' is not defined

I know that FrameLocalsProxy has a few undocumented limitations, I am not sure if those are relevant here - I am linking the relevant issue just in case if that helps maintainers to refresh the context:

• Document FrameLocalsProxy behaviour quirks #125731

I am coming here from IPython, to explore if the solution belongs in IPython, or in CPython repo:

• a) could exec handle FrameLocalsProxy specially in some way?
• b) as one option, could the same code as added for pdb in gh-83151: Make closure work on pdb #111094 (_exec_in_closure method) be included in the default exec implementation for when FrameLocalsProxy is passed?
• c) if adding _exec_in_closure into exec proper does not make sense, is it a good idea to explore moving _exec_in_closure out of pdb and making it public?

The fix/workaround from #111094 is not perfect yet, though I think most of the problems could be resolved with a little bit more work. These issues were reported in:

• limitations of new multiline code and closure support in pdb #126958

If _exec_in_closure were exposed as public, there would be a bigger incentive for community to fix issues outlined in #126958, basically centralising the effort to make that work well.

On the other hand, if there are plans to inline generators in the future (I saw that PEP 709 leaved that as a possibility), maybe _exec_in_closure would no longer be necessary in the first place.

As an alternative to reusing _exec_in_closure, IPython could use locals() call to populate locals_ns which get passed down to exec via locals argument when in embed mode; I am somewhat apprehensive to make this change as I worry it might break downstream code.

However, if you all advise that _exec_in_closure shall remain a private pdb utility, this would likely tilt the the trade-off towards using the locals() call, as otherwise IPython would need to maintain its own version of _exec_in_closure adding to maintenance cost on already over-stretched team.

Has this already been discussed elsewhere?

This is a minor feature, which does not need previous discussion elsewhere

Links to previous discussion of this feature:

Not directly, but this is the third oldest unresolved issue in IPython dating back over 15 years:

• Global variables undefined in interactive use of embedded ipython shell ipython/ipython#62
• IPythonShellEmbed fails to recognize local variables ipython/ipython#136
• Variable not recognized in nested or filtered list comprehension under IPython.embed() ipython/ipython#12199

CC @gaogaotiantian if I may, as author of #111094 and assignee on #126958

[gaogaotiantian]

First of all, could you make the example code show your actual ideas next time? Your current code will just raise an exception because there's no f_back. A helpful reproducible example is something that we can copy & paste then execute.

To answer your questions. I don't believe this has anything to do with FrameLocalsProxy. It will have the exact same issue if you pass locals(). It's the code you compile - the generator believes x is a global variable but it does not exist in the global name space.

So your question a) is not valid - it's not the FrameLocalsProxy that caused the problem.

Can we use the pdb's closure implementation in exec? No we can't do that. That would break backwards compatibility. Also it's very implicit. The fact exec() takes a closure argument shows that you should deal with it in your debugger - it's an advanced function and we want to keep it relatively straightforward. You have the option to do advanced things, but the default behavior does not assume the code was executed in a potential closure environment.

And we probably won't expose the internal function of pdb either - as you said, it's not perfect. It's kind of a workaround and we don't want to make a promise to it.

Actually, making it to work is very difficult. I must disagree on your thought that "most of the problems could be resolved with a little bit more work", because I've spent plenty of time into it and did not find any good solution without hacking heavily on the code.

The problem will not be fully solved even if generators are inlined - the user can always create a function scope in their code.

I understand that I've talked plenty but did not give any helpful directions, sorry about that. I did put plenty of thoughts into this and if you have a better idea of how to make it work better, I'm happy to look at the solution. I did look at how the original author of that issue solved it in their project - it's a lot of code and full of black magics. It's impossible for us to maintain that in stdlib.

[krassowski]

First of all, could you make the example code show your actual ideas next time? Your current code will just raise an exception because there's no f_back.

Apologies, you are probably right. This is what I see

I do not understand why you say there is no f_back:

A helpful reproducible example is something that we can copy & paste then execute.

I know that. This is what I tried to achieve, I understand this is not the example you would expect. Apologies for annoying you but it is hard for me to isolate the thing from all of the IPython codebase.

I don't believe this has anything to do with FrameLocalsProxy. It will have the exact same issue if you pass locals(). It's the code you compile - the generator believes x is a global variable but it does not exist in the global name space.

I don't understand why you say that.

[gaogaotiantian]

Your locals() and f_locals are not equivalent. Your f_locals is not the local scope of your interpreter, it's the pyrepl object. So you are running everything in the interpreter and that makes it a bit complicated.

If you do

call_frame = sys._getframe(0)

instead of

call_frame = sys._getframe(0).f_back

You'll find that everything just works - that's the equivalent form as your locals() solution. It's not about FrameLocalsProxy, you are just using different frames.

Normally when we try to reproduce an example, we copy the lines in a file, save it and run it with python. If you do that you'll realize that call_frame = sys._getframe(0).f_back gives None because sys._getframe(0) is already at the bottom. It does not raise an exception because of how current Python shell works.

[krassowski]

Your f_locals is not the local scope of your interpreter, it's the pyrepl object.

I know. I think it would make more sense to you in context of ipython/ipython#62

So you are running everything in the interpreter and that makes it a bit complicated

Yes, I know. I said I am coming from IPython :)

Normally when we try to reproduce an example, we copy the lines in a file, save it and run it with python. If you do that you'll realize that call_frame = sys._getframe(0).f_back gives None because sys._getframe(0) is already at the bottom. It does not raise an exception because of how current Python shell works.

Fine, indeed my reproducer assumed that one would be running it in a Python shell. It still stands that this is a real issue, and this is the smallest reproducer I was able to create.

Anyways, since it seems you are not very enthusiastic about adding a code path to exec which would handle generators well (whether that would be triggered by FrameLocalsProxy type or a new argument or whatever - you know better than me), feel free to close this issue.

[gaogaotiantian]

This is a real issue, it just has nothing to do with FrameLocalsProxy. The problem is that when we compile a piece of code, we have no way to know that it is supposed to be in another function (closured). We can't change that behavior because it's not backward compatible - it's not that we don't want to make it better, we can't make it better without breaking a lot of the old code.

The way pdb deals with it is very hacky and should only live in a debugger. You need to change a lot of things to make exec really work in such scenario - I think you probably need to change the compiler.