Generator expression behavior changed in 3.13.4 - it does not throw exception anymore

[Roman513]

Bug report

Bug description:

In 3.13.3 and before

>>> bool(_ for item in False)
Traceback (most recent call last):
  File "<python-input-14>", line 1, in <module>
    bool(_ for item in False)
                       ^^^^^
TypeError: 'bool' object is not iterable
>>> (_ for item in False)
Traceback (most recent call last):
  File "<python-input-15>", line 1, in <module>
    (_ for item in False)
                   ^^^^^
TypeError: 'bool' object is not iterable

Now in 3.13.4

>>> bool(_ for item in False)
True
>>> (_ for item in False)
<generator object <genexpr> at 0xffffa9c02200>

This is very fundamental change. Is it expected?

CPython versions tested on:

3.13

Operating systems tested on:

Linux

Linked PRs

• [3.14] GH-135171: Fix generator expressions one last time (hopefully) #135225
• [3.13] GH-135171: Fix generator expressions one last time (hopefully) (GH-135225) #135293
• gh-135171: Update documentation for the generator expression #135351
• [3.14] GH-135171: Revert async generator expressions behavior #135352
• [3.13] GH-135171: Roll back all fixes for GH-127682 as they are not suitable for 3.13 #135390
• [3.13] GH-135171: Roll back fix for GH-125038 as it's not suitable for 3.13 #135403
• [3.14] gh-135171: Update documentation for the generator expression (GH-135351) #135524
• [3.13] gh-135171: Update documentation for the generator expression (GH-135351) #135525

[Roman513]

Most probably related to fix #132384 of #127682

[serhiy-storchaka]

We should restore an old behavior which was intentional: GET_ITER should be called before passing argument to generator expression, not after.

[Yhg1s]

Sounds like the change should be reverted. We're doing an expedited 3.13.5 for other reasons, and the rollback can go in there.

[serhiy-storchaka]

We need two rollbacks -- one for the change that removed GET_ITER before calling the generator expression, and other for the change that added GET_ITER in a generator expression (that may not be trivial).

[markshannon]

What is the bug here?

bool(_ for item in False) should be True. All generators are truthy.

If the intent was to write any(_ for item in False) or all(_ for item in False) then the expected TypeError is raised.

Consider the equivalent generator function:

def gen():
     for item in False:
        yield _

creating the generator does not raise an exception, but iterating over it does.

>>> bool(gen())
True

Iterating over an iterable is a two step process.:

1. Convert the iterable into an iterator (using the GET_ITER instruction)
2. Repeatedly get the next item from the iterator (using the FOR_ITER instruction)

For for loops and generators, the FOR_ITER and GET_ITER are together and everything works fine.

Historically, for reasons I am unaware of, in generator expressions the conversion from iterable to iterator occurred when the generator was created, not when used. This left a gap between conversion and the (unchecked) iteration which could mean that iteration could crash.
#125038.

We attempted to fix that by putting a GET_ITER in the generator expression as well, but that breaks iterators without __iter__ methods, or where the __iter__ has side effects. #127682
It also introduced a crash, but in SWIG generated code: rpm-software-management/libdnf#1682

So we must do one of the following:

1. Allow a crash. I'm strongly opposed to this option
2. Break working iterators that do not implement __iter__, which have always worked.
3. Allow the creation of some generator expressions that was previously not allowed, but will still fail as expected when iterated over.

Option 3 is what we now have and seems to me to be easily the best option. It breaks no working code, and produces the same exceptions for faulty code in a way that is largely indistinguishable from the behavior in 3.12.

That generator expressions and generator functions behaved differently could also be considered a bug.

See also #127682 (comment)

[serhiy-storchaka]

It should be a TypeError, because False is not iterable. It's been that way since the beginning, and I believe it was intentional, and that the existing code depends on it. It helps to detect bugs. Even if we want to change the behavior, it should be preceded by a general discussion, and we can only do that in a new Python version, not in a bugfix release.

We just have to return the old behavior. The crash you mentioned is only possible after low level hacking. The behavior change is visible to common users.

[Roman513]

@markshannon I do not insist that it must be fixed, but I thought it could be considered for a major release. I encountered this issue in legacy code, and there’s no question it should be refactored to avoid this. The exception is used there to exit recursion while parsing nested dict-like structures, if you need a real-world scenario.
Casting to bool is not important, a more generic example would be something like x = (_ for _ in None). If the examples in the bug description seem confusing, I can replace them.

[markshannon]

@Roman513 the problem is not that your examples are hard to use, but they appear contrived. What was the original issue you found (in actual code)?

[markshannon]

@serhiy-storchaka What about the other issues I listed, do they do not deserve to be fixed? Why this issue and not those?