Inconsistency in Cython object handling

[x42005e1f]

Bug report

Bug description:

There seems to be no consensus in the codebase on how to handle Cython objects. Both inspect.iscoroutinefunction() and inspect.iscoroutine() are defined in the documentation for async def functions and their objects, respectively. Therefore, the following code should be valid for them:

#!/usr/bin/env python3

import inspect


async def coroutine_function() -> None:
    pass


def main() -> None:
    print(inspect.iscoroutinefunction(coroutine_function))  # `True`
    print(inspect.iscoroutine(coro := coroutine_function()))  # `True`
    coro.close()  # to avoid `RuntimeWarning`


if __name__ == "__main__":
    main()

The same semantics are also expected by typeshed, which defines annotations as follows (and thus declares support for introspection):

...
@overload
def iscoroutinefunction(obj: object) -> TypeGuard[Callable[..., CoroutineType[Any, Any, Any]]]: ...
...
def iscoroutine(object: object) -> TypeIs[CoroutineType[Any, Any, Any]]: ...
...

However, after cythonize -i -3, the code changes its behavior:

True  # `inspect.iscoroutinefunction()`
False  # `inspect.iscoroutine()`

This is because Cython uses its own objects for compiled functions and generators/coroutines/asynchronous generators. inspect.is<object>() returns True only for uncompiled objects, as it relies on the types module, but inspect.is<object>function() relies on checking function flags and handles function-like objects in a special way (for compatibility with Cython). As a result, we get the contradictory "coroutine function does not return a coroutine".

The problem is even deeper than one might imagine:

1. Cython objects do not actually correspond completely to their counterparts from CPython. See [BUG] inspect.get*state() does not work for compiled objects on Python ≥3.11 cython/cython#7448.
2. Mypy avoids using the types module to infer the return type due to possible incompatibility with Cython. See mypy should use CoroutineType instead of Coroutine for return types of async def functions mypy#18635 and Allow types.GeneratorType to be used instead of collections.abc.Generator as the return type of generator functions mypy#20522.
3. If the user wants to handle objects regardless of whether the module has been compiled or not, they have to come up with their own way of checking objects. See Add a collection.abc.Function ABC for function-like objects #131983.

What if we want to define a universal decorator that works for both regular and asynchronous functions? We could use inspect.iscoroutinefunction(), but it has very limited detection (in particular, it cannot detect object with async def __call__()), and library developers will only be able to rely on inspect.markcoroutinefunction() after the Python 3.11's EOF. A frequently suggested alternative is to check the return value's type using inspect.isawaitable(), but this is actually shooting oneself in the foot, as it will lead to false positives for future-like objects (such as asyncio future objects, which are also awaitable). The most logical solution would be to use inspect.iscoroutine(), but it does not support Cython (and in fact, it is even less useful; see GrahamDumpleton/wrapt#236 (comment)).

It seems to me that the question of whether the standard library (and related tools) should be Cython-aware has not been fully explored, which leads to such contradictions. I marked the issue as a bug, since it violates the expected behavior for any consistent interpretation.

CPython versions tested on:

3.9, 3.10, 3.11, 3.12, 3.13, 3.14

Operating systems tested on:

Linux

[ZeroIntensity]

A frequently suggested alternative is to check the return value's type using inspect.isawaitable(), but this is actually shooting oneself in the foot, as it will lead to false positives for future-like objects

Why is this necessarily a bad thing? If all you want to do is await something, I don't see why a future would be bad.

Anyways, the ideal solution is to avoid using inspect entirely. inspect.iscoroutine is only meant to detect native coroutines (types.CoroutineType instances), not other coroutine types. If a library wants to support compiled coroutines from Cython or something like that, then they should be using isinstance on collections.abc.Coroutine.

It seems to me that the question of whether the standard library (and related tools) should be Cython-aware has not been fully explored, which leads to such contradictions. I marked the issue as a bug, since it violates the expected behavior for any consistent interpretation.

I don't think CPython should add special cases solely for Cython. That would lead to an increase in maintenance burden when Cython inevitably changes how they implement coroutines or similar, and we don't have a great way to test it without adding Cython to the test suite.

If Cython wants new generic APIs for coroutines, that seems feasible, but I don't see any such proposal here.

[x42005e1f]

If a library wants to support compiled coroutines from Cython or something like that, then they should be using isinstance on collections.abc.Coroutine.

Why should a user library care about Cython support when it works with Python-level code? Any Python code can be compiled via cythonize. The fact that Cython objects do not inherit from types.<Object>Type (while Nuitka does) is an intentional design decision, and both CPython and Mypy already take care of this for some reason (why?).

I don't think CPython should add special cases solely for Cython. That would lead to an increase in maintenance burden when Cython inevitably changes how they implement coroutines or similar, and we don't have a great way to test it without adding Cython to the test suite.

But it already has them, does not it?

[x42005e1f]

To be clear, I am not proposing anything in this issue. Neither to strengthen support for Cython nor to remove it completely. I am merely summarizing that, because the question of Cython support has apparently never been seriously discussed, the implementation contains conflicting decisions, and this uncertainty significantly complicates typing and introspection (and violates the expected behavior).

Mypy avoids using types.<Object>Type because they are not compatible with Cython. typeshed uses types.<Object>Type in standard library annotations, which is not compatible with Cython. It seems like the same Python, but they contradict each other. And the same thing happens in CPython itself.

[ZeroIntensity]

Why should a user library care about Cython support when it works with Python-level code? Any Python code can be compiled via cythonize.

I don't know, but this is a better question for the Cython people. If they want a better API, then we can consider adding something to CPython.

But it already has them, does not it?

None of these comments are actually special to Cython. They're merely generalizations, and Cython is an example of where they're hit.

To be clear, I am not proposing anything in this issue. Neither to strengthen support for Cython nor to remove it completely. I am merely summarizing that, because the question of Cython support has apparently never been seriously discussed, the implementation contains conflicting decisions, and this uncertainty significantly complicates typing and introspection (and violates the expected behavior).

This sounds like something that should be a DPO thread, then. Our issue tracker isn't a good place for discussion/summaries. If there's problems with typeshed's annotations, then go report it over there.

[x42005e1f]

None of these comments are actually special to Cython. They're merely generalizations, and Cython is an example of where they're hit.

I think we have strayed from the point. Why does iscoroutinefunction() have a generalized implementation, but iscoroutine() does not? The former could rely on isfunction(), but it does not. At the same time, both are defined for async def functions (see the documentation). Whether we are considering Cython here or not is secondary.

This sounds like something that should be a DPO thread, then. Our issue tracker isn't a good place for discussion/summaries. If there's problems with typeshed's annotations, then go report it over there.

I posted this as an issue because I considered the observed behavior to be inconsistent. I used the annotation problem as an example (after all, type checkers reflect runtime behavior). That is a consequence, not a cause.

I apologize for not being able to convey my thoughts to you clearly. If you believe that the issue should be closed, that is fine.

[x42005e1f]

None of these comments are actually special to Cython.

And in general, what do you consider "special to Cython"? In the pursuit of generalization, almost any case can be generalized to "object-like" things. That is how #68504 was resolved. People will avoid using implementation-specific names, attributes, and types, but the intention remains the same.

[ZeroIntensity]

Why does iscoroutinefunction() have a generalized implementation, but iscoroutine() does not?

"generalized" is doing some heavy lifting here. iscoroutinefunction is basically defined as "if I call this object, will the return value be awaitable?" whereas iscoroutine is defined as "is this object a native coroutine?"

In hindsight, iscoroutine should have been named isnativecoroutine (or perhaps not added at all and send users to isinstance instead), but that ship has sailed.

I apologize for not being able to convey my thoughts to you clearly. If you believe that the issue should be closed, that is fine.

My concern is that there's not really anything that is actionable here. It's fine to discuss whether something is a bug, but that should happen on DPO so it gets more eyes. Issues fly too low under the radar.

And in general, what do you consider "special to Cython"? In the pursuit of generalization, almost any case can be generalized to "object-like" things.

I meant "special to Cython" as in implementing fixes that are only applicable to Cython. If something is a problem across multiple C coroutine implementations, then that's probably worth fixing because that would be indicative of a more general problem regarding non-native coroutines. But, as far as I can tell, this issue is more about "how should we handle Cython coroutines?" The answer is we shouldn't; we should handle non-native coroutines where we can, with Cython as an example use case.

---

I'm going to close this on the basis that it will likely just collect dust on the issue tracker, and it doesn't look like there's any actionable items here anyway. Please create a thread on the "Core Development" category of DPO so we can get some more opinions. (For that thread though, please focus the question more on "how do we support other types of coroutines?" rather than "how do we support Cython?")