[Feature Request] Option to Recurse into All Subdirectories with Namespace Packages

[jtschoonhoven]

Feature Request
An option to recursively type-check all .py files under a given directory when using namespace_packages (without __init__.py).

Use Case
Given the following directory structure:

my-py3-repo/
├── hello/
│   ├── services/
│   │   └── hello_service.py
│   └── hello.py
├── scripts/
│   ├── db/
│   │   └── migrate.py
│   └── manage.py
└── tests/
    └── hello/
        ├── services/
        │   └── hello_service_test.py
        └── hello_test.py

Assuming that hello.py imports hello_service.py, everything under the hello namespace will be type checked as expected with mypy ./hello.

However test discovery with pytest, nose, django et al works differently and hello_test.py would not usually import hello_service_test.py. There is currently no way for Mypy to discover hello_service_test.py with mypy ./tests (if not using __init__.py).

Similarly, everything under the scripts directory would suffer the same problem.

If Mypy supported a --recursive -r option (or similar) that would cause it to automatically recurse into subdirectories, this would solve these common use cases.

Why not just use __init__.py?
To quote iScrE4m's comment from #1645 (comment)_,

In future, there will be python programmers who never heard of pre PEP420 era of __init__.py and that's good. What's not good is mypy forcing them to create dummy files.

Configuration

# Pipfile
[dev-packages]
mypy = "==0.670"
[requires]
python_version = "3.7"

# setup.cfg
[mypy]
python_version = 3.7
ignore_missing_imports = True
namespace_packages = True

[JohnHBrock]

Which of the following would be a better solution: (1) changing the behavior of --namespace-packages so that it recurses into all subdirectories, or (2) adding a new flag (such as --recursive) that has the same effect as --namespace-packages, except it recurses into all subdirectories?

My vote is for option 1 just to keep things simple by having as few command line args as necessary, and because I find the current behavior of --namespace-packages unintuitive (I was surprised to discover I have a repo where some .py files weren't getting checked by mypy simply because they were too deep in a directory hierarchy without __init__.py files). I'm guessing there are other projects out there right now where people are running mypy foo/, not getting any errors, and therefore assuming everything is fine with their type annotations, when in reality they have .py files in the hierarchy under foo/ that aren't getting checked.

Of course, the flip side is that changing the current behavior of --namespace-packages will break some people's mypy/CI scripts when they update mypy (e.g., in @jtschoonhoven's example above, someone might be running mypy ./tests and truly want it to skip hello_service_test.py).

[gvanrossum]

Like everything with Python import, stuff here is more complicated than it seems.

Mypy has two different ways of finding files: one for the initial set off target files specified on the command line, and a separate set of rules for following imports.

The --namespace-packages flag applies to following imports only. But here we’re concerned with the file targets specified on the command line.

I think it’s actually a bug that when you specify a directory on the command line it doesn’t really recurse into that directory, but does a shallow directory listing.

I propose to fix that bug and replace the shallow directory listing with a proper walking of the directory hierarchy starting at that point.

Note that the -p or --packages flag is still different: its argument is not a directory relative to the current directory, but specifies a search in the initial $MYPYPATH, so that’s unrelated.

[JohnHBrock]

I see, thanks for the clarification. I'll work on a PR that properly walks the directory hierarchy for targets specified on the command line.

[Coder-Sharon]

+1

I've been hoping for mypy to recursively traverse the given directory for some time. I look forward to this fix!

[JohnHBrock]

I'm investigating what it will take to make this change and found a problem:

SourceFinder.expand_dir() uses crawl_up_dir() to figure out the top module and base directory for the package (link to code). crawl_up_dir() works by recursively checking the parent directory until it finds one without an __init__.py file.

How should mypy determine the base directory and top module for namespace packages since there are no __init__.py files for crawl_up_dir() to rely upon?

[gvanrossum]

Hm, looking more at that code it seems that expand_dir() does recurse down subdirectories, but only if they have __init__.py files (there's a recursive invocation on L87 that will return [] immediately if there isn't one, see L67-L69). That's a bit inconsistent since it doesn't insist on a __init__.py file for the toplevel directory.

This puts us in a difficult position. The current code seems to be treating the toplevel directory as a member of $MYPYPATH, and every module it finds is rooted there. For example, if we have this directory:

foo/
 +---- bar.py
 +---- baz/
        +---- __init__.py
        +---- yo.py

and we run mypy foo, then the global module namespace contains bar, baz and baz.yo.

And if we wanted to simply extend this to namespace packages (probably only when using --namespace-packages), we would expect to obtain the same hierarchy if we left out foo/baz/__init__.py.

But an alternative interpretation might be that foo itself would be a namespace package, and then we'd expect the global module namespace to have foo, foo.bar, foo.baz, and foo.baz.yo. (Same as if there was also a file foo/__init__.py.)

So which should it be? I'm not sure.

It might be easier to expand the meaning of -p when combined with --namespace-packages to have the second interpretation, so that mypy --namespace-packages -p foo would use the second interpretation above, regardless of whether foo/__init__.py exists. (BTW: It actually does this when foo/__init__.py is absent, but when that file exists, it seems to do something different. Worth investigating.)

But this doesn't provide exactly the same functionality: the argument to -p must be a valid module name, whereas when running mypy <directory> the directory could contain slashes.

Oh wait: another difference is that with mypy -p <package> only the package's __init__.py and anything it imports is checked, while with mypy <directory> all files in the directory are checked. But with --namespace-packages, -p seems to also dive into subdirectory. I don't have time to investigate all this -- clearly I've let the complexity of this code get away from me, and it will take some time to sort it out.

(In the meantime, a useful diversion might be to implement #7672 first.)

[aldanor]

@JohnHBrock wondering if there any news or any help needed?

In the current implementation, mypy pretty much discourages users from going with pep420; I personally had to go and add a bunch of those pesky empty __init__.py files to existing projects just so they could be mypy'd.

One other use case to note, for namespace packages it's not uncommon to place the source code under src/ folder, so that setuptools.find_namespace_packages() is easier to use without having to provide explicit globs (by setting package_dir={'': 'src'}).

This way, you could have something like

src/
+-- foo/
   +-- bar.py
   +-- baz/
      +-- __init__.py
      +-- yo.py

and, ideally, one would expect things to "just work" by running

env MYPYPATH=src mypy --namespace-packages -p foo 

which would then traverse foo, foo.baz and foo.baz.yo.

[JohnHBrock]

@aldanor No updates on my end. It's not clear to me how to proceed until the questions raised by Guido above are resolved.