Mypy 1.20 crashes on concurrent runs

[dvarrazzo]

Crash Report

Mypy 1.20 crashes on concurrent usage because parallel process fight on accessing a sqlite database, at a first glance.

Traceback

version: 1.20.0
Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/mypy/__main__.py", line 15, in console_entry
    main()
  File "mypy/main.py", line 141, in main
  File "mypy/main.py", line 225, in run_build
  File "mypy/build.py", line 388, in build
  File "mypy/build.py", line 485, in build_inner
  File "mypy/build.py", line 3664, in dispatch
  File "mypy/build.py", line 3938, in load_graph
  File "mypy/build.py", line 2431, in new_state
  File "mypy/build.py", line 1717, in find_cache_meta
  File "mypy/build.py", line 1527, in _load_ff_file
  File "mypy/metastore.py", line 200, in read
  File "mypy/metastore.py", line 187, in _query
OperationalError: database is locked

To Reproduce

A bit difficult to reproduce: it fails regularly on github.

The cause of the failure is pre-commit running 4 concurrent runs of mypy each with about 60 input files. Adding a wrapper script in this run you can see the timings of the run and the list of files passed. Trying the same on my local machine doesn't reproduce because pre-commit decides to pass 20 files per run and the issue is not triggered. I don't see an easy way to configure the batch size used by pre-commit, but it wouldn't be a reliable test anyway.

Looking at the timing of the concurrent processes in the failed run above:

pid: 3032 t0: 1775044872.4642754 started
pid: 3030 t0: 1775044872.4644167 started
pid: 3033 t0: 1775044872.4644256 started
pid: 3034 t0: 1775044872.4649644 started
pid: 3034 tf: 1775044879.1738014 success
pid: 3030 tf: 1775044882.4480267 success
pid: 3032 tf: 1775044884.7451327 failed 
pid: 3033 tf: 1775044885.9920673 success

Your Environment

• Mypy version used: 1.20
• Mypy command-line flags: mypy --pretty --follow-imports=silent --show-traceback file1 file2 ... for long lists of files and with concurrent runs.
• Mypy configuration options from mypy.ini (and other config files): here
• Python version used: 3.11.15
• Operating system and version: Ubuntu 24.04

[ilevkivskyi]

@dvarrazzo Yeah, running multiple mypy processes with shared cache in parallel is not supported (and unlikely to be supported). If you need to invoke multiple mypy processes for some reason, you should specify separate (ideally stable) --cache-dir for each.

Btw, @asottile as we are getting reports like this for some time, I was trying to understand why pre-commit runs multiple mypy processes in parallel, is this for performance reasons? Also is this the default behaviour, or users can opt-in to it? It would be great if you can clarify the situation, so we can make mypy experience better for pre-commit users. Thanks!

[dvarrazzo]

Looks like --no-sqlite-cache is a valid workaround.

[ilevkivskyi]

Looks like --no-sqlite-cache is a valid workaround.

Same issue exists without SQLite cache, we have been getting such reports for a while, see my comment above. SQLite cache simply makes it easier to hit this.

[dvarrazzo]

Something must have changed in 1.20 to make the issue more visible too then: As you can see from the previous runs in our project this has never been an issue before 1.20.

[ilevkivskyi]

Something must have changed in 1.20

We simply turned SQLite cache on by default in 1.20, see CHANGELOG.

[brianschubert]

@dvarrazzo I think a proper workaround would be setting require_serial: true in your system hook. This is what's done in the pre-commit mirror hook: https://github.com/pre-commit/mirrors-mypy/blob/8e5c80792e2ec0c87804d8ef915bf35e2caea6da/.pre-commit-hooks.yaml#L8

[dvarrazzo]

@brandtbucher I was just testing with that pre-commit option, yes. Finding different issues but probably unrelated.

Edit: passing all the file names hits #17362 because there are both .py and a .pyi files with the same name... wtaf.

[ilevkivskyi]

@brianschubert Btw do you know what is the exact semantics of require_serial is? In the sense how does pre-commit decide which mypy command(s) to run if it is on vs off?

[dvarrazzo]

Good luck with the flood of reports you'll have on this.

As far as I am concerned, disabling the sqlite cache is an adequate workaround because it restores the pre-1.20 behaviour, which maybe was brittle but mostly working. It makes the error from 100%-reprodicible to possible-but-never-seen-before and I am comfortable with dealing with occasional issues in the pre-commit hook or in the CI job.

running multiple mypy processes with shared cache in parallel is not supported (and unlikely to be supported).

Probably nobody had noticed this before and entire pyramids have been built on the assumption that, instead, mypy supports parallel run. I think you should lock your cache and only allow serialized access, which might slow down something to someone but at least would keep working.

The responsible alternative would be to write a pid in a lockfile in the cache and raise an exception as soon as a parallel run is detected.

[ilevkivskyi]

Oh I just found #13916, I guess we can continue discussion there.

[ilevkivskyi]

@dvarrazzo

Good luck with the flood of reports you'll have on this.

I really don't appreciate this tone.