mypy shows type error with json.dump(s) and kwargs

[pganssle]

I have noticed that in the seemingly common situation of passing **kwargs to json.dump or json.dumps, mypy fails the type checks unless the kwargs are explicitly annotated as Dict[str, Any]. A simple example:

import json

json_kwargs = dict(indent=2)

json.dumps({}, **json_kwargs)

Which results in:

$ mypy test.py 
test.py:6: error: Argument 2 to "dumps" has incompatible type "**Dict[str, int]"; expected "bool"
test.py:6: error: Argument 2 to "dumps" has incompatible type "**Dict[str, int]"; expected "Optional[Type[JSONEncoder]]"
test.py:6: error: Argument 2 to "dumps" has incompatible type "**Dict[str, int]"; expected "Optional[Tuple[str, str]]"
test.py:6: error: Argument 2 to "dumps" has incompatible type "**Dict[str, int]"; expected "Optional[Callable[[Any], Any]]"
Found 4 errors in 1 file (checked 1 source file)

If I change the example to this:

import json

json_kwargs = dict(indent=2, separators=(",", ";"))

json.dumps({}, **json_kwargs)

The errors are the same, except the inferred type changes to **Dict[str, object]. I can work around it like so:

import json
from typing import Any, Dict

json_kwargs : Dict[str, Any] = dict(indent=2, separators=(",", ";"))

json.dumps({}, **json_kwargs)

Using Python 3.8 and mypy == 0.770 on Arch Linux. The error is still present when using the current master (commit 2a3de7b).

I have also tested this with pytype and pytype does not have the same failure.

[lulivi]

Same happens when passing **kwargs to a class constructor in Python3.7 with same version of mypy:

from typing import Dict


class A:
    """Class A."""

    def __init__(self, a: str, b: str, c: str, d: int = 2) -> None:
        """Construct A object."""
        self.a: str = a
        self.b: str = b
        self.c: str = c
        self.d: int = d


def main() -> None:
    """Run main method."""
    argument_a: str = "argument_a"
    arguments_b_c: Dict[str, str] = {"b": "argument_b", "c": "argument_c"}

    object_a: A = A(a=argument_a, **arguments_b_c)

    print(object_a)


if __name__ == "__main__":
    main()

Gives the error:

> mypy a.py
a.py:21: error: Argument 2 to "A" has incompatible type "**Dict[str, str]"; expected "int"
Found 1 error in 1 file (checked 1 source file)

Edit: Added error.

[JelleZijlstra]

I feel like there's two issues here (neither of which is directly related to json or class constructors, and neither of which is necessarily a bug):

• mypy by default infers a type like Dict[str, object] for a heterogeneous dict. This is a reasonable inference, but sometimes a TypedDict may better. To get a TypedDict type you need to explicitly declare it.
• mypy is very strict about the type of dicts passed as **kwargs to a function. This is type-safe in theory but means the type of a **kwargs dict in most practical cases has to be Dict[str, Any] or a TypedDict.

[rra]

To elaborate on the last point (and to check that I understand the semantics here), I believe that mypy is enforcing that the type of the value of the expanded dict must match every possible keyword argument to the function called. Therefore, if the function's keyword arguments take incompatible types, only a dict whose values are typed as Any (or a TypedDict where mypy knows the types of the values of specific keys) is permitted.

I was instead expecting mypy to err on the side of generosity and instead assume that if the value type of the dict matches the type of any of the keyword arguments of the function, the caller probably knows what they're doing.

I'm now not sure which approach is better. The approach I was expecting is less safe, but mypy's current approach is going to result in a lot of false positives. I suppose the workaround of typing the dict with Any values isn't too bad. Setting up a TypedDict here is a bit tedious, although obviously the most correct.

[msullivan]

Yeah, my inclination here is to say not-a-bug. Though if somebody wants to open a bug to argue for looser interpretation of kwargs typing, that would be reasonable.

[rra]

Would it be possible to mention this directly in the documentation under common issues? I've seen multiple people get confused by this behavior, and it can be quite difficult to understand what's happening because of the way the type mismatch is reported.

[JukkaL]

Added #8874 about improving the generated error messages.

[dliu-fn]

I would argue to lean towards less safe, but assuming the caller probably knows what they're doing. Ask forgiveness, not permission.

[hauntsaninja]

I think aspects of this behaviour have been changed on master, so if you have complaints, please make sure they're up to date.