Direct assignment from lambda doesn't narrow type correctly

[jacksonriley]

(Might be a duplicate of #10993)

Bug Report

Assigning a lambda function to a variable typed as e.g. Callable[[], int] doesn't always work if the lambda returns a variable whose type has been previously narrowed to e.g. int. This behaviour does not remain if the lambda returns a literal e.g. int, or if an intermediate variable is introduced.
To Reproduce

from collections.abc import Callable
from typing import Union

def foo(bar: Union[int, Callable[[], int]]) -> int:
    int_fn: Callable[[], int]
    if callable(bar):
        int_fn = bar
    else:
        reveal_type(bar)
        int_fn = lambda: bar
    return int_fn()

Expected Behavior

There should be no mypy errors.

Actual Behavior

There were mypy errors (I've left the reveal_type in for illustration):

test.py:9: note: Revealed type is "builtins.int"
test.py:10: error: Incompatible types in assignment (expression has type "Callable[[], Union[int, Callable[[], int]]]", variable has type "Callable[[], int]")
test.py:10: error: Incompatible return value type (got "Union[int, Callable[[], int]]", expected "int")

These errors do not remain if line 10 is replaced with e.g. int_fn = lambda: 1.
In addition, these errors do not remain if line 10 is replaced with an intermediate assignment, e.g.:

    intermediate = lambda: bar
    int_fn = intermediate

Your Environment

• Python 3.9.4
• mypy 0.910
• Mypy command-line flags: None
• Mypy configuration options from mypy.ini (and other config files): None
• Operating system and version: Linux, RHEL 7.8

[erictraut]

In general, it's not safe to apply narrowed types to variables that are captured and used in an independent execution scope because the execution time of that execution scope is not known. Consider, for example, if you made the following change to your sample.

def foo(bar: Union[int, Callable[[], int]]) -> int:
    int_fn: Callable[[], int]
    if callable(bar):
        int_fn = bar
    else:
        reveal_type(bar)
        int_fn = lambda: bar
        bar = lambda: 3 # Reassign bar here
    return int_fn()

The recommended workaround in this case is to assign the narrowed type to a local variable, which can then be captured by the lambda.

def foo(bar: Union[int, Callable[[], int]]) -> int:
    int_fn: Callable[[], int]
    if callable(bar):
        int_fn = bar
    else:
        x = bar
        int_fn = lambda: x
    return int_fn()

[jacksonriley]

Ah thank you Eric, that's very interesting! Sounds like this isn't a bug in that case - I'll close the issue :)

[jacksonriley]

Apologies @erictraut, I thought that I understood why the workaround that I described in my original report works, but I'm now convinced that it actually is a bug.

Consider this example

from collections.abc import Callable
from typing import Union

def foo(bar: Union[int, Callable[[], int]]) -> int:
    int_fn: Callable[[], int]
    if callable(bar):
        int_fn = bar
    else:
        intermediate = lambda: bar
        int_fn = intermediate
        bar = lambda: 3 # Reassign bar here
    return int_fn()

print(foo(2))

Mypy doesn't complain about this code, but foo(2) evaluates to e.g. <function foo.<locals>.<lambda> at 0x7f53aa510310>.
Therefore, a function that claims to return an int (and is passed by mypy) does not do so.

This does seem like a bug?

[erictraut]

Yes, I agree that's a bug, but this appears to be different from the one you originally reported. In your original example, mypy was not applying narrowing in a situation where it should not, so it was doing the right thing. In this case, it is applying narrowing in a situation where it should not — namely, when inferring the type of intermediate.

For comparison, mypy infers the type of intermediate to be () -> int, but pyright infers the type of intermediate to be () -> (int | () -> int). Pyright therefore emits an error when attempting to assign intermediate to int_fn. I think pyright is doing the right thing here, and mypy is not. As your example points out, this code is not type safe.

[jacksonriley]

Yes I agree that this is different from what I originally reported - apologies. Do you reckon I should close this issue and raise a new one?

[hauntsaninja]

The original (not actually a bug) issue is a duplicate of #2608. The actually a bug issue is a duplicate of #10993