Support recursive types

[o11c]

The following in particular would be useful:

Callback = Callable[[str], 'Callback']
Foo = Union[str, List['Foo']]

[JukkaL]

If (or when) mypy will have structural subtyping, recursive types would also be useful there.

[JukkaL]

My current plan is to postpone recursive types until simple structural subtyping is in and reconsider them later. After thinking about them more they are going to increase complexity a lot of I haven't seen much evidence for them being needed that often.

[oconnor663]

If I'm understanding this issue right, I'm running into it for classes that want a fluent interface. So for example, if I want callers to do something like this:

myfoo.add(bar).add(baz).finish()

Then the definition of the Foo class and the add method need to look something like this:

class Foo:
    def add(self, x) -> Foo:  # Python chokes on this line!
        # do stuff
        return self

Another place where Python commonly does return self is in the __enter__ method for context managers. Is mypy able to typecheck those right now?

[refi64]

@oconnor663 Try:

class Foo:
    def add(self, x) -> 'Foo':
        # do stuff
        return self

[oconnor663]

Ah, thank you.

[oconnor663]

@kirbyfan64, do you know if there are standard functions anywhere that understand this convention? Like, if I wanted to introspect a couple functions and compare the types of their arguments, should I handle the Foo == "Foo" case explicitly? That seems doable, but a string-aware version of say isinstance seems harder.

[refi64]

@oconnor663 I don't think there's anything like that. If you're introspecting the functions via a decorator, you could try accessing the caller's globals and locals.

[gvanrossum]

You're aware of typing.get_type_hints(obj)right? It is similar to obj.annotations` but expands forward references.
https://docs.python.org/3/library/typing.html?highlight=typing#typing.get_type_hints

There used to be an instance() implementation in typing.py but Mark Shannon
made me take it out. It's being deleted in this rev:
python/typing@ac7494f

On Fri, Oct 16, 2015 at 9:37 AM, Ryan Gonzalez notifications@github.com
wrote:

@oconnor663 https://github.com/oconnor663 I don't think there's
anything like that. If you're introspecting the functions via a decorator,
you could try accessing the caller's globals and locals.

—
Reply to this email directly or view it on GitHub
#731 (comment).

--Guido van Rossum (python.org/~guido)

[oconnor663]

@gvanrossum that's exactly what I was looking for, thanks. Sorry for the n00b questions today, but awesome that all this is supported.

[JukkaL]

Mypy should detect missing string literal escapes (see #948).

[dmoisset]

Going back to the original point on the issue, I found a case in the stdlib where this would be needed; the type for isinstance() is currently:

def isinstance(o: object, t: Union[type, Tuple[type, ...]]) -> bool: ...

but it should actually be:

ClassInfo = Union[type, Tuple['ClassInfo', ...]]
def isinstance(o: object, t: ClassInfo) -> bool: ...

Because according to https://docs.python.org/3/library/functions.html#isinstance the tuples can be nested. I found an actual example of this while typechecking django.http.response.HttpResponse.content

[srittau]

I have come across this while trying to define a generic JSON type:

JSON = Union[Dict[str, "JSON"], List["JSON"], str, int, float, bool, None]

So consider this a +1 for supporting this use case.

[graingert]

@srittau JSON needs to be Any because it is recursive and you can give json.loads a custom JSONEncoder class:

_PlainJSON = Union[Dict[str, "_PlainJSON"], List["_PlainJSON"], str, int, float, bool, None]
_T = TypeVar('_T')
JSON = Union[_PlainJSON, _T, Dict[str, "JSON"], List["JSON"]]
def loads(data: str, cls: Type[JSONEncoder[_T]]) -> JSON: ...

of course recursive types and Type[JSONEncoder[_T]] types arn't supported.

[DustinWehr]

The following pattern seems to be good enough for my purposes. The boilerplate is tolerable for me.

class BTree(NamedTuple):
    val: int
    left_ : Any
    right_ : Any

    # typed constructor
    @staticmethod
    def c(val: int, left: Optional['BTree'] = None, right: Optional['BTree'] = None) -> 'BTree':
        return BTree(val, left, right)

    # typed accessors
    @property
    def left(self) -> Optional['BTree']:
        return cast(Optional[BTree], self.left_)
    @property
    def right(self) -> Optional['BTree']:
        return cast(Optional[BTree], self.right_)

atree = BTree.c(1, BTree.c(2, BTree.c(3), BTree.c(4)), BTree.c(5))
atree2 = BTree.c(1, BTree.c(2, BTree.c(3), BTree.c(4)), BTree.c(5))
assert atree == atree2
assert isinstance(atree,BTree) and isinstance(atree.left,BTree) and isinstance(atree.left.left,BTree)

[DustinWehr]

Latest version of pattern. We use this example at Legalese for interacting with SMT solvers (the SMTLIB language).

I found that I ended up forgetting to use the typed .c static method instead of the untyped constructor in the BTree example above. This version addresses that. It's only very minor deficits are:

1. Boilerplate (which I don't care about if the datatype is significant enough for us to care about the difference between an immutable recursive datatype and Tuple[Any,...])
2. The constructor SMTExprNonatom and the type SMTExprNonatom_ do not share the same name. But this is only aesthetic; You won't use one when you mean the other, since with this naming convention, SMTExprNonatom will come up first in autocomplete, and only SMTExprNonatom_ can be used in a type position.

# Immutable recursive datatypes pattern
SMTAtom = Union[str, int, float, bool]
SMTExpr = Union['SMTExprNonatom_',SMTAtom]
class SMTExprNonatom_(NamedTuple):  
    symb: str
    args_: Tuple[Any,...] # see `args` below
    @staticmethod  # typed constructor, which we alias to `SMTExprNonatom` after the class def
    def c(symb:str, args:Iterable[SMTExpr]) -> 'SMTExprNonatom_': return SMTExprNonatom_(symb, tuple(args))
    @property # typed accessor
    def args(self) -> Tuple[SMTExpr]: return cast(Tuple[SMTExpr], self.args_)
SMTExprNonatom = SMTExprNonatom_.c
SMTCommand = NewType('SMTCommand', SMTExprNonatom_)

[bartfeenstra]

It appears that v0.990/v0.991 may not have addressed the entirety of the challenge that is recursive/cyclic types: #14219

[jamesbraza]

For those reading in posterity:

1. mypy==0.981 added --enable-recursive-alias: Enable recursive type aliases behind a flag #13297
2. mypy==0.990 started a deprecation cycle for --enable-recursive-alias: Temporarily put back --enable-recursive-aliases (as depreceated) #13852
3. mypy==1.7.0 completed the deprecation cycle, removing --enable-recursive-alias: Delete recursive aliases flags #16346

So with mypy>=1.7, recursive type support is built into mypy ✅

[jorenham]

It doesn't work in this case:

from collections.abc import Sequence
from typing import TypeAlias

FloatVector: TypeAlias = Sequence[float]
FloatTensor: TypeAlias = FloatVector | Sequence["FloatTensor"]

rejected_float_vector: FloatVector = "ok"       # OK: rejected

accepted_float_tensor: FloatTensor = [[[3.14]]] # OK: accepted
rejected_float_vector1: FloatTensor = object()  # OK: rejected
rejected_float_vector2: FloatTensor = "fail"    # FAIL: accepted

https://mypy-play.net/?mypy=latest&python=3.12&flags=strict&gist=da0434e6062956cfd667a9e8c8bd3ff9

[brianschubert]

@jorenham Thanks for the report! Can you please open a new issue?

[randolf-scholz]

@jorenham Beware that this TypeAlias actually allows mixed lists, like [[1.2, 3.4], [[1.0, 0.0], [0.0, 1.0]]]. You might want to consider a non-recursive alias like S[float] | S[S[float]] | S[S[S[float]]] | S[S[S[S[float]]]] for a sufficient number of levels instead. The infinite union you probably want is currently not representable in the type system, I think.

EDIT: we would need inductive types. If typing.Map is introduced
(python/typing#1216, python/typing#1383, pre-pep discussion), then we should be able to express this as

Tensor = Sequence[float] | Map[Sequence, Tensor]

[jorenham]

You might want to consider a non-recursive alias like S[float] | S[S[float]] | S[S[S[float]]] | S[S[S[S[float]]]] for a sufficient number of levels instead.

In numpy the current maximum number of dimensions is 64, so that would indeed be possible. But in order to keep error messages from getting to long, and the type-checking from becoming to slow, I decided to go for this concise yet "broader" approach instead.

[jorenham]

@jorenham Thanks for the report! Can you please open a new issue?

🫡 #18184

[jorenham]

@jorenham Beware that this TypeAlias actually allows mixed lists, like [[1.2, 3.4], [[1.0, 0.0], [0.0, 1.0]]].

This made me realize that some of the numpy array-like type aliases in optype (that are heavily used in scipy-stubs) could be improved. Specifically, this affected the optype.numpy.To{Bool,JustInt,Int,Float,Complex,Array}{2,3}D type aliases (docs), which also accepted the mixed lists that you mentioned, @randolf-scholz. So now this has been fixed in optype 0.7.2 👌🏻.