Confusing behavior when subclassing a generic type

[marienz]

I'm trying to use mypy with Urwid, and ran into something I can't quite get to work. I'm not sure if this is a bug, a missing feature, or me expecting too much magic.

Urwid provides a hierarchy of Widget classes. One of them is WidgetWrap: it wraps a widget, exposes the wrapped widget as self._w, and forwards the methods of the Widget base class to the wrapped widget. It is intended to be subclassed, with the subclass methods accessing the wrapped widget but keeping it hidden from other code.

I'm writing an application that uses Urwid, so I can't change Urwid itself.

With an urwid.pyi of:

from typing import TypeVar, Generic, Any

class Widget: ...

class Text(Widget):
    def __init__(self, markup: Any) -> None: ...
    def set_text(self, text: str) -> None: ...

W = TypeVar('W', bound=Widget)

class WidgetWrap(Widget, Generic[W]):
    def __init__(self, w: W) -> None: ...
    _w: W

and some example code:

import typing
import urwid

class WrappedText(urwid.WidgetWrap):

    def __init__(self):
        # Using super() behaves the same way.
        urwid.WidgetWrap.__init__(self, urwid.Text(''))

    def update(self):
        self._w.set_text('Ni!')
        self._w.no_such_method()


raw_wrap = urwid.WidgetWrap(urwid.Text(''))

if typing.TYPE_CHECKING:
    reveal_type(raw_wrap)
    reveal_type(raw_wrap._w)

raw_wrap._w.set_text('This is ok.')
try:
    raw_wrap._w.no_such_method()
except AttributeError:
    pass

class_wrap = WrappedText()

if typing.TYPE_CHECKING:
    reveal_type(class_wrap)
    reveal_type(class_wrap._w)

class_wrap.update()

I want mypy to catch the two calls to no_such_method.

Mypy 0.540 catches the first one, but not the second one:

wid.py:19: error: Revealed type is 'urwid.WidgetWrap[urwid.Text*]'
wid.py:20: error: Revealed type is 'urwid.Text*'
wid.py:24: error: "Text" has no attribute "no_such_method"
wid.py:31: error: Revealed type is 'wid.WrappedText'
wid.py:32: error: Revealed type is 'Any'

So when instantiating WidgetWrap directly, mypy correctly infers the type of _w, and catches the call to a nonexistent method. But when subclassing WidgetWrap, the type for _w is Any (this surprises me: I was at least expecting the bound of Widget to apply), so it cannot detect the subclass doing something wrong.

I tried to work around this by explicit use of Generic in my code, not just in the .pyi stub:

class WidgetWrap(urwid.WidgetWrap, Generic[W]):
    def __init__(self, widget: W) -> None:
        super().__init__(widget)
    _w: W

class WrappedText(WidgetWrap[urwid.Text]):
    # as before

This works great with mypy, but runs afoul of python/typing#449 (Urwid uses metaclasses internally, so mixing in Generic triggers a metaclass conflict).

I found a workaround for that, but it's a bit ugly:

if TYPE_CHECKING:
    class WidgetWrap(urwid.WidgetWrap, Generic[W]):
        # as in previous code snippet
else:
    WidgetWrap = collections.defaultdict(lambda: urwid.WidgetWrap)

class WrappedText(WidgetWrap[urwid.Text]):

This works at runtime and seems to work with mypy.

Is there a neat way of dealing with this class I'm overlooking? And is the different behaviour from mypy when a subclass passes fixed arguments to the superclass __init__ versus directly instantiating the class expected?

[gvanrossum]

I think that WrappedText, by inheriting from WidgetWrap, really inherits from WidgetWrap[Any], and that suppresses all errors. Can you inherit from WidgetWrap[Text] instead?

[ilevkivskyi]

@gvanrossum

I think that WrappedText, by inheriting from WidgetWrap, really inherits from WidgetWrap[Any], and that suppresses all errors. Can you inherit from WidgetWrap[Text] instead?

IIUC the problem is that the actual implementation is not generic (because of the metaclass conflict).
WidgetWrap is only generic in the stub .pyi file, therefore WidgetWrap[Text] will fail at runtime.

@marienz There are two possible solutions, one is the one you found with TYPE_CHECKING, you can play with it more to make it more beautiful, for example:

if TYPE_CHECKING:
    class _WrappedText(urwid.WidgetWrap[urwid.Text]):
        pass
else:
    _WrappedText = urwid.WidgetWrap

class WrappedText(_WrappedText):
    # actual implementation
    ...

The second option is to somehow play with metaclasses:

from typing import GenericMeta

class AdvWrapperMeta(GenericMeta, WrapperMeta):
    pass

and then use explicit metaclass=AdvWrapperMeta where needed. I hope this is a temporary workaround, and will be fixed in Python 3.7.

@gvanrossum I am going to post the updated version of PEP 560 (implementing suggestions form python-ideas) very soon. Everyone seems to be supportive, so I hope that PEP 560 can be accepted and merged soon, and we can move forward with corresponding changes to typing.

[gvanrossum]

I am supportive of PEP 560 too!

[marienz]

Thanks! I'll stick to a variant of the TYPE_CHECKING-conditional code for now, as I do not really want to ensure the two metaclasses play nice together. Looks like PEP 560 is the way forward.