test failures are really really difficult to read

[asottile]

here's an example failure I had to debug recently:

$ pip install . -qq --no-deps && python -m pytest tests -k test_emits_error_for_unresolved_managers
============================= test session starts ==============================
platform linux -- Python 3.10.12, pytest-8.3.1, pluggy-1.5.0 -- /tmp/django-stubs/venv/bin/python
cachedir: .pytest_cache
rootdir: /tmp/django-stubs
configfile: pytest.ini
plugins: mypy-plugins-3.1.2, shard-0.1.2
collected 415 items / 414 deselected / 1 selected                              
Running 1 items in this shard: tests/typecheck/managers/test_managers.yml::test_emits_error_for_unresolved_managers

tests/typecheck/managers/test_managers.yml::test_emits_error_for_unresolved_managers FAILED

=================================== FAILURES ===================================
___________________ test_emits_error_for_unresolved_managers ___________________
/tmp/django-stubs/tests/typecheck/managers/test_managers.yml:495: 
E   pytest_mypy_plugins.utils.TypecheckAssertionError: Invalid output: 
E   Actual:
E     ...
E     myapp/models:36: note: Revealed type is "django.db.models.manager.Manager[myapp.models.User]" (diff)
E     myapp/models:37: note: Revealed type is "django.db.models.manager.Manager[myapp.models.User]" (diff)
E     myapp/models:39: note: Revealed type is "myapp.models.UnknownManager[myapp.models.Booking]" (diff)
E     myapp/models:40: note: Revealed type is "django.db.models.manager.Manager[myapp.models.Booking]" (diff)
E     myapp/models:42: note: Revealed type is "myapp.models.UnknownManager[myapp.models.TwoUnresolvable]" (diff)
E     myapp/models:43: note: Revealed type is "myapp.models.UnknownManager[myapp.models.TwoUnresolvable]" (diff)
E     myapp/models:44: note: Revealed type is "django.db.models.manager.Manager[myapp.models.TwoUnresolvable]" (diff)
E     myapp/models:46: note: Revealed type is "myapp.models.UnknownManager[myapp.models.AbstractUnresolvable]" (diff)
E     myapp/models:47: note: Revealed type is "django.db.models.manager.Manager[myapp.models.InvisibleUnresolvable]" (diff)
E     myapp/models:49: note: Revealed type is "django.db.models.fields.related_descriptors.RelatedManager[myapp.models.Booking]" (diff)
E     myapp/models:50: note: Revealed type is "django.db.models.fields.related_descriptors.RelatedManager[myapp.models.Booking]" (diff)
E     myapp/models:53: note: Revealed type is "def () -> myapp.models.UnknownQuerySet[myapp.models.Booking, myapp.models.Booking]" (diff)
E     myapp/models:54: note: Revealed type is "Any" (diff)
E     myapp/models:55: note: Revealed type is "def (*args: Any, **kwargs: Any) -> myapp.models.UnknownQuerySet[myapp.models.Booking, myapp.models.Booking]" (diff)
E     myapp/models:56: note: Revealed type is "Any" (diff)
E     myapp/models:57: note: Revealed type is "Union[myapp.models.Booking, None]" (diff)
E     myapp/models:58: note: Revealed type is "myapp.models.Booking" (diff)
E     myapp/models:59: note: Revealed type is "builtins.list[myapp.models.Booking]" (diff)
E     myapp/models:60: note: Revealed type is "builtins.list[myapp.models.Booking]" (diff)
E     myapp/models:64: note: Revealed type is "def () -> django.db.models.query.QuerySet[myapp.models.Booking, myapp.models.Booking]" (diff)
E     myapp/models:65: error: "RelatedManager[Booking]" has no attribute "custom"  [attr-defined] (diff)
E     myapp/models:65: note: Revealed type is "Any" (diff)
E     myapp/models:66: note: Revealed type is "def (*args: Any, **kwargs: Any) -> django.db.models.query.QuerySet[myapp.models.Booking, myapp.models.Booking]" (diff)
E     myapp/models:67: error: "QuerySet[Booking, Booking]" has no attribute "custom"  [attr-defined] (diff)
E     myapp/models:67: note: Revealed type is "Any" (diff)
E     myapp/models:68: note: Revealed type is "Union[myapp.models.Booking, None]" (diff)
E   Expected:
E     ...
E     myapp/models:32: error: Could not resolve manager type for "myapp.models.InvisibleUnresolvable.objects"  [django-manager-missing] (diff)
E     myapp/models:36: note: Revealed type is "django.db.models.manager.Manager[myapp.models.User]" (diff)
E     myapp/models:37: note: Revealed type is "django.db.models.manager.Manager[myapp.models.User]" (diff)
E     myapp/models:39: note: Revealed type is "myapp.models.UnknownManager[myapp.models.Booking]" (diff)
E     myapp/models:40: note: Revealed type is "django.db.models.manager.Manager[myapp.models.Booking]" (diff)
E     myapp/models:42: note: Revealed type is "myapp.models.UnknownManager[myapp.models.TwoUnresolvable]" (diff)
E     myapp/models:43: note: Revealed type is "myapp.models.UnknownManager[myapp.models.TwoUnresolvable]" (diff)
E     myapp/models:44: note: Revealed type is "django.db.models.manager.Manager[myapp.models.TwoUnresolvable]" (diff)
E     myapp/models:46: note: Revealed type is "myapp.models.UnknownManager[myapp.models.InvisibleUnresolvable]" (diff)
E     myapp/models:47: note: Revealed type is "django.db.models.manager.Manager[myapp.models.InvisibleUnresolvable]" (diff)
E     myapp/models:49: note: Revealed type is "django.db.models.fields.related_descriptors.RelatedManager[myapp.models.Booking]" (diff)
E     myapp/models:50: note: Revealed type is "django.db.models.fields.related_descriptors.RelatedManager[myapp.models.Booking]" (diff)
E     myapp/models:53: note: Revealed type is "def () -> myapp.models.UnknownQuerySet[myapp.models.Booking, myapp.models.Booking]" (diff)
E     myapp/models:54: note: Revealed type is "Any" (diff)
E     myapp/models:55: note: Revealed type is "def (*args: Any, **kwargs: Any) -> myapp.models.UnknownQuerySet[myapp.models.Booking, myapp.models.Booking]" (diff)
E     myapp/models:56: note: Revealed type is "Any" (diff)
E     myapp/models:57: note: Revealed type is "Union[myapp.models.Booking, None]" (diff)
E     myapp/models:58: note: Revealed type is "myapp.models.Booking" (diff)
E     myapp/models:59: note: Revealed type is "builtins.list[myapp.models.Booking]" (diff)
E     myapp/models:60: note: Revealed type is "builtins.list[myapp.models.Booking]" (diff)
E     myapp/models:64: note: Revealed type is "def () -> django.db.models.query.QuerySet[myapp.models.Booking, myapp.models.Booking]" (diff)
E     myapp/models:65: error: "RelatedManager[Booking]" has no attribute "custom"  [attr-defined] (diff)
E     myapp/models:65: note: Revealed type is "Any" (diff)
E     myapp/models:66: note: Revealed type is "def (*args: Any, **kwargs: Any) -> django.db.models.query.QuerySet[myapp.models.Booking, myapp.models.Booking]" (diff)
E     myapp/models:67: error: "QuerySet[Booking, Booking]" has no attribute "custom"  [attr-defined] (diff)
E     myapp/models:67: note: Revealed type is "Any" (diff)
E     myapp/models:68: note: Revealed type is "Union[myapp.models.Booking, None]" (diff)
E   Alignment of first line difference:
E     E: myapp/models:32: error: Could not resolve manager type for "myapp.models...
E     A: myapp/models:36: note: Revealed type is "django.db.models.manager.Manage...
E                      ^
=========================== short test summary info ============================
FAILED tests/typecheck/managers/test_managers.yml::test_emits_error_for_unresolved_managers - 
====================== 1 failed, 414 deselected in 3.89s =======================

the "Alignment of first line difference" part is "useful" at least, but every line is marked as (diff) -- I ended up copying Actual and Expected to a file and using diff -u:

$ diff -u expected actual
--- expected	2024-07-25 15:31:06.184912437 -0400
+++ actual	2024-07-25 15:31:08.888821824 -0400
@@ -1,4 +1,3 @@
-E     myapp/models:32: error: Could not resolve manager type for "myapp.models.InvisibleUnresolvable.objects"  [django-manager-missing] (diff)
 E     myapp/models:36: note: Revealed type is "django.db.models.manager.Manager[myapp.models.User]" (diff)
 E     myapp/models:37: note: Revealed type is "django.db.models.manager.Manager[myapp.models.User]" (diff)
 E     myapp/models:39: note: Revealed type is "myapp.models.UnknownManager[myapp.models.Booking]" (diff)
@@ -6,7 +5,7 @@
 E     myapp/models:42: note: Revealed type is "myapp.models.UnknownManager[myapp.models.TwoUnresolvable]" (diff)
 E     myapp/models:43: note: Revealed type is "myapp.models.UnknownManager[myapp.models.TwoUnresolvable]" (diff)
 E     myapp/models:44: note: Revealed type is "django.db.models.manager.Manager[myapp.models.TwoUnresolvable]" (diff)
-E     myapp/models:46: note: Revealed type is "myapp.models.UnknownManager[myapp.models.InvisibleUnresolvable]" (diff)
+E     myapp/models:46: note: Revealed type is "myapp.models.UnknownManager[myapp.models.AbstractUnresolvable]" (diff)
 E     myapp/models:47: note: Revealed type is "django.db.models.manager.Manager[myapp.models.InvisibleUnresolvable]" (diff)
 E     myapp/models:49: note: Revealed type is "django.db.models.fields.related_descriptors.RelatedManager[myapp.models.Booking]" (diff)
 E     myapp/models:50: note: Revealed type is "django.db.models.fields.related_descriptors.RelatedManager[myapp.models.Booking]" (diff)

much much more readable! and at least as a programmer I'm used to looking at unified diffs and would have been able to spot the difference much quicker!

[sobolevn]

Yes, this is really hard. It took me several minutes just to review one line change (I had to count lines separately, because lines in IDE do not match lines in main.py example).

I am open to suggestions.