0.3.3: pytest fails in tests/test_syntactic.py::test_black_autoformatter_from_grammar unit

[kloczek]

I'm packaging your module as an rpm package so I'm using the typical PEP517 based build, install and test cycle used on building packages from non-root account.

• python3 -sBm build -w --no-isolation
• because I'm calling build with --no-isolation I'm using during all processes only locally installed modules
• install .whl file in </install/prefix> using installer module
• run pytest with $PYTHONPATH pointing to sitearch and sitelib inside </install/prefix>
• build is performed in env which is cut off from access to the public network (pytest is executed with -m "not network")

Here is pytest output:

+ PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-hypothesmith-0.3.3-2.fc36.x86_64/usr/lib64/python3.9/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-hypothesmith-0.3.3-2.fc36.x86_64/usr/lib/python3.9/site-packages
+ /usr/bin/pytest -ra -m 'not network'
============================= test session starts ==============================
platform linux -- Python 3.9.18, pytest-8.1.1, pluggy-1.4.0
rootdir: /home/tkloczko/rpmbuild/BUILD/hypothesmith-0.3.3
configfile: tox.ini
plugins: hypothesis-6.99.6
collected 181 items

tests/test_cst.py ....s................................................. [ 29%]
...................................F....F..........................s.... [ 69%]
............................F............x...                            [ 94%]
tests/test_syntactic.py xx.....                                          [ 98%]
tests/test_version.py ...                                                [100%]

=================================== FAILURES ===================================
________________ test_source_code_from_libcst_node_type[Match] _________________
tests/test_cst.py:25: in test_source_code_from_libcst_node_type
    @given(data=st.data())
E   hypothesis.errors.Unsatisfiable: Unable to satisfy assumptions of test_source_code_from_libcst_node_type
---------------------------------- Hypothesis ----------------------------------
You can add @seed(41049388112553499489934747796162445915) to this test or run pytest with --hypothesis-seed=41049388112553499489934747796162445915 to reproduce this failure.
______________ test_source_code_from_libcst_node_type[MatchList] _______________
tests/test_cst.py:25: in test_source_code_from_libcst_node_type
    @given(data=st.data())
E   hypothesis.errors.Unsatisfiable: Unable to satisfy assumptions of test_source_code_from_libcst_node_type
---------------------------------- Hypothesis ----------------------------------
You can add @seed(189423066481151525614644471526471958627) to this test or run pytest with --hypothesis-seed=189423066481151525614644471526471958627 to reproduce this failure.
_______________ test_source_code_from_libcst_node_type[TryStar] ________________
tests/test_cst.py:25: in test_source_code_from_libcst_node_type
    @given(data=st.data())
E   hypothesis.errors.Unsatisfiable: Unable to satisfy assumptions of test_source_code_from_libcst_node_type
---------------------------------- Hypothesis ----------------------------------
You can add @seed(94447068965149540313353865934997293123) to this test or run pytest with --hypothesis-seed=94447068965149540313353865934997293123 to reproduce this failure.
================================== XFAILURES ===================================
_____________________ test_black_autoformatter_from_nodes ______________________
/usr/lib/python3.9/site-packages/black/__init__.py:1526: in assert_equivalent
    src_ast = parse_ast(src)
/usr/lib/python3.9/site-packages/black/parsing.py:148: in parse_ast
    raise SyntaxError(first_error)
E   SyntaxError: invalid character '▒' (U+2592) (<unknown>, line 1)

The above exception was the direct cause of the following exception:
tests/test_cst.py:57: in test_black_autoformatter_from_nodes
    @example("A\u2592", black.Mode())
/usr/lib/python3.9/site-packages/hypothesis/core.py:1277: in _raise_to_user
    raise the_error_hypothesis_found
tests/test_cst.py:69: in test_black_autoformatter_from_nodes
    result = black.format_file_contents(source_code, fast=False, mode=mode)
/usr/lib/python3.9/site-packages/black/__init__.py:1083: in format_file_contents
    check_stability_and_equivalence(
/usr/lib/python3.9/site-packages/black/__init__.py:1057: in check_stability_and_equivalence
    assert_equivalent(src_contents, dst_contents)
/usr/lib/python3.9/site-packages/black/__init__.py:1528: in assert_equivalent
    raise ASTSafetyError(
E   black.parsing.ASTSafetyError: cannot use --safe with this file; failed to parse source file AST: invalid character '▒' (U+2592) (<unknown>, line 1)
E   This could be caused by running Black with an older Python version that does not support new syntax used in your source file.
E   Falsifying explicit example: test_black_autoformatter_from_nodes(
E       source_code='A▒',
E       mode=Mode(target_versions=set(), line_length=88, string_normalization=True, is_pyi=False, is_ipynb=False, skip_source_first_line=False, magic_trailing_comma=True, python_cell_magics=set(), preview=False, unstable=False, enabled_features=set()),
E   )
________________________ test_tokenize_round_trip_bytes ________________________
  + Exception Group Traceback (most recent call last):
  |   File "/usr/lib/python3.9/site-packages/_pytest/runner.py", line 340, in from_call
  |     result: Optional[TResult] = func()
  |   File "/usr/lib/python3.9/site-packages/_pytest/runner.py", line 240, in <lambda>
  |     lambda: runtest_hook(item=item, **kwds), when=when, reraise=reraise
  |   File "/usr/lib/python3.9/site-packages/pluggy/_hooks.py", line 501, in __call__
  |     return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
  |   File "/usr/lib/python3.9/site-packages/pluggy/_manager.py", line 119, in _hookexec
  |     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  |   File "/usr/lib/python3.9/site-packages/pluggy/_callers.py", line 181, in _multicall
  |     return outcome.get_result()
  |   File "/usr/lib/python3.9/site-packages/pluggy/_result.py", line 99, in get_result
  |     raise exc.with_traceback(exc.__traceback__)
  |   File "/usr/lib/python3.9/site-packages/pluggy/_callers.py", line 166, in _multicall
  |     teardown.throw(outcome._exception)
  |   File "/usr/lib/python3.9/site-packages/_pytest/threadexception.py", line 87, in pytest_runtest_call
  |     yield from thread_exception_runtest_hook()
  |   File "/usr/lib/python3.9/site-packages/_pytest/threadexception.py", line 63, in thread_exception_runtest_hook
  |     yield
  |   File "/usr/lib/python3.9/site-packages/pluggy/_callers.py", line 166, in _multicall
  |     teardown.throw(outcome._exception)
  |   File "/usr/lib/python3.9/site-packages/_pytest/unraisableexception.py", line 90, in pytest_runtest_call
  |     yield from unraisable_exception_runtest_hook()
  |   File "/usr/lib/python3.9/site-packages/_pytest/unraisableexception.py", line 65, in unraisable_exception_runtest_hook
  |     yield
  |   File "/usr/lib/python3.9/site-packages/pluggy/_callers.py", line 166, in _multicall
  |     teardown.throw(outcome._exception)
  |   File "/usr/lib/python3.9/site-packages/_pytest/logging.py", line 849, in pytest_runtest_call
  |     yield from self._runtest_for(item, "call")
  |   File "/usr/lib/python3.9/site-packages/_pytest/logging.py", line 832, in _runtest_for
  |     yield
  |   File "/usr/lib/python3.9/site-packages/pluggy/_callers.py", line 166, in _multicall
  |     teardown.throw(outcome._exception)
  |   File "/usr/lib/python3.9/site-packages/_pytest/capture.py", line 883, in pytest_runtest_call
  |     return (yield)
  |   File "/usr/lib/python3.9/site-packages/pluggy/_callers.py", line 166, in _multicall
  |     teardown.throw(outcome._exception)
  |   File "/usr/lib/python3.9/site-packages/_pytest/skipping.py", line 256, in pytest_runtest_call
  |     return (yield)
  |   File "/usr/lib/python3.9/site-packages/pluggy/_callers.py", line 102, in _multicall
  |     res = hook_impl.function(*args)
  |   File "/usr/lib/python3.9/site-packages/_pytest/runner.py", line 182, in pytest_runtest_call
  |     raise e
  |   File "/usr/lib/python3.9/site-packages/_pytest/runner.py", line 172, in pytest_runtest_call
  |     item.runtest()
  |   File "/usr/lib/python3.9/site-packages/_pytest/python.py", line 1777, in runtest
  |     self.ihook.pytest_pyfunc_call(pyfuncitem=self)
  |   File "/usr/lib/python3.9/site-packages/pluggy/_hooks.py", line 501, in __call__
  |     return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
  |   File "/usr/lib/python3.9/site-packages/pluggy/_manager.py", line 119, in _hookexec
  |     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  |   File "/usr/lib/python3.9/site-packages/pluggy/_callers.py", line 138, in _multicall
  |     raise exception.with_traceback(exception.__traceback__)
  |   File "/usr/lib/python3.9/site-packages/pluggy/_callers.py", line 102, in _multicall
  |     res = hook_impl.function(*args)
  |   File "/usr/lib/python3.9/site-packages/_pytest/python.py", line 200, in pytest_pyfunc_call
  |     result = testfunction(**testargs)
  |   File "/home/tkloczko/rpmbuild/BUILD/hypothesmith-0.3.3/tests/test_syntactic.py", line 23, in test_tokenize_round_trip_bytes
  |     @example("#")
  |   File "/usr/lib/python3.9/site-packages/hypothesis/core.py", line 1573, in wrapped_test
  |     _raise_to_user(errors, state.settings, [], " in explicit examples")
  |   File "/usr/lib/python3.9/site-packages/hypothesis/core.py", line 1277, in _raise_to_user
  |     raise the_error_hypothesis_found
  | exceptiongroup.ExceptionGroup: Hypothesis found 2 distinct failures in explicit examples. (2 sub-exceptions)
  +-+---------------- 1 ----------------
    | Traceback (most recent call last):
    |   File "/home/tkloczko/rpmbuild/BUILD/hypothesmith-0.3.3/tests/test_syntactic.py", line 32, in test_tokenize_round_trip_bytes
    |     tokens = list(tokenize.tokenize(io.BytesIO(source).readline))
    |   File "/usr/lib64/python3.9/tokenize.py", line 521, in _tokenize
    |     raise TokenError("EOF in multi-line statement", (lnum, 0))
    | tokenize.TokenError: ('EOF in multi-line statement', (3, 0))
    | Falsifying explicit example: test_tokenize_round_trip_bytes(
    |     source_code='\n\\\n',
    | )
    +---------------- 2 ----------------
    | Traceback (most recent call last):
    |   File "/home/tkloczko/rpmbuild/BUILD/hypothesmith-0.3.3/tests/test_syntactic.py", line 35, in test_tokenize_round_trip_bytes
    |     assert [(t.type, t.string) for t in tokens] == [(t.type, t.string) for t in output]
    | AssertionError: assert [(62, 'utf-8'...4, '\n'), ...] == [(62, 'utf-8'...60, '#'), ...]
    |
    |   At index 3 diff: (1, 'pass') != (5, ' ')
    |   Right contains 2 more items, first extra item: (6, '')
    |   Use -v to get more diff
    | Falsifying explicit example: test_tokenize_round_trip_bytes(
    |     source_code='#\n\x0cpass#\n',
    | )
    +------------------------------------
_______________________ test_tokenize_round_trip_string ________________________
  + Exception Group Traceback (most recent call last):
  |   File "/usr/lib/python3.9/site-packages/_pytest/runner.py", line 340, in from_call
  |     result: Optional[TResult] = func()
  |   File "/usr/lib/python3.9/site-packages/_pytest/runner.py", line 240, in <lambda>
  |     lambda: runtest_hook(item=item, **kwds), when=when, reraise=reraise
  |   File "/usr/lib/python3.9/site-packages/pluggy/_hooks.py", line 501, in __call__
  |     return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
  |   File "/usr/lib/python3.9/site-packages/pluggy/_manager.py", line 119, in _hookexec
  |     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  |   File "/usr/lib/python3.9/site-packages/pluggy/_callers.py", line 181, in _multicall
  |     return outcome.get_result()
  |   File "/usr/lib/python3.9/site-packages/pluggy/_result.py", line 99, in get_result
  |     raise exc.with_traceback(exc.__traceback__)
  |   File "/usr/lib/python3.9/site-packages/pluggy/_callers.py", line 166, in _multicall
  |     teardown.throw(outcome._exception)
  |   File "/usr/lib/python3.9/site-packages/_pytest/threadexception.py", line 87, in pytest_runtest_call
  |     yield from thread_exception_runtest_hook()
  |   File "/usr/lib/python3.9/site-packages/_pytest/threadexception.py", line 63, in thread_exception_runtest_hook
  |     yield
  |   File "/usr/lib/python3.9/site-packages/pluggy/_callers.py", line 166, in _multicall
  |     teardown.throw(outcome._exception)
  |   File "/usr/lib/python3.9/site-packages/_pytest/unraisableexception.py", line 90, in pytest_runtest_call
  |     yield from unraisable_exception_runtest_hook()
  |   File "/usr/lib/python3.9/site-packages/_pytest/unraisableexception.py", line 65, in unraisable_exception_runtest_hook
  |     yield
  |   File "/usr/lib/python3.9/site-packages/pluggy/_callers.py", line 166, in _multicall
  |     teardown.throw(outcome._exception)
  |   File "/usr/lib/python3.9/site-packages/_pytest/logging.py", line 849, in pytest_runtest_call
  |     yield from self._runtest_for(item, "call")
  |   File "/usr/lib/python3.9/site-packages/_pytest/logging.py", line 832, in _runtest_for
  |     yield
  |   File "/usr/lib/python3.9/site-packages/pluggy/_callers.py", line 166, in _multicall
  |     teardown.throw(outcome._exception)
  |   File "/usr/lib/python3.9/site-packages/_pytest/capture.py", line 883, in pytest_runtest_call
  |     return (yield)
  |   File "/usr/lib/python3.9/site-packages/pluggy/_callers.py", line 166, in _multicall
  |     teardown.throw(outcome._exception)
  |   File "/usr/lib/python3.9/site-packages/_pytest/skipping.py", line 256, in pytest_runtest_call
  |     return (yield)
  |   File "/usr/lib/python3.9/site-packages/pluggy/_callers.py", line 102, in _multicall
  |     res = hook_impl.function(*args)
  |   File "/usr/lib/python3.9/site-packages/_pytest/runner.py", line 182, in pytest_runtest_call
  |     raise e
  |   File "/usr/lib/python3.9/site-packages/_pytest/runner.py", line 172, in pytest_runtest_call
  |     item.runtest()
  |   File "/usr/lib/python3.9/site-packages/_pytest/python.py", line 1777, in runtest
  |     self.ihook.pytest_pyfunc_call(pyfuncitem=self)
  |   File "/usr/lib/python3.9/site-packages/pluggy/_hooks.py", line 501, in __call__
  |     return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
  |   File "/usr/lib/python3.9/site-packages/pluggy/_manager.py", line 119, in _hookexec
  |     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  |   File "/usr/lib/python3.9/site-packages/pluggy/_callers.py", line 138, in _multicall
  |     raise exception.with_traceback(exception.__traceback__)
  |   File "/usr/lib/python3.9/site-packages/pluggy/_callers.py", line 102, in _multicall
  |     res = hook_impl.function(*args)
  |   File "/usr/lib/python3.9/site-packages/_pytest/python.py", line 200, in pytest_pyfunc_call
  |     result = testfunction(**testargs)
  |   File "/home/tkloczko/rpmbuild/BUILD/hypothesmith-0.3.3/tests/test_syntactic.py", line 41, in test_tokenize_round_trip_string
  |     @example("#")
  |   File "/usr/lib/python3.9/site-packages/hypothesis/core.py", line 1573, in wrapped_test
  |     _raise_to_user(errors, state.settings, [], " in explicit examples")
  |   File "/usr/lib/python3.9/site-packages/hypothesis/core.py", line 1277, in _raise_to_user
  |     raise the_error_hypothesis_found
  | exceptiongroup.ExceptionGroup: Hypothesis found 2 distinct failures in explicit examples. (2 sub-exceptions)
  +-+---------------- 1 ----------------
    | Traceback (most recent call last):
    |   File "/home/tkloczko/rpmbuild/BUILD/hypothesmith-0.3.3/tests/test_syntactic.py", line 46, in test_tokenize_round_trip_string
    |     tokens = list(tokenize.generate_tokens(io.StringIO(source_code).readline))
    |   File "/usr/lib64/python3.9/tokenize.py", line 521, in _tokenize
    |     raise TokenError("EOF in multi-line statement", (lnum, 0))
    | tokenize.TokenError: ('EOF in multi-line statement', (3, 0))
    | Falsifying explicit example: test_tokenize_round_trip_string(
    |     source_code='\n\\\n',
    | )
    +---------------- 2 ----------------
    | Traceback (most recent call last):
    |   File "/home/tkloczko/rpmbuild/BUILD/hypothesmith-0.3.3/tests/test_syntactic.py", line 49, in test_tokenize_round_trip_string
    |     assert [(t.type, t.string) for t in tokens] == [(t.type, t.string) for t in output]
    | AssertionError: assert [(60, '#'), (...\n'), (0, '')] == [(60, '#'), (...4, '\n'), ...]
    |
    |   At index 2 diff: (1, 'pass') != (5, ' ')
    |   Right contains 2 more items, first extra item: (6, '')
    |   Use -v to get more diff
    | Falsifying explicit example: test_tokenize_round_trip_string(
    |     source_code='#\n\x0cpass#\n',
    | )
    +------------------------------------
=========================== short test summary info ============================
SKIPPED [2] tests/test_cst.py:43: codegen not supported yet, e.g. Annotation
XFAIL tests/test_cst.py::test_black_autoformatter_from_nodes
XFAIL tests/test_syntactic.py::test_tokenize_round_trip_bytes
XFAIL tests/test_syntactic.py::test_tokenize_round_trip_string
FAILED tests/test_cst.py::test_source_code_from_libcst_node_type[Match] - hyp...
FAILED tests/test_cst.py::test_source_code_from_libcst_node_type[MatchList]
FAILED tests/test_cst.py::test_source_code_from_libcst_node_type[TryStar] - h...
======= 3 failed, 173 passed, 2 skipped, 3 xfailed in 3907.94s (1:05:07) =======

List of installed modules in build env:

Package            Version
------------------ -----------
attrs              23.2.0
black              24.3.0
build              1.1.1
click              8.1.7
exceptiongroup     1.1.3
hypothesis         6.99.11
importlib_metadata 7.0.1
iniconfig          2.0.0
installer          0.7.0
lark               1.1.9
libcst             1.2.0
mypy_extensions    1.0.0
packaging          24.0
parso              0.8.3
pathspec           0.12.1
platformdirs       4.2.0
pluggy             1.4.0
pyproject_hooks    1.0.0
pytest             8.1.1
python-dateutil    2.9.0.post0
PyYAML             6.0.1
setuptools         69.1.1
sortedcontainers   2.4.0
tokenize_rt        5.2.0
tomli              2.0.1
typing_extensions  4.10.0
typing_inspect     0.9.0
wheel              0.43.0
zipp               3.17.0

Please let me know if you need more details or want me to perform some diagnostics.

[Zac-HD]

Hmm, looks like these might be real bugs in Black: I've seen at least one before in psf/black#1012