POC: start adding "repro snippets" for failed assertions

[ev-br]

It is not always easy to tell what exactly failed with hypothesis (gh-379). Thus add a failing snippet to the exceptions' output.

With this PR, the OP example from gh-379 preserves whatever output pytest and hypothesis generated, and adds an explicit incantation that triggered the error:

$ ARRAY_API_TESTS_MODULE=array_api_compat.numpy pytest array_api_tests/test_manipulation_functions.py::test_repeat

... snip ...

array_api_tests/test_manipulation_functions.py:288: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
array_api_tests/test_manipulation_functions.py:318: in test_repeat
    out = xp.repeat(x, repeats, **kw)
../../miniforge3/envs/array-api/lib/python3.12/site-packages/numpy/_core/fromnumeric.py:506: in repeat
    return _wrapfunc(a, 'repeat', repeats, axis=axis)
../../miniforge3/envs/array-api/lib/python3.12/site-packages/numpy/_core/fromnumeric.py:66: in _wrapfunc
    return _wrapit(obj, method, *args, **kwds)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

obj = array([], dtype=bool), method = 'repeat', args = (array([], dtype=uint64),)
kwds = {'axis': None}
conv = <numpy._core._multiarray_umath._array_converter object at 0x7fd5f2252c10>
arr = array([], dtype=bool)

    def _wrapit(obj, method, *args, **kwds):
        conv = _array_converter(obj)
        # As this already tried the method, subok is maybe quite reasonable here
        # but this follows what was done before. TODO: revisit this.
        arr, = conv.as_arrays(subok=False)
>       result = getattr(arr, method)(*args, **kwds)
E       TypeError: Cannot cast array data from dtype('uint64') to dtype('int64') according to the rule 'safe'
E       
E       ========== FAILING CODE SNIPPET:
E       xp.repeat(array([], dtype=bool),array([], dtype=uint64), **kw) with kw={}
E       ====================
E       
E       Falsifying example: test_repeat(
E           x=array([], dtype=bool),
E           kw={},
E           data=data(...),
E       )
E       Draw 1 (repeats): array([], dtype=uint64)

../../miniforge3/envs/array-api/lib/python3.12/site-packages/numpy/_core/fromnumeric.py:46: TypeError
=============================== short test summary info ================================
FAILED array_api_tests/test_manipulation_functions.py::test_repeat - TypeError: Cannot cast array data from dtype('uint64') to dtype('int64') according ...
================================== 1 failed in 0.24s ===================================

The approach here is a bit manual, and I don't see a generic way to make it less so.

[ev-br]

One more example:

$ ARRAY_API_TESTS_MODULE=array_api_compat.torch pytest array_api_tests/test_array_object.py::test_getitem

.... snip ....


array_api_tests/test_array_object.py:78: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

shape = (0, 2), dtype = torch.bool, data = data(...)

    @given(shape=hh.shapes(), dtype=hh.all_dtypes, data=st.data())
    def test_getitem(shape, dtype, data):
        zero_sided = any(side == 0 for side in shape)
        if zero_sided:
            x = xp.zeros(shape, dtype=dtype)
        else:
            obj = data.draw(scalar_objects(dtype, shape), label="obj")
            x = xp.asarray(obj, dtype=dtype)
        note(f"{x=}")
        key = data.draw(xps.indices(shape=shape, allow_newaxis=True), label="key")
    
        repro_snippet = ph.format_snippet(f"{x!r}[{key!r}]")
     ###   breakpoint()
    
        try:
>           out = x[key]
E           ValueError: step must be greater than zero
E           
E           ========== FAILING CODE SNIPPET:
E           tensor([], size=(0, 2), dtype=torch.bool)[(slice(None, None, None), slice(1, 0, -1))]
E           ====================
E           
E           Falsifying example: test_getitem(
E               shape=(0, 2),
E               dtype=torch.bool,  # or any other generated value
E               data=data(...),
E           )
E           x=tensor([], size=(0, 2), dtype=torch.bool)
E           Draw 1 (key): (slice(None, None, None), slice(1, 0, -1))
E           Explanation:
E               These lines were always and only run by failing examples:
E                   /home/br/repos/array-api-tests/array_api_tests/test_array_object.py:109

array_api_tests/test_array_object.py:92: ValueError

[ev-br]

• test_array_object.py
• test_constants.py
• test_creation_functions.py
• test_data_type_functions.py
• test_fft.py
• test_has_names.py
• test_indexing_functions.py
• test_inspection_functions.py
• test_linalg.py
• test_manipulation_functions.py
• test_operators_and_elementwise_functions.py
• test_searching_functions.py
• test_set_functions.py
• test_signatures.py
• test_sorting_functions.py
• test_special_cases.py
• test_statistical_functions.py
• test_utility_functions.py

[ev-br]

The shape of this POC is getting clear now. I've converted one of the test modules, test_manipulation_functions.py. The patch size is mostly whitespace/indentation, and is best viewed with the "hide whitespace" option.

Converting the rest is a bit of mechanical work, but is totally doable.

Downsides:

• git blame gets a bit messier, will need git blame -M -C -C or some such to help git see past trivial indentation changes.
• this will only catch failures in the functions under tests and test assertions. Tricky failures inside hypothesis (either "essential functions" of Providing reproduction code for failed test cases #379 (comment) or obscure "flaky" failures of Skipped flaky tests #300) will not be caught.