Flaky UI tests for ToggleIcon and Tabulator on MacOS

[cdeil]

With latest 9404b43 on MacOS I get a few UI test fails.

One run see full log where these fail:

FAILED panel/tests/ui/pane/test_textual.py::test_textual_app - TimeoutError: wait_until timed out in 5000 milliseconds
FAILED panel/tests/ui/widgets/test_tabulator.py::test_tabulator_patch_no_height_resize - TimeoutError: wait_until timed out in 5000 milliseconds
FAILED panel/tests/ui/widgets/test_tabulator.py::test_selection_indices_on_paginated_sorted_and_filtered_data[remote] - TimeoutError: wait_until timed out in 5000 milliseconds
FAILED panel/tests/ui/widgets/test_tabulator.py::test_tabulator_edit_event_and_header_filters_same_column[index-True] - playwright._impl._errors.TimeoutError: Locator.fill: Timeout 20000ms exceeded.
FAILED panel/tests/ui/widgets/test_tabulator.py::test_tabulator_edit_event_and_header_filters_same_column[index-False] - playwright._impl._errors.TimeoutError: Locator.fill: Timeout 20000ms exceeded.
FAILED panel/tests/ui/widgets/test_tabulator.py::test_tabulator_edit_event_and_header_filters_same_column[foo-False] - playwright._impl._errors.TimeoutError: Locator.fill: Timeout 20000ms exceeded.
FAILED panel/tests/ui/widgets/test_tabulator.py::test_tabulator_edit_event_and_header_filters_same_column[foo-True] - playwright._impl._errors.TimeoutError: Locator.fill: Timeout 20000ms exceeded.

Another run see full log where these fail:

FAILED panel/tests/ui/io/test_reload.py::test_reload_app_on_local_module_change - TimeoutError: wait_until timed out in 5000 milliseconds
FAILED panel/tests/ui/pane/test_textual.py::test_textual_app - TimeoutError: wait_until timed out in 5000 milliseconds
FAILED panel/tests/ui/widgets/test_tabulator.py::test_tabulator_patch_no_height_resize - TimeoutError: wait_until timed out in 5000 milliseconds
FAILED panel/tests/ui/widgets/test_tabulator.py::test_selection_indices_on_paginated_sorted_and_filtered_data[remote] - TimeoutError: wait_until timed out in 5000 milliseconds
FAILED panel/tests/ui/widgets/test_tabulator.py::test_tabulator_edit_event_and_header_filters_same_column[index-True] - playwright._impl._errors.TimeoutError: Locator.fill: Timeout 20000ms exceeded.
=

I tried turning xdist off via

$ pixi run -e test-ui pytest --ui panel/tests/ui/widgets/test_icon.py -v --browser chromium -n logical --dist no -n 0

but still got test fails ( see full log ):

FAILED panel/tests/ui/pane/test_textual.py::test_textual_app - TimeoutError: wait_until timed out in 5000 milliseconds
FAILED panel/tests/ui/pane/test_vizzu.py::test_vizzu_click - TimeoutError: wait_until timed out in 5000 milliseconds
FAILED panel/tests/ui/template/test_editabletemplate.py::test_editable_template_drag_item - TimeoutError: wait_until timed out in 5000 milliseconds
FAILED panel/tests/ui/widgets/test_icon.py::test_toggle_icon_width_height - TimeoutError: wait_until timed out in 5000 milliseconds
FAILED panel/tests/ui/widgets/test_icon.py::test_toggle_icon_size - TimeoutError: wait_until timed out in 5000 milliseconds
FAILED panel/tests/ui/widgets/test_tabulator.py::test_tabulator_patch_no_height_resize - TimeoutError: wait_until timed out in 5000 milliseconds
FAILED panel/tests/ui/widgets/test_tabulator.py::test_tabulator_header_filter_no_horizontal_rescroll[remote] - AssertionError: assert {'height': 20...: 714, 'y': 9} == {'height': 20...: 264, 'y': 9}
FAILED panel/tests/ui/widgets/test_tabulator.py::test_tabulator_edit_event_and_header_filters_same_column[index-True] - playwright._impl._errors.TimeoutError: Locator.fill: Timeout 20000ms exceeded.
FAILED panel/tests/ui/widgets/test_tabulator.py::test_tabulator_edit_event_and_header_filters_same_column[index-False] - playwright._impl._errors.TimeoutError: Locator.fill: Timeout 20000ms exceeded.
FAILED panel/tests/ui/widgets/test_tabulator.py::test_tabulator_edit_event_and_header_filters_same_column[foo-True] - playwright._impl._errors.TimeoutError: Locator.fill: Timeout 20000ms exceeded.
FAILED panel/tests/ui/widgets/test_tabulator.py::test_tabulator_edit_event_and_header_filters_same_column[foo-False] - playwright._impl._errors.TimeoutError: Locator.fill: Timeout 20000ms exceeded.
FAILED panel/tests/ui/widgets/test_tabulator.py::test_selection_indices_on_paginated_sorted_and_filtered_data[remote] - TimeoutError: wait_until timed out in 5000 milliseconds
ERROR panel/tests/ui/widgets/test_tabulator.py::test_tabulator_header_filter_no_horizontal_rescroll[None] - pluggy.PluggyTeardownRaisedWarning: A plugin raised an exception during an old-style hookwrapper teardown.

The textual fail is due to a recent breaking API change - see #7117

The others are flaky tests I think, although this seems to fail for me consistently now:

panel $ pixi run -e test-ui pytest --ui panel/tests/ui/widgets/test_icon.py -k test_toggle_icon_size -v
=============================================================================================== test session starts ================================================================================================
platform darwin -- Python 3.12.5, pytest-7.4.4, pluggy-1.5.0 -- /Users/cdeil/code/oss/panel/.pixi/envs/test-ui/bin/python3.12
cachedir: .pytest_cache
rootdir: /Users/cdeil/code/oss/panel
configfile: pyproject.toml
plugins: asyncio-0.23.8, cov-5.0.0, github-actions-annotate-failures-0.2.0, playwright-0.5.0, rerunfailures-14.0, anyio-4.4.0, base-url-2.1.0, xdist-3.6.1
asyncio: mode=Mode.AUTO
collected 17 items / 16 deselected / 1 selected                                                                                                                                                                    

panel/tests/ui/widgets/test_icon.py::test_toggle_icon_size FAILED                                                                                                                                            [100%]

===================================================================================================== FAILURES =====================================================================================================
______________________________________________________________________________________________ test_toggle_icon_size _______________________________________________________________________________________________

page = <Page url='http://localhost:65348/'>

    def test_toggle_icon_size(page):
        icon = ToggleIcon(size="120px")
        serve_component(page, icon)
    
        # test defaults
        assert icon.icon == "heart"
        assert not icon.value
        icon_element = page.locator(".ti-heart")
    
>       wait_until(lambda: icon_element.bounding_box()["width"] == 120)
E       TimeoutError: wait_until timed out in 5000 milliseconds

panel/tests/ui/widgets/test_icon.py:66: TimeoutError
----------------------------------------------------------------------------------------------- Captured stdout call -----------------------------------------------------------------------------------------------
Launching server at http://localhost:65348
----------------------------------------------------------------------------------------------- Captured stderr call -----------------------------------------------------------------------------------------------
INFO:bokeh.server.server:Starting Bokeh server version 3.5.1 (running on Tornado 6.4.1)
INFO:bokeh.server.tornado:User authentication hooks NOT provided (default user enabled)
INFO:bokeh.server.views.ws:WebSocket connection opened
INFO:bokeh.server.views.ws:ServerConnection created
------------------------------------------------------------------------------------------------ Captured log call -------------------------------------------------------------------------------------------------
INFO     tornado.access:web.py:2348 200 GET /liveness (127.0.0.1) 0.39ms
INFO     tornado.access:web.py:2348 200 GET / (::1) 17.99ms
INFO     tornado.access:web.py:2348 200 GET /static/js/bokeh.min.js?v=276377ed021e1611c60311b355033c865900f31a918aa4565aba37a78700f17b017100a8a618bded4140c6ad247a0b0237d3a02bee9fd722ce67a459479522dc (::1) 1.99ms
INFO     tornado.access:web.py:2348 200 GET /static/extensions/panel/bundled/reactiveesm/es-module-shims@%5E1.10.0/dist/es-module-shims.min.js (::1) 2.09ms
INFO     tornado.access:web.py:2348 200 GET /static/js/bokeh-gl.min.js?v=70bc1a9856b732e888ed6b2a8e9b6382bf538fee3ec9f1145b8db1778158fd51e478dbe0600650e30d5a0083b12fc43961bc7b2ef3e9f366000199b83b9a1644 (::1) 0.38ms
INFO     tornado.access:web.py:2348 200 GET /static/extensions/panel/panel.min.js?v=a91daab4668e3299f59ed231b5da2e657f5e65d10a1d501ff0a660306b1fdb79 (::1) 4.22ms
INFO     tornado.access:web.py:2348 200 GET /static/js/bokeh-widgets.min.js?v=8541420c1bb1dbde534df1d9b2be7c8248f61fca353a821ffc4d459b08b79c4b39f0ea1dd6960aa3b734bea988cf822dc6993c786de844db80e4f258dd90727f (::1) 1.91ms
INFO     tornado.access:web.py:2348 200 GET /static/js/bokeh-tables.min.js?v=26281191594de496d010d87b3a56c1679330da29fcf72d3dab91ac4a45479c16b36e82ce4325f4217df4614fad13927fd7f1e1be64cf838e4a18a60852e2be0e (::1) 2.00ms
INFO     tornado.access:web.py:2348 101 GET /ws (::1) 0.32ms
INFO     tornado.access:web.py:2348 200 GET /static/extensions/panel/css/loading.css?v=1.5.0-b.3 (::1) 1.10ms
INFO     tornado.access:web.py:2348 200 GET /static/extensions/panel/css/icon.css?v=1.5.0-b.3 (::1) 1.52ms
INFO     tornado.access:web.py:2348 200 GET /static/extensions/panel/bundled/theme/default.css?v=1.5.0-b.3 (::1) 4.11ms
INFO     tornado.access:web.py:2348 200 GET /static/extensions/panel/bundled/theme/native.css?v=1.5.0-b.3 (::1) 9.85ms
--------------------------------------------------------------------------------------------- Captured stderr teardown ---------------------------------------------------------------------------------------------
INFO:bokeh.server.views.ws:WebSocket connection closed: code=1001, reason=None
============================================================================================= short test summary info ==============================================================================================
FAILED panel/tests/ui/widgets/test_icon.py::test_toggle_icon_size - TimeoutError: wait_until timed out in 5000 milliseconds
========================================================================================= 1 failed, 16 deselected in 6.08s =========================================================================================

[cdeil]

I see similar fails in CI: https://github.com/holoviz/panel/actions/runs/10333902177/job/28606768301?pr=7120

[philippjfr]

I've tried to mitigate some of these but it is indeed a game of whack-a-mole. I also couldn't reproduce a bunch of them so these are the ones I focused on:

FAILED panel/tests/ui/widgets/test_tabulator.py::test_tabulator_patch_no_height_resize - TimeoutError: wait_until timed out in 5000 milliseconds
FAILED panel/tests/ui/widgets/test_tabulator.py::test_selection_indices_on_paginated_sorted_and_filtered_data[remote] - TimeoutError: wait_until timed out in 5000 milliseconds
FAILED panel/tests/ui/widgets/test_tabulator.py::test_tabulator_edit_event_and_header_filters_same_column[index-True] - playwright._impl._errors.TimeoutError: Locator.fill: Timeout 20000ms exceeded.

[cdeil]

Wow, thanks!

Maybe mark remaining flaky UI tests on MacOS only like this to remove the noise?

pytest.mark.skipif(sys.platform == 'darwin', strict=False, reason="Flaky, see GH 7118")

See https://docs.pytest.org/en/7.1.x/explanation/flaky.html

Or alternatively - do you think it should be possible to get reliable tests?
Or is there something fundamental in Panel / Bokeh / Python async & threading / MacOS / Pywright / etc that prevents this?

I saw yesterday that Bokeh doesn't use Playwright and do much UI testing on MacOS probably because they've run into similar issues?

[philippjfr]

Yes, it should be possible to get more reliable tests, I'm 99% certain this is just about how the tests are structured. Specifically Playwright operates much faster than any real world usage ever would, so that causes some issues that aren't visible otherwise. By restructuring the tests and/or adding a bunch of additional timeouts we could probably make them more reliable. You could try to test that theory by re-running the UI tests with --slowmo 100 or so, which adds 100 ms timeouts between all interactions.