rerun failures for `shiny` workflow

[dangotbanned]

No idea when the flakiness started, but the most recent one I had was today

Show logs

=================================== FAILURES ===================================
______________ test_validate_row_selection_in_edit_mode[chromium] ______________
[gw0] linux -- Python 3.12.11 /home/runner/work/narwhals/narwhals/py-shiny/.venv/bin/python3

page = <Page url='http://127.0.0.1:2956/'>
local_app = <shiny.run._run.ShinyAppProc object at 0x7f34b31086e0>

    @skip_if_not_chrome
    @pytest.mark.flaky(reruns=reruns, reruns_delay=reruns_delay)
    def test_validate_row_selection_in_edit_mode(
        page: Page, local_app: ShinyAppProc
    ) -> None:
        page.set_viewport_size({"width": 1920 * 2, "height": 1080 * 2})
        page.goto(local_app.url)
    
        # Select (and verify) a row. Edit a cell content in that row.
        # Verify the row is not focused. Hit escape key. Verify the cell value is not updated.
        # Verify the row is focused. Hit escape key again.
        # Verify the row is not focused. (Possibly verify the container div is focused?)
        data_frame = controller.OutputDataFrame(page, "penguins_df")
    
        data_frame.expect_cell("N1A2", row=1, col=6)
        data_frame._edit_cell_no_save("N2A2", row=1, col=6)
        data_frame._expect_row_focus_state(False, row=1)
        data_frame.expect_class_state("editing", row=1, col=6)
        data_frame.expect_selected_num_rows(1)
        data_frame.expect_selected_rows([1])
        data_frame.set_cell("N3A2", row=1, col=6, finish_key="Escape")
        data_frame.expect_cell("N1A2", row=1, col=6)
        data_frame._expect_row_focus_state(True, row=1)
        page.keyboard.press("Escape")
        data_frame._expect_row_focus_state(False, row=1)
    
        # Enable rows selection and editable.
        # Select (and verify) a row. Edit a cell content in that row.
        # Click a cell in another row. Verify the new row is selected and focused.
        # Verify the old row is not selected. Verify the old row cell value was updated.
        data_frame.expect_cell("N1A2", row=1, col=6)
        data_frame._edit_cell_no_save("N2A2", row=1, col=6)
        data_frame._expect_row_focus_state(False, row=1)
        data_frame.expect_class_state("editing", row=1, col=6)
        data_frame.cell_locator(row=2, col=6).click()
        data_frame._expect_row_focus_state(True, row=2)
        data_frame._expect_row_focus_state(False, row=1)
        data_frame.expect_cell("N2A2", row=1, col=6)
    
        # Enable rows selection and editable.
        # Select (and verify) a row. Hit enter to edit the first cell in that row.
        # Hit escape key. Verify the same row is focused.
        # Scroll right and display an html column in the left part of the view.
        # Hit enter to edit the first visible non-html cell in that row.
        # Verify that cell is editing.
        data_frame.cell_locator(row=1, col=2).click()
        page.keyboard.press("Enter")
        data_frame._expect_row_focus_state(False, row=1)
        page.keyboard.press("Escape")
        data_frame._expect_row_focus_state(True, row=1)
        page.keyboard.press("Escape")
        data_frame._edit_cell_no_save("Temp value", row=1, col=16)
        page.keyboard.press("Escape")
        # Wait for the row to be focused again after escaping edit mode
        data_frame._expect_row_focus_state(True, row=1)
        # Also ensure the cell is no longer in editing state
        data_frame.expect_class_state("ready", row=1, col=16)
        page.keyboard.press("Enter")
        data_frame.expect_class_state(
            "editing",
            row=1,
            col=0,
        )  # Stage column begins to be edited.
    
        # Click outside the table/Press Escape to exit row focus.
        # Tab to the column name, hit enter. Verify the table becomes sorted.
        # Tab to an HTML column name, hit enter. Verify the sort does not update.
        page.keyboard.press("Escape")
        page.keyboard.press("Escape")
        page.keyboard.press("Tab")
        page.keyboard.press("Tab")  # tab to sample number
        page.keyboard.press("Enter")
>       data_frame.expect_cell("152", row=0, col=1)

tests/playwright/shiny/components/data_frame/validate_row_selection_edit_mode/test_validate_row_selection_edit_mode.py:82: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <shiny.playwright.controller._output.OutputDataFrame object at 0x7f34b310b2f0>
value = '152'

    def expect_cell(
        self,
        value: PatternOrStr,
        *,
        row: int,
        col: int,
        timeout: Timeout = None,
    ) -> None:
        """
        Expects the cell in the data frame to have the specified text.
    
        Parameters
        ----------
        value
            The expected text in the cell.
        row
            The row number of the cell.
        col
            The column number of the cell.
        timeout
            The maximum time to wait for the expectation to pass. Defaults to `None`.
        """
        if not isinstance(row, int):
            raise TypeError("`row` must be an integer.")
        if not isinstance(col, int):
            raise TypeError("`col` must be an integer.")
        self._cell_scroll_if_needed(row=row, col=col, timeout=timeout)
>       playwright_expect(self.cell_locator(row, col)).to_have_text(
            value, timeout=timeout
        )
E       AssertionError: Locator expected to have text '152'
E       Actual value: 1 
E       Call log:
E         - Expect "to_have_text" with timeout 5000ms
E         - waiting for locator("#penguins_df.html-fill-item").locator("xpath=.").filter(has=locator("> div > div.shiny-data-grid")).locator("> div > div.shiny-data-grid").locator("> table").locator("> tbody").locator("> tr[data-index='0']").locator("> td, > th").locator("xpath=.").filter(has=locator("xpath=self::*[not(contains(@class, 'row-number'))]")).nth(1)
E           9 × locator resolved to <td class="cell-editable">1</td>
E             - unexpected value "1"

shiny/playwright/controller/_output.py:845: AssertionError
---------------------------- Captured log teardown -----------------------------
WARNING  root:_run.py:359 Application output:
/home/runner/work/narwhals/narwhals/py-shiny/.venv/lib/python3.12/site-packages/palmerpenguins/penguins.py:2: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
  import pkg_resources
INFO:     Started server process [6813]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://127.0.0.1:2956/ (Press CTRL+C to quit)
INFO:     127.0.0.1:51722 - "WebSocket /websocket/" [accepted]
INFO:     connection open
INFO:     connection closed
INFO:     127.0.0.1:52822 - "WebSocket /websocket/" [accepted]
INFO:     connection open
INFO:     connection closed
INFO:     127.0.0.1:39504 - "WebSocket /websocket/" [accepted]
INFO:     connection open
INFO:     connection closed
INFO:     127.0.0.1:39530 - "WebSocket /websocket/" [accepted]
INFO:     connection open
============================= slowest 6 durations ==============================
16.73s call     tests/playwright/shiny/components/data_frame/html_columns/test_html_columns.py::test_validate_html_columns[chromium-pandas]
15.16s call     tests/playwright/shiny/components/data_frame/html_columns/test_html_columns.py::test_validate_html_columns[chromium-polars]
6.97s setup    tests/playwright/shiny/components/data_frame/example/test_data_frame.py::test_grid_mode[chromium]
6.57s call     tests/playwright/shiny/components/data_frame/validate_row_selection_edit_mode/test_validate_row_selection_edit_mode.py::test_validate_row_selection_in_edit_mode[chromium]
6.47s call     tests/playwright/shiny/components/data_frame/validate_row_selection_edit_mode/test_validate_row_selection_edit_mode.py::test_validate_row_selection_in_edit_mode[chromium]
6.26s call     tests/playwright/shiny/components/data_frame/validate_row_selection_edit_mode/test_validate_row_selection_edit_mode.py::test_validate_row_selection_in_edit_mode[chromium]
=========================== short test summary info ============================
FAILED tests/playwright/shiny/components/data_frame/validate_row_selection_edit_mode/test_validate_row_selection_edit_mode.py::test_validate_row_selection_in_edit_mode[chromium] - AssertionError: Locator expected to have text '152'
Actual value: 1 
Call log:
  - Expect "to_have_text" with timeout 5000ms
  - waiting for locator("#penguins_df.html-fill-item").locator("xpath=.").filter(has=locator("> div > div.shiny-data-grid")).locator("> div > div.shiny-data-grid").locator("> table").locator("> tbody").locator("> tr[data-index='0']").locator("> td, > th").locator("xpath=.").filter(has=locator("xpath=self::*[not(contains(@class, 'row-number'))]")).nth(1)
    9 × locator resolved to <td class="cell-editable">1</td>
      - unexpected value "1"
=============== 1 failed, 35 passed, 3 rerun in 74.44s (0:01:14) ===============
make[1]: *** [Makefile:168: playwright] Error 1
make[1]: Leaving directory '/home/runner/work/narwhals/narwhals/py-shiny'
make: *** [Makefile:263: narwhals-test-integration] Error 2

Pretty sure this is the same thing 2 months ago

Always seems to work after re-running the whole workflow.

It'd be great if we could configure pytest-rerunfailures to do more of this for us

[dangotbanned]

Been digging and found

• chore: Use shiny's Makefile to allow for test commands to change over time #1161

It seems like we might not be able to configure much on our end

narwhals/.github/workflows/downstream_tests.yml

Lines 200 to 210 in 5309dd5

	. .venv/bin/activate
	uv pip freeze
	- name: Run `make narwhals-test-integration`
	run: |
	cd py-shiny
	. .venv/bin/activate
	# Isort seems to behave slightly differently in CI
	# so we ignore its output
	make format -s
	make narwhals-test-integration

That command ends up at

https://github.com/posit-dev/py-shiny/blob/ef6649b0677f736b710580acfe9218c0d67a05c5/Makefile#L259-L263

@schloerke is this something you might be able to help out with? 🙏

[FBruzzesi]

One thing I would consider is: do we need to run the entire test suite? If the answer is no, then I think there are two options:

• The library make use of a "narwhals" pytest marker, and we run pytest -m "narwhals". In this way the maintainers are free to change, enable and disable whatever they want, whenever they want. I did a similar proposal in fairlearn: TEST Add pytest marker for narwhals fairlearn/fairlearn#1555
• We run a subset of tests agreed beforehand. This is what we do for darts. Honestly this is a bit unstable and I would prefer option one

[schloerke]

Sorry about the headache y'all. The failure seems unrelated to what you're doing.

---

Currently the tests are type checking tests and wholistic tests that eventually touch narwhals. Wholistic tests that do not involve narwhals are not executed. So while not a marked test, the intent is already similar to Option 1.

What do you think about reducing shiny's test suite to just type checking? This should be stable and only produce valid type check errors.

---

Side solution if other packages join the flaky club... R runs reverse dependency checks and only fails the check if the results are worse than running the tests with the latest release version [of narwhals].

I don't know how to easily test this within python, but it helps with a moving target that you do not have full control.

[karangattu]

I have a PR that runs a test that is much targeted to Narwhals integration within Shiny. You can see the PR here.
cc: @dangotbanned

[FBruzzesi]

posit-dev/py-shiny#2067 to reduce the scope of narwhals tests has been merged, and we didn't get failures in a while. I will close this issue for now. Thanks everyone 🎉