Improve test docs for LLMs

[MarcSkovMadsen]

Providing LLMs with the skills to quickly and robustly test Panel apps is key for LLMs to create working Panel Apps.

I do believe the current documentation of tests can be significantly improved. I will collect my experience in this Feature Request and the plan to make a Pull Request when I can find the time.

Smoke Tests

"""Smoke tests for docstring example apps in tmp/.

Each test imports the module and verifies it doesn't crash on import.
"""

import importlib
import sys
from pathlib import Path

import pytest

TMP_DIR = Path(__file__).resolve().parent.parent / "tmp"

APP_FILES = sorted(TMP_DIR.glob("app_*.py"))
APP_IDS = [p.stem for p in APP_FILES]


@pytest.fixture(autouse=True)
def _clean_module_cache():
    """Remove imported app modules after each test to avoid side-effects."""
    yield
    to_remove = [name for name in sys.modules if name.startswith("app_")]
    for name in to_remove:
        del sys.modules[name]


@pytest.mark.parametrize("app_path", APP_FILES, ids=APP_IDS)
def test_app_imports_without_error(app_path):
    """Verify the app module can be imported without raising an exception."""
    spec = importlib.util.spec_from_file_location(app_path.stem, app_path)
    mod = importlib.util.module_from_spec(spec)
    spec.loader.exec_module(mod)

UI Test

Improvements to https://panel.holoviz.org/how_to/test/uitests.html

Coming up:

• Start Server
• Test with Playwright
• Take screenshots

[MarcSkovMadsen]

Triage: panel #8411 — Improve test docs for LLMs

Type: Task | Labels: type: docs

Summary

Feature request to improve Panel's testing documentation for LLM-friendliness, including smoke test patterns for dynamic app module imports and Playwright UI testing docs. During triage, a concrete bug was found in the existing pytest how-to guide: doc/how_to/test/pytest.md line 117 uses sleep_delay as a parameter name, but the App class defines run_delay instead, causing a TypeError if copied verbatim.

Next Steps

• Fix the sleep_delay → run_delay typo at doc/how_to/test/pytest.md line 117 (quick standalone PR or bundle with broader improvements)
• The author plans to contribute the broader test docs improvements — coordinate scope

Details

Issue: panel#8411
Author: @MarcSkovMadsen | Created: 2026-02-09 | State: open

Duplicate / Related Issues

• Simple Example Apps in Docs #8223 — Simple Example Apps in Docs (open, same author): Closely related — focuses on adding simple example apps to docs, overlaps with the goal of making docs more accessible for LLMs.
• Discourse: Defining failure modes and benchmarks for LLM use in the HoloViz ecosystem (https://discourse.holoviz.org/t/defining-failure-modes-and-benchmarks-for-llm-use-in-the-holoviz-ecosystem/9086): Broader context about a NumFOCUS-funded effort to understand LLM-friendliness of HoloViz projects.
• Discourse: Looking for tutorials on writing unit test for Panels (https://discourse.holoviz.org/t/looking-for-tutorials-on-writing-unit-test-for-panels/4050): Community interest in better testing tutorials.

No exact duplicates found.

Documentation Search

The existing test documentation consists of three how-to guides:

1. doc/how_to/test/pytest.md — Unit and performance testing with pytest. Contains a sleep_delay bug (line 117).
2. doc/how_to/test/uitests.md — UI testing with Playwright. Covers basic setup but is incomplete (missing server startup patterns, screenshot-taking).
3. doc/how_to/test/loadtests.md — Load testing with Locust and Loadwright.

The docs do not currently cover: smoke test patterns, screenshot comparison testing, or LLM-oriented testing guidance.

Root Cause

Two aspects:

1. Docs bug: doc/how_to/test/pytest.md line 117 uses sleep_delay instead of run_delay in the pytest fixture. Simple typo — the App class defines run_delay but the fixture uses the non-existent sleep_delay.

2. Feature gap: The test documentation does not cover smoke testing patterns (dynamic module imports), screenshot comparison testing, or LLM-oriented testing guidance.

Potential Solution

1. Immediate fix: Change sleep_delay to run_delay on line 117 of doc/how_to/test/pytest.md.
2. Feature request: The broader improvements (smoke tests, UI test enhancements) require the author's planned PR. Scope and structure should be decided by the maintainer.

Reproduction

Test file: .triaging/triage_panel_8411_20260218_164522_7fd26112-8aa2-4b6c-87b1-b647af04a191_test.py

Run tests:

uvx --with panel pytest .triaging/triage_panel_8411_20260218_164522_7fd26112-8aa2-4b6c-87b1-b647af04a191_test.py -v

Pytest result: 1 failed, 4 passed in 1.65s
Reproduces on main: Yes

The test_docs_fixture_uses_invalid_param_sleep_delay test fails with TypeError: App.__init__() got an unexpected keyword argument 'sleep_delay', confirming the docs bug.

"""Reproduction tests for panel#8411.

This issue is a feature request to improve test documentation for LLMs.
During triage we discovered a concrete bug in the existing pytest docs:
the fixture uses `sleep_delay` which is not a valid App parameter (should
be `run_delay`).  We also verify that the smoke-test pattern proposed in
the issue works correctly, and that the existing docs App class is
functional with the correct parameter names.
"""
import importlib
import sys
import textwrap
import time
from pathlib import Path

import panel as pn
import param
import pytest


# Inline copy of the App class from the docs (doc/how_to/test/pytest.md)

class App(pn.viewable.Viewer):
    run = param.Event(doc="Runs for click_delay seconds when clicked")
    runs = param.Integer(doc="The number of runs")
    status = param.String(default="No runs yet")
    load_delay = param.Number(default=0.5)
    run_delay = param.Number(default=0.5)

    def __init__(self, **params):
        super().__init__(**params)
        result = self._load()
        self._time = time.time()
        self._status_pane = pn.pane.Markdown(
            self.status, height=40, align="start", margin=(0, 5, 10, 5)
        )
        self._result_pane = pn.Column(result)
        self._view = pn.Column(
            pn.Row(
                pn.widgets.Button.from_param(self.param.run, sizing_mode="fixed"),
                self._status_pane,
            ),
            self._result_pane,
        )

    def __panel__(self):
        return self._view

    def _start_run(self):
        self.status = "Running ..."
        self._time = time.time()

    def _stop_run(self):
        now = time.time()
        duration = round(now - self._time, 3)
        self._time = now
        self.runs += 1
        self.status = f"Finished run {self.runs} in {duration}sec"

    @param.depends("run", watch=True)
    def _run_with_status_update(self):
        self._start_run()
        self._result_pane[:] = [self._run()]
        self._stop_run()

    @param.depends("status", watch=True)
    def _update_status_pane(self):
        self._status_pane.object = self.status

    def _load(self):
        time.sleep(self.load_delay)
        return "Loaded"

    def _run(self):
        time.sleep(self.run_delay)
        return f"Result {self.runs + 1}"


def test_docs_fixture_uses_invalid_param_sleep_delay():
    """The docs fixture at pytest.md:117 uses sleep_delay (invalid).
    This test FAILS now and PASSES once docs are corrected to run_delay."""
    App(sleep_delay=0.001, load_delay=0.001)


def test_docs_fixture_correct_param_run_delay():
    """Using the correct parameter name run_delay works as expected."""
    app = App(run_delay=0.001, load_delay=0.001)
    assert app.run_delay == 0.001
    assert app.load_delay == 0.001


def test_smoke_test_pattern_works(tmp_path):
    """The dynamic-import smoke test pattern from the issue works."""
    app_code = textwrap.dedent("""\
        import panel as pn
        slider = pn.widgets.IntSlider(name="Value", start=0, end=10, value=5)
        text = pn.pane.Markdown(f"Current: {slider.value}")
        layout = pn.Column(slider, text)
    """)
    app_file = tmp_path / "app_example.py"
    app_file.write_text(app_code)
    spec = importlib.util.spec_from_file_location(app_file.stem, app_file)
    mod = importlib.util.module_from_spec(spec)
    spec.loader.exec_module(mod)
    assert hasattr(mod, "slider")
    assert hasattr(mod, "layout")


def test_app_constructor_defaults():
    """App should have correct defaults as described in docs."""
    app = App(load_delay=0.001, run_delay=0.001)
    assert app.runs == 0
    assert app.status == "No runs yet"


def test_app_run_updates_state():
    """App state should update when run is triggered."""
    app = App(load_delay=0.001, run_delay=0.001)
    app.param.trigger("run")
    assert app.runs == 1
    assert app.status.startswith("Finished run 1 in")

Fix Plan

Files to Modify

File	Change
doc/how_to/test/pytest.md	Line 117: change sleep_delay=0.001 to run_delay=0.001

Verification

• Run reproduction tests: uvx --with panel pytest .triaging/triage_panel_8411_20260218_164522_7fd26112-8aa2-4b6c-87b1-b647af04a191_test.py -v — note: test_docs_fixture_uses_invalid_param_sleep_delay is designed to FAIL while the bug exists; after the fix, this test should be updated to use run_delay (or rely on test_docs_fixture_correct_param_run_delay which already passes)
• Verify the corrected fixture in the docs matches the App class parameters

Risk Assessment

• No runtime code changes — docs-only fix
• The broader feature request (smoke tests, UI test improvements) should be addressed in a separate PR by the author