Improve relative app path handling in test generation

Refactored SYSTEM_PROMPT_testing.md to clarify rules for relative app paths in create_app_fixture. Updated main.py to compute and rewrite app fixture paths to be relative to the test file directory, ensuring generated tests use correct relative paths. Simplified workflow logic and dependency installation in verify-testing-docs-on-change.yml.

Author: karangattu

.github/workflows/verify-testing-docs-on-change.yml

@@ -21,36 +21,20 @@ jobs:
         with:
           fetch-depth: 0
 
-      - name: Check for controller changes
-        id: check-controller
-        run: |
-          # Check if any files in shiny/playwright/controller have changed
-          if git diff --name-only origin/${{ github.base_ref }}...HEAD | grep -q '^shiny/playwright/controller/'; then
-            echo "controller_changed=true" >> $GITHUB_OUTPUT
-            echo "Changes detected in shiny/playwright/controller directory"
-          else
-            echo "controller_changed=false" >> $GITHUB_OUTPUT
-            echo "No changes detected in shiny/playwright/controller directory"
-          fi
-
       - name: Set up Python
-        if: steps.check-controller.outputs.controller_changed == 'true'
         uses: actions/setup-python@v5
         with:
           python-version: '3.13'
 
       - name: Install uv
-        if: steps.check-controller.outputs.controller_changed == 'true'
         uses: astral-sh/setup-uv@v4
 
       - name: Install dependencies
-        if: steps.check-controller.outputs.controller_changed == 'true'
         run: |
           uv pip install --system --upgrade pip
-          uv pip install --system -e ".[dev,test]"
+          uv pip install --system -e ".[dev,test,doc]"
 
       - name: Update testing docs and check for changes
-        if: steps.check-controller.outputs.controller_changed == 'true'
         id: check-docs-changes
         run: |
           # Store the current state of the documentation file
@@ -59,10 +43,14 @@ jobs:
           # Run the make command to update testing docs
           make update-testing-docs
 
+          if [[ ! -f documentation_testing_before.json || ! -f shiny/pytest/generate/data/docs/documentation_testing.json ]]; then
+          echo "One or both documentation files are missing."
+          exit 1
+          fi
+
           # Check if the documentation file has changed
           if ! diff -q documentation_testing_before.json shiny/pytest/generate/data/docs/documentation_testing.json > /dev/null 2>&1; then
             echo "docs_changed=true" >> $GITHUB_OUTPUT
-            echo "Documentation file has changed after running make update-testing-docs"
             echo "The generated documentation is out of sync with the current controller changes."
             exit 1
           else
@@ -100,7 +88,7 @@ jobs:
             *This comment was automatically generated by the validate_testing_docs workflow.*
 
       - name: Remove comment when no controller changes or docs are up to date
-        if: steps.check-controller.outputs.controller_changed == 'false' || (steps.check-controller.outputs.controller_changed == 'true' && steps.check-docs-changes.outputs.docs_changed == 'false')
+        if: steps.check-docs-changes.outputs.docs_changed == 'false'
         uses: marocchino/sticky-pull-request-comment@v2
         with:
           header: testing-docs-update

shiny/pytest/generate/data/prompts/SYSTEM_PROMPT_testing.md

@@ -7,7 +7,14 @@ For non-Shiny Python code, respond: "This framework is for Shiny for Python only
 
 ## Core Rules
 
-1. **Dynamic App File**: Use exact filename from prompt in `create_app_fixture(["filename.py"])`
+1. **Dynamic App File**: When generating code that uses `create_app_fixture`, follow these rules:
+   - Use the exact filename provided in the prompt.
+   - If the test file is under `app_dir/tests`, make the app path relative to the tests directory.
+
+   - ✅ `app = create_app_fixture(["../app.py"])`
+   - ❌ `app = create_app_fixture(["app.py"])`
+
+   - If the provided filename is in a different path, adjust the path accordingly while keeping it relative.
 
 2. **Controller Classes Only**: Always use official controllers, never `page.locator()`
    - ✅ `controller.InputSlider(page, "my_slider")`
@@ -33,11 +40,11 @@ For non-Shiny Python code, respond: "This framework is for Shiny for Python only
 
 8. **Skip plots**: Do not test plot content or functionality i.e. using OutputPlot controller.
 
-9. **Keyword-Only Args**: Always pass every argument as a keyword for every controller method.
+9.  **Keyword-Only Args**: Always pass every argument as a keyword for every controller method.
    - ✅  `expect_cell(value="0", row=1, col=2)`
    - ❌  `expect_cell("0", 1, 2)`
 
-10. **Newline at End**: Always end files with a newline.
+10.  **Newline at End**: Always end files with a newline.
 
 ## Examples
 

shiny/pytest/generate/main.py

@@ -1,5 +1,6 @@
 import importlib.resources
 import logging
+import os
 import re
 import sys
 from dataclasses import dataclass
@@ -219,6 +220,44 @@ def extract_test(self, response: str) -> str:
         match = self.CODE_PATTERN.search(response)
         return match.group(1).strip() if match else ""
 
+    def _compute_relative_app_path(
+        self, app_file_path: Path, test_file_path: Path
+    ) -> str:
+        """Compute POSIX-style relative path from the test file directory to the app file."""
+        rel = os.path.relpath(str(app_file_path), start=str(test_file_path.parent))
+        return Path(rel).as_posix()
+
+    def _rewrite_fixture_path(self, test_code: str, relative_app_path: str) -> str:
+        """Rewrite create_app_fixture path to be relative to the test file directory.
+
+        Handles common patterns like:
+        - create_app_fixture(["app.py"]) -> create_app_fixture(["../app.py"]) (or appropriate)
+        - create_app_fixture("app.py") -> create_app_fixture("../app.py")
+        Keeps other arguments intact if present.
+        """
+        # Pattern for list form: create_app_fixture(["app.py"]) or with spaces
+        pattern_list = re.compile(
+            r"(create_app_fixture\(\s*\[\s*)(['\"])([^'\"]+)(\2)(\s*)([,\]])",
+            re.DOTALL,
+        )
+
+        def repl_list(m: re.Match) -> str:
+            return f"{m.group(1)}{m.group(2)}{relative_app_path}{m.group(2)}{m.group(5)}{m.group(6)}"
+
+        new_code, _ = pattern_list.subn(repl_list, test_code)
+
+        # Pattern for direct string form: create_app_fixture("app.py")
+        pattern_str = re.compile(
+            r"(create_app_fixture\(\s*)(['\"])([^'\"]+)(\2)(\s*)([,\)])",
+            re.DOTALL,
+        )
+
+        def repl_str(m: re.Match) -> str:
+            return f"{m.group(1)}{m.group(2)}{relative_app_path}{m.group(2)}{m.group(5)}{m.group(6)}"
+
+        new_code2, _ = pattern_str.subn(repl_str, new_code)
+        return new_code2
+
     def _create_test_prompt(self, app_text: str, app_file_name: str) -> str:
         """Create test generation prompt with app file name"""
         return (
@@ -227,8 +266,10 @@ def _create_test_prompt(self, app_text: str, app_file_name: str) -> str:
             "Do not add tests for ones that do not have an existing ids since controllers need IDs to locate elements.\n"
             "and server functionality of this app. Include appropriate assertions \n"
             "and test cases to verify the app's behavior.\n"
-            f"IMPORTANT: Use the exact app file name '{app_file_name}' in the create_app_fixture call like this:\n"
-            f'app = create_app_fixture(["{app_file_name}"])\n'
+            "IMPORTANT: In the create_app_fixture call, pass a RELATIVE path from the test file's directory to the app file.\n"
+            "For example, if the test lives under a 'tests/' subfolder next to the app file, use '../"
+            + app_file_name
+            + "'. Do not use absolute paths.\n"
             "IMPORTANT: Only output the Python test code in a single code block. Do not include any explanation, justification, or extra text."
         )
 
@@ -315,6 +356,14 @@ def generate_test(
                 inferred_app_path, output_dir_path
             )
 
+        try:
+            relative_app_path = self._compute_relative_app_path(
+                inferred_app_path, test_file_path
+            )
+            test_code = self._rewrite_fixture_path(test_code, relative_app_path)
+        except Exception:
+            pass
+
         test_file_path.parent.mkdir(parents=True, exist_ok=True)
         test_file_path.write_text(test_code, encoding="utf-8")