Jt playwrite

CrestApps.OrchardCore.slnx

@@ -49,6 +49,9 @@
     <Project Path="src/Modules/CrestApps.OrchardCore.AI.Mcp.Resources.Sftp/CrestApps.OrchardCore.AI.Mcp.Resources.Sftp.csproj" />
     <Project Path="src/Modules/CrestApps.OrchardCore.AI.A2A/CrestApps.OrchardCore.AI.A2A.csproj" />
     <Project Path="src/Modules/CrestApps.OrchardCore.AI.Mcp/CrestApps.OrchardCore.AI.Mcp.csproj" />
+    <Project Path="src/Modules/CrestApps.OrchardCore.AI.Playwright/CrestApps.OrchardCore.AI.Playwright.csproj">
+      <Build Solution="Debug|*" Project="false" />
+    </Project>
     <Project Path="src/Modules/CrestApps.OrchardCore.AI.Prompting/CrestApps.OrchardCore.AI.Prompting.csproj" />
     <Project Path="src/Modules/CrestApps.OrchardCore.AI/CrestApps.OrchardCore.AI.csproj" />
     <Project Path="src/Modules/CrestApps.OrchardCore.AzureAIInference/CrestApps.OrchardCore.AzureAIInference.csproj" />

Directory.Packages.props

@@ -7,6 +7,7 @@
   </PropertyGroup>
   <ItemGroup>
     <!-- Miscellaneous Packages -->
+    <PackageVersion Include="Microsoft.Playwright" Version="1.58.0" />
     <PackageVersion Include="CrestApps.AgentSkills.OrchardCore" Version="1.0.0-preview-0002" />
     <PackageVersion Include="CrestApps.AgentSkills.Mcp.OrchardCore" Version="1.0.0-preview-0002" />
     <PackageVersion Include="DocumentFormat.OpenXml" Version="3.4.1" />

src/CrestApps.OrchardCore.Documentations/docs/ai/index.md

@@ -41,6 +41,7 @@ The **Orchestrator** (`IOrchestrator`) is the central runtime that manages AI co
 | [AI Chat](chat) | Admin and frontend chat interfaces |
 | [AI Chat Interactions](chat-interactions) | Ad-hoc chat with configurable parameters and document upload |
 | [AI Prompt Templates](prompt-templates) | Centralized prompt management with Liquid templates and file-based discovery |
+| [AI Playwright Browser Automation](playwright) | Persistent browser workflows and live page observation for Orchard Core admin |
 | [Consuming AI Services](consuming-ai-services) | Using AI services programmatically via code |
 | [Copilot Integration](copilot) | GitHub Copilot SDK-based orchestration |
 | [Data Sources](data-sources/) | Retrieval-augmented generation (RAG) / knowledge base indexing and vector search |

src/CrestApps.OrchardCore.Documentations/docs/ai/playwright.md

@@ -0,0 +1,109 @@
+---
+sidebar_label: Playwright
+sidebar_position: 11
+title: AI Playwright Browser Automation
+description: Configure CrestApps.OrchardCore.AI.Playwright for multi-step browser workflows and interactive page observation in Orchard Core admin.
+---
+
+# AI Playwright Browser Automation
+
+The `CrestApps.OrchardCore.AI.Playwright` module gives an AI profile a dedicated browser that can work inside Orchard Core admin and stay available for follow-up tasks.
+
+## What It Does
+
+- Opens a dedicated Playwright browser window for the current AI profile
+- Reuses that browser across follow-up tasks instead of closing it after every answer
+- Keeps its own sign-in state and can use the saved Playwright login when the admin login page appears
+- Lets the AI inspect the live page before answering questions such as "do you see the HtmlBody widget?"
+- Lets the AI continue from the current page when you ask for another task
+- Shows a visible AI target indicator in the browser before clicks and typing so users can see what the operator is about to act on
+
+## Session Behavior
+
+When Playwright is enabled on an AI profile:
+
+- The browser stays open after a task completes
+- The assistant is expected to ask whether you have another task for the same browser session
+- If you continue, the next task reuses the same browser and current page context
+- If you are done, the browser remains available until the inactivity timeout closes it naturally
+
+The session timeout follows the AI profile session inactivity timeout when the profile provides one. Otherwise, Playwright uses a 30 minute inactivity timeout.
+
+## Interactive Page Guidance
+
+The Playwright operator supports live page observation, not just one-shot execution.
+
+Examples:
+
+- `Do you see the HtmlBody widget?`
+- `What widgets do you see on this page?`
+- `Can you edit it?`
+- `Take a screenshot of the current page.`
+
+When the user asks an observation question, the operator should inspect the live page first and answer from the current browser state instead of guessing from memory.
+
+## Built-In OrchardCore Tools
+
+The Playwright operator now exposes OrchardCore-specific tools for admin workflows and live-page follow-up work:
+
+| Tool | Purpose |
+| --- | --- |
+| `playwright_capture_state` | Capture the current URL, title, heading, toast, validation messages, and visible buttons |
+| `playwright_open_content_items` | Open the Orchard content items list |
+| `playwright_list_content_items` | List visible Orchard content items from the current content items screen |
+| `playwright_open_content_item_editor` | Open an existing Orchard content item editor by title using the current list first |
+| `playwright_open_editor_tab` | Open an Orchard editor tab, summary, or section by name |
+| `playwright_set_content_title` | Set the Orchard title field |
+| `playwright_set_field_value` | Update typed Orchard fields such as text, textarea, select, and checkbox |
+| `playwright_set_body_field` | Update HtmlBody and other body-like fields with append or replace behavior |
+| `playwright_save_draft` | Save the current Orchard content item as a draft |
+| `playwright_publish_content` | Publish the current Orchard content item and return verification evidence |
+| `playwright_get_page_content` | Return the visible text content of the current page |
+| `playwright_find_element` | Find visible elements that match a text snippet, label, or widget name |
+| `playwright_check_element_exists` | Check whether a requested widget, control, or text snippet is visible |
+| `playwright_get_visible_widgets` | List visible widget-like cards, headings, and editor sections |
+| `playwright_take_screenshot` | Save a screenshot of the current page and return the saved file path |
+
+## Content Editing Reliability
+
+The Orchard admin automation now uses content-aware editing behavior instead of generic browser actions:
+
+- When the assistant needs to edit an existing content item from the content list, it scopes the action to the row that contains the requested title before clicking `Edit`
+- If one visible content item is already the clear match, the assistant should open it directly instead of asking for extra confirmation
+- When the assistant is already on the content items screen, follow-up requests should continue from that screen instead of restarting from admin home
+- When you ask to list content items, the assistant should list the visible titles directly instead of asking whether it should list them
+- When you ask to open an existing content item, the assistant can use the current list and a best-match title search before falling back to broader navigation
+- When a field is hidden behind an Orchard editor tab or section, the assistant should open that tab or section before declaring the field missing
+- Typed fields such as text, textarea, select, and checkbox are handled through Orchard-specific field tools instead of a generic fill tool
+- When the assistant fills `HtmlBody` or other body-like fields, it checks whether the live editor is a textarea, rich text surface, contenteditable region, TinyMCE instance, or iframe editor
+- For body-like fields, the assistant supports both append and replace behavior explicitly
+- Field edits do not automatically save or publish. The assistant should pause after editing unless you explicitly ask it to save or publish.
+- Publish results are only reported as successful after the tool returns verification signals from the resulting page state
+- If Orchard returns an application error such as `database is locked`, the assistant should report that exact app failure instead of pretending the browser action succeeded
+- If a retry is needed, the assistant should summarize the final verified state instead of sending conflicting success and failure messages
+
+## Configuration
+
+In the AI Profile editor, keep the Playwright configuration simple:
+
+- Enable Playwright browser automation
+- Provide the Playwright login username and password if the browser should sign in automatically when it reaches Orchard admin login
+- Optionally enable publish-by-default if you want task completion to publish unless you explicitly ask for a draft
+
+The browser interaction style is fixed to the dedicated Playwright browser workflow. End users do not need to choose browser modes.
+
+## Multi-Step Workflow Example
+
+1. `Create a new SitePage titled "Launch Draft".`
+2. The assistant completes the task and asks whether there is another task for the same browser session.
+3. You reply: `Yes. Open the Content tab and update HtmlBody.`
+4. The assistant opens the Orchard editor tab, updates the body field, and pauses until you ask to save or publish.
+5. You reply: `Publish it.`
+6. The assistant publishes the item and reports the final verified publish state.
+
+## Notes
+
+- The operator should prefer Orchard-specific tools instead of exposing generic browser click or fill tools to the AI.
+- Observation questions should use the inspection tools before answering.
+- The browser is not closed automatically after every successful task anymore.
+- The browser window now shows a visible indicator for click and typing attempts to make the live automation easier to follow.

src/CrestApps.OrchardCore.Documentations/docs/changelog/v2.0.0.md

@@ -191,14 +191,30 @@ A new suite of modules for multi-channel communication:
 
 ## Improvements
 
+### AI Playwright Browser Automation (`CrestApps.OrchardCore.AI.Playwright`)
+
+- **Persistent multi-step browser sessions** — Playwright browser windows now remain open across follow-up tasks instead of closing immediately after each completed assistant reply.
+- **Inactivity-based cleanup** — Inactive Playwright browser sessions are now closed by a background cleanup task using the configured profile session timeout when available, or a 30 minute default timeout.
+- **Interactive page observation tools** — Added deterministic live-page tools for `playwright_get_page_content`, `playwright_find_element`, `playwright_check_element_exists`, `playwright_get_visible_widgets`, and `playwright_take_screenshot`.
+- **Interactive completion flow** — The Playwright operator prompt now tells the assistant to ask whether the user has another task after each completed workflow and to answer page-observation questions from the live browser state instead of guessing.
+
+- **Row-scoped content list editing** — Content item list actions now resolve the row that contains the requested title before invoking row-level actions such as `Edit`, avoiding misclicks when multiple items are visible.
+- **Smart `HtmlBody` editor detection** — `playwright_fill_by_label` now detects visible textareas, hidden source textareas, rich text surfaces, contenteditable editors, and iframe-backed editors for fields such as `HtmlBody`.
+- **Append-first body editing** — Body-like fields now append new text by default instead of overwriting existing content unless the user explicitly asks to replace the content.
+- **Stricter completion reporting** — The Playwright operator prompt now instructs the assistant to avoid conflicting success and failure statements in the same reply and to report save or publish success only from the latest verified page observation.
+- **Visible content list skills** — Added `playwright_list_content_items` and `playwright_open_content_item_editor` so the assistant can stay on the current Orchard content list, report visible items directly, and open an editor by title without restarting from admin home.
+
+- **Visible action indicator** — The dedicated Playwright browser now shows an in-page highlight ring and label before clicks and typing so users can see what the operator is targeting.
+- **Current-page-first editing flow** — The Playwright operator now opens a clear visible content-item match directly, stays on the current page for follow-up requests, and avoids auto-saving after a field edit unless the user explicitly asks.
+- **Clearer Orchard failure reporting** — Save and publish flows now surface application error headlines such as `database is locked` from the returned Orchard page state, and Playwright tool failures are returned as structured results instead of raw tool exceptions.
+
 ### Configuration-Based AI Deployments
 
 - **appsettings.json deployments** — AI deployments defined in configuration (both connection-based `Deployments` arrays and the new standalone `CrestApps_AI:Deployments` array) are now automatically available at runtime. This allows site owners to define deployments in `appsettings.json` so they are shared across all tenants without per-tenant configuration.
 - **Non-connection deployments via config** — Contained-connection providers (e.g., Azure Speech) can now be configured in `appsettings.json` as standalone deployment objects with a `ProviderName`, enabling speech-to-text and other contained-connection deployments to be defined centrally rather than through the admin UI on each tenant.
 - **Read-only and ephemeral** — Configuration-sourced deployments appear in dropdown menus and API queries alongside database-managed deployments, but are not persisted to the database. Removing them from configuration removes them from the system.
 - **Deterministic IDs** — Configuration deployments receive stable, deterministic IDs derived from their provider, connection, and deployment names, ensuring consistent references across application restarts.
 
----
 
 ### MCP Client Authentication (`CrestApps.OrchardCore.AI.Mcp`)
 
@@ -213,8 +229,6 @@ A new suite of modules for multi-channel communication:
 
 **Breaking:** `IMcpClientTransportProvider.Get()` has been renamed to `GetAsync()` and now returns `Task<IClientTransport>` instead of `IClientTransport`. Custom transport provider implementations must update their method signature.
 
----
-
 ### Unified Citation & Reference System
 
 The citation and reference system has been completely reworked so that **every AI provider** (Azure OpenAI, OpenAI, Ollama, Azure AI Inference) now returns the same citation references. Previously, citations only worked with Azure OpenAI's native data-sources feature (`GetMessageContext()`); since we now inject context ourselves via preemptive RAG and tool-based search, that approach no longer applied.