diff --git a/CrestApps.OrchardCore.slnx b/CrestApps.OrchardCore.slnx index 82b357632..4a3dbeda2 100644 --- a/CrestApps.OrchardCore.slnx +++ b/CrestApps.OrchardCore.slnx @@ -49,6 +49,9 @@ + + + diff --git a/Directory.Packages.props b/Directory.Packages.props index a51356514..e7ab1052d 100644 --- a/Directory.Packages.props +++ b/Directory.Packages.props @@ -7,6 +7,7 @@ + diff --git a/src/CrestApps.OrchardCore.Documentations/docs/ai/index.md b/src/CrestApps.OrchardCore.Documentations/docs/ai/index.md index 82c783721..fc351ed55 100644 --- a/src/CrestApps.OrchardCore.Documentations/docs/ai/index.md +++ b/src/CrestApps.OrchardCore.Documentations/docs/ai/index.md @@ -41,6 +41,7 @@ The **Orchestrator** (`IOrchestrator`) is the central runtime that manages AI co | [AI Chat](chat) | Admin and frontend chat interfaces | | [AI Chat Interactions](chat-interactions) | Ad-hoc chat with configurable parameters and document upload | | [AI Prompt Templates](prompt-templates) | Centralized prompt management with Liquid templates and file-based discovery | +| [AI Playwright Browser Automation](playwright) | Persistent browser workflows and live page observation for Orchard Core admin | | [Consuming AI Services](consuming-ai-services) | Using AI services programmatically via code | | [Copilot Integration](copilot) | GitHub Copilot SDK-based orchestration | | [Data Sources](data-sources/) | Retrieval-augmented generation (RAG) / knowledge base indexing and vector search | diff --git a/src/CrestApps.OrchardCore.Documentations/docs/ai/playwright.md b/src/CrestApps.OrchardCore.Documentations/docs/ai/playwright.md new file mode 100644 index 000000000..d31ec8ec7 --- /dev/null +++ b/src/CrestApps.OrchardCore.Documentations/docs/ai/playwright.md @@ -0,0 +1,109 @@ +--- +sidebar_label: Playwright +sidebar_position: 11 +title: AI Playwright Browser Automation +description: Configure CrestApps.OrchardCore.AI.Playwright for multi-step browser workflows and interactive page observation in Orchard Core admin. +--- + +# AI Playwright Browser Automation + +The `CrestApps.OrchardCore.AI.Playwright` module gives an AI profile a dedicated browser that can work inside Orchard Core admin and stay available for follow-up tasks. + +## What It Does + +- Opens a dedicated Playwright browser window for the current AI profile +- Reuses that browser across follow-up tasks instead of closing it after every answer +- Keeps its own sign-in state and can use the saved Playwright login when the admin login page appears +- Lets the AI inspect the live page before answering questions such as "do you see the HtmlBody widget?" +- Lets the AI continue from the current page when you ask for another task +- Shows a visible AI target indicator in the browser before clicks and typing so users can see what the operator is about to act on + +## Session Behavior + +When Playwright is enabled on an AI profile: + +- The browser stays open after a task completes +- The assistant is expected to ask whether you have another task for the same browser session +- If you continue, the next task reuses the same browser and current page context +- If you are done, the browser remains available until the inactivity timeout closes it naturally + +The session timeout follows the AI profile session inactivity timeout when the profile provides one. Otherwise, Playwright uses a 30 minute inactivity timeout. + +## Interactive Page Guidance + +The Playwright operator supports live page observation, not just one-shot execution. + +Examples: + +- `Do you see the HtmlBody widget?` +- `What widgets do you see on this page?` +- `Can you edit it?` +- `Take a screenshot of the current page.` + +When the user asks an observation question, the operator should inspect the live page first and answer from the current browser state instead of guessing from memory. + +## Built-In OrchardCore Tools + +The Playwright operator now exposes OrchardCore-specific tools for admin workflows and live-page follow-up work: + +| Tool | Purpose | +| --- | --- | +| `playwright_capture_state` | Capture the current URL, title, heading, toast, validation messages, and visible buttons | +| `playwright_open_content_items` | Open the Orchard content items list | +| `playwright_list_content_items` | List visible Orchard content items from the current content items screen | +| `playwright_open_content_item_editor` | Open an existing Orchard content item editor by title using the current list first | +| `playwright_open_editor_tab` | Open an Orchard editor tab, summary, or section by name | +| `playwright_set_content_title` | Set the Orchard title field | +| `playwright_set_field_value` | Update typed Orchard fields such as text, textarea, select, and checkbox | +| `playwright_set_body_field` | Update HtmlBody and other body-like fields with append or replace behavior | +| `playwright_save_draft` | Save the current Orchard content item as a draft | +| `playwright_publish_content` | Publish the current Orchard content item and return verification evidence | +| `playwright_get_page_content` | Return the visible text content of the current page | +| `playwright_find_element` | Find visible elements that match a text snippet, label, or widget name | +| `playwright_check_element_exists` | Check whether a requested widget, control, or text snippet is visible | +| `playwright_get_visible_widgets` | List visible widget-like cards, headings, and editor sections | +| `playwright_take_screenshot` | Save a screenshot of the current page and return the saved file path | + +## Content Editing Reliability + +The Orchard admin automation now uses content-aware editing behavior instead of generic browser actions: + +- When the assistant needs to edit an existing content item from the content list, it scopes the action to the row that contains the requested title before clicking `Edit` +- If one visible content item is already the clear match, the assistant should open it directly instead of asking for extra confirmation +- When the assistant is already on the content items screen, follow-up requests should continue from that screen instead of restarting from admin home +- When you ask to list content items, the assistant should list the visible titles directly instead of asking whether it should list them +- When you ask to open an existing content item, the assistant can use the current list and a best-match title search before falling back to broader navigation +- When a field is hidden behind an Orchard editor tab or section, the assistant should open that tab or section before declaring the field missing +- Typed fields such as text, textarea, select, and checkbox are handled through Orchard-specific field tools instead of a generic fill tool +- When the assistant fills `HtmlBody` or other body-like fields, it checks whether the live editor is a textarea, rich text surface, contenteditable region, TinyMCE instance, or iframe editor +- For body-like fields, the assistant supports both append and replace behavior explicitly +- Field edits do not automatically save or publish. The assistant should pause after editing unless you explicitly ask it to save or publish. +- Publish results are only reported as successful after the tool returns verification signals from the resulting page state +- If Orchard returns an application error such as `database is locked`, the assistant should report that exact app failure instead of pretending the browser action succeeded +- If a retry is needed, the assistant should summarize the final verified state instead of sending conflicting success and failure messages + +## Configuration + +In the AI Profile editor, keep the Playwright configuration simple: + +- Enable Playwright browser automation +- Provide the Playwright login username and password if the browser should sign in automatically when it reaches Orchard admin login +- Optionally enable publish-by-default if you want task completion to publish unless you explicitly ask for a draft + +The browser interaction style is fixed to the dedicated Playwright browser workflow. End users do not need to choose browser modes. + +## Multi-Step Workflow Example + +1. `Create a new SitePage titled "Launch Draft".` +2. The assistant completes the task and asks whether there is another task for the same browser session. +3. You reply: `Yes. Open the Content tab and update HtmlBody.` +4. The assistant opens the Orchard editor tab, updates the body field, and pauses until you ask to save or publish. +5. You reply: `Publish it.` +6. The assistant publishes the item and reports the final verified publish state. + +## Notes + +- The operator should prefer Orchard-specific tools instead of exposing generic browser click or fill tools to the AI. +- Observation questions should use the inspection tools before answering. +- The browser is not closed automatically after every successful task anymore. +- The browser window now shows a visible indicator for click and typing attempts to make the live automation easier to follow. diff --git a/src/CrestApps.OrchardCore.Documentations/docs/changelog/v2.0.0.md b/src/CrestApps.OrchardCore.Documentations/docs/changelog/v2.0.0.md index 0eaab2138..2a7abbaf7 100644 --- a/src/CrestApps.OrchardCore.Documentations/docs/changelog/v2.0.0.md +++ b/src/CrestApps.OrchardCore.Documentations/docs/changelog/v2.0.0.md @@ -191,6 +191,23 @@ A new suite of modules for multi-channel communication: ## Improvements +### AI Playwright Browser Automation (`CrestApps.OrchardCore.AI.Playwright`) + +- **Persistent multi-step browser sessions** — Playwright browser windows now remain open across follow-up tasks instead of closing immediately after each completed assistant reply. +- **Inactivity-based cleanup** — Inactive Playwright browser sessions are now closed by a background cleanup task using the configured profile session timeout when available, or a 30 minute default timeout. +- **Interactive page observation tools** — Added deterministic live-page tools for `playwright_get_page_content`, `playwright_find_element`, `playwright_check_element_exists`, `playwright_get_visible_widgets`, and `playwright_take_screenshot`. +- **Interactive completion flow** — The Playwright operator prompt now tells the assistant to ask whether the user has another task after each completed workflow and to answer page-observation questions from the live browser state instead of guessing. + +- **Row-scoped content list editing** — Content item list actions now resolve the row that contains the requested title before invoking row-level actions such as `Edit`, avoiding misclicks when multiple items are visible. +- **Smart `HtmlBody` editor detection** — `playwright_fill_by_label` now detects visible textareas, hidden source textareas, rich text surfaces, contenteditable editors, and iframe-backed editors for fields such as `HtmlBody`. +- **Append-first body editing** — Body-like fields now append new text by default instead of overwriting existing content unless the user explicitly asks to replace the content. +- **Stricter completion reporting** — The Playwright operator prompt now instructs the assistant to avoid conflicting success and failure statements in the same reply and to report save or publish success only from the latest verified page observation. +- **Visible content list skills** — Added `playwright_list_content_items` and `playwright_open_content_item_editor` so the assistant can stay on the current Orchard content list, report visible items directly, and open an editor by title without restarting from admin home. + +- **Visible action indicator** — The dedicated Playwright browser now shows an in-page highlight ring and label before clicks and typing so users can see what the operator is targeting. +- **Current-page-first editing flow** — The Playwright operator now opens a clear visible content-item match directly, stays on the current page for follow-up requests, and avoids auto-saving after a field edit unless the user explicitly asks. +- **Clearer Orchard failure reporting** — Save and publish flows now surface application error headlines such as `database is locked` from the returned Orchard page state, and Playwright tool failures are returned as structured results instead of raw tool exceptions. + ### Configuration-Based AI Deployments - **appsettings.json deployments** — AI deployments defined in configuration (both connection-based `Deployments` arrays and the new standalone `CrestApps_AI:Deployments` array) are now automatically available at runtime. This allows site owners to define deployments in `appsettings.json` so they are shared across all tenants without per-tenant configuration. @@ -198,7 +215,6 @@ A new suite of modules for multi-channel communication: - **Read-only and ephemeral** — Configuration-sourced deployments appear in dropdown menus and API queries alongside database-managed deployments, but are not persisted to the database. Removing them from configuration removes them from the system. - **Deterministic IDs** — Configuration deployments receive stable, deterministic IDs derived from their provider, connection, and deployment names, ensuring consistent references across application restarts. ---- ### MCP Client Authentication (`CrestApps.OrchardCore.AI.Mcp`) @@ -213,8 +229,6 @@ A new suite of modules for multi-channel communication: **Breaking:** `IMcpClientTransportProvider.Get()` has been renamed to `GetAsync()` and now returns `Task` instead of `IClientTransport`. Custom transport provider implementations must update their method signature. ---- - ### Unified Citation & Reference System The citation and reference system has been completely reworked so that **every AI provider** (Azure OpenAI, OpenAI, Ollama, Azure AI Inference) now returns the same citation references. Previously, citations only worked with Azure OpenAI's native data-sources feature (`GetMessageContext()`); since we now inject context ourselves via preemptive RAG and tool-based search, that approach no longer applied. diff --git a/src/Modules/CrestApps.OrchardCore.AI.Playwright/AITemplates/Prompts/playwright-operator.md b/src/Modules/CrestApps.OrchardCore.AI.Playwright/AITemplates/Prompts/playwright-operator.md new file mode 100644 index 000000000..e4466af91 --- /dev/null +++ b/src/Modules/CrestApps.OrchardCore.AI.Playwright/AITemplates/Prompts/playwright-operator.md @@ -0,0 +1,146 @@ +--- +Title: Playwright Operator +Description: Deterministic OrchardCore admin browser guidance for Playwright-enabled sessions. +Parameters: + - baseUrl: string + - adminBaseUrl: string + - publishBehavior: string +IsListable: false +Category: Orchestration +--- + +You are the CrestApps OrchardCore Playwright operator. + +Your job is to complete OrchardCore admin workflows safely, deterministically, and with OrchardCore-specific reasoning. +Do not treat OrchardCore admin like a generic website. +Treat it like a structured CMS editor with content lists, editor tabs, Orchard fields, widget containers, publish workflows, and verification rules. + +Current Playwright session: +- Browser session: dedicated Playwright browser that keeps its own sign-in state +- Base URL: {{ baseUrl }} +- Admin URL: {{ adminBaseUrl }} +- Default completion behavior: {{ publishBehavior }} +- Session lifecycle: keep the browser open across follow-up tasks until the user is finished or the inactivity timeout closes it naturally + +Operating model: + +1. Plan the next OrchardCore step before calling a tool. +2. First identify the current page type from live evidence whenever possible: login, dashboard, content list, content editor, content definition editor, settings page, or preview page. +3. Prefer the highest-level OrchardCore-specific tool that matches the task. +4. Execute one verified action at a time. +5. After each mutating action, inspect the returned URL, title, heading, toast, validation messages, status indicators, or screenshot evidence before continuing. +6. Continue from the current page whenever possible instead of restarting navigation. +7. Never guess when the live page can be inspected. + +Tool selection order: + +1. Use OrchardCore navigation and content workflow tools first: + - `playwright_capture_state` + - `playwright_open_admin_home` + - `playwright_open_content_items` + - `playwright_list_content_items` + - `playwright_open_content_item_editor` + - `playwright_open_new_content_item` + - `playwright_open_editor_tab` + - `playwright_set_content_title` + - `playwright_set_field_value` + - `playwright_set_body_field` + - `playwright_save_draft` + - `playwright_publish_content` + - `playwright_publish_and_verify` +2. Use Orchard-aware inspection tools next: + - `playwright_get_page_content` + - `playwright_find_element` + - `playwright_check_element_exists` + - `playwright_get_visible_widgets` + - `playwright_take_screenshot` + - `playwright_diagnose_orchard_action` +3. Do not improvise raw browser selectors. If the current Orchard-specific tool surface cannot solve the task safely, stop and say what Orchard-specific tool is missing. + +Selector rules: + +1. Prefer dedicated OrchardCore tools over selectors. +2. Keep selector logic inside Orchard services, not in the AI plan. +3. Scope actions to OrchardCore containers such as the current content row, current editor tab, current fieldset, current card, or current widget container. +4. Do not fall back to raw browser selectors from the prompt. + +OrchardCore workflow skills: + +1. Admin navigation skill + - Use the configured `adminBaseUrl`. + - Never assume the admin prefix is `/Admin`. + - If the current page already contains the needed context, stay on that page. +2. Content list skill + - Treat the content items screen as the authoritative source for visible titles, types, status, and row actions. + - When the user asks to find or edit a specific content item, locate the row containing that title first. + - If one visible match is clearly best, use it directly and state which title was matched. + - If the match is weak, report the closest visible titles instead of editing the wrong row. +3. Content editor skill + - Wait for the content editor to be clearly open before editing. + - Prefer the visible `Title` label first, then Orchard title fallbacks. + - Use `playwright_open_editor_tab` before declaring a tabbed or collapsed field missing. + - Use `playwright_set_field_value` for typed fields. + - Use `playwright_set_body_field` for HtmlBody, Summary, and other rich or body-like editors. + - Changing a field is not the same as saving. +4. Widget and nested editor skill + - OrchardCore widgets may appear as cards, panels, legends, summaries, repeated items, `data-widget-type` surfaces, or heading-based editor sections. + - For `FlowPart`, `BagPart`, and nested widget workflows, identify the parent container first, then the target widget card, then the field inside that widget. + - Expand collapsed sections before claiming the widget is missing. + - If the current tool layer cannot identify the widget deterministically, capture state, list visible widgets, and explain the limitation instead of guessing. +5. Publish and verification skill + - `Save Draft` and `Publish` are different outcomes. + - Do not publish unless the user explicitly asks for publish or the profile is configured to publish by default. + - Prefer `playwright_publish_and_verify` when the task needs proof that publish completed successfully. + - If evidence is weak, take a screenshot or capture state again before reporting success. +6. Screenshot and diagnostics skill + - Use screenshots for debugging, visual confirmation, ambiguous layouts, and multi-step checkpoints. + - Use page content and element inspection for grounded answers to live UI questions. + - Report the verified final state only. + +Rules: + +1. Prefer Orchard-specific Playwright skills over generic browser actions. +2. When the current page is unclear, capture the current state before making a risky move. +3. Execute one verified step at a time. +4. Trust the configured URLs and the current observation, not assumptions. +5. Prefer these Orchard admin flows: + - ensure admin home is available + - open content items + - open an existing content item editor by title + - open a new content item for the requested content type + - open an editor tab or section before editing hidden fields + - set the content title + - set typed field values + - set body-like field values with append or replace semantics + - save draft or publish according to the user's request +6. When the user asks to find the `Edit` button for a specific item, identify the correct row first and use that row's action. +7. When the user asks to find `Publish`, `Save`, `Save Draft`, or `Preview`, search the current editor state first and scope the action to the visible editor surface. +8. If the user did not explicitly ask to publish, do not publish unless the profile is configured to publish by default. When in doubt, save a draft. +9. If login is still required after saved credentials were attempted, stop and explain what happened instead of guessing. +10. If validation errors, blocking messages, or Orchard application errors appear, summarize them and stop instead of guessing. +11. Do not close the browser just because one task is complete. +12. After a task is successfully completed, explicitly ask whether the user has another task for the same browser session or wants to stop. +13. When the user asks interactive observation questions such as `do you see X?`, `what widgets do you see?`, `can you check whether Y is visible?`, or `did that save?`, inspect the live page first using the observation tools. +14. If the page does not clearly show the requested widget, control, or definition, say exactly that and summarize the closest visible matches. +15. When filling body-like fields such as `HtmlBody`, respect append versus replace mode explicitly. +16. When a field is likely hidden behind a tab, accordion, card, or section, open that container before declaring the field missing. +17. When the task involves `FlowPart`, `BagPart`, widgets, or nested content structures, keep every action scoped to the nearest visible parent container. +18. Do not save or publish after editing unless the user explicitly asked for save or publish, or the task clearly requires completion and the profile says to publish by default. +19. If the user says `pause`, stop after the requested action is complete and ask what they want to do next from the current page. +20. Do not report conflicting outcomes in the same answer. After a failed step or retry, summarize only the final verified state from the latest observation or the latest tool error. +21. Do not claim publish succeeded unless the latest observation or structured verification evidence confirms it. +22. If a tool returns an application or database error such as `database is locked`, say that plainly, explain that Orchard failed the save or publish request, and stop instead of retrying automatically. +23. Follow-up instructions should continue from the current browser page whenever possible. +24. When the user says `list them` or asks what content items are visible, use the content-list tool and report the visible titles directly. +25. When the user gives an approximate title or says `use your best judgment`, choose the best visible or searchable title match, state which title you matched, and continue unless the page evidence is too weak. +26. When the user asks about content types, parts, fields, or definitions and no dedicated tool exists, move cautiously through the admin UI, capture state after each navigation step, and report exactly which definition screen you reached. +27. Use screenshots after important state changes, before risky follow-up debugging, and when the user asks for visual confirmation. +28. When an OrchardCore action such as `Edit`, `Publish Now`, `Save Draft`, `Preview`, `Delete`, `Clone`, or a widget action cannot be located through normal inspection tools, call `playwright_diagnose_orchard_action` with the exact action label before declaring the action missing. Report the structured evidence from the result including the attempted locator strategies, any captured screenshots, and the current page URL and title so the user can understand why the action was not found. + +Success criteria: + +- the requested OrchardCore admin action is completed +- the correct content item, definition, widget, or field was targeted +- the requested title or target value matches exactly +- the final observation confirms completion through URL, heading, toast, status, visible widget state, page content, or screenshot evidence +- after task completion, the assistant asks whether there is another task for the same browser session diff --git a/src/Modules/CrestApps.OrchardCore.AI.Playwright/BackgroundTasks/PlaywrightSessionCleanupBackgroundTask.cs b/src/Modules/CrestApps.OrchardCore.AI.Playwright/BackgroundTasks/PlaywrightSessionCleanupBackgroundTask.cs new file mode 100644 index 000000000..1b44f6936 --- /dev/null +++ b/src/Modules/CrestApps.OrchardCore.AI.Playwright/BackgroundTasks/PlaywrightSessionCleanupBackgroundTask.cs @@ -0,0 +1,24 @@ +using CrestApps.OrchardCore.AI.Playwright.Services; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Logging; +using OrchardCore.BackgroundTasks; + +namespace CrestApps.OrchardCore.AI.Playwright.BackgroundTasks; + +[BackgroundTask( + Title = "Playwright Session Cleanup", + Schedule = "*/5 * * * *", + Description = "Closes inactive Playwright browser sessions after their inactivity timeout elapses.", + LockTimeout = 5_000, + LockExpiration = 300_000)] +public sealed class PlaywrightSessionCleanupBackgroundTask : IBackgroundTask +{ + public async Task DoWorkAsync(IServiceProvider serviceProvider, CancellationToken cancellationToken) + { + var logger = serviceProvider.GetRequiredService>(); + var sessionManager = serviceProvider.GetRequiredService(); + + logger.LogDebug("Running Playwright session cleanup."); + await sessionManager.CloseInactiveSessionsAsync(cancellationToken); + } +} diff --git a/src/Modules/CrestApps.OrchardCore.AI.Playwright/Controllers/PlaywrightSessionApiController.cs b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Controllers/PlaywrightSessionApiController.cs new file mode 100644 index 000000000..850a4f36e --- /dev/null +++ b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Controllers/PlaywrightSessionApiController.cs @@ -0,0 +1,66 @@ +using System.Security.Claims; +using CrestApps.OrchardCore.AI.Playwright.Models; +using CrestApps.OrchardCore.AI.Playwright.Services; +using Microsoft.AspNetCore.Authorization; +using Microsoft.AspNetCore.Mvc; + +namespace CrestApps.OrchardCore.AI.Playwright.Controllers; + +/// +/// Lightweight REST surface used by the admin widget JS to query and control Playwright sessions. +/// All endpoints require the user to be authenticated. +/// +[ApiController] +[Authorize] +[Route("api/playwright")] +public sealed class PlaywrightSessionApiController : ControllerBase +{ + private readonly IPlaywrightSessionManager _sessionManager; + + public PlaywrightSessionApiController(IPlaywrightSessionManager sessionManager) + { + _sessionManager = sessionManager; + } + + [HttpGet("{chatSessionId}/status")] + public IActionResult GetStatus(string chatSessionId) + { + var session = _sessionManager.GetSession(chatSessionId, GetOwnerId()); + + return session == null + ? Ok(PlaywrightStatusResponse.Inactive(chatSessionId)) + : Ok(PlaywrightStatusResponse.FromSession(chatSessionId, session)); + } + + [HttpGet("active")] + public IActionResult GetActiveSessions() + { + var sessions = _sessionManager.GetActiveSessions(GetOwnerId()); + var result = sessions.Select(kvp => PlaywrightStatusResponse.FromSession(kvp.Key, kvp.Value)); + + return Ok(result); + } + + [HttpPost("{chatSessionId}/stop")] + public IActionResult Stop(string chatSessionId) + { + _sessionManager.Stop(chatSessionId, GetOwnerId()); + + var session = _sessionManager.GetSession(chatSessionId, GetOwnerId()); + + return session == null + ? Ok(PlaywrightStatusResponse.Inactive(chatSessionId)) + : Ok(PlaywrightStatusResponse.FromSession(chatSessionId, session)); + } + + [HttpPost("{chatSessionId}/close")] + public async Task Close(string chatSessionId) + { + await _sessionManager.CloseAsync(chatSessionId, GetOwnerId()); + + return Ok(PlaywrightStatusResponse.Inactive(chatSessionId)); + } + + private string GetOwnerId() + => User.FindFirstValue(ClaimTypes.NameIdentifier) ?? User.Identity?.Name; +} diff --git a/src/Modules/CrestApps.OrchardCore.AI.Playwright/CrestApps.OrchardCore.AI.Playwright.csproj b/src/Modules/CrestApps.OrchardCore.AI.Playwright/CrestApps.OrchardCore.AI.Playwright.csproj new file mode 100644 index 000000000..34dfc38e9 --- /dev/null +++ b/src/Modules/CrestApps.OrchardCore.AI.Playwright/CrestApps.OrchardCore.AI.Playwright.csproj @@ -0,0 +1,38 @@ + + + + $(MSBuildProjectName) + true + CrestApps OrchardCore AI Playwright Module + + $(CrestAppsDescription) + + Enables the AI Chat Admin Widget agent to drive a real browser via Playwright, + allowing admins to watch, stop, and interact with browser automation tasks. + + $(PackageTags) OrchardCoreCMS AI Playwright + + + + + + + + + + + + + + + + + + + + + + diff --git a/src/Modules/CrestApps.OrchardCore.AI.Playwright/Drivers/ChatInteractionPlaywrightSettingsDisplayDriver.cs b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Drivers/ChatInteractionPlaywrightSettingsDisplayDriver.cs new file mode 100644 index 000000000..d975ba955 --- /dev/null +++ b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Drivers/ChatInteractionPlaywrightSettingsDisplayDriver.cs @@ -0,0 +1,36 @@ +using CrestApps.OrchardCore.AI.Models; +using CrestApps.OrchardCore.AI.Playwright.Models; +using CrestApps.OrchardCore.AI.Playwright.ViewModels; +using Microsoft.Extensions.Localization; +using OrchardCore.DisplayManagement.Handlers; +using OrchardCore.DisplayManagement.Views; +using OrchardCore.Entities; + +namespace CrestApps.OrchardCore.AI.Playwright.Drivers; + +internal sealed class ChatInteractionPlaywrightSettingsDisplayDriver : DisplayDriver +{ + internal readonly IStringLocalizer S; + + public ChatInteractionPlaywrightSettingsDisplayDriver(IStringLocalizer stringLocalizer) + { + S = stringLocalizer; + } + + public override IDisplayResult Edit(ChatInteraction interaction, BuildEditorContext context) + { + return Initialize("ChatInteractionPlaywrightSettings_Edit", model => + { + var metadata = interaction.As() ?? new PlaywrightSessionMetadata(); + + model.PlaywrightEnabled = metadata.Enabled; + model.PlaywrightUsername = metadata.Username; + model.HasSavedPassword = !string.IsNullOrWhiteSpace(metadata.ProtectedPassword); + model.PlaywrightBaseUrl = metadata.BaseUrl; + model.PlaywrightAdminBaseUrl = metadata.AdminBaseUrl; + model.PlaywrightPersistentProfilePath = metadata.PersistentProfilePath; + model.PlaywrightHeadless = metadata.Headless; + model.PlaywrightPublishByDefault = metadata.PublishByDefault; + }).Location("Parameters:6#Settings;1"); + } +} diff --git a/src/Modules/CrestApps.OrchardCore.AI.Playwright/Drivers/PlaywrightProfileSettingsDisplayDriver.cs b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Drivers/PlaywrightProfileSettingsDisplayDriver.cs new file mode 100644 index 000000000..3d14023e4 --- /dev/null +++ b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Drivers/PlaywrightProfileSettingsDisplayDriver.cs @@ -0,0 +1,81 @@ +using CrestApps.OrchardCore.AI.Models; +using CrestApps.OrchardCore.AI.Playwright.Models; +using CrestApps.OrchardCore.AI.Playwright.Settings; +using CrestApps.OrchardCore.AI.Playwright.ViewModels; +using Microsoft.AspNetCore.DataProtection; +using Microsoft.Extensions.Localization; +using OrchardCore.DisplayManagement.Handlers; +using OrchardCore.DisplayManagement.Views; +using OrchardCore.Entities; + +namespace CrestApps.OrchardCore.AI.Playwright.Drivers; + +/// +/// Adds Playwright configuration to the AI Profile editor. +/// +public class PlaywrightProfileSettingsDisplayDriver : DisplayDriver +{ + private readonly IDataProtectionProvider _dataProtectionProvider; + internal readonly IStringLocalizer S; + + public PlaywrightProfileSettingsDisplayDriver( + IDataProtectionProvider dataProtectionProvider, + IStringLocalizer stringLocalizer) + { + _dataProtectionProvider = dataProtectionProvider; + S = stringLocalizer; + } + + public override IDisplayResult Edit(AIProfile profile, BuildEditorContext context) + { + return Initialize("PlaywrightProfileSettings_Edit", model => + { + var metadata = profile.As() ?? new PlaywrightSessionMetadata(); + var legacySettings = profile.GetSettings(); + + model.Enabled = metadata.Enabled || legacySettings.Enabled; + model.Username = metadata.Username; + model.HasSavedPassword = !string.IsNullOrWhiteSpace(metadata.ProtectedPassword); + model.BaseUrl = metadata.BaseUrl; + model.AdminBaseUrl = metadata.AdminBaseUrl; + model.PersistentProfilePath = metadata.PersistentProfilePath; + model.Headless = metadata.Headless; + model.PublishByDefault = metadata.PublishByDefault; + }).Location("Content:50#PlaywrightBrowserAutomation;10"); + } + + public override async Task UpdateAsync(AIProfile profile, UpdateEditorContext context) + { + var model = new PlaywrightProfileSettingsViewModel(); + await context.Updater.TryUpdateModelAsync(model, Prefix); + + var existingMetadata = profile.As() ?? new PlaywrightSessionMetadata(); + var protector = _dataProtectionProvider.CreateProtector(PlaywrightConstants.ProtectorName); + var protectedPassword = existingMetadata.ProtectedPassword; + + if (!string.IsNullOrWhiteSpace(model.Password)) + { + protectedPassword = protector.Protect(model.Password); + } + + profile.Put(new PlaywrightSessionMetadata + { + Enabled = model.Enabled, + BrowserMode = PlaywrightBrowserMode.PersistentContext, + Username = model.Username?.Trim(), + ProtectedPassword = protectedPassword, + BaseUrl = model.BaseUrl?.Trim(), + AdminBaseUrl = model.AdminBaseUrl?.Trim(), + PersistentProfilePath = model.PersistentProfilePath?.Trim(), + Headless = model.Headless, + PublishByDefault = model.PublishByDefault, + }); + + profile.AlterSettings(settings => + { + settings.Enabled = model.Enabled; + }); + + return Edit(profile, context); + } +} diff --git a/src/Modules/CrestApps.OrchardCore.AI.Playwright/Filters/PlaywrightAdminWidgetFilter.cs b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Filters/PlaywrightAdminWidgetFilter.cs new file mode 100644 index 000000000..600e244e6 --- /dev/null +++ b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Filters/PlaywrightAdminWidgetFilter.cs @@ -0,0 +1,60 @@ +using Microsoft.AspNetCore.Mvc; +using Microsoft.AspNetCore.Mvc.Filters; +using Microsoft.AspNetCore.Mvc.RazorPages; +using Microsoft.Extensions.Options; +using OrchardCore.Admin; +using OrchardCore.DisplayManagement; +using OrchardCore.DisplayManagement.Layout; + +namespace CrestApps.OrchardCore.AI.Playwright.Filters; + +/// +/// Injects the Playwright status bar into every admin page's Footer zone. +/// The bar is hidden by default and shown by JS once a Playwright session becomes active. +/// +public sealed class PlaywrightAdminWidgetFilter : IAsyncResultFilter +{ + private readonly ILayoutAccessor _layoutAccessor; + private readonly IShapeFactory _shapeFactory; + private readonly AdminOptions _adminOptions; + + public PlaywrightAdminWidgetFilter( + ILayoutAccessor layoutAccessor, + IShapeFactory shapeFactory, + IOptions adminOptions) + { + _layoutAccessor = layoutAccessor; + _shapeFactory = shapeFactory; + _adminOptions = adminOptions.Value; + } + + public async Task OnResultExecutionAsync(ResultExecutingContext context, ResultExecutionDelegate next) + { + if (!IsAdminPage(context) || context.HttpContext.User.Identity?.IsAuthenticated != true) + { + await next(); + return; + } + + var shape = await _shapeFactory.CreateAsync("PlaywrightAdminWidget"); + + var layout = await _layoutAccessor.GetLayoutAsync(); + + // Place just after the AI chat widget (which is at "999"). + await layout.Zones["Footer"].AddAsync(shape, "1000"); + + await next(); + } + + private bool IsAdminPage(ResultExecutingContext context) + { + if (context.Result is not (ViewResult or PageResult)) + { + return false; + } + + return context.HttpContext.Request.Path.StartsWithSegments( + '/' + _adminOptions.AdminUrlPrefix, + StringComparison.OrdinalIgnoreCase); + } +} diff --git a/src/Modules/CrestApps.OrchardCore.AI.Playwright/FunctionTools.md b/src/Modules/CrestApps.OrchardCore.AI.Playwright/FunctionTools.md new file mode 100644 index 000000000..5bef81367 --- /dev/null +++ b/src/Modules/CrestApps.OrchardCore.AI.Playwright/FunctionTools.md @@ -0,0 +1,165 @@ +# OrchardCore Playwright Skills and Function Tools + +This file defines the OrchardCore-specific Playwright tool surface for `CrestApps.OrchardCore.AI.Playwright`. + +Keep the layers separate: + +- Skill: the AI playbook for an OrchardCore workflow. +- Tool: the deterministic OrchardCore action exposed to the AI. +- Service: the reusable C# implementation behind the tool. +- Tenant or app: the orchestrator that loads the right OrchardCore skill and tool set for the current session. + +## Operating Principles + +- Treat OrchardCore admin as a structured CMS, not a generic website. +- Expose OrchardCore-specific tools only. +- Keep generic browser click and fill behavior inside services, not in the public AI tool surface. +- Verify every mutating step with page state, toast, heading, URL, status, or screenshot evidence. +- Scope actions to OrchardCore structures such as content rows, editor tabs, fieldsets, cards, summaries, and widget containers. +- Continue from the current page whenever possible. + +## Current Tool Architecture + +These are the OrchardCore tools that should remain exposed to the AI. + +| Tool | Primary C# service | Use for | Verify after call | +| --- | --- | --- | --- | +| `playwright_capture_state` | `IOrchardAdminPlaywrightService.CaptureStateAsync()` | Ground the next step from the live page before risky actions | URL, title, heading, toast, validation messages, visible buttons | +| `playwright_open_admin_home` | `IOrchardAdminPlaywrightService.OpenAdminHomeAsync()` | Ensure the session is inside the correct Orchard admin shell | Admin URL, authenticated state, admin heading | +| `playwright_open_content_items` | `IOrchardAdminPlaywrightService.OpenContentItemsAsync()` | Move to the content items list without guessing routes | Content list heading and URL | +| `playwright_list_content_items` | `IOrchardAdminPlaywrightService.ListVisibleContentItemsAsync()` | Inspect visible content rows, titles, types, status, and edit availability | Returned titles, status text, content type, item count | +| `playwright_open_content_item_editor` | `IOrchardAdminPlaywrightService.OpenContentItemEditorAsync()` | Open the correct content item by title using row-scoped actions | Matched title, editor URL, editor heading, post-open state | +| `playwright_open_new_content_item` | `IOrchardAdminPlaywrightService.OpenNewContentItemAsync()` | Start the create flow for a content type from the admin list | Editor heading and title field visibility | +| `playwright_open_editor_tab` | `IOrchardAdminPlaywrightService.OpenEditorTabAsync()` | Open Orchard editor tabs, accordion sections, summaries, and visible editor panels by name | Matched tab or section and resulting editor state | +| `playwright_set_content_title` | `IOrchardAdminPlaywrightService.SetContentTitleAsync()` | Set the Orchard title field reliably | Updated editor state and visible title input | +| `playwright_set_field_value` | `IOrchardAdminPlaywrightService.SetFieldValueAsync()` | Update typed Orchard fields such as text, textarea, select, and checkbox | Field result plus updated observation | +| `playwright_set_body_field` | `IOrchardAdminPlaywrightService.SetBodyFieldAsync()` | Update body-like Orchard fields with append or replace semantics | Resolved editor type and updated observation | +| `playwright_save_draft` | `IOrchardAdminPlaywrightService.SaveDraftAsync()` | Save without publishing | Toast, URL, heading, status message | +| `playwright_publish_content` | `IOrchardAdminPlaywrightService.PublishContentAsync()` | Publish the current content item when the user asked for publish | Toast, URL, heading, and post-publish page state | +| `playwright_publish_and_verify` | `IOrchardAdminPlaywrightService.PublishAndVerifyAsync()` | Publish the current content item and return structured Orchard verification evidence | Verification signals and final observation | +| `playwright_get_page_content` | `IPlaywrightPageInspectionService.GetPageContentAsync()` | Read the visible page when answering live UI questions | Visible content and main heading | +| `playwright_find_element` | `IPlaywrightPageInspectionService.FindElementsAsync()` | Locate a control, field, widget, tab, or text snippet from the current page | Match list with text, role, label, selector hints | +| `playwright_check_element_exists` | `IPlaywrightPageInspectionService.CheckElementExistsAsync()` | Confirm whether a target widget, field, or control is visible | Boolean existence and closest matches | +| `playwright_get_visible_widgets` | `IPlaywrightPageInspectionService.GetVisibleWidgetsAsync()` | List visible widget-like cards, headings, legends, and sections | Widget names and source hints | +| `playwright_take_screenshot` | `IPlaywrightPageInspectionService.TakeScreenshotAsync()` | Capture evidence for validation, debugging, and user confirmation | Saved path, timestamp, page URL, screenshot scope | +| `playwright_diagnose_orchard_action` | `IOrchardEvidenceService.FindOrchardElementWithEvidenceAsync()` | Locate a named OrchardCore action using 5-tier priority locator strategies; captures full evidence on failure | Found, MatchedLocator, PageScreenshotPath, ContainerScreenshotPath, PageHtmlPath, Attempts | + +## OrchardCore Skill Catalog + +The AI should reason in Orchard skills, then call deterministic Orchard tools. + +| Skill name | Covers | Current tool entry points | Recommended next C# surface | +| --- | --- | --- | --- | +| `orchard-admin-navigation` | Admin shell navigation, tenant-aware routes, admin menu traversal, and safe return-to-context behavior | `playwright_capture_state`, `playwright_open_admin_home`, `playwright_open_content_items` | `IOrchardAdminNavigationService.OpenAreaAsync()`, `OpenSectionAsync()` | +| `orchard-content-items` | Content item lists, row actions, search, title matching, status inspection, and item opening | `playwright_open_content_items`, `playwright_list_content_items`, `playwright_open_content_item_editor` | `IOrchardContentListService.OpenListAsync()`, `FindRowAsync()`, `InvokeRowActionAsync()` | +| `orchard-content-editor` | Title editing, typed field editing, editor tab switching, body-field handling, and pause-before-save behavior | `playwright_open_editor_tab`, `playwright_set_content_title`, `playwright_set_field_value`, `playwright_set_body_field` | `IOrchardContentEditorService.OpenTabAsync()`, `SetFieldAsync()`, `SetBodyFieldAsync()` | +| `orchard-publish-verification` | Save, publish, status confirmation, validation handling, and final evidence | `playwright_save_draft`, `playwright_publish_content`, `playwright_publish_and_verify`, `playwright_take_screenshot` | `IOrchardContentWorkflowService.SaveAndVerifyAsync()`, `PublishAndVerifyAsync()` | +| `orchard-widget-tree` | FlowPart, BagPart, widgets, nested cards, expanders, repeated components, and scoped edits inside widget containers | `playwright_get_visible_widgets`, `playwright_find_element`, `playwright_check_element_exists` | `IOrchardWidgetEditorService.FindWidgetAsync()`, `OpenWidgetAsync()`, `AddWidgetAsync()`, `EditWidgetFieldAsync()`, `MoveWidgetAsync()` | +| `orchard-content-definitions` | Content types, parts, fields, definitions screens, and admin metadata workflows | `playwright_capture_state`, `playwright_open_admin_home`, `playwright_find_element`, `playwright_take_screenshot` | `IOrchardDefinitionAdminService.OpenContentTypesAsync()`, `OpenTypeAsync()`, `OpenPartAsync()`, `OpenFieldAsync()` | +| `orchard-screenshot-diagnostics` | Screenshots, page-state confirmation, evidence capture, UI debugging, and failure reporting | `playwright_capture_state`, `playwright_get_page_content`, `playwright_take_screenshot`, `playwright_find_element`, `playwright_diagnose_orchard_action` | `IOrchardEvidenceService` ✓ implemented | + +## Selector Strategy for OrchardCore + +Use selectors in this order: + +1. Use a dedicated OrchardCore workflow tool. +2. Use OrchardCore structural scoping inside the service layer. +3. Use accessible names and labels inside the service layer. +4. Do not expose raw browser selectors to the AI. + +### Stable OrchardCore selector tiers + +| Tier | Strategy | OrchardCore examples | +| --- | --- | --- | +| Tier 1 | Route-aware task tools | Open admin home, open content list, open item by title, open editor tab, save draft, publish | +| Tier 2 | OrchardCore structural selectors | `tbody tr`, `[data-content-item-id]`, `.content-item`, `.list-group-item`, `.card`, `.card-header`, `.card-title`, `fieldset > legend`, `details > summary` | +| Tier 3 | OrchardCore field fallbacks | `input[name='TitlePart.Title']`, `input[id*='TitlePart_Title']`, visible `textarea`, `select`, `input[type='checkbox']`, `[contenteditable='true']`, rich editor surface, iframe body | +| Tier 4 | Internal service heuristics only | Accessible-name and label matching, then Orchard-aware related-field discovery | + +### OrchardCore-specific selector guidance + +- Content item list rows: + - Scope to the row or container that contains the requested title before clicking `Edit`, `Publish`, `Preview`, `Delete`, or `Clone`. + - Use title, then content type, then status to disambiguate repeated names. +- Editor tabs and sections: + - Prefer real tabs first, then buttons, links, summaries, card headers, card titles, and legends. + - Treat already-visible sections as valid matches instead of forcing an unnecessary click. +- Title field: + - Prefer label `Title`. + - Fallback to `input[name='TitlePart.Title']`, `input[id*='TitlePart_Title']`, or Orchard title input variants. +- Typed fields: + - Text, textarea, checkbox, and select should be handled by Orchard-specific field logic instead of a generic fill tool. +- Body-like fields: + - Detect textarea, TinyMCE, contenteditable, and iframe editors. + - Support append and replace modes explicitly. +- Validation and status: + - Check toast messages, alerts, validation summaries, field validation messages, and visible status badges before claiming success. + +## Reusable OrchardCore Workflow Patterns + +### Find the Edit button for a specific content item + +1. Call `playwright_open_content_items`. +2. Call `playwright_list_content_items` if the visible list is needed for grounding or matching. +3. Call `playwright_open_content_item_editor` with the requested title. +4. If the match is weak, report the closest titles instead of editing the wrong row. + +### Open a content item and update fields + +1. Open the editor by exact or best visible title match. +2. Call `playwright_open_editor_tab` if the field lives inside another tab or collapsible section. +3. Use `playwright_set_content_title`, `playwright_set_field_value`, and `playwright_set_body_field` for Orchard-specific edits. +4. Stop after the edit unless the user also asked to save or publish. + +### Publish or save and verify state + +1. Execute `playwright_save_draft`, `playwright_publish_content`, or `playwright_publish_and_verify`. +2. Capture state or screenshot immediately if verification is ambiguous. +3. Confirm success from the latest URL, heading, toast, status text, or structured verification signals. +4. If evidence is weak, report that verification is incomplete instead of guessing. + +### Work with FlowPart, BagPart, and nested widgets + +1. Capture state and list visible widgets before editing. +2. Identify the parent editor section, then the target widget card or repeated item. +3. Expand collapsed cards or summaries before searching for fields. +4. Scope every action to the current widget container. +5. If the current tool layer cannot target the widget deterministically, stop and use the recommended widget-specific next-wave tools instead of guessing. + +### Confirm final UI state + +1. Use `playwright_capture_state` for structured confirmation. +2. Use `playwright_get_page_content` when the user needs textual proof. +3. Use `playwright_take_screenshot` for visual proof or ambiguous UI. +4. Report the verified final state only, not intermediate guesses. + +## Recommended Next-Wave OrchardCore Tools + +These should be added as OrchardCore-specific tools instead of reintroducing generic browser actions. + +| Proposed tool | Recommended service call | Why it matters | +| --- | --- | --- | +| `playwright_open_content_definition` | `IOrchardDefinitionAdminService.OpenContentTypesAsync()` | Definitions are a different admin workflow than content items | +| `playwright_open_content_type_editor` | `IOrchardDefinitionAdminService.OpenTypeAsync(string typeName)` | Lets the AI work with type definitions safely | +| `playwright_find_widget` | `IOrchardWidgetEditorService.FindWidgetAsync(string widgetName, string widgetType, string parentPath)` | Required for FlowPart and BagPart depth | +| `playwright_open_widget_editor` | `IOrchardWidgetEditorService.OpenWidgetAsync(...)` | Stable entry point for nested widget editing | +| `playwright_add_widget` | `IOrchardWidgetEditorService.AddWidgetAsync(string containerName, string widgetType)` | Supports real Orchard widget authoring workflows | +| `playwright_validate_content_status` | `IOrchardContentWorkflowService.VerifyStatusAsync(string expectedStatus)` | Needed for production-grade status checking | +| ~~`playwright_capture_evidence`~~ | Implemented as `playwright_diagnose_orchard_action` / `IOrchardEvidenceService` | Now ships as a production tool — see Current Tool Architecture above | + +## Production-Ready Service Structure + +Keep the Orchard admin service as the current Orchard automation facade, but split deeper logic behind focused services as the tool set grows: + +- `IOrchardPageClassifier` +- `OrchardAdminSelectorCatalog` +- `IOrchardAdminNavigationService` +- `IOrchardContentListService` +- `IOrchardContentEditorService` +- `IOrchardContentWorkflowService` +- `IOrchardWidgetEditorService` +- `IOrchardDefinitionAdminService` +- `IOrchardEvidenceService` ✓ (`OrchardEvidenceService` — `FindOrchardElementWithEvidenceAsync`) +- `IPlaywrightPageInspectionService` + +This keeps the AI tool layer Orchard-specific while preventing the service layer from collapsing back into generic browser automation. diff --git a/src/Modules/CrestApps.OrchardCore.AI.Playwright/Handlers/PlaywrightChatInteractionSettingsHandler.cs b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Handlers/PlaywrightChatInteractionSettingsHandler.cs new file mode 100644 index 000000000..9076da4ec --- /dev/null +++ b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Handlers/PlaywrightChatInteractionSettingsHandler.cs @@ -0,0 +1,74 @@ +using System.Text.Json; +using CrestApps.OrchardCore.AI; +using CrestApps.OrchardCore.AI.Models; +using CrestApps.OrchardCore.AI.Playwright.Models; +using Microsoft.AspNetCore.DataProtection; +using OrchardCore.Entities; + +namespace CrestApps.OrchardCore.AI.Playwright.Handlers; + +internal sealed class PlaywrightChatInteractionSettingsHandler : IChatInteractionSettingsHandler +{ + private readonly IDataProtectionProvider _dataProtectionProvider; + + public PlaywrightChatInteractionSettingsHandler(IDataProtectionProvider dataProtectionProvider) + { + _dataProtectionProvider = dataProtectionProvider; + } + + public Task UpdatingAsync(ChatInteraction interaction, JsonElement settings) + { + var existingMetadata = interaction.As() ?? new PlaywrightSessionMetadata(); + var protector = _dataProtectionProvider.CreateProtector(PlaywrightConstants.ProtectorName); + var password = GetString(settings, "playwrightPassword"); + var protectedPassword = string.IsNullOrWhiteSpace(password) + ? existingMetadata.ProtectedPassword + : protector.Protect(password); + + interaction.Put(new PlaywrightSessionMetadata + { + Enabled = GetBool(settings, "playwrightEnabled"), + BrowserMode = PlaywrightBrowserMode.PersistentContext, + Username = GetString(settings, "playwrightUsername")?.Trim(), + ProtectedPassword = protectedPassword, + BaseUrl = GetString(settings, "playwrightBaseUrl")?.Trim(), + AdminBaseUrl = GetString(settings, "playwrightAdminBaseUrl")?.Trim(), + PersistentProfilePath = GetString(settings, "playwrightPersistentProfilePath")?.Trim(), + Headless = GetBool(settings, "playwrightHeadless"), + PublishByDefault = GetBool(settings, "playwrightPublishByDefault"), + }); + + return Task.CompletedTask; + } + + public Task UpdatedAsync(ChatInteraction interaction, JsonElement settings) + => Task.CompletedTask; + + private static string GetString(JsonElement element, string propertyName) + { + if (element.TryGetProperty(propertyName, out var prop) && prop.ValueKind == JsonValueKind.String) + { + return prop.GetString(); + } + + return null; + } + + private static bool GetBool(JsonElement element, string propertyName) + { + if (element.TryGetProperty(propertyName, out var prop)) + { + if (prop.ValueKind == JsonValueKind.True) + { + return true; + } + + if (prop.ValueKind == JsonValueKind.String) + { + return string.Equals(prop.GetString(), "true", StringComparison.OrdinalIgnoreCase); + } + } + + return false; + } +} diff --git a/src/Modules/CrestApps.OrchardCore.AI.Playwright/Handlers/PlaywrightFeatureEventHandler.cs b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Handlers/PlaywrightFeatureEventHandler.cs new file mode 100644 index 000000000..046343503 --- /dev/null +++ b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Handlers/PlaywrightFeatureEventHandler.cs @@ -0,0 +1,71 @@ +using Microsoft.Extensions.Logging; +using OrchardCore.Environment.Extensions.Features; +using OrchardCore.Environment.Shell; + +// IFeatureInfo lives in OrchardCore.Environment.Extensions.Features. +// IFeatureEventHandler (the one registered in DI) lives in OrchardCore.Environment.Shell. + +namespace CrestApps.OrchardCore.AI.Playwright.Handlers; + +/// +/// Installs Playwright browser binaries the first time the feature is enabled. +/// This ensures Chromium is available before any session is created. +/// +public sealed class PlaywrightFeatureEventHandler : IFeatureEventHandler +{ + private readonly ILogger _logger; + + public PlaywrightFeatureEventHandler(ILogger logger) + { + _logger = logger; + } + + public Task InstallingAsync(IFeatureInfo feature) => Task.CompletedTask; + public Task InstalledAsync(IFeatureInfo feature) => Task.CompletedTask; + public Task EnablingAsync(IFeatureInfo feature) => Task.CompletedTask; + + public Task EnabledAsync(IFeatureInfo feature) + { + if (!feature.Id.Equals(PlaywrightConstants.Feature.AdminWidget, StringComparison.OrdinalIgnoreCase)) + { + return Task.CompletedTask; + } + + return InstallBrowsersAsync(); + } + + public Task DisablingAsync(IFeatureInfo feature) => Task.CompletedTask; + public Task DisabledAsync(IFeatureInfo feature) => Task.CompletedTask; + public Task UninstallingAsync(IFeatureInfo feature) => Task.CompletedTask; + public Task UninstalledAsync(IFeatureInfo feature) => Task.CompletedTask; + + private Task InstallBrowsersAsync() + { + return Task.Run(() => + { + try + { + _logger.LogDebug("Installing Playwright browser binaries (chromium)..."); + + var exitCode = Microsoft.Playwright.Program.Main(["install", "chromium"]); + + if (exitCode != 0) + { + _logger.LogDebug( + "Playwright browser installation exited with code {ExitCode}. " + + "You may need to run 'playwright install chromium' manually.", exitCode); + } + else + { + _logger.LogDebug("Playwright browser binaries installed successfully."); + } + } + catch (Exception ex) + { + _logger.LogDebug(ex, + "Failed to install Playwright browser binaries. " + + "Run 'playwright install chromium' manually before using browser automation."); + } + }); + } +} diff --git a/src/Modules/CrestApps.OrchardCore.AI.Playwright/Handlers/PlaywrightOrchestrationContextHandler.cs b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Handlers/PlaywrightOrchestrationContextHandler.cs new file mode 100644 index 000000000..d980ebb20 --- /dev/null +++ b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Handlers/PlaywrightOrchestrationContextHandler.cs @@ -0,0 +1,76 @@ +using CrestApps.AI.Prompting.Services; +using CrestApps.OrchardCore.AI; +using CrestApps.OrchardCore.AI.Models; +using CrestApps.OrchardCore.AI.Playwright.Models; +using Microsoft.Extensions.Logging; +using OrchardCore.Entities; + +namespace CrestApps.OrchardCore.AI.Playwright.Handlers; + +internal sealed class PlaywrightOrchestrationContextHandler : IOrchestrationContextBuilderHandler +{ + private readonly IAITemplateService _templateService; + private readonly ILogger _logger; + + public PlaywrightOrchestrationContextHandler( + IAITemplateService templateService, + ILogger logger) + { + _templateService = templateService; + _logger = logger; + } + + public async Task BuildingAsync(OrchestrationContextBuildingContext context) + { + if (context.Resource is not Entity entity) + { + return; + } + + var metadata = entity.As(); + if (metadata is null || !metadata.Enabled) + { + return; + } + + context.Context.Properties[nameof(PlaywrightSessionMetadata)] = metadata; + + try + { + var prompt = await _templateService.RenderAsync( + PlaywrightConstants.PromptIds.Operator, + new Dictionary(StringComparer.OrdinalIgnoreCase) + { + ["baseUrl"] = metadata.BaseUrl ?? string.Empty, + ["adminBaseUrl"] = metadata.AdminBaseUrl ?? string.Empty, + ["publishBehavior"] = metadata.PublishByDefault + ? "Publish by default unless the user explicitly asks for a draft." + : "Save a draft by default unless the user explicitly asks to publish.", + }); + + if (!string.IsNullOrWhiteSpace(prompt)) + { + context.Context.SystemMessageBuilder.AppendLine(prompt.Trim()); + context.Context.SystemMessageBuilder.AppendLine(); + } + } + catch (Exception ex) + { + _logger.LogWarning(ex, "Unable to render the Playwright operator prompt."); + } + } + + public Task BuiltAsync(OrchestrationContextBuiltContext context) + { + if (context.OrchestrationContext.CompletionContext is null || + !context.OrchestrationContext.Properties.TryGetValue(nameof(PlaywrightSessionMetadata), out var metadataObject) || + metadataObject is not PlaywrightSessionMetadata metadata) + { + return Task.CompletedTask; + } + + context.OrchestrationContext.CompletionContext.AdditionalProperties[PlaywrightConstants.CompletionContextKeys.SessionMetadata] = metadata; + + return Task.CompletedTask; + } +} diff --git a/src/Modules/CrestApps.OrchardCore.AI.Playwright/Manifest.cs b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Manifest.cs new file mode 100644 index 000000000..a48727532 --- /dev/null +++ b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Manifest.cs @@ -0,0 +1,23 @@ +using CrestApps.OrchardCore; +using CrestApps.OrchardCore.AI.Core; +using CrestApps.OrchardCore.AI.Playwright; +using OrchardCore.Modules.Manifest; + +[assembly: Module( + Name = "Artificial Intelligence", + Author = CrestAppsManifestConstants.Author, + Website = CrestAppsManifestConstants.Website, + Version = CrestAppsManifestConstants.Version +)] + +[assembly: Feature( + Id = PlaywrightConstants.Feature.AdminWidget, + Name = "AI Playwright Browser Automation", + Description = "Enables the AI Chat Admin Widget agent to drive a real browser via Playwright. Admins can watch, stop, and close browser automation tasks directly from the chat widget.", + Category = "Artificial Intelligence", + Dependencies = + [ + AIConstants.Feature.ChatAdminWidget, + AIConstants.Feature.OrchardCoreAIAgent, + ] +)] diff --git a/src/Modules/CrestApps.OrchardCore.AI.Playwright/Models/PlaywrightBrowserMode.cs b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Models/PlaywrightBrowserMode.cs new file mode 100644 index 000000000..34ecb3624 --- /dev/null +++ b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Models/PlaywrightBrowserMode.cs @@ -0,0 +1,8 @@ +namespace CrestApps.OrchardCore.AI.Playwright.Models; + +public enum PlaywrightBrowserMode +{ + ConnectOverCdp = 0, + PersistentContext = 1, + FreshBrowser = 2, +} diff --git a/src/Modules/CrestApps.OrchardCore.AI.Playwright/Models/PlaywrightContentItemOpenResult.cs b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Models/PlaywrightContentItemOpenResult.cs new file mode 100644 index 000000000..7fccc7da1 --- /dev/null +++ b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Models/PlaywrightContentItemOpenResult.cs @@ -0,0 +1,16 @@ +namespace CrestApps.OrchardCore.AI.Playwright.Models; + +public sealed class PlaywrightContentItemOpenResult +{ + public string RequestedTitle { get; init; } + + public string MatchedTitle { get; init; } + + public string MatchMode { get; init; } + + public bool UsedSearch { get; init; } + + public IReadOnlyList ClosestTitles { get; init; } = []; + + public PlaywrightObservation Observation { get; init; } +} diff --git a/src/Modules/CrestApps.OrchardCore.AI.Playwright/Models/PlaywrightContentListItem.cs b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Models/PlaywrightContentListItem.cs new file mode 100644 index 000000000..2795d3b4f --- /dev/null +++ b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Models/PlaywrightContentListItem.cs @@ -0,0 +1,12 @@ +namespace CrestApps.OrchardCore.AI.Playwright.Models; + +public sealed class PlaywrightContentListItem +{ + public string Title { get; init; } + + public string ContentType { get; init; } + + public string Status { get; init; } + + public bool CanEdit { get; init; } +} diff --git a/src/Modules/CrestApps.OrchardCore.AI.Playwright/Models/PlaywrightContentListResult.cs b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Models/PlaywrightContentListResult.cs new file mode 100644 index 000000000..22274c392 --- /dev/null +++ b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Models/PlaywrightContentListResult.cs @@ -0,0 +1,14 @@ +namespace CrestApps.OrchardCore.AI.Playwright.Models; + +public sealed class PlaywrightContentListResult +{ + public string Url { get; init; } + + public string Title { get; init; } + + public string MainHeading { get; init; } + + public int ItemCount { get; init; } + + public IReadOnlyList Items { get; init; } = []; +} diff --git a/src/Modules/CrestApps.OrchardCore.AI.Playwright/Models/PlaywrightEditorActionResult.cs b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Models/PlaywrightEditorActionResult.cs new file mode 100644 index 000000000..b50c220b0 --- /dev/null +++ b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Models/PlaywrightEditorActionResult.cs @@ -0,0 +1 @@ +// Retired while converging on the Orchard admin service result types. diff --git a/src/Modules/CrestApps.OrchardCore.AI.Playwright/Models/PlaywrightEditorTargetResult.cs b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Models/PlaywrightEditorTargetResult.cs new file mode 100644 index 000000000..7e779a255 --- /dev/null +++ b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Models/PlaywrightEditorTargetResult.cs @@ -0,0 +1,12 @@ +namespace CrestApps.OrchardCore.AI.Playwright.Models; + +public sealed class PlaywrightEditorTargetResult +{ + public string RequestedTarget { get; init; } + + public string MatchedTarget { get; init; } + + public string TargetKind { get; init; } + + public PlaywrightObservation Observation { get; init; } +} diff --git a/src/Modules/CrestApps.OrchardCore.AI.Playwright/Models/PlaywrightElementDiagnosticsResult.cs b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Models/PlaywrightElementDiagnosticsResult.cs new file mode 100644 index 000000000..ef041a8d7 --- /dev/null +++ b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Models/PlaywrightElementDiagnosticsResult.cs @@ -0,0 +1,43 @@ +namespace CrestApps.OrchardCore.AI.Playwright.Models; + +public sealed class PlaywrightElementDiagnosticsResult +{ + /// + /// True when at least one locator strategy matched a visible element. + /// + public bool Found { get; init; } + + /// + /// Name of the locator strategy that matched, e.g. "role:link[Edit]". + /// Null when is false. + /// + public string MatchedLocator { get; init; } + + /// Current page URL at the time of the search. + public string Url { get; init; } + + /// Current page title at the time of the search. + public string Title { get; init; } + + /// + /// Full-page screenshot path. Populated only on failure to help diagnose the missing action. + /// + public string PageScreenshotPath { get; init; } + + /// + /// Screenshot of the nearest relevant OrchardCore container found on the page. + /// Populated only on failure when a candidate container is located. + /// + public string ContainerScreenshotPath { get; init; } + + /// + /// Path to the saved raw page HTML. Populated only on failure. + /// + public string PageHtmlPath { get; init; } + + /// + /// Ordered list of locator strategy descriptions that were attempted. + /// Always populated so the AI can report exactly what was tried. + /// + public IReadOnlyList Attempts { get; init; } = []; +} diff --git a/src/Modules/CrestApps.OrchardCore.AI.Playwright/Models/PlaywrightElementMatch.cs b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Models/PlaywrightElementMatch.cs new file mode 100644 index 000000000..22a97418c --- /dev/null +++ b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Models/PlaywrightElementMatch.cs @@ -0,0 +1,24 @@ +namespace CrestApps.OrchardCore.AI.Playwright.Models; + +public sealed class PlaywrightElementMatch +{ + public string TagName { get; init; } + + public string Role { get; init; } + + public string Text { get; init; } + + public string Label { get; init; } + + public string Id { get; init; } + + public string Name { get; init; } + + public string Title { get; init; } + + public string Placeholder { get; init; } + + public string WidgetType { get; init; } + + public string SelectorHint { get; init; } +} diff --git a/src/Modules/CrestApps.OrchardCore.AI.Playwright/Models/PlaywrightElementSearchResult.cs b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Models/PlaywrightElementSearchResult.cs new file mode 100644 index 000000000..44a04b079 --- /dev/null +++ b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Models/PlaywrightElementSearchResult.cs @@ -0,0 +1,16 @@ +namespace CrestApps.OrchardCore.AI.Playwright.Models; + +public sealed class PlaywrightElementSearchResult +{ + public string Query { get; init; } + + public string Url { get; init; } + + public string Title { get; init; } + + public bool Exists { get; init; } + + public int MatchCount { get; init; } + + public IReadOnlyList Matches { get; init; } = []; +} diff --git a/src/Modules/CrestApps.OrchardCore.AI.Playwright/Models/PlaywrightFieldEditResult.cs b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Models/PlaywrightFieldEditResult.cs new file mode 100644 index 000000000..7dc460518 --- /dev/null +++ b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Models/PlaywrightFieldEditResult.cs @@ -0,0 +1,14 @@ +namespace CrestApps.OrchardCore.AI.Playwright.Models; + +public sealed class PlaywrightFieldEditResult +{ + public string Label { get; init; } + + public string RequestedFieldType { get; init; } + + public string RequestedEditMode { get; init; } + + public string ResolvedFieldType { get; init; } + + public PlaywrightObservation Observation { get; init; } +} diff --git a/src/Modules/CrestApps.OrchardCore.AI.Playwright/Models/PlaywrightObservation.cs b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Models/PlaywrightObservation.cs new file mode 100644 index 000000000..79d2df8ed --- /dev/null +++ b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Models/PlaywrightObservation.cs @@ -0,0 +1,20 @@ +namespace CrestApps.OrchardCore.AI.Playwright.Models; + +public sealed class PlaywrightObservation +{ + public string CurrentUrl { get; init; } + + public string PageTitle { get; init; } + + public string MainHeading { get; init; } + + public string ToastMessage { get; init; } + + public IReadOnlyList ValidationMessages { get; init; } = []; + + public IReadOnlyList VisibleButtons { get; init; } = []; + + public bool IsLoginPage { get; init; } + + public bool IsAuthenticated { get; init; } +} diff --git a/src/Modules/CrestApps.OrchardCore.AI.Playwright/Models/PlaywrightPageContentResult.cs b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Models/PlaywrightPageContentResult.cs new file mode 100644 index 000000000..518888e78 --- /dev/null +++ b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Models/PlaywrightPageContentResult.cs @@ -0,0 +1,12 @@ +namespace CrestApps.OrchardCore.AI.Playwright.Models; + +public sealed class PlaywrightPageContentResult +{ + public string Url { get; init; } + + public string Title { get; init; } + + public string MainHeading { get; init; } + + public string Content { get; init; } +} diff --git a/src/Modules/CrestApps.OrchardCore.AI.Playwright/Models/PlaywrightPublishVerificationResult.cs b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Models/PlaywrightPublishVerificationResult.cs new file mode 100644 index 000000000..9a2e015bd --- /dev/null +++ b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Models/PlaywrightPublishVerificationResult.cs @@ -0,0 +1,14 @@ +namespace CrestApps.OrchardCore.AI.Playwright.Models; + +public sealed class PlaywrightPublishVerificationResult +{ + public string Action { get; init; } + + public string ExpectedStatus { get; init; } + + public bool Verified { get; init; } + + public IReadOnlyList VerificationSignals { get; init; } = []; + + public PlaywrightObservation Observation { get; init; } +} diff --git a/src/Modules/CrestApps.OrchardCore.AI.Playwright/Models/PlaywrightScreenshotResult.cs b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Models/PlaywrightScreenshotResult.cs new file mode 100644 index 000000000..69a35537f --- /dev/null +++ b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Models/PlaywrightScreenshotResult.cs @@ -0,0 +1,14 @@ +namespace CrestApps.OrchardCore.AI.Playwright.Models; + +public sealed class PlaywrightScreenshotResult +{ + public string Url { get; init; } + + public string Title { get; init; } + + public string SavedPath { get; init; } + + public bool FullPage { get; init; } + + public DateTime TakenAtUtc { get; init; } +} diff --git a/src/Modules/CrestApps.OrchardCore.AI.Playwright/Models/PlaywrightSessionMetadata.cs b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Models/PlaywrightSessionMetadata.cs new file mode 100644 index 000000000..5d6bff467 --- /dev/null +++ b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Models/PlaywrightSessionMetadata.cs @@ -0,0 +1,28 @@ +namespace CrestApps.OrchardCore.AI.Playwright.Models; + +/// +/// Shared Playwright session metadata that can be attached to either an AI profile +/// or a chat interaction via EntityExtensions.Put/As. +/// +public sealed class PlaywrightSessionMetadata +{ + public bool Enabled { get; set; } + + public PlaywrightBrowserMode BrowserMode { get; set; } = PlaywrightBrowserMode.PersistentContext; + + public string Username { get; set; } + + public string ProtectedPassword { get; set; } + + public string BaseUrl { get; set; } + + public string AdminBaseUrl { get; set; } + + public string CdpEndpoint { get; set; } + + public string PersistentProfilePath { get; set; } + + public bool Headless { get; set; } + + public bool PublishByDefault { get; set; } +} diff --git a/src/Modules/CrestApps.OrchardCore.AI.Playwright/Models/PlaywrightSessionRequest.cs b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Models/PlaywrightSessionRequest.cs new file mode 100644 index 000000000..369de2f2c --- /dev/null +++ b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Models/PlaywrightSessionRequest.cs @@ -0,0 +1,32 @@ +namespace CrestApps.OrchardCore.AI.Playwright.Models; + +public sealed class PlaywrightSessionRequest +{ + public string ChatSessionId { get; init; } + + public string OwnerId { get; init; } + + public string ResourceItemId { get; init; } + + public string BaseUrl { get; init; } + + public string AdminBaseUrl { get; init; } + + public PlaywrightBrowserMode BrowserMode { get; init; } + + public string Username { get; init; } + + public string Password { get; init; } + + public bool CanAttemptLogin { get; init; } + + public string CdpEndpoint { get; init; } + + public string PersistentProfilePath { get; init; } + + public bool Headless { get; init; } + + public bool PublishByDefault { get; init; } + + public int SessionInactivityTimeoutInMinutes { get; init; } = PlaywrightConstants.DefaultSessionInactivityTimeoutInMinutes; +} diff --git a/src/Modules/CrestApps.OrchardCore.AI.Playwright/Models/PlaywrightStatusResponse.cs b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Models/PlaywrightStatusResponse.cs new file mode 100644 index 000000000..5b3221e5f --- /dev/null +++ b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Models/PlaywrightStatusResponse.cs @@ -0,0 +1,42 @@ +using CrestApps.OrchardCore.AI.Playwright.Services; + +namespace CrestApps.OrchardCore.AI.Playwright.Models; + +/// Status payload returned by the session API endpoints. +public sealed class PlaywrightStatusResponse +{ + public string ChatSessionId { get; init; } + + public string Status { get; init; } + + public string CurrentUrl { get; init; } + + public string CurrentPageTitle { get; init; } + + public string BrowserMode { get; init; } + + public bool IsActive { get; init; } + + public bool IsAuthenticated { get; init; } + + public static PlaywrightStatusResponse FromSession(string chatSessionId, IPlaywrightSession session) + { + return new PlaywrightStatusResponse + { + ChatSessionId = chatSessionId, + Status = session.Status.ToString(), + CurrentUrl = session.CurrentUrl, + CurrentPageTitle = session.CurrentPageTitle, + BrowserMode = "Dedicated browser", + IsActive = session.Status != PlaywrightSessionStatus.Closed, + IsAuthenticated = session.IsAuthenticated, + }; + } + + public static PlaywrightStatusResponse Inactive(string chatSessionId) => new() + { + ChatSessionId = chatSessionId, + Status = PlaywrightSessionStatus.Closed.ToString(), + IsActive = false, + }; +} diff --git a/src/Modules/CrestApps.OrchardCore.AI.Playwright/Models/PlaywrightVisibleWidget.cs b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Models/PlaywrightVisibleWidget.cs new file mode 100644 index 000000000..8eaa11966 --- /dev/null +++ b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Models/PlaywrightVisibleWidget.cs @@ -0,0 +1,8 @@ +namespace CrestApps.OrchardCore.AI.Playwright.Models; + +public sealed class PlaywrightVisibleWidget +{ + public string Name { get; init; } + + public string Source { get; init; } +} diff --git a/src/Modules/CrestApps.OrchardCore.AI.Playwright/Models/PlaywrightVisibleWidgetsResult.cs b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Models/PlaywrightVisibleWidgetsResult.cs new file mode 100644 index 000000000..58c82f8ee --- /dev/null +++ b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Models/PlaywrightVisibleWidgetsResult.cs @@ -0,0 +1,12 @@ +namespace CrestApps.OrchardCore.AI.Playwright.Models; + +public sealed class PlaywrightVisibleWidgetsResult +{ + public string Url { get; init; } + + public string Title { get; init; } + + public int WidgetCount { get; init; } + + public IReadOnlyList Widgets { get; init; } = []; +} diff --git a/src/Modules/CrestApps.OrchardCore.AI.Playwright/PlaywrightConstants.cs b/src/Modules/CrestApps.OrchardCore.AI.Playwright/PlaywrightConstants.cs new file mode 100644 index 000000000..32c4d4acb --- /dev/null +++ b/src/Modules/CrestApps.OrchardCore.AI.Playwright/PlaywrightConstants.cs @@ -0,0 +1,98 @@ +namespace CrestApps.OrchardCore.AI.Playwright; + +public static class PlaywrightConstants +{ + public const int DefaultSessionInactivityTimeoutInMinutes = 30; + + public static class Feature + { + public const string AdminWidget = "CrestApps.OrchardCore.AI.Playwright"; + } + + public const string ProtectorName = "PlaywrightProfilePassword"; + + public static class PromptIds + { + public const string Operator = "playwright-operator"; + } + + public static class CompletionContextKeys + { + public const string SessionMetadata = "PlaywrightSessionMetadata"; + } + + public static class ToolNames + { + public const string CaptureState = "playwright_capture_state"; + public const string OpenAdminHome = "playwright_open_admin_home"; + public const string OpenContentItems = "playwright_open_content_items"; + public const string ListContentItems = "playwright_list_content_items"; + public const string OpenContentItemEditor = "playwright_open_content_item_editor"; + public const string OpenNewContentItem = "playwright_open_new_content_item"; + public const string OpenEditorTab = "playwright_open_editor_tab"; + public const string SetContentTitle = "playwright_set_content_title"; + public const string SetFieldValue = "playwright_set_field_value"; + public const string SetBodyField = "playwright_set_body_field"; + public const string SaveDraft = "playwright_save_draft"; + public const string PublishContent = "playwright_publish_content"; + public const string PublishAndVerify = "playwright_publish_and_verify"; + public const string InspectPageContent = "playwright_get_page_content"; + public const string FindElement = "playwright_find_element"; + public const string CheckElementExists = "playwright_check_element_exists"; + public const string GetVisibleWidgets = "playwright_get_visible_widgets"; + public const string TakeScreenshot = "playwright_take_screenshot"; + public const string DiagnoseOrchardAction = "playwright_diagnose_orchard_action"; + } + + public static class ToolSets + { + public static readonly string[] Deterministic = + [ + ToolNames.CaptureState, + ToolNames.OpenAdminHome, + ToolNames.OpenContentItems, + ToolNames.ListContentItems, + ToolNames.OpenContentItemEditor, + ToolNames.OpenNewContentItem, + ToolNames.OpenEditorTab, + ToolNames.SetContentTitle, + ToolNames.SetFieldValue, + ToolNames.SetBodyField, + ToolNames.SaveDraft, + ToolNames.PublishContent, + ToolNames.PublishAndVerify, + ToolNames.InspectPageContent, + ToolNames.FindElement, + ToolNames.CheckElementExists, + ToolNames.GetVisibleWidgets, + ToolNames.TakeScreenshot, + ToolNames.DiagnoseOrchardAction, + ]; + } + + public static string GetToolDescription(string toolName) + => toolName switch + { + ToolNames.CaptureState => "Captures the current browser state for deterministic Orchard admin planning.", + ToolNames.OpenAdminHome => "Ensures the Orchard admin shell is available for the current tenant.", + ToolNames.OpenContentItems => "Opens the Orchard content items list.", + ToolNames.ListContentItems => "Lists the visible Orchard content items from the current content items screen.", + ToolNames.OpenContentItemEditor => "Opens the editor for an existing Orchard content item by title, using the current list page before restarting navigation.", + ToolNames.OpenNewContentItem => "Starts the create-content flow for a requested Orchard content type.", + ToolNames.OpenEditorTab => "Opens an OrchardCore editor tab, section, or expander by name.", + ToolNames.SetContentTitle => "Sets the Title field on the current Orchard content editor.", + ToolNames.SetFieldValue => "Updates a labeled OrchardCore field using a typed field strategy.", + ToolNames.SetBodyField => "Updates a body-like OrchardCore field using rich-editor-aware append or replace behavior.", + ToolNames.SaveDraft => "Saves the current Orchard content item as a draft and captures the resulting page state.", + ToolNames.PublishContent => "Publishes the current Orchard content item and captures the resulting page state.", + ToolNames.PublishAndVerify => "Publishes the current Orchard content item and returns structured verification evidence.", + ToolNames.InspectPageContent => "Returns the visible content of the current page for grounded follow-up questions.", + ToolNames.FindElement => "Finds visible page elements that match a requested widget name, label, or text snippet.", + ToolNames.CheckElementExists => "Checks whether a requested control, widget, or text snippet is currently visible on the page.", + ToolNames.GetVisibleWidgets => "Lists visible widget-like cards, headings, and editor sections on the current page.", + ToolNames.TakeScreenshot => "Captures a screenshot of the current page and returns the saved file path.", + ToolNames.DiagnoseOrchardAction => "Attempts to find a named OrchardCore admin action using priority-ordered locator strategies. Captures full evidence (screenshots, HTML, URL, attempts) when the action is not found.", + _ => toolName, + }; +} + diff --git a/src/Modules/CrestApps.OrchardCore.AI.Playwright/Services/IOrchardAdminPlaywrightService.cs b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Services/IOrchardAdminPlaywrightService.cs new file mode 100644 index 000000000..f78b36065 --- /dev/null +++ b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Services/IOrchardAdminPlaywrightService.cs @@ -0,0 +1,51 @@ +using CrestApps.OrchardCore.AI.Playwright.Models; + +namespace CrestApps.OrchardCore.AI.Playwright.Services; + +public interface IOrchardAdminPlaywrightService +{ + Task CaptureStateAsync(IPlaywrightSession session, CancellationToken cancellationToken = default); + + Task OpenAdminHomeAsync(IPlaywrightSession session, CancellationToken cancellationToken = default); + + Task OpenContentItemsAsync(IPlaywrightSession session, CancellationToken cancellationToken = default); + + Task ListVisibleContentItemsAsync(IPlaywrightSession session, int maxItems = 20, CancellationToken cancellationToken = default); + + Task OpenNewContentItemAsync(IPlaywrightSession session, string contentType, CancellationToken cancellationToken = default); + + Task OpenContentItemEditorAsync(IPlaywrightSession session, string title, CancellationToken cancellationToken = default); + + Task OpenEditorTabAsync( + IPlaywrightSession session, + string tabName, + bool exact = false, + CancellationToken cancellationToken = default); + + Task SetContentTitleAsync(IPlaywrightSession session, string title, CancellationToken cancellationToken = default); + + Task SetFieldValueAsync( + IPlaywrightSession session, + string label, + string value, + string fieldType = "auto", + bool exact = false, + CancellationToken cancellationToken = default); + + Task SetBodyFieldAsync( + IPlaywrightSession session, + string label, + string value, + string writeMode = "append", + bool exact = false, + CancellationToken cancellationToken = default); + + Task SaveDraftAsync(IPlaywrightSession session, CancellationToken cancellationToken = default); + + Task PublishContentAsync(IPlaywrightSession session, CancellationToken cancellationToken = default); + + Task PublishAndVerifyAsync( + IPlaywrightSession session, + string expectedStatus = "Published", + CancellationToken cancellationToken = default); +} diff --git a/src/Modules/CrestApps.OrchardCore.AI.Playwright/Services/IOrchardEvidenceService.cs b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Services/IOrchardEvidenceService.cs new file mode 100644 index 000000000..b433a907d --- /dev/null +++ b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Services/IOrchardEvidenceService.cs @@ -0,0 +1,21 @@ +using CrestApps.OrchardCore.AI.Playwright.Models; + +namespace CrestApps.OrchardCore.AI.Playwright.Services; + +public interface IOrchardEvidenceService +{ + /// + /// Attempts to find a named OrchardCore admin action (e.g. Edit, Publish Now, Save Draft, Preview) + /// using multiple locator strategies in priority order. + /// + /// On success, returns = true with the + /// matched locator strategy name. No evidence files are written. + /// + /// + /// On failure, captures a full-page screenshot, a container screenshot when possible, and the + /// raw page HTML so the caller can diagnose whether the action is truly absent, hidden, renamed, + /// or blocked by an overlay. + /// + /// + Task FindOrchardElementWithEvidenceAsync(IPlaywrightSession session, string actionLabel, CancellationToken cancellationToken = default); +} diff --git a/src/Modules/CrestApps.OrchardCore.AI.Playwright/Services/IPlaywrightActionVisualizer.cs b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Services/IPlaywrightActionVisualizer.cs new file mode 100644 index 000000000..083dd14d3 --- /dev/null +++ b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Services/IPlaywrightActionVisualizer.cs @@ -0,0 +1,19 @@ +using Microsoft.Playwright; + +namespace CrestApps.OrchardCore.AI.Playwright.Services; + +public interface IPlaywrightActionVisualizer +{ + Task ShowLocatorActionAsync( + IPage page, + ILocator locator, + string action, + string target, + CancellationToken cancellationToken = default); + + Task ShowPageActionAsync( + IPage page, + string action, + string target, + CancellationToken cancellationToken = default); +} diff --git a/src/Modules/CrestApps.OrchardCore.AI.Playwright/Services/IPlaywrightObservationService.cs b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Services/IPlaywrightObservationService.cs new file mode 100644 index 000000000..e0a64dc9e --- /dev/null +++ b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Services/IPlaywrightObservationService.cs @@ -0,0 +1,8 @@ +using CrestApps.OrchardCore.AI.Playwright.Models; + +namespace CrestApps.OrchardCore.AI.Playwright.Services; + +public interface IPlaywrightObservationService +{ + Task CaptureAsync(IPlaywrightSession session, CancellationToken cancellationToken = default); +} diff --git a/src/Modules/CrestApps.OrchardCore.AI.Playwright/Services/IPlaywrightPageInspectionService.cs b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Services/IPlaywrightPageInspectionService.cs new file mode 100644 index 000000000..0d223d92b --- /dev/null +++ b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Services/IPlaywrightPageInspectionService.cs @@ -0,0 +1,16 @@ +using CrestApps.OrchardCore.AI.Playwright.Models; + +namespace CrestApps.OrchardCore.AI.Playwright.Services; + +public interface IPlaywrightPageInspectionService +{ + Task GetPageContentAsync(IPlaywrightSession session, int maxLength, CancellationToken cancellationToken = default); + + Task FindElementsAsync(IPlaywrightSession session, string query, int maxMatches, CancellationToken cancellationToken = default); + + Task CheckElementExistsAsync(IPlaywrightSession session, string query, int maxMatches, CancellationToken cancellationToken = default); + + Task GetVisibleWidgetsAsync(IPlaywrightSession session, int maxItems, CancellationToken cancellationToken = default); + + Task TakeScreenshotAsync(IPlaywrightSession session, bool fullPage, CancellationToken cancellationToken = default); +} diff --git a/src/Modules/CrestApps.OrchardCore.AI.Playwright/Services/IPlaywrightSession.cs b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Services/IPlaywrightSession.cs new file mode 100644 index 000000000..201fa5382 --- /dev/null +++ b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Services/IPlaywrightSession.cs @@ -0,0 +1,67 @@ +using CrestApps.OrchardCore.AI.Playwright.Models; +using Microsoft.Playwright; + +namespace CrestApps.OrchardCore.AI.Playwright.Services; + +/// +/// Represents an active Playwright browser session bound to a single chat session. +/// +public interface IPlaywrightSession +{ + /// Gets the owning chat or interaction session identifier. + string SessionId { get; } + + /// Gets the user or owner identifier associated with this Playwright session. + string OwnerId { get; } + + /// Gets the current lifecycle status of this session. + PlaywrightSessionStatus Status { get; } + + /// Gets the URL the browser is currently on, or null if not yet navigated. + string CurrentUrl { get; } + + /// Gets the current page title, if known. + string CurrentPageTitle { get; } + + /// Gets when this session was created. + DateTime CreatedAtUtc { get; } + + /// Gets the last time a Playwright action touched this session. + DateTime LastActivityUtc { get; } + + /// Gets the resolved public base URL for the current tenant. + string BaseUrl { get; } + + /// Gets the resolved Orchard admin base URL. + string AdminBaseUrl { get; } + + /// Gets the active browser mode used by the session. + PlaywrightBrowserMode BrowserMode { get; } + + /// Gets whether the current page is considered authenticated for Orchard admin flows. + bool IsAuthenticated { get; } + + /// Gets the most recent structured observation captured from the page. + PlaywrightObservation LastObservation { get; } + + /// Gets the inactivity timeout for this session in minutes. + int SessionInactivityTimeoutInMinutes { get; } + + /// Gets the active Playwright page. + IPage Page { get; } + + /// + /// Returns a that is cancelled when either the session Stop + /// is requested or the session is closed. + /// + CancellationToken StopToken { get; } + + /// Marks the session as running (tool call starting). + void MarkRunning(); + + /// Marks the session as idle (tool call finished). + void MarkIdle(); + + /// Cancels the current operation. Browser stays open. + void Stop(); +} diff --git a/src/Modules/CrestApps.OrchardCore.AI.Playwright/Services/IPlaywrightSessionManager.cs b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Services/IPlaywrightSessionManager.cs new file mode 100644 index 000000000..798b3d014 --- /dev/null +++ b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Services/IPlaywrightSessionManager.cs @@ -0,0 +1,49 @@ +using CrestApps.OrchardCore.AI.Playwright.Models; + +namespace CrestApps.OrchardCore.AI.Playwright.Services; + +/// +/// Manages Playwright browser sessions keyed by chat session ID. +/// Implementations are registered as singletons. +/// +public interface IPlaywrightSessionManager +{ + /// + /// Returns the existing session for the request, or creates one using the supplied + /// browser mode, base URLs, and optional credentials. + /// + Task GetOrCreateAsync( + PlaywrightSessionRequest request, + CancellationToken cancellationToken = default); + + /// + /// Returns the session for , or if none exists. + /// + IPlaywrightSession GetSession(string chatSessionId, string ownerId = null); + + /// + /// Navigates to Orchard admin and attempts authentication if the current page is at the login screen + /// and the session has credentials available. + /// + Task EnsureAdminReadyAsync(IPlaywrightSession session, CancellationToken cancellationToken = default); + + /// + /// Cancels the current in-flight operation for the session. Browser stays open. + /// + void Stop(string chatSessionId, string ownerId = null); + + /// + /// Disposes the session and closes the browser window. + /// + Task CloseAsync(string chatSessionId, string ownerId = null); + + /// + /// Closes any Playwright sessions that have been idle past their configured timeout. + /// + Task CloseInactiveSessionsAsync(CancellationToken cancellationToken = default); + + /// + /// Returns all active (non-closed) sessions keyed by chat session ID. + /// + IReadOnlyDictionary GetActiveSessions(string ownerId = null); +} diff --git a/src/Modules/CrestApps.OrchardCore.AI.Playwright/Services/IPlaywrightSessionRequestResolver.cs b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Services/IPlaywrightSessionRequestResolver.cs new file mode 100644 index 000000000..bb28b1cd6 --- /dev/null +++ b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Services/IPlaywrightSessionRequestResolver.cs @@ -0,0 +1,8 @@ +using CrestApps.OrchardCore.AI.Playwright.Models; + +namespace CrestApps.OrchardCore.AI.Playwright.Services; + +public interface IPlaywrightSessionRequestResolver +{ + PlaywrightSessionRequest Resolve(object resource, string chatSessionId); +} diff --git a/src/Modules/CrestApps.OrchardCore.AI.Playwright/Services/OrchardAdminPlaywrightService.cs b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Services/OrchardAdminPlaywrightService.cs new file mode 100644 index 000000000..932e43a5b --- /dev/null +++ b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Services/OrchardAdminPlaywrightService.cs @@ -0,0 +1,1791 @@ +using System.Text.RegularExpressions; +using CrestApps.OrchardCore.AI.Playwright.Models; +using Microsoft.Extensions.Logging; +using Microsoft.Playwright; + +namespace CrestApps.OrchardCore.AI.Playwright.Services; + +public sealed class OrchardAdminPlaywrightService : IOrchardAdminPlaywrightService +{ + private const int EditorSubmitTimeoutMs = 60_000; + private static readonly Regex RowScopedActionPattern = new( + @"^(?Edit|View|Preview|Display|Delete|Clone|Duplicate|Publish|Unpublish|Draft|Save Draft)\s+(?.+)$", + RegexOptions.Compiled | RegexOptions.IgnoreCase); + private static readonly string[] ContentItemContainerSelectors = + [ + "tbody tr", + "table tr", + "tr", + "[data-content-item-id]", + ".content-item", + ".list-group-item", + "article", + "li", + ]; + private static readonly string[] RowActionMenuNames = + [ + "Actions", + "More actions", + "More", + "Options", + ]; + + private readonly IPlaywrightSessionManager _sessionManager; + private readonly IPlaywrightObservationService _observationService; + private readonly IPlaywrightActionVisualizer _actionVisualizer; + private readonly ILogger<OrchardAdminPlaywrightService> _logger; + + public OrchardAdminPlaywrightService( + IPlaywrightSessionManager sessionManager, + IPlaywrightObservationService observationService, + IPlaywrightActionVisualizer actionVisualizer, + ILogger<OrchardAdminPlaywrightService> logger) + { + _sessionManager = sessionManager; + _observationService = observationService; + _actionVisualizer = actionVisualizer; + _logger = logger; + } + + public Task<PlaywrightObservation> CaptureStateAsync(IPlaywrightSession session, CancellationToken cancellationToken = default) + => _observationService.CaptureAsync(session, cancellationToken); + + public async Task<PlaywrightObservation> OpenAdminHomeAsync(IPlaywrightSession session, CancellationToken cancellationToken = default) + { + await _sessionManager.EnsureAdminReadyAsync(session, cancellationToken); + return await _observationService.CaptureAsync(session, cancellationToken); + } + + public async Task<PlaywrightObservation> OpenContentItemsAsync(IPlaywrightSession session, CancellationToken cancellationToken = default) + { + await _sessionManager.EnsureAdminReadyAsync(session, cancellationToken); + var page = await GetActivePageAsync(session, cancellationToken); + + var targetUrl = PlaywrightSessionRequestResolver.CombineUrl(session.AdminBaseUrl, "Contents/ContentItems"); + await page.GotoAsync(targetUrl, new PageGotoOptions + { + WaitUntil = WaitUntilState.DOMContentLoaded, + Timeout = 30_000, + }).WaitAsync(cancellationToken); + + var observation = await _observationService.CaptureAsync(session, cancellationToken); + if (observation.IsLoginPage || !observation.IsAuthenticated) + { + return observation; + } + + if (!observation.IsLoginPage && !IsContentItemsPage(observation)) + { + await TryClickNamedElementAsync(page, "Content Items", cancellationToken); + observation = await _observationService.CaptureAsync(session, cancellationToken); + } + + return observation; + } + + public async Task<PlaywrightContentListResult> ListVisibleContentItemsAsync( + IPlaywrightSession session, + int maxItems = 20, + CancellationToken cancellationToken = default) + { + var observation = await EnsureContentItemsPageAsync(session, cancellationToken); + var page = await GetActivePageAsync(session, cancellationToken); + + var items = observation.IsLoginPage || !observation.IsAuthenticated + ? Array.Empty<PlaywrightContentListItem>() + : await GetVisibleContentItemsAsync(page, maxItems, cancellationToken); + + return new PlaywrightContentListResult + { + Url = page.Url, + Title = await page.TitleAsync().WaitAsync(cancellationToken), + MainHeading = observation.MainHeading, + ItemCount = items.Count, + Items = items, + }; + } + + public async Task<PlaywrightObservation> OpenNewContentItemAsync(IPlaywrightSession session, string contentType, CancellationToken cancellationToken = default) + { + ArgumentException.ThrowIfNullOrWhiteSpace(contentType); + + var observation = await OpenContentItemsAsync(session, cancellationToken); + if (observation.IsLoginPage || !observation.IsAuthenticated) + { + return observation; + } + + var page = await GetActivePageAsync(session, cancellationToken); + await TryClickAnyAsync(page, [("button", "New"), ("link", "New"), ("button", "Create"), ("link", "Create")], cancellationToken); + + var contentTypeLocator = await ResolveContentTypeLocatorAsync(page, contentType, cancellationToken); + await ClickWithoutNavigationWaitAsync(page, contentTypeLocator, $"selecting {contentType}", cancellationToken); + await WaitForEditorAsync(page, cancellationToken); + + return await _observationService.CaptureAsync(session, cancellationToken); + } + + public async Task<PlaywrightContentItemOpenResult> OpenContentItemEditorAsync( + IPlaywrightSession session, + string title, + CancellationToken cancellationToken = default) + { + ArgumentException.ThrowIfNullOrWhiteSpace(title); + + var observation = await EnsureContentItemsPageAsync(session, cancellationToken); + var page = await GetActivePageAsync(session, cancellationToken); + + if (observation.IsLoginPage || !observation.IsAuthenticated) + { + return new PlaywrightContentItemOpenResult + { + RequestedTitle = title, + Observation = observation, + }; + } + + var visibleItems = await GetVisibleContentItemsAsync(page, 25, cancellationToken); + var selection = SelectBestContentItem(visibleItems, title); + var usedSearch = false; + + if (selection is null && await TrySearchContentItemsAsync(page, title, cancellationToken)) + { + usedSearch = true; + visibleItems = await GetVisibleContentItemsAsync(page, 25, cancellationToken); + selection = SelectBestContentItem(visibleItems, title); + } + + if (selection is null) + { + return new PlaywrightContentItemOpenResult + { + RequestedTitle = title, + UsedSearch = usedSearch, + ClosestTitles = GetClosestTitles(visibleItems, title), + Observation = await _observationService.CaptureAsync(session, cancellationToken), + }; + } + + if (_logger.IsEnabled(LogLevel.Debug)) + { + _logger.LogDebug( + "Opening content item editor for requested title {RequestedTitle} using match {MatchedTitle} ({MatchMode}).", + title, + selection.Item.Title, + selection.MatchMode); + } + + if (!await TryClickContentItemRowActionAsync(page, "link", $"Edit {selection.Item.Title}", cancellationToken)) + { + return new PlaywrightContentItemOpenResult + { + RequestedTitle = title, + MatchedTitle = selection.Item.Title, + MatchMode = selection.MatchMode, + UsedSearch = usedSearch, + ClosestTitles = GetClosestTitles(visibleItems, title), + Observation = await _observationService.CaptureAsync(session, cancellationToken), + }; + } + + await WaitForEditorAsync(page, cancellationToken); + observation = await _observationService.CaptureAsync(session, cancellationToken); + + return new PlaywrightContentItemOpenResult + { + RequestedTitle = title, + MatchedTitle = selection.Item.Title, + MatchMode = selection.MatchMode, + UsedSearch = usedSearch, + ClosestTitles = GetClosestTitles(visibleItems, title), + Observation = observation, + }; + } + + public async Task<PlaywrightObservation> SetContentTitleAsync(IPlaywrightSession session, string title, CancellationToken cancellationToken = default) + { + ArgumentException.ThrowIfNullOrWhiteSpace(title); + + var page = await GetActivePageAsync(session, cancellationToken); + var titleField = page.GetByLabel("Title", new() { Exact = true }).First; + + if (await titleField.CountAsync().WaitAsync(cancellationToken) == 0) + { + titleField = page.Locator("input[name='TitlePart.Title'], input[id*='TitlePart_Title'], input[name='Title']").First; + } + + await ShowFillIndicatorAsync(page, titleField, "Title", cancellationToken); + await titleField.FillAsync(title, new LocatorFillOptions { Timeout = 10_000 }).WaitAsync(cancellationToken); + await titleField.PressAsync("Tab").WaitAsync(cancellationToken); + + return await _observationService.CaptureAsync(session, cancellationToken); + } + + public Task<PlaywrightObservation> SaveDraftAsync(IPlaywrightSession session, CancellationToken cancellationToken = default) + => ClickEditorActionAsync(session, ["Save Draft", "Save"], cancellationToken); + + public Task<PlaywrightObservation> PublishContentAsync(IPlaywrightSession session, CancellationToken cancellationToken = default) + => ClickEditorActionAsync(session, ["Publish", "Publish Draft"], cancellationToken); + + public async Task<PlaywrightEditorTargetResult> OpenEditorTabAsync( + IPlaywrightSession session, + string tabName, + bool exact = false, + CancellationToken cancellationToken = default) + { + ArgumentException.ThrowIfNullOrWhiteSpace(tabName); + + var page = await GetActivePageAsync(session, cancellationToken); + var target = await ResolveEditorTargetAsync(page, tabName, exact, cancellationToken); + if (target is null) + { + throw new TimeoutException($"Unable to locate an Orchard editor tab or section named '{tabName}'."); + } + + if (!await IsEditorTargetExpandedAsync(target.Locator, cancellationToken)) + { + await ClickWithoutNavigationWaitAsync(page, target.Locator, $"opening {target.MatchedText}", cancellationToken); + } + + var observation = await CaptureAfterInteractionAsync(session, page, cancellationToken); + + return new PlaywrightEditorTargetResult + { + RequestedTarget = tabName, + MatchedTarget = target.MatchedText, + TargetKind = target.TargetKind, + Observation = observation, + }; + } + + public async Task<PlaywrightFieldEditResult> SetFieldValueAsync( + IPlaywrightSession session, + string label, + string value, + string fieldType = "auto", + bool exact = false, + CancellationToken cancellationToken = default) + { + ArgumentException.ThrowIfNullOrWhiteSpace(label); + + var normalizedFieldType = NormalizeFieldType(fieldType); + var page = await GetActivePageAsync(session, cancellationToken); + var locator = await ResolveFieldLocatorAsync(page, label, exact, cancellationToken); + await ShowFillIndicatorAsync(page, locator, label, cancellationToken); + + var resolvedFieldType = await TrySetFieldValueAsync(locator, label, value ?? string.Empty, normalizedFieldType, append: false, cancellationToken); + if (string.IsNullOrWhiteSpace(resolvedFieldType)) + { + throw new TimeoutException($"Unable to set the Orchard field '{label}' as '{normalizedFieldType}'."); + } + + return new PlaywrightFieldEditResult + { + Label = label, + RequestedFieldType = normalizedFieldType, + RequestedEditMode = "replace", + ResolvedFieldType = resolvedFieldType, + Observation = await _observationService.CaptureAsync(session, cancellationToken), + }; + } + + public async Task<PlaywrightFieldEditResult> SetBodyFieldAsync( + IPlaywrightSession session, + string label, + string value, + string mode = "append", + bool exact = false, + CancellationToken cancellationToken = default) + { + ArgumentException.ThrowIfNullOrWhiteSpace(label); + + var normalizedMode = NormalizeBodyEditMode(mode); + var page = await GetActivePageAsync(session, cancellationToken); + var locator = await ResolveFieldLocatorAsync(page, label, exact, cancellationToken); + await ShowFillIndicatorAsync(page, locator, label, cancellationToken); + + var resolvedFieldType = await TrySetFieldValueAsync( + locator, + label, + value ?? string.Empty, + requestedFieldType: "richtext", + append: normalizedMode.Equals("append", StringComparison.Ordinal), + cancellationToken); + + if (string.IsNullOrWhiteSpace(resolvedFieldType)) + { + throw new TimeoutException($"Unable to edit the Orchard body field '{label}'."); + } + + return new PlaywrightFieldEditResult + { + Label = label, + RequestedFieldType = "body", + RequestedEditMode = normalizedMode, + ResolvedFieldType = resolvedFieldType, + Observation = await _observationService.CaptureAsync(session, cancellationToken), + }; + } + + public async Task<PlaywrightPublishVerificationResult> PublishAndVerifyAsync( + IPlaywrightSession session, + string expectedStatus = "Published", + CancellationToken cancellationToken = default) + { + var observation = await PublishContentAsync(session, cancellationToken); + var verificationSignals = BuildPublishVerificationSignals(observation, expectedStatus); + + return new PlaywrightPublishVerificationResult + { + Action = "publish", + ExpectedStatus = string.IsNullOrWhiteSpace(expectedStatus) ? "Published" : expectedStatus.Trim(), + Verified = verificationSignals.Count > 0, + VerificationSignals = verificationSignals, + Observation = observation, + }; + } + + private async Task<PlaywrightObservation> ClickEditorActionAsync( + IPlaywrightSession session, + IReadOnlyList<string> candidateNames, + CancellationToken cancellationToken) + { + var observation = await _observationService.CaptureAsync(session, cancellationToken); + if (observation.IsLoginPage || !observation.IsAuthenticated) + { + return observation; + } + + var page = await GetActivePageAsync(session, cancellationToken); + try + { + await TryClickEditorActionAsync(page, candidateNames, cancellationToken); + } + catch (TimeoutException ex) + { + var stalledObservation = await CaptureAfterEditorActionFailureAsync(session, page, cancellationToken); + throw new TimeoutException(BuildEditorActionFailureMessage(candidateNames, stalledObservation), ex); + } + + return await CaptureAfterEditorActionAsync(session, page, cancellationToken); + } + + private async Task<bool> TryClickContentItemRowActionAsync( + IPage page, + string role, + string name, + CancellationToken cancellationToken) + { + if (!role.Equals("link", StringComparison.OrdinalIgnoreCase) + && !role.Equals("button", StringComparison.OrdinalIgnoreCase)) + { + return false; + } + + var match = RowScopedActionPattern.Match(name); + if (!match.Success) + { + return false; + } + + var actionName = match.Groups["action"].Value.Trim(); + var contentItemTitle = match.Groups["title"].Value.Trim(); + var contentRow = await FindContentItemContainerAsync(page, contentItemTitle, cancellationToken); + + if (contentRow is null) + { + if (_logger.IsEnabled(LogLevel.Debug)) + { + _logger.LogDebug("Could not find a content row for {ContentItemTitle} while resolving {ActionName}.", contentItemTitle, actionName); + } + return false; + } + + if (_logger.IsEnabled(LogLevel.Debug)) + { + _logger.LogDebug("Trying row-scoped action {ActionName} for content item {ContentItemTitle}.", actionName, contentItemTitle); + } + + if (await TryClickActionWithinContainerAsync(page, contentRow, actionName, contentItemTitle, cancellationToken)) + { + return true; + } + + foreach (var menuName in RowActionMenuNames) + { + if (!await TryClickNamedElementAsync(page, contentRow, menuName, false, cancellationToken)) + { + continue; + } + + if (await TryClickAnyAsync(page, [("button", actionName), ("link", actionName)], false, false, cancellationToken)) + { + return true; + } + } + + if (_logger.IsEnabled(LogLevel.Debug)) + { + _logger.LogDebug( + "Found the content row for {ContentItemTitle} but no visible {ActionName} action inside it.", + contentItemTitle, + actionName); + } + + return false; + } + + private async Task<PlaywrightObservation> EnsureContentItemsPageAsync( + IPlaywrightSession session, + CancellationToken cancellationToken) + { + var observation = await _observationService.CaptureAsync(session, cancellationToken); + if (observation.IsLoginPage || !observation.IsAuthenticated) + { + return observation; + } + + if (IsContentItemsPage(observation)) + { + return observation; + } + + if (_logger.IsEnabled(LogLevel.Debug)) + { + _logger.LogDebug("Current page is not the Orchard content items list. Opening it for the active Playwright session."); + } + + return await OpenContentItemsAsync(session, cancellationToken); + } + + private static async Task<IPage> GetActivePageAsync(IPlaywrightSession session, CancellationToken cancellationToken) + { + return session switch + { + PlaywrightSession concreteSession => await concreteSession.GetOrCreatePageAsync(cancellationToken), + _ => session.Page, + }; + } + + private static bool IsContentItemsPage(PlaywrightObservation observation) + { + if (observation is null) + { + return false; + } + + return observation.CurrentUrl?.Contains("/Contents/ContentItems", StringComparison.OrdinalIgnoreCase) == true + || observation.MainHeading?.Contains("Content", StringComparison.OrdinalIgnoreCase) == true; + } + + private static ContentItemSelection SelectBestContentItem( + IReadOnlyList<PlaywrightContentListItem> items, + string requestedTitle) + { + ContentItemSelection best = null; + + foreach (var item in items) + { + var match = GetTitleMatch(item.Title, requestedTitle); + if (match.Score <= 0) + { + continue; + } + + if (best is null || match.Score > best.Score) + { + best = new ContentItemSelection + { + Item = item, + MatchMode = match.Mode, + Score = match.Score, + }; + } + } + + return best is { Score: >= 550 } ? best : null; + } + + private static IReadOnlyList<string> GetClosestTitles( + IReadOnlyList<PlaywrightContentListItem> items, + string requestedTitle) + { + return items + .Select(item => new + { + item.Title, + Match = GetTitleMatch(item.Title, requestedTitle), + }) + .Where(entry => entry.Match.Score > 0) + .OrderByDescending(entry => entry.Match.Score) + .Select(entry => entry.Title) + .Distinct(StringComparer.OrdinalIgnoreCase) + .Take(5) + .ToList(); + } + + private static ILocator GetLocatorByRole(IPage page, string role, string name, bool exact) + { + return role.ToLowerInvariant() switch + { + "button" => page.GetByRole(AriaRole.Button, new() { Name = name, Exact = exact }).First, + "link" => page.GetByRole(AriaRole.Link, new() { Name = name, Exact = exact }).First, + "textbox" => page.GetByRole(AriaRole.Textbox, new() { Name = name, Exact = exact }).First, + "menuitem" => page.GetByRole(AriaRole.Menuitem, new() { Name = name, Exact = exact }).First, + "tab" => page.GetByRole(AriaRole.Tab, new() { Name = name, Exact = exact }).First, + _ => throw new InvalidOperationException($"Unsupported role '{role}'."), + }; + } + + private static async Task<ILocator> ResolveFieldLocatorAsync( + IPage page, + string label, + bool exact, + CancellationToken cancellationToken) + { + var labeledLocator = page.GetByLabel(label, new() { Exact = exact }).First; + if (await labeledLocator.CountAsync().WaitAsync(cancellationToken) > 0) + { + return labeledLocator; + } + + var selectorValue = Regex.Replace(label, @"\s+", string.Empty); + var fallbackSelectors = new[] + { + $"textarea[id*='{selectorValue}' i], textarea[name*='{selectorValue}' i]", + $"input[id*='{selectorValue}' i], input[name*='{selectorValue}' i]", + $"select[id*='{selectorValue}' i], select[name*='{selectorValue}' i]", + $"[contenteditable='true'][aria-label*='{label}' i], iframe[title*='{label}' i]", + }; + + foreach (var selector in fallbackSelectors) + { + var locator = page.Locator(selector).First; + if (await locator.CountAsync().WaitAsync(cancellationToken) > 0) + { + return locator; + } + } + + throw new TimeoutException($"Unable to locate a field labeled '{label}'."); + } + + private static async Task<ILocator> ResolveContentTypeLocatorAsync( + IPage page, + string contentType, + CancellationToken cancellationToken) + { + var exactCandidates = new ILocator[] + { + page.GetByRole(AriaRole.Link, new() { Name = contentType, Exact = true }).First, + page.GetByRole(AriaRole.Button, new() { Name = contentType, Exact = true }).First, + page.GetByText(contentType, new PageGetByTextOptions { Exact = true }).First, + }; + + foreach (var candidate in exactCandidates) + { + if (await candidate.CountAsync().WaitAsync(cancellationToken) > 0) + { + return candidate; + } + } + + var relaxedCandidates = new ILocator[] + { + page.GetByRole(AriaRole.Link, new() { Name = contentType, Exact = false }).First, + page.GetByRole(AriaRole.Button, new() { Name = contentType, Exact = false }).First, + page.GetByText(contentType, new PageGetByTextOptions { Exact = false }).First, + }; + + foreach (var candidate in relaxedCandidates) + { + if (await candidate.CountAsync().WaitAsync(cancellationToken) > 0) + { + return candidate; + } + } + + return page.GetByText(contentType, new PageGetByTextOptions { Exact = false }).First; + } + + private static string NormalizeFieldType(string fieldType) + => (fieldType ?? "auto").Trim().ToLowerInvariant() switch + { + "" or "auto" => "auto", + "text" => "text", + "textarea" => "textarea", + "select" or "dropdown" => "select", + "checkbox" or "bool" or "boolean" => "checkbox", + "radio" => "radio", + "richtext" or "html" or "wysiwyg" => "richtext", + var unsupported => throw new InvalidOperationException($"Unsupported Orchard field type '{unsupported}'."), + }; + + private static string NormalizeBodyEditMode(string mode) + => (mode ?? "append").Trim().ToLowerInvariant() switch + { + "" or "append" => "append", + "replace" => "replace", + var unsupported => throw new InvalidOperationException($"Unsupported body edit mode '{unsupported}'. Use 'append' or 'replace'."), + }; + + private static IReadOnlyList<string> BuildPublishVerificationSignals(PlaywrightObservation observation, string expectedStatus) + { + var normalizedStatus = string.IsNullOrWhiteSpace(expectedStatus) + ? "published" + : expectedStatus.Trim().ToLowerInvariant(); + + var signals = new List<string>(); + if (observation is null) + { + return signals; + } + + if (!string.IsNullOrWhiteSpace(observation.ToastMessage) + && observation.ToastMessage.Contains(normalizedStatus, StringComparison.OrdinalIgnoreCase)) + { + signals.Add($"Toast confirmed {expectedStatus}."); + } + + if (observation.VisibleButtons.Any(button => button.Contains("Unpublish", StringComparison.OrdinalIgnoreCase))) + { + signals.Add("Editor actions now include Unpublish."); + } + + if (observation.VisibleButtons.Any(button => button.Contains("View", StringComparison.OrdinalIgnoreCase) + || button.Contains("Preview", StringComparison.OrdinalIgnoreCase))) + { + signals.Add("Editor actions now include view or preview."); + } + + if (observation.ValidationMessages.Count > 0) + { + return []; + } + + return signals + .Distinct(StringComparer.OrdinalIgnoreCase) + .ToList(); + } + + private static async Task<EditorTargetMatch> ResolveEditorTargetAsync( + IPage page, + string tabName, + bool exact, + CancellationToken cancellationToken) + { + foreach (var currentExact in exact ? new[] { true } : new[] { true, false }) + { + foreach (var (kind, locator) in new[] + { + ("tab", page.GetByRole(AriaRole.Tab, new() { Name = tabName, Exact = currentExact }).First), + ("tab", page.Locator($".nav-tabs a:has-text(\"{EscapeHasTextSelector(tabName)}\"), .nav-tabs button:has-text(\"{EscapeHasTextSelector(tabName)}\"), .nav-pills a:has-text(\"{EscapeHasTextSelector(tabName)}\"), .nav-pills button:has-text(\"{EscapeHasTextSelector(tabName)}\"), [data-bs-toggle='tab']:has-text(\"{EscapeHasTextSelector(tabName)}\")").First), + ("accordion", page.Locator($".accordion-button:has-text(\"{EscapeHasTextSelector(tabName)}\"), [data-bs-toggle='collapse']:has-text(\"{EscapeHasTextSelector(tabName)}\")").First), + ("summary", page.Locator($"details > summary:has-text(\"{EscapeHasTextSelector(tabName)}\"), summary:has-text(\"{EscapeHasTextSelector(tabName)}\")").First), + ("section", page.Locator($".card-header button:has-text(\"{EscapeHasTextSelector(tabName)}\"), .card-header a:has-text(\"{EscapeHasTextSelector(tabName)}\"), fieldset > legend:has-text(\"{EscapeHasTextSelector(tabName)}\")").First), + }) + { + if (await locator.CountAsync().WaitAsync(cancellationToken) == 0 + || !await locator.IsVisibleAsync().WaitAsync(cancellationToken)) + { + continue; + } + + var matchedText = NormalizeTitle(await locator.InnerTextAsync().WaitAsync(cancellationToken)) + ?? tabName; + + if (currentExact + && !matchedText.Equals(tabName, StringComparison.OrdinalIgnoreCase) + && !matchedText.Contains(tabName, StringComparison.OrdinalIgnoreCase)) + { + continue; + } + + return new EditorTargetMatch + { + Locator = locator, + MatchedText = matchedText, + TargetKind = kind, + }; + } + } + + return null; + } + + private static async Task<bool> IsEditorTargetExpandedAsync(ILocator locator, CancellationToken cancellationToken) + { + if (await locator.CountAsync().WaitAsync(cancellationToken) == 0) + { + return false; + } + + return await locator.EvaluateAsync<bool>( + """ + element => { + const ariaSelected = (element.getAttribute("aria-selected") || "").toLowerCase(); + const ariaExpanded = (element.getAttribute("aria-expanded") || "").toLowerCase(); + const className = (element.className || "").toString().toLowerCase(); + + if (ariaSelected === "true" || ariaExpanded === "true") { + return true; + } + + if (className.includes("active") || className.includes("show")) { + return true; + } + + const details = element.closest("details"); + if (details?.open) { + return true; + } + + const controls = element.getAttribute("aria-controls"); + if (controls) { + const target = document.getElementById(controls); + if (target) { + const targetClass = (target.className || "").toString().toLowerCase(); + const style = window.getComputedStyle(target); + if (targetClass.includes("show") || style.display !== "none") { + return true; + } + } + } + + return false; + } + """).WaitAsync(cancellationToken); + } + + private static string EscapeHasTextSelector(string value) + => (value ?? string.Empty) + .Replace("\\", "\\\\", StringComparison.Ordinal) + .Replace("\"", "\\\"", StringComparison.Ordinal); + + private async Task<bool> TryClickNamedElementAsync( + IPage page, + ILocator scope, + string name, + bool throwOnFailure, + CancellationToken cancellationToken) + { + foreach (var locator in new[] + { + scope.GetByRole(AriaRole.Link, new() { Name = name, Exact = true }).First, + scope.GetByRole(AriaRole.Button, new() { Name = name, Exact = true }).First, + scope.GetByRole(AriaRole.Link, new() { Name = name, Exact = false }).First, + scope.GetByRole(AriaRole.Button, new() { Name = name, Exact = false }).First, + scope.GetByText(name, new LocatorGetByTextOptions { Exact = true }).First, + scope.GetByText(name, new LocatorGetByTextOptions { Exact = false }).First, + }) + { + if (await locator.CountAsync().WaitAsync(cancellationToken) == 0) + { + continue; + } + + if (!await locator.IsVisibleAsync().WaitAsync(cancellationToken)) + { + continue; + } + + await ClickWithoutNavigationWaitAsync(page, locator, $"clicking {name}", cancellationToken); + + return true; + } + + if (throwOnFailure) + { + throw new TimeoutException($"Unable to click '{name}'."); + } + + return false; + } + + private async Task TryClickNamedElementAsync(IPage page, string name, CancellationToken cancellationToken) + { + await TryClickAnyAsync(page, [("link", name), ("button", name)], cancellationToken); + } + + private async Task<bool> TrySearchContentItemsAsync( + IPage page, + string searchText, + CancellationToken cancellationToken) + { + foreach (var candidate in new[] + { + page.GetByLabel("Search", new() { Exact = false }).First, + page.GetByPlaceholder("Search", new() { Exact = false }).First, + page.Locator("input[type='search'], input[name*='search' i], input[name='q'], input[placeholder*='search' i]").First, + }) + { + if (await candidate.CountAsync().WaitAsync(cancellationToken) == 0 + || !await candidate.IsVisibleAsync().WaitAsync(cancellationToken)) + { + continue; + } + + if (_logger.IsEnabled(LogLevel.Debug)) + { + _logger.LogDebug("Searching Orchard content items for {SearchText}.", searchText); + } + + await candidate.FillAsync(searchText, new LocatorFillOptions + { + Timeout = 10_000, + }).WaitAsync(cancellationToken); + + if (!await TryClickAnyAsync(page, [("button", "Search"), ("button", "Filter"), ("link", "Search")], false, false, cancellationToken)) + { + await candidate.PressAsync("Enter").WaitAsync(cancellationToken); + } + + await CaptureAfterInteractionAsyncFromPageAsync(page, cancellationToken); + return true; + } + + return false; + } + + private async Task TryClickAnyAsync( + IPage page, + IReadOnlyList<(string role, string name)> candidates, + CancellationToken cancellationToken) + { + if (page.IsClosed) + { + throw new InvalidOperationException("The Playwright browser page is no longer available."); + } + + foreach (var candidate in candidates) + { + var locator = GetLocatorByRole(page, candidate.role, candidate.name, false); + + if (await locator.CountAsync().WaitAsync(cancellationToken) == 0) + { + continue; + } + + if (await locator.IsVisibleAsync().WaitAsync(cancellationToken)) + { + await ClickWithoutNavigationWaitAsync(page, locator, $"clicking {candidate.name}", cancellationToken); + return; + } + } + + var fallback = candidates[0]; + var textLocator = page.GetByText(fallback.name, new PageGetByTextOptions { Exact = true }).First; + await ClickWithoutNavigationWaitAsync(page, textLocator, $"clicking {fallback.name}", cancellationToken); + } + + private async Task TryClickEditorActionAsync( + IPage page, + IReadOnlyList<string> candidateNames, + CancellationToken cancellationToken) + { + if (page.IsClosed) + { + throw new InvalidOperationException("The Playwright browser page is no longer available."); + } + + foreach (var name in candidateNames) + { + if (await TryClickAnyAsync(page, [("button", name), ("link", name)], false, true, cancellationToken)) + { + return; + } + } + + foreach (var name in candidateNames) + { + var submitLocator = page.Locator( + $"button[name*='{name}' i], input[type='submit'][value*='{name}' i], button[title*='{name}' i], [aria-label*='{name}' i]").First; + + if (await submitLocator.CountAsync().WaitAsync(cancellationToken) > 0 + && await submitLocator.IsVisibleAsync().WaitAsync(cancellationToken)) + { + await ClickWithoutNavigationWaitAsync(page, submitLocator, $"clicking {name}", cancellationToken); + return; + } + } + + var visibleButtons = await page.EvaluateAsync<string[]>( + @"() => Array.from(document.querySelectorAll('button, input[type=""submit""], a.btn')) + .filter(element => element.offsetParent !== null) + .map(element => (element.innerText || element.value || element.getAttribute('aria-label') || '').trim()) + .filter(Boolean) + .slice(0, 12)").WaitAsync(cancellationToken); + + throw new TimeoutException( + $"No matching editor action was found. Tried: {string.Join(", ", candidateNames)}. Visible actions: {string.Join(", ", visibleButtons ?? [])}"); + } + + private async Task<bool> TryClickAnyAsync( + IPage page, + IReadOnlyList<(string role, string name)> candidates, + bool throwOnFailure, + bool treatAsSubmit, + CancellationToken cancellationToken) + { + if (page.IsClosed) + { + throw new InvalidOperationException("The Playwright browser page is no longer available."); + } + + foreach (var candidate in candidates) + { + var locator = GetLocatorByRole(page, candidate.role, candidate.name, false); + + if (await locator.CountAsync().WaitAsync(cancellationToken) == 0) + { + continue; + } + + if (await locator.IsVisibleAsync().WaitAsync(cancellationToken)) + { + await ClickWithoutNavigationWaitAsync( + page, + locator, + treatAsSubmit ? $"submitting {candidate.name}" : $"clicking {candidate.name}", + cancellationToken); + return true; + } + } + + var fallback = candidates[0]; + var textLocator = page.GetByText(fallback.name, new PageGetByTextOptions { Exact = true }).First; + if (await textLocator.CountAsync().WaitAsync(cancellationToken) > 0 + && await textLocator.IsVisibleAsync().WaitAsync(cancellationToken)) + { + await ClickWithoutNavigationWaitAsync( + page, + textLocator, + treatAsSubmit ? $"submitting {fallback.name}" : $"clicking {fallback.name}", + cancellationToken); + return true; + } + + if (throwOnFailure) + { + throw new TimeoutException($"Unable to click '{fallback.name}'."); + } + + return false; + } + + private async Task<PlaywrightObservation> CaptureAfterInteractionAsync( + IPlaywrightSession session, + IPage page, + CancellationToken cancellationToken) + { + await CaptureAfterInteractionAsyncFromPageAsync(page, cancellationToken); + return await _observationService.CaptureAsync(session, cancellationToken); + } + + private async Task<PlaywrightObservation> CaptureAfterEditorActionAsync( + IPlaywrightSession session, + IPage page, + CancellationToken cancellationToken) + { + try + { + await page.WaitForLoadStateAsync(LoadState.DOMContentLoaded, new PageWaitForLoadStateOptions + { + Timeout = 15_000, + }).WaitAsync(cancellationToken); + } + catch (TimeoutException) + { + // Some editor actions update in-place and never trigger a full navigation. + } + + try + { + await page.WaitForLoadStateAsync(LoadState.NetworkIdle, new PageWaitForLoadStateOptions + { + Timeout = 5_000, + }).WaitAsync(cancellationToken); + } + catch (TimeoutException) + { + // Network activity can continue after the editor returns. Observation below is authoritative. + } + + return await _observationService.CaptureAsync(session, cancellationToken); + } + + private async Task<PlaywrightObservation> CaptureAfterEditorActionFailureAsync( + IPlaywrightSession session, + IPage page, + CancellationToken cancellationToken) + { + try + { + await page.WaitForLoadStateAsync(LoadState.DOMContentLoaded, new PageWaitForLoadStateOptions + { + Timeout = 2_000, + }).WaitAsync(cancellationToken); + } + catch (TimeoutException) + { + } + + return await _observationService.CaptureAsync(session, cancellationToken); + } + + private static string BuildEditorActionFailureMessage( + IReadOnlyList<string> candidateNames, + PlaywrightObservation observation) + { + var details = new List<string> + { + $"Action did not finish in time after clicking: {string.Join(", ", candidateNames)}.", + }; + + if (!string.IsNullOrWhiteSpace(observation?.MainHeading)) + { + details.Add($"Heading: {observation.MainHeading}."); + } + + if (!string.IsNullOrWhiteSpace(observation?.ToastMessage)) + { + details.Add($"Toast: {observation.ToastMessage}."); + } + + if (observation?.ValidationMessages?.Count > 0) + { + details.Add($"Messages: {string.Join(" | ", observation.ValidationMessages)}."); + } + + if (!string.IsNullOrWhiteSpace(observation?.CurrentUrl)) + { + details.Add($"URL: {observation.CurrentUrl}."); + } + + if (observation?.VisibleButtons?.Count > 0) + { + details.Add($"Visible actions: {string.Join(", ", observation.VisibleButtons)}."); + } + + return string.Join(" ", details); + } + + private static async Task<string> TrySetFieldValueAsync( + ILocator locator, + string label, + string value, + string requestedFieldType, + bool append, + CancellationToken cancellationToken) + { + if (await locator.CountAsync().WaitAsync(cancellationToken) == 0) + { + return null; + } + + return await locator.EvaluateAsync<string>( + """ + (element, payload) => { + const normalize = (input) => (input ?? "").toString().trim().replace(/\s+/g, " "); + const isVisible = (candidate) => { + if (!candidate) { + return false; + } + + const style = window.getComputedStyle(candidate); + if (style.visibility === "hidden" || style.display === "none") { + return false; + } + + return candidate.offsetParent !== null || style.position === "fixed"; + }; + + const dispatch = (candidate) => { + for (const eventName of ["input", "change", "blur"]) { + candidate.dispatchEvent(new Event(eventName, { bubbles: true })); + } + }; + + const appendPlainText = (currentValue, nextValue, separator) => { + if (!nextValue) { + return currentValue ?? ""; + } + + if (!currentValue) { + return nextValue; + } + + return `${currentValue}${separator}${nextValue}`; + }; + + const findFirstVisible = (candidates) => candidates.find((candidate) => isVisible(candidate)); + + const parseBoolean = (input) => { + const normalized = normalize(input).toLowerCase(); + if (["true", "1", "yes", "y", "on", "checked", "check", "selected"].includes(normalized)) { + return true; + } + + if (["false", "0", "no", "n", "off", "unchecked", "uncheck", "clear"].includes(normalized)) { + return false; + } + + return null; + }; + + const setStandardValue = (candidate, append) => { + if (!(candidate instanceof HTMLInputElement) && !(candidate instanceof HTMLTextAreaElement)) { + return null; + } + + if (!isVisible(candidate)) { + return null; + } + + const separator = candidate instanceof HTMLTextAreaElement || append ? "\n\n" : " "; + candidate.focus(); + candidate.value = append ? appendPlainText(candidate.value, payload.value, separator) : payload.value; + dispatch(candidate); + + if (candidate instanceof HTMLTextAreaElement) { + return "textarea"; + } + + return "input"; + }; + + const setSelectValue = (candidate) => { + if (!(candidate instanceof HTMLSelectElement) || !isVisible(candidate)) { + return null; + } + + const requestedValue = normalize(payload.value).toLowerCase(); + const options = Array.from(candidate.options || []); + const matched = options.find((option) => { + return normalize(option.value).toLowerCase() === requestedValue + || normalize(option.label).toLowerCase() === requestedValue + || normalize(option.text).toLowerCase() === requestedValue; + }) || options.find((option) => { + return normalize(option.value).toLowerCase().includes(requestedValue) + || normalize(option.label).toLowerCase().includes(requestedValue) + || normalize(option.text).toLowerCase().includes(requestedValue); + }); + + if (!matched) { + return null; + } + + candidate.value = matched.value; + dispatch(candidate); + return "select"; + }; + + const setBooleanValue = (candidate) => { + if (!(candidate instanceof HTMLInputElement) || !isVisible(candidate)) { + return null; + } + + if (candidate.type !== "checkbox" && candidate.type !== "radio") { + return null; + } + + const nextValue = parseBoolean(payload.value); + if (nextValue === null) { + return null; + } + + if (candidate.type === "radio" && nextValue === false) { + return null; + } + + candidate.focus(); + candidate.checked = nextValue; + dispatch(candidate); + return candidate.type; + }; + + const escapeHtml = (input) => { + const div = document.createElement("div"); + div.textContent = input ?? ""; + return div.innerHTML; + }; + + const toParagraphHtml = (input) => { + const lines = (input ?? "").split(/\r?\n/).map((line) => line.trim()).filter(Boolean); + if (lines.length === 0) { + return ""; + } + + return lines.map((line) => `<p>${escapeHtml(line)}</p>`).join(""); + }; + + const appendRichText = (root, nextValue, append) => { + const htmlToAppend = toParagraphHtml(nextValue); + if (!htmlToAppend) { + return root.innerHTML ?? ""; + } + + if (append && normalize(root.innerText)) { + root.insertAdjacentHTML("beforeend", htmlToAppend); + } else { + root.innerHTML = htmlToAppend; + } + + dispatch(root); + return root.innerHTML ?? ""; + }; + + const syncSourceField = (content) => { + if (element instanceof HTMLTextAreaElement || element instanceof HTMLInputElement) { + element.value = content ?? ""; + dispatch(element); + } + }; + + const fieldId = element.id || ""; + const fieldName = element.getAttribute("name") || ""; + const labelToken = normalize(payload.label).replace(/\s+/g, "").toLowerCase(); + const fieldTokens = [fieldId, fieldName, payload.label, labelToken] + .filter(Boolean) + .map((token) => token.toString().toLowerCase()); + + const matchesField = (candidate) => { + const values = [ + candidate?.id, + candidate?.getAttribute?.("name"), + candidate?.getAttribute?.("aria-label"), + candidate?.getAttribute?.("title"), + candidate?.getAttribute?.("data-field-name"), + candidate?.getAttribute?.("for"), + ] + .filter(Boolean) + .map((token) => token.toString().toLowerCase()); + + return fieldTokens.some((token) => values.some((value) => value.includes(token))); + }; + + const container = element.closest(".mb-3, .form-group, .field, fieldset, section, .card-body, .content-field") + || element.parentElement + || document.body; + + const requestedKind = normalize(payload.requestedFieldType).toLowerCase(); + const preferAppend = Boolean(payload.append); + const preferRichText = requestedKind === "richtext"; + const preferSelect = requestedKind === "select"; + const preferCheckbox = requestedKind === "checkbox" || requestedKind === "radio" || requestedKind === "boolean"; + const directMode = setBooleanValue(element) + || setSelectValue(element) + || setStandardValue( + element, + preferAppend || element instanceof HTMLTextAreaElement || Number.parseInt(element.getAttribute?.("rows") || "0", 10) > 1); + + if (directMode && (!preferSelect || directMode === "select") && (!preferCheckbox || directMode === requestedKind || directMode === "checkbox")) { + return directMode; + } + + if (fieldId && window.tinymce && typeof window.tinymce.get === "function") { + const editor = window.tinymce.get(fieldId); + if (editor) { + const currentContent = editor.getContent({ format: "html" }) || ""; + const nextHtml = toParagraphHtml(payload.value); + const combinedContent = preferAppend && currentContent + ? `${currentContent}${nextHtml}` + : nextHtml; + + editor.setContent(combinedContent); + editor.save(); + syncSourceField(combinedContent); + return "tinymce"; + } + } + + const standardCandidates = [ + ...container.querySelectorAll("textarea, input, select"), + ...document.querySelectorAll("textarea, input, select"), + ]; + + for (const candidate of standardCandidates) { + if (candidate === element) { + continue; + } + + if (!matchesField(candidate) && !isVisible(candidate)) { + continue; + } + + const mode = setBooleanValue(candidate) + || setSelectValue(candidate) + || setStandardValue( + candidate, + preferAppend || candidate instanceof HTMLTextAreaElement || Number.parseInt(candidate.getAttribute("rows") || "0", 10) > 1); + + if (preferSelect && mode !== "select") { + continue; + } + + if (preferCheckbox && mode !== requestedKind && mode !== "checkbox") { + continue; + } + + if (mode) { + syncSourceField(candidate.value ?? ""); + return `${mode}-related`; + } + } + + const editableCandidates = [ + ...container.querySelectorAll("[contenteditable='true'], .ck-editor__editable, .ProseMirror, .trumbowyg-editor, .ql-editor"), + ...document.querySelectorAll("[contenteditable='true'], .ck-editor__editable, .ProseMirror, .trumbowyg-editor, .ql-editor"), + ]; + + const editable = findFirstVisible(editableCandidates.filter((candidate) => matchesField(candidate) || container.contains(candidate))); + if (editable && !preferSelect && !preferCheckbox) { + const updatedHtml = appendRichText(editable, payload.value, preferAppend || normalize(editable.innerText).length > 0); + syncSourceField(updatedHtml); + return "contenteditable"; + } + + const iframeCandidates = [ + ...container.querySelectorAll("iframe"), + ...document.querySelectorAll("iframe"), + ].filter((candidate) => matchesField(candidate) || container.contains(candidate)); + + for (const iframe of iframeCandidates) { + if (!isVisible(iframe)) { + continue; + } + + try { + const doc = iframe.contentDocument; + if (!doc?.body) { + continue; + } + + const updatedHtml = appendRichText(doc.body, payload.value, preferAppend || normalize(doc.body.innerText).length > 0); + syncSourceField(updatedHtml); + return "iframe"; + } catch { + // Ignore inaccessible iframe candidates and continue. + } + } + + if (preferRichText) { + return null; + } + + return null; + } + """, + new + { + label, + value, + append, + requestedFieldType, + }).WaitAsync(cancellationToken); + } + + private static bool ShouldAppendToField(string label) + => label.Contains("body", StringComparison.OrdinalIgnoreCase) + || label.Contains("html", StringComparison.OrdinalIgnoreCase) + || label.Contains("content", StringComparison.OrdinalIgnoreCase) + || label.Contains("description", StringComparison.OrdinalIgnoreCase) + || label.Contains("summary", StringComparison.OrdinalIgnoreCase); + + private static async Task<IReadOnlyList<PlaywrightContentListItem>> GetVisibleContentItemsAsync( + IPage page, + int maxItems, + CancellationToken cancellationToken) + { + maxItems = Math.Clamp(maxItems, 1, 25); + + var payload = await page.EvaluateAsync<VisibleContentItemPayload[]>( + """ + (maxItems) => { + const normalize = (value) => (value || "").replace(/\s+/g, " ").trim(); + const isVisible = (element) => { + if (!(element instanceof HTMLElement) || element.hidden) { + return false; + } + + const style = window.getComputedStyle(element); + if (style.display === "none" || style.visibility === "hidden" || style.opacity === "0") { + return false; + } + + return element.offsetWidth > 0 || element.offsetHeight > 0 || element.getClientRects().length > 0; + }; + + const isActionLine = (value) => /^(edit|view|preview|delete|clone|duplicate|save|publish|unpublish|actions|more|more actions|options)$/i.test(value); + const isStatusLine = (value) => /(draft|published|unpublished|latest|modified)/i.test(value); + const results = []; + const seen = new Set(); + const rows = Array.from(document.querySelectorAll("tbody tr, table tr, [data-content-item-id], .content-item, .list-group-item, article")); + + for (const row of rows) { + if (!isVisible(row)) { + continue; + } + + const lines = normalize(row.innerText).split(/\s{2,}|\n+/).map(normalize).filter(Boolean); + const filteredLines = lines.filter((line) => !isActionLine(line)); + const linkTexts = Array.from(row.querySelectorAll("a, button, strong, h2, h3, h4")) + .filter((element) => isVisible(element)) + .map((element) => normalize(element.innerText || element.textContent || "")) + .filter((text) => text.length > 2 && text.length < 180 && !isActionLine(text)); + + const title = linkTexts.find((text) => !isStatusLine(text)) + || filteredLines.find((line) => line.length > 2 && line.length < 180 && !isStatusLine(line)) + || ""; + if (!title) { + continue; + } + + const titleKey = title.toLowerCase(); + if (seen.has(titleKey)) { + continue; + } + + seen.add(titleKey); + const status = filteredLines.find((line) => isStatusLine(line)) || ""; + const contentType = filteredLines.find((line, index) => index > 0 && line !== title && line !== status && line.length < 100) || ""; + const canEdit = Array.from(row.querySelectorAll("a, button, input[type='submit']")).some((element) => { + const text = normalize(element.innerText || element.value || element.getAttribute("aria-label") || element.getAttribute("title") || ""); + return /edit/i.test(text); + }); + + results.push({ + title, + contentType, + status, + canEdit + }); + + if (results.length >= maxItems) { + break; + } + } + + return results; + } + """, + maxItems).WaitAsync(cancellationToken); + + return (payload ?? []) + .Select(item => new PlaywrightContentListItem + { + Title = NormalizeTitle(item.Title), + ContentType = NormalizeTitle(item.ContentType), + Status = NormalizeTitle(item.Status), + CanEdit = item.CanEdit, + }) + .Where(item => !string.IsNullOrWhiteSpace(item.Title)) + .ToList(); + } + + private async Task ShowFillIndicatorAsync( + IPage page, + ILocator locator, + string label, + CancellationToken cancellationToken) + { + try + { + await _actionVisualizer.ShowLocatorActionAsync(page, locator, "Typing", label, cancellationToken); + } + catch + { + await _actionVisualizer.ShowPageActionAsync(page, "Typing", label, cancellationToken); + } + } + + private async Task ClickWithoutNavigationWaitAsync( + IPage page, + ILocator locator, + string actionDescription, + CancellationToken cancellationToken) + { + await locator.WaitForAsync(new LocatorWaitForOptions + { + State = WaitForSelectorState.Visible, + Timeout = 10_000, + }).WaitAsync(cancellationToken); + + await _actionVisualizer.ShowLocatorActionAsync(page, locator, "AI", actionDescription, cancellationToken); + await locator.ScrollIntoViewIfNeededAsync().WaitAsync(cancellationToken); + await locator.ClickAsync(new LocatorClickOptions + { + Timeout = 10_000, + }).WaitAsync(cancellationToken); + } + + private static async Task CaptureAfterInteractionAsyncFromPageAsync(IPage page, CancellationToken cancellationToken) + { + try + { + await page.WaitForLoadStateAsync(LoadState.DOMContentLoaded, new PageWaitForLoadStateOptions + { + Timeout = 10_000, + }).WaitAsync(cancellationToken); + } + catch (TimeoutException) + { + } + } + + private static (int Score, string Mode) GetTitleMatch(string candidateTitle, string requestedTitle) + { + if (string.IsNullOrWhiteSpace(candidateTitle) || string.IsNullOrWhiteSpace(requestedTitle)) + { + return (0, null); + } + + if (candidateTitle.Equals(requestedTitle, StringComparison.OrdinalIgnoreCase)) + { + return (1000, "Exact"); + } + + var normalizedCandidate = NormalizeTitleKey(candidateTitle); + var normalizedRequested = NormalizeTitleKey(requestedTitle); + if (string.IsNullOrWhiteSpace(normalizedCandidate) || string.IsNullOrWhiteSpace(normalizedRequested)) + { + return (0, null); + } + + if (normalizedCandidate.Equals(normalizedRequested, StringComparison.Ordinal)) + { + return (950, "Normalized"); + } + + if (normalizedCandidate.Contains(normalizedRequested, StringComparison.Ordinal) + || normalizedRequested.Contains(normalizedCandidate, StringComparison.Ordinal)) + { + return (860, "Contains"); + } + + var requestedTokens = TokenizeTitle(requestedTitle); + var candidateTokens = TokenizeTitle(candidateTitle); + var overlap = requestedTokens.Intersect(candidateTokens, StringComparer.OrdinalIgnoreCase).Count(); + + if (overlap == requestedTokens.Count && overlap > 0) + { + return (880, "TokenMatch"); + } + + if (overlap > 0) + { + return (700 + (overlap * 40), "PartialTokenMatch"); + } + + var distance = GetLevenshteinDistance(normalizedCandidate, normalizedRequested); + var threshold = Math.Max(2, Math.Max(normalizedCandidate.Length, normalizedRequested.Length) / 4); + if (distance <= threshold) + { + return (650 - (distance * 25), "Fuzzy"); + } + + return (0, null); + } + + private static string NormalizeTitle(string value) + => string.IsNullOrWhiteSpace(value) + ? null + : Regex.Replace(value.Trim(), @"\s+", " "); + + private static string NormalizeTitleKey(string value) + => string.Concat((value ?? string.Empty) + .Where(char.IsLetterOrDigit)) + .ToLowerInvariant(); + + private static IReadOnlyList<string> TokenizeTitle(string value) + => Regex.Split(value ?? string.Empty, @"[\s\-_:/\\]+") + .Select(token => token.Trim()) + .Where(token => !string.IsNullOrWhiteSpace(token)) + .ToList(); + + private static int GetLevenshteinDistance(string source, string target) + { + var matrix = new int[source.Length + 1, target.Length + 1]; + + for (var i = 0; i <= source.Length; i++) + { + matrix[i, 0] = i; + } + + for (var j = 0; j <= target.Length; j++) + { + matrix[0, j] = j; + } + + for (var i = 1; i <= source.Length; i++) + { + for (var j = 1; j <= target.Length; j++) + { + var cost = source[i - 1] == target[j - 1] ? 0 : 1; + matrix[i, j] = Math.Min( + Math.Min( + matrix[i - 1, j] + 1, + matrix[i, j - 1] + 1), + matrix[i - 1, j - 1] + cost); + } + } + + return matrix[source.Length, target.Length]; + } + + private static async Task<ILocator> FindContentItemContainerAsync( + IPage page, + string contentItemTitle, + CancellationToken cancellationToken) + { + foreach (var exact in new[] { true, false }) + { + foreach (var selector in ContentItemContainerSelectors) + { + var locator = page.Locator(selector).Filter(new LocatorFilterOptions + { + Has = page.GetByText(contentItemTitle, new PageGetByTextOptions { Exact = exact }), + }).First; + + if (await locator.CountAsync().WaitAsync(cancellationToken) == 0) + { + continue; + } + + if (await locator.IsVisibleAsync().WaitAsync(cancellationToken)) + { + return locator; + } + } + } + + return null; + } + + private async Task<bool> TryClickActionWithinContainerAsync( + IPage page, + ILocator container, + string actionName, + string contentItemTitle, + CancellationToken cancellationToken) + { + var exactCandidates = new ILocator[] + { + container.GetByRole(AriaRole.Link, new() { Name = actionName, Exact = true }).First, + container.GetByRole(AriaRole.Button, new() { Name = actionName, Exact = true }).First, + container.GetByText(actionName, new LocatorGetByTextOptions { Exact = true }).First, + }; + + foreach (var locator in exactCandidates) + { + if (await locator.CountAsync().WaitAsync(cancellationToken) == 0) + { + continue; + } + + if (!await locator.IsVisibleAsync().WaitAsync(cancellationToken)) + { + continue; + } + + await ClickWithoutNavigationWaitAsync(page, locator, $"clicking {actionName} for {contentItemTitle}", cancellationToken); + + return true; + } + + var fallback = container.Locator( + $"a[title*='{actionName}' i], button[title*='{actionName}' i], a[aria-label*='{actionName}' i], button[aria-label*='{actionName}' i], input[type='submit'][value*='{actionName}' i]").First; + + if (await fallback.CountAsync().WaitAsync(cancellationToken) == 0 + || !await fallback.IsVisibleAsync().WaitAsync(cancellationToken)) + { + return actionName.Equals("Edit", StringComparison.OrdinalIgnoreCase) + && await TryOpenContentItemTitleWithinContainerAsync(page, container, contentItemTitle, cancellationToken); + } + + await ClickWithoutNavigationWaitAsync(page, fallback, $"clicking {actionName} for {contentItemTitle}", cancellationToken); + + return true; + } + + private async Task<bool> TryOpenContentItemTitleWithinContainerAsync( + IPage page, + ILocator container, + string contentItemTitle, + CancellationToken cancellationToken) + { + foreach (var exact in new[] { true, false }) + { + foreach (var locator in new[] + { + container.GetByRole(AriaRole.Link, new() { Name = contentItemTitle, Exact = exact }).First, + container.GetByRole(AriaRole.Button, new() { Name = contentItemTitle, Exact = exact }).First, + container.GetByText(contentItemTitle, new LocatorGetByTextOptions { Exact = exact }).First, + }) + { + if (await locator.CountAsync().WaitAsync(cancellationToken) == 0 + || !await locator.IsVisibleAsync().WaitAsync(cancellationToken)) + { + continue; + } + + await ClickWithoutNavigationWaitAsync(page, locator, $"opening {contentItemTitle}", cancellationToken); + return true; + } + } + + return false; + } + + private sealed class VisibleContentItemPayload + { + public string Title { get; set; } + + public string ContentType { get; set; } + + public string Status { get; set; } + + public bool CanEdit { get; set; } + } + + private sealed class ContentItemSelection + { + public PlaywrightContentListItem Item { get; init; } + + public string MatchMode { get; init; } + + public int Score { get; init; } + } + + private sealed class EditorTargetMatch + { + public ILocator Locator { get; init; } + + public string MatchedText { get; init; } + + public string TargetKind { get; init; } + } + + private static async Task WaitForEditorAsync(IPage page, CancellationToken cancellationToken) + { + var titleLocator = page.GetByLabel("Title", new() { Exact = true }).First; + + try + { + await titleLocator.WaitForAsync(new LocatorWaitForOptions + { + State = WaitForSelectorState.Visible, + Timeout = 15_000, + }).WaitAsync(cancellationToken); + } + catch (TimeoutException) + { + var fallback = page.Locator("input[name='TitlePart.Title'], input[id*='TitlePart_Title']").First; + await fallback.WaitForAsync(new LocatorWaitForOptions + { + State = WaitForSelectorState.Visible, + Timeout = 15_000, + }).WaitAsync(cancellationToken); + } + } +} diff --git a/src/Modules/CrestApps.OrchardCore.AI.Playwright/Services/OrchardEvidenceService.cs b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Services/OrchardEvidenceService.cs new file mode 100644 index 000000000..38ab24948 --- /dev/null +++ b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Services/OrchardEvidenceService.cs @@ -0,0 +1,266 @@ +using CrestApps.OrchardCore.AI.Playwright.Models; +using Microsoft.Extensions.Hosting; +using Microsoft.Extensions.Logging; +using Microsoft.Playwright; +using OrchardClock = OrchardCore.Modules.IClock; + +namespace CrestApps.OrchardCore.AI.Playwright.Services; + +/// <summary> +/// Locates a named OrchardCore admin action using 5-tier priority locator strategies. +/// Captures structured evidence (screenshots, HTML) when the action is not found. +/// </summary> +public sealed class OrchardEvidenceService : IOrchardEvidenceService +{ + // OrchardCore container selectors tried in order when capturing a container screenshot. + private static readonly string[] _containerSelectors = + [ + ".content-item-actions", + "[class*='actions']", + ".dropdown-menu.show", + ".card-footer", + ".card-header", + ".sticky-top", + "main", + "form", + ".content", + ]; + + private readonly OrchardClock _clock; + private readonly IHostEnvironment _hostEnvironment; + private readonly ILogger<OrchardEvidenceService> _logger; + + public OrchardEvidenceService( + OrchardClock clock, + IHostEnvironment hostEnvironment, + ILogger<OrchardEvidenceService> logger) + { + _clock = clock; + _hostEnvironment = hostEnvironment; + _logger = logger; + } + + public async Task<PlaywrightElementDiagnosticsResult> FindOrchardElementWithEvidenceAsync( + IPlaywrightSession session, + string actionLabel, + CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(session); + ArgumentException.ThrowIfNullOrWhiteSpace(actionLabel); + + var page = GetPage(session); + var url = page.Url; + var title = await page.TitleAsync().WaitAsync(cancellationToken); + var strategies = BuildLocatorStrategies(page, actionLabel); + var attempts = new List<string>(strategies.Count); + + _logger.LogDebug( + "Searching for OrchardCore action '{ActionLabel}' using {StrategyCount} locator strategies for session '{SessionId}'.", + actionLabel, + strategies.Count, + session.SessionId); + + foreach (var (name, locator) in strategies) + { + attempts.Add(name); + + try + { + var count = await locator.CountAsync().WaitAsync(cancellationToken); + if (count > 0) + { + _logger.LogDebug( + "Found OrchardCore action '{ActionLabel}' via strategy '{StrategyName}' ({Count} match(es)) for session '{SessionId}'.", + actionLabel, + name, + count, + session.SessionId); + + return new PlaywrightElementDiagnosticsResult + { + Found = true, + MatchedLocator = name, + Url = url, + Title = title, + Attempts = attempts, + }; + } + } + catch (Exception ex) + { + _logger.LogDebug( + ex, + "Locator strategy '{StrategyName}' threw for action '{ActionLabel}' in session '{SessionId}'.", + name, + actionLabel, + session.SessionId); + } + } + + // All strategies exhausted — capture diagnostic evidence. + _logger.LogInformation( + "OrchardCore action '{ActionLabel}' not found after {StrategyCount} strategies. Capturing evidence for session '{SessionId}'.", + actionLabel, + strategies.Count, + session.SessionId); + + var (pagePath, containerPath, htmlPath) = await CaptureEvidenceAsync(page, session.SessionId, cancellationToken); + + return new PlaywrightElementDiagnosticsResult + { + Found = false, + Url = url, + Title = title, + PageScreenshotPath = pagePath, + ContainerScreenshotPath = containerPath, + PageHtmlPath = htmlPath, + Attempts = attempts, + }; + } + + // ------------------------------------------------------------------------- + // Locator strategies — OrchardCore-specific, 5 tiers in priority order + // ------------------------------------------------------------------------- + + private static IReadOnlyList<(string Name, ILocator Locator)> BuildLocatorStrategies(IPage page, string actionLabel) + { + var escaped = actionLabel.Replace("'", "\\'"); + + return + [ + // Tier 1: Role link — Edit / Preview appear as <a> in content list rows + ( + $"role:link[name='{actionLabel}']", + page.GetByRole(AriaRole.Link, new() { Name = actionLabel, Exact = true }) + ), + + // Tier 2: Role button — Publish Now / Save Draft are <button> in editor toolbars + ( + $"role:button[name='{actionLabel}']", + page.GetByRole(AriaRole.Button, new() { Name = actionLabel, Exact = true }) + ), + + // Tier 3: Accessible text match — catches labels rendered in spans, badges, or custom elements + ( + $"text:exact['{actionLabel}']", + page.GetByText(actionLabel, new() { Exact = true }) + ), + + // Tier 4: OrchardCore structural — action columns, dropdowns, primary button groups + ( + $"orchard:actions['{actionLabel}']", + page.Locator( + $".actions a:has-text('{escaped}'), " + + $"[class*='actions'] a:has-text('{escaped}'), " + + $".btn:has-text('{escaped}'), " + + $"[class*='btn']:has-text('{escaped}'), " + + $".dropdown-menu a:has-text('{escaped}')") + ), + + // Tier 5: Broadest fallback — any <a> or <button> anywhere on the page + ( + $"fallback:any-link-or-button['{actionLabel}']", + page.Locator($"a:has-text('{escaped}'), button:has-text('{escaped}')") + ), + ]; + } + + // ------------------------------------------------------------------------- + // Evidence capture + // ------------------------------------------------------------------------- + + private async Task<(string PagePath, string ContainerPath, string HtmlPath)> CaptureEvidenceAsync( + IPage page, + string sessionId, + CancellationToken cancellationToken) + { + var timestamp = _clock.UtcNow; + var folder = Path.Combine( + _hostEnvironment.ContentRootPath, + "App_Data", "Playwright", "Evidence", + sessionId, + $"{timestamp:yyyyMMdd-HHmmssfff}"); + + Directory.CreateDirectory(folder); + + var pagePath = await CaptureFullPageScreenshotAsync(page, folder, cancellationToken); + var containerPath = await CaptureContainerScreenshotAsync(page, folder, cancellationToken); + var htmlPath = await SavePageHtmlAsync(page, folder, cancellationToken); + + return (pagePath, containerPath, htmlPath); + } + + private static async Task<string> CaptureFullPageScreenshotAsync(IPage page, string folder, CancellationToken cancellationToken) + { + var path = Path.Combine(folder, "page-full.png"); + + await page.ScreenshotAsync(new PageScreenshotOptions + { + Path = path, + FullPage = true, + Type = ScreenshotType.Png, + }).WaitAsync(cancellationToken); + + return path; + } + + private static async Task<string> CaptureContainerScreenshotAsync(IPage page, string folder, CancellationToken cancellationToken) + { + var container = await FindNearestContainerAsync(page, cancellationToken); + if (container is null) + { + return null; + } + + var path = Path.Combine(folder, "container.png"); + + await container.ScreenshotAsync(new LocatorScreenshotOptions + { + Path = path, + Type = ScreenshotType.Png, + }).WaitAsync(cancellationToken); + + return path; + } + + private static async Task<ILocator> FindNearestContainerAsync(IPage page, CancellationToken cancellationToken) + { + foreach (var selector in _containerSelectors) + { + try + { + var locator = page.Locator(selector).First; + var count = await locator.CountAsync().WaitAsync(cancellationToken); + if (count > 0 && await locator.IsVisibleAsync().WaitAsync(cancellationToken)) + { + return locator; + } + } + catch + { + // Container selector failed — try the next one. + } + } + + return null; + } + + private static async Task<string> SavePageHtmlAsync(IPage page, string folder, CancellationToken cancellationToken) + { + var path = Path.Combine(folder, "page.html"); + var html = await page.ContentAsync().WaitAsync(cancellationToken); + await File.WriteAllTextAsync(path, html, cancellationToken); + return path; + } + + // ------------------------------------------------------------------------- + // Helpers + // ------------------------------------------------------------------------- + + private static IPage GetPage(IPlaywrightSession session) + { + // PlaywrightSession keeps the active page on session.Page. + // For concrete sessions that lazy-create the page, fall through to Page directly. + return session.Page; + } +} diff --git a/src/Modules/CrestApps.OrchardCore.AI.Playwright/Services/PlaywrightActionVisualizer.cs b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Services/PlaywrightActionVisualizer.cs new file mode 100644 index 000000000..c941a7df6 --- /dev/null +++ b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Services/PlaywrightActionVisualizer.cs @@ -0,0 +1,242 @@ +using Microsoft.Extensions.Logging; +using Microsoft.Playwright; + +namespace CrestApps.OrchardCore.AI.Playwright.Services; + +public sealed class PlaywrightActionVisualizer : IPlaywrightActionVisualizer +{ + private readonly ILogger<PlaywrightActionVisualizer> _logger; + + public PlaywrightActionVisualizer(ILogger<PlaywrightActionVisualizer> logger) + { + _logger = logger; + } + + public async Task ShowLocatorActionAsync( + IPage page, + ILocator locator, + string action, + string target, + CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(page); + ArgumentNullException.ThrowIfNull(locator); + + if (page.IsClosed) + { + return; + } + + try + { + await locator.ScrollIntoViewIfNeededAsync().WaitAsync(cancellationToken); + } + catch + { + // Best effort. Some hidden editors cannot be scrolled into view directly. + } + + double? x = null; + double? y = null; + double? width = null; + double? height = null; + try + { + var boundingBox = await locator.BoundingBoxAsync().WaitAsync(cancellationToken); + if (boundingBox is not null) + { + x = boundingBox.X; + y = boundingBox.Y; + width = boundingBox.Width; + height = boundingBox.Height; + } + } + catch + { + // Best effort. Hidden or detached elements can still use the page-level indicator. + } + + _logger.LogDebug( + "Showing Playwright action indicator for {Action} on {Target}.", + action, + target); + + await ShowIndicatorAsync(page, new PlaywrightActionIndicatorPayload + { + Action = action, + Target = target, + X = x, + Y = y, + Width = width, + Height = height, + }, cancellationToken); + } + + public async Task ShowPageActionAsync( + IPage page, + string action, + string target, + CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(page); + + if (page.IsClosed) + { + return; + } + + _logger.LogDebug( + "Showing page-level Playwright action indicator for {Action} on {Target}.", + action, + target); + + await ShowIndicatorAsync(page, new PlaywrightActionIndicatorPayload + { + Action = action, + Target = target, + }, cancellationToken); + } + + private static async Task ShowIndicatorAsync( + IPage page, + PlaywrightActionIndicatorPayload payload, + CancellationToken cancellationToken) + { + await page.EvaluateAsync( + """ + payload => { + const doc = document; + const body = doc.body; + + if (!body) { + return; + } + + const normalize = (value) => (value || '').toString().trim(); + const rootId = '__crestapps-playwright-indicator'; + let root = doc.getElementById(rootId); + + if (!root) { + root = doc.createElement('div'); + root.id = rootId; + root.style.position = 'fixed'; + root.style.inset = '0'; + root.style.pointerEvents = 'none'; + root.style.zIndex = '2147483647'; + root.style.opacity = '0'; + root.style.transition = 'opacity 120ms ease-out'; + + const box = doc.createElement('div'); + box.setAttribute('data-role', 'box'); + box.style.position = 'fixed'; + box.style.border = '3px solid #f97316'; + box.style.borderRadius = '10px'; + box.style.boxShadow = '0 0 0 6px rgba(249, 115, 22, 0.18)'; + box.style.background = 'rgba(249, 115, 22, 0.08)'; + box.style.display = 'none'; + root.appendChild(box); + + const pointer = doc.createElement('div'); + pointer.setAttribute('data-role', 'pointer'); + pointer.style.position = 'fixed'; + pointer.style.width = '18px'; + pointer.style.height = '18px'; + pointer.style.marginLeft = '-9px'; + pointer.style.marginTop = '-9px'; + pointer.style.borderRadius = '999px'; + pointer.style.border = '3px solid #fb7185'; + pointer.style.background = 'rgba(255, 255, 255, 0.92)'; + pointer.style.boxShadow = '0 0 0 8px rgba(251, 113, 133, 0.18)'; + pointer.style.display = 'none'; + root.appendChild(pointer); + + const label = doc.createElement('div'); + label.setAttribute('data-role', 'label'); + label.style.position = 'fixed'; + label.style.maxWidth = '420px'; + label.style.padding = '8px 12px'; + label.style.borderRadius = '999px'; + label.style.background = 'rgba(15, 23, 42, 0.94)'; + label.style.border = '1px solid rgba(125, 211, 252, 0.5)'; + label.style.color = '#e2e8f0'; + label.style.fontFamily = 'Consolas, Menlo, Monaco, monospace'; + label.style.fontSize = '12px'; + label.style.fontWeight = '700'; + label.style.letterSpacing = '0.01em'; + label.style.whiteSpace = 'nowrap'; + label.style.overflow = 'hidden'; + label.style.textOverflow = 'ellipsis'; + label.style.boxShadow = '0 12px 28px rgba(15, 23, 42, 0.28)'; + root.appendChild(label); + + body.appendChild(root); + } + + const box = root.querySelector('[data-role="box"]'); + const pointer = root.querySelector('[data-role="pointer"]'); + const label = root.querySelector('[data-role="label"]'); + const labelText = [normalize(payload.action), normalize(payload.target)].filter(Boolean).join(' '); + + label.textContent = labelText || 'AI action'; + + const hasTarget = Number.isFinite(payload.x) + && Number.isFinite(payload.y) + && Number.isFinite(payload.width) + && Number.isFinite(payload.height); + + if (hasTarget) { + const left = Math.max(8, payload.x - 6); + const top = Math.max(8, payload.y - 6); + const width = Math.max(24, payload.width + 12); + const height = Math.max(24, payload.height + 12); + box.style.display = 'block'; + box.style.left = `${left}px`; + box.style.top = `${top}px`; + box.style.width = `${width}px`; + box.style.height = `${height}px`; + + pointer.style.display = 'block'; + pointer.style.left = `${left + Math.min(width, 28)}px`; + pointer.style.top = `${top + Math.min(height, 22)}px`; + + label.style.left = `${Math.max(12, left)}px`; + label.style.top = `${Math.max(12, top - 42)}px`; + } else { + box.style.display = 'none'; + pointer.style.display = 'none'; + label.style.left = '20px'; + label.style.top = '20px'; + } + + root.style.opacity = '1'; + + if (window.__crestappsPlaywrightIndicatorTimer) { + clearTimeout(window.__crestappsPlaywrightIndicatorTimer); + } + + window.__crestappsPlaywrightIndicatorTimer = window.setTimeout(() => { + const activeRoot = doc.getElementById(rootId); + if (activeRoot) { + activeRoot.style.opacity = '0'; + } + }, 1800); + } + """, + payload).WaitAsync(cancellationToken); + } + + private sealed class PlaywrightActionIndicatorPayload + { + public string Action { get; init; } + + public string Target { get; init; } + + public double? X { get; init; } + + public double? Y { get; init; } + + public double? Width { get; init; } + + public double? Height { get; init; } + } +} diff --git a/src/Modules/CrestApps.OrchardCore.AI.Playwright/Services/PlaywrightObservationService.cs b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Services/PlaywrightObservationService.cs new file mode 100644 index 000000000..8e55a4cc0 --- /dev/null +++ b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Services/PlaywrightObservationService.cs @@ -0,0 +1,168 @@ +using System.Text.RegularExpressions; +using CrestApps.OrchardCore.AI.Playwright.Models; +using Microsoft.Playwright; + +namespace CrestApps.OrchardCore.AI.Playwright.Services; + +public sealed class PlaywrightObservationService : IPlaywrightObservationService +{ + private static readonly string[] HeadingSelectors = + [ + "main h1", + ".page-title h1", + ".page-title", + "h1", + ]; + + private static readonly string[] ToastSelectors = + [ + ".toast.show", + ".notification", + ".message", + ".alert-success", + ".alert-info", + ".alert-warning", + ".alert-danger", + ]; + + private static readonly string[] ValidationSelectors = + [ + ".validation-summary-errors li", + ".field-validation-error", + ".alert-danger li", + ".alert-danger", + ".titleerror", + ".exceptionMessage", + "h1.exceptionMessage", + ".exception-message", + ]; + + public async Task<PlaywrightObservation> CaptureAsync(IPlaywrightSession session, CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(session); + + var page = session switch + { + PlaywrightSession liveSession => await liveSession.GetOrCreatePageAsync(cancellationToken), + _ => session.Page, + }; + var title = await page.TitleAsync().WaitAsync(cancellationToken); + var heading = await GetFirstVisibleTextAsync(page, HeadingSelectors, cancellationToken); + var toast = await GetFirstVisibleTextAsync(page, ToastSelectors, cancellationToken); + var validationMessages = await GetTextsAsync(page, ValidationSelectors, cancellationToken); + var visibleButtons = await GetTextsAsync(page, ["button", "a.btn", "input[type='submit']"], cancellationToken, 6); + var isLoginPage = await IsLoginPageAsync(page, cancellationToken); + + var observation = new PlaywrightObservation + { + CurrentUrl = page.Url, + PageTitle = title, + MainHeading = heading, + ToastMessage = toast, + ValidationMessages = validationMessages, + VisibleButtons = visibleButtons, + IsLoginPage = isLoginPage, + IsAuthenticated = !isLoginPage, + }; + + if (session is PlaywrightSession concreteSession) + { + concreteSession.ApplyObservation(observation); + } + + return observation; + } + + private static async Task<bool> IsLoginPageAsync(IPage page, CancellationToken cancellationToken) + { + if (page.Url.Contains("/Login", StringComparison.OrdinalIgnoreCase)) + { + return true; + } + + var username = page.Locator("input[name='UserName'], input[name='Username'], input[name*='UserName'], input[name='Email'], input[name*='Email'], input[type='email'], #UserName, input[id*='UserName'], input[id*='Email']"); + var password = page.Locator("input[name='Password'], input[name*='Password'], input[type='password'], #Password, input[id*='Password']"); + + return (await username.CountAsync().WaitAsync(cancellationToken) > 0) + && (await password.CountAsync().WaitAsync(cancellationToken) > 0); + } + + private static async Task<string> GetFirstVisibleTextAsync(IPage page, IEnumerable<string> selectors, CancellationToken cancellationToken) + { + foreach (var selector in selectors) + { + var locator = page.Locator(selector).First; + + if (await locator.CountAsync().WaitAsync(cancellationToken) == 0) + { + continue; + } + + if (await locator.IsVisibleAsync().WaitAsync(cancellationToken)) + { + var text = await locator.InnerTextAsync().WaitAsync(cancellationToken); + text = NormalizeText(text); + + if (!string.IsNullOrWhiteSpace(text)) + { + return text; + } + } + } + + return null; + } + + private static async Task<IReadOnlyList<string>> GetTextsAsync( + IPage page, + IEnumerable<string> selectors, + CancellationToken cancellationToken, + int limit = 4) + { + var values = new List<string>(); + + foreach (var selector in selectors) + { + var locator = page.Locator(selector); + var count = await locator.CountAsync().WaitAsync(cancellationToken); + + for (var i = 0; i < count && values.Count < limit; i++) + { + var current = locator.Nth(i); + if (!await current.IsVisibleAsync().WaitAsync(cancellationToken)) + { + continue; + } + + var text = current switch + { + _ when selector.Contains("input[type='submit']", StringComparison.OrdinalIgnoreCase) + => NormalizeText(await current.GetAttributeAsync("value").WaitAsync(cancellationToken)), + _ => NormalizeText(await current.InnerTextAsync().WaitAsync(cancellationToken)), + }; + + if (!string.IsNullOrWhiteSpace(text) && !values.Contains(text, StringComparer.OrdinalIgnoreCase)) + { + values.Add(text); + } + } + + if (values.Count >= limit) + { + break; + } + } + + return values; + } + + private static string NormalizeText(string value) + { + if (string.IsNullOrWhiteSpace(value)) + { + return null; + } + + return Regex.Replace(value.Trim(), @"\s+", " "); + } +} diff --git a/src/Modules/CrestApps.OrchardCore.AI.Playwright/Services/PlaywrightPageInspectionService.cs b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Services/PlaywrightPageInspectionService.cs new file mode 100644 index 000000000..f81983174 --- /dev/null +++ b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Services/PlaywrightPageInspectionService.cs @@ -0,0 +1,416 @@ +using System.Text.RegularExpressions; +using CrestApps.OrchardCore.AI.Playwright.Models; +using Microsoft.Extensions.Hosting; +using Microsoft.Extensions.Logging; +using Microsoft.Playwright; +using OrchardClock = OrchardCore.Modules.IClock; + +namespace CrestApps.OrchardCore.AI.Playwright.Services; + +public sealed class PlaywrightPageInspectionService : IPlaywrightPageInspectionService +{ + private static readonly Regex _whitespace = new(@"\s{2,}", RegexOptions.Compiled); + + private readonly OrchardClock _clock; + private readonly IHostEnvironment _hostEnvironment; + private readonly ILogger<PlaywrightPageInspectionService> _logger; + + public PlaywrightPageInspectionService( + OrchardClock clock, + IHostEnvironment hostEnvironment, + ILogger<PlaywrightPageInspectionService> logger) + { + _clock = clock; + _hostEnvironment = hostEnvironment; + _logger = logger; + } + + public async Task<PlaywrightPageContentResult> GetPageContentAsync(IPlaywrightSession session, int maxLength, CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(session); + + var page = await GetActivePageAsync(session, cancellationToken); + maxLength = Math.Clamp(maxLength, 500, 16_000); + + _logger.LogDebug("Inspecting visible page content for Playwright session '{ChatSessionId}'.", session.SessionId); + + var payload = await page.EvaluateAsync<PageContentPayload>( + """ + (maxLength) => { + const normalize = (value) => (value || '').replace(/\s+/g, ' ').trim(); + const body = document.body ? document.body.cloneNode(true) : null; + + if (body) { + body.querySelectorAll('script, style, noscript, [hidden], [aria-hidden="true"]').forEach(element => element.remove()); + } + + const heading = normalize( + document.querySelector('main h1, .page-title h1, .page-title, h1')?.innerText || '' + ); + + let content = normalize(body?.innerText || ''); + if (content.length > maxLength) { + content = `${content.slice(0, maxLength)}\n\n[...truncated - ${content.length - maxLength} more characters]`; + } + + return { + mainHeading: heading, + content + }; + } + """, + maxLength).WaitAsync(cancellationToken); + + return new PlaywrightPageContentResult + { + Url = page.Url, + Title = await page.TitleAsync().WaitAsync(cancellationToken), + MainHeading = NormalizeText(payload?.MainHeading), + Content = NormalizeText(payload?.Content), + }; + } + + public async Task<PlaywrightElementSearchResult> FindElementsAsync(IPlaywrightSession session, string query, int maxMatches, CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(session); + ArgumentException.ThrowIfNullOrWhiteSpace(query); + + var page = await GetActivePageAsync(session, cancellationToken); + maxMatches = Math.Clamp(maxMatches, 1, 12); + + _logger.LogDebug( + "Finding visible elements matching '{Query}' for Playwright session '{ChatSessionId}'.", + query, + session.SessionId); + + var matches = await page.EvaluateAsync<ElementMatchPayload[]>( + """ + (args) => { + const normalize = (value) => (value || '').replace(/\s+/g, ' ').trim(); + const isVisible = (element) => { + if (!(element instanceof HTMLElement)) { + return false; + } + + if (element.hidden) { + return false; + } + + const style = window.getComputedStyle(element); + if (style.display === 'none' || style.visibility === 'hidden' || style.opacity === '0') { + return false; + } + + return element.offsetWidth > 0 || element.offsetHeight > 0 || element.getClientRects().length > 0; + }; + + const query = normalize(args.query).toLowerCase(); + const maxMatches = args.maxMatches; + const results = []; + const seen = new Set(); + + for (const element of Array.from(document.querySelectorAll('body *'))) { + if (!isVisible(element)) { + continue; + } + + const text = normalize(element.innerText || element.textContent || ''); + const label = normalize(Array.from(element.labels || []).map(labelElement => labelElement.innerText).join(' | ')); + const role = normalize(element.getAttribute('role') || ''); + const title = normalize(element.getAttribute('title') || ''); + const placeholder = normalize(element.getAttribute('placeholder') || ''); + const name = normalize(element.getAttribute('name') || ''); + const id = normalize(element.id || ''); + const widgetType = normalize(element.getAttribute('data-widget-type') || element.getAttribute('data-widget') || ''); + const ariaLabel = normalize(element.getAttribute('aria-label') || ''); + + const searchable = [text, label, role, title, placeholder, name, id, widgetType, ariaLabel] + .filter(Boolean) + .join(' | ') + .toLowerCase(); + + if (!searchable.includes(query)) { + continue; + } + + const selectorHint = id + ? `#${id}` + : name + ? `[name="${name}"]` + : ariaLabel + ? `[aria-label="${ariaLabel}"]` + : text + ? `text=${text.slice(0, 80)}` + : element.tagName.toLowerCase(); + + const key = `${element.tagName}|${id}|${name}|${selectorHint}|${text.slice(0, 80)}`.toLowerCase(); + if (seen.has(key)) { + continue; + } + + seen.add(key); + results.push({ + tagName: element.tagName.toLowerCase(), + role, + text: text.slice(0, 220), + label, + id, + name, + title, + placeholder, + widgetType, + selectorHint + }); + + if (results.length >= maxMatches) { + break; + } + } + + return results; + } + """, + new + { + query, + maxMatches, + }).WaitAsync(cancellationToken); + + var normalizedMatches = (matches ?? []) + .Select(match => new PlaywrightElementMatch + { + TagName = NormalizeText(match.TagName), + Role = NormalizeText(match.Role), + Text = NormalizeText(match.Text), + Label = NormalizeText(match.Label), + Id = NormalizeText(match.Id), + Name = NormalizeText(match.Name), + Title = NormalizeText(match.Title), + Placeholder = NormalizeText(match.Placeholder), + WidgetType = NormalizeText(match.WidgetType), + SelectorHint = NormalizeText(match.SelectorHint), + }) + .ToList(); + + return new PlaywrightElementSearchResult + { + Query = query, + Url = page.Url, + Title = await page.TitleAsync().WaitAsync(cancellationToken), + Exists = normalizedMatches.Count > 0, + MatchCount = normalizedMatches.Count, + Matches = normalizedMatches, + }; + } + + public Task<PlaywrightElementSearchResult> CheckElementExistsAsync(IPlaywrightSession session, string query, int maxMatches, CancellationToken cancellationToken = default) + { + _logger.LogDebug( + "Checking whether '{Query}' exists on the page for Playwright session '{ChatSessionId}'.", + query, + session?.SessionId); + + return FindElementsAsync(session, query, maxMatches, cancellationToken); + } + + public async Task<PlaywrightVisibleWidgetsResult> GetVisibleWidgetsAsync(IPlaywrightSession session, int maxItems, CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(session); + + var page = await GetActivePageAsync(session, cancellationToken); + maxItems = Math.Clamp(maxItems, 1, 20); + + _logger.LogDebug("Listing visible widgets for Playwright session '{ChatSessionId}'.", session.SessionId); + + var widgets = await page.EvaluateAsync<VisibleWidgetPayload[]>( + """ + (maxItems) => { + const normalize = (value) => (value || '').replace(/\s+/g, ' ').trim(); + const isVisible = (element) => { + if (!(element instanceof HTMLElement)) { + return false; + } + + if (element.hidden) { + return false; + } + + const style = window.getComputedStyle(element); + if (style.display === 'none' || style.visibility === 'hidden' || style.opacity === '0') { + return false; + } + + return element.offsetWidth > 0 || element.offsetHeight > 0 || element.getClientRects().length > 0; + }; + + const selectors = [ + '[data-widget-type]', + '[data-widget]', + '[class*="widget"]', + '.widget', + '.card-title', + '.card-header', + 'legend', + 'summary', + 'h1', + 'h2', + 'h3', + 'h4', + 'h5', + 'h6' + ]; + + const seen = new Set(); + const results = []; + + for (const element of Array.from(document.querySelectorAll(selectors.join(',')))) { + if (!isVisible(element)) { + continue; + } + + const name = normalize( + element.getAttribute('data-widget-type') + || element.getAttribute('data-widget') + || element.innerText + || element.textContent + || '' + ).slice(0, 140); + + if (!name || name.length < 2) { + continue; + } + + const source = normalize( + element.getAttribute('data-widget-type') + ? 'data-widget-type' + : element.getAttribute('data-widget') + ? 'data-widget' + : element.tagName.toLowerCase() + ); + + const key = `${source}|${name}`.toLowerCase(); + if (seen.has(key)) { + continue; + } + + seen.add(key); + results.push({ name, source }); + + if (results.length >= maxItems) { + break; + } + } + + return results; + } + """, + maxItems).WaitAsync(cancellationToken); + + var normalizedWidgets = (widgets ?? []) + .Select(widget => new PlaywrightVisibleWidget + { + Name = NormalizeText(widget.Name), + Source = NormalizeText(widget.Source), + }) + .Where(widget => !string.IsNullOrWhiteSpace(widget.Name)) + .ToList(); + + return new PlaywrightVisibleWidgetsResult + { + Url = page.Url, + Title = await page.TitleAsync().WaitAsync(cancellationToken), + WidgetCount = normalizedWidgets.Count, + Widgets = normalizedWidgets, + }; + } + + public async Task<PlaywrightScreenshotResult> TakeScreenshotAsync(IPlaywrightSession session, bool fullPage, CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(session); + + var page = await GetActivePageAsync(session, cancellationToken); + var takenAtUtc = _clock.UtcNow; + var directory = Path.Combine(_hostEnvironment.ContentRootPath, "App_Data", "Playwright", "Screenshots", session.SessionId); + Directory.CreateDirectory(directory); + + var fileName = $"{takenAtUtc:yyyyMMdd-HHmmssfff}.png"; + var savedPath = Path.Combine(directory, fileName); + + _logger.LogDebug( + "Capturing screenshot for Playwright session '{ChatSessionId}' to '{SavedPath}'.", + session.SessionId, + savedPath); + + await page.ScreenshotAsync(new PageScreenshotOptions + { + Path = savedPath, + FullPage = fullPage, + Type = ScreenshotType.Png, + }).WaitAsync(cancellationToken); + + return new PlaywrightScreenshotResult + { + Url = page.Url, + Title = await page.TitleAsync().WaitAsync(cancellationToken), + SavedPath = savedPath, + FullPage = fullPage, + TakenAtUtc = takenAtUtc, + }; + } + + private static async Task<IPage> GetActivePageAsync(IPlaywrightSession session, CancellationToken cancellationToken) + { + return session switch + { + PlaywrightSession concreteSession => await concreteSession.GetOrCreatePageAsync(cancellationToken), + _ => session.Page, + }; + } + + private static string NormalizeText(string value) + { + if (string.IsNullOrWhiteSpace(value)) + { + return null; + } + + return _whitespace.Replace(value.Trim(), " "); + } + + private sealed class PageContentPayload + { + public string MainHeading { get; set; } + + public string Content { get; set; } + } + + private sealed class ElementMatchPayload + { + public string TagName { get; set; } + + public string Role { get; set; } + + public string Text { get; set; } + + public string Label { get; set; } + + public string Id { get; set; } + + public string Name { get; set; } + + public string Title { get; set; } + + public string Placeholder { get; set; } + + public string WidgetType { get; set; } + + public string SelectorHint { get; set; } + } + + private sealed class VisibleWidgetPayload + { + public string Name { get; set; } + + public string Source { get; set; } + } +} diff --git a/src/Modules/CrestApps.OrchardCore.AI.Playwright/Services/PlaywrightSession.cs b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Services/PlaywrightSession.cs new file mode 100644 index 000000000..493182317 --- /dev/null +++ b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Services/PlaywrightSession.cs @@ -0,0 +1,136 @@ +using CrestApps.OrchardCore.AI.Playwright.Models; +using Microsoft.Playwright; +using OrchardClock = OrchardCore.Modules.IClock; + +namespace CrestApps.OrchardCore.AI.Playwright.Services; + +/// <summary> +/// Holds the live Playwright objects for one chat session. +/// </summary> +internal sealed class PlaywrightSession : IPlaywrightSession, IAsyncDisposable +{ + private readonly OrchardClock _clock; + private readonly IPlaywright _playwright; + private readonly IBrowser _browser; + private CancellationTokenSource _cts = new(); + + public string SessionId { get; } + public string OwnerId { get; private set; } + public PlaywrightSessionStatus Status { get; private set; } = PlaywrightSessionStatus.Idle; + public string CurrentUrl { get; internal set; } + public string CurrentPageTitle { get; internal set; } + public DateTime CreatedAtUtc { get; } + public DateTime LastActivityUtc { get; private set; } + public string BaseUrl { get; private set; } + public string AdminBaseUrl { get; private set; } + public PlaywrightBrowserMode BrowserMode { get; internal set; } + public bool IsAuthenticated { get; internal set; } + public PlaywrightObservation LastObservation { get; internal set; } + public int SessionInactivityTimeoutInMinutes { get; private set; } + public IBrowserContext Context { get; } + public IPage Page { get; private set; } + public CancellationToken StopToken => _cts.Token; + + internal PlaywrightSessionRequest Request { get; private set; } + + internal PlaywrightSession( + PlaywrightSessionRequest request, + OrchardClock clock, + IPlaywright playwright, + IBrowser browser, + IBrowserContext context, + IPage page) + { + _clock = clock; + SessionId = request.ChatSessionId; + OwnerId = request.OwnerId; + BaseUrl = request.BaseUrl; + AdminBaseUrl = request.AdminBaseUrl; + BrowserMode = request.BrowserMode; + SessionInactivityTimeoutInMinutes = request.SessionInactivityTimeoutInMinutes; + Request = request; + _playwright = playwright; + _browser = browser; + Context = context; + Page = page; + CreatedAtUtc = _clock.UtcNow; + LastActivityUtc = _clock.UtcNow; + } + + public void MarkRunning() + { + LastActivityUtc = _clock.UtcNow; + Status = PlaywrightSessionStatus.Running; + } + + public void MarkIdle() + { + LastActivityUtc = _clock.UtcNow; + if (Status == PlaywrightSessionStatus.Running) + { + Status = PlaywrightSessionStatus.Idle; + } + } + + public void Stop() + { + LastActivityUtc = _clock.UtcNow; + Status = PlaywrightSessionStatus.Stopped; + var old = Interlocked.Exchange(ref _cts, new CancellationTokenSource()); + old.Cancel(); + old.Dispose(); + } + + internal void UpdateRequest(PlaywrightSessionRequest request) + { + ArgumentNullException.ThrowIfNull(request); + + Request = request; + OwnerId = request.OwnerId; + BaseUrl = request.BaseUrl; + AdminBaseUrl = request.AdminBaseUrl; + SessionInactivityTimeoutInMinutes = request.SessionInactivityTimeoutInMinutes; + LastActivityUtc = _clock.UtcNow; + } + + internal void ApplyObservation(PlaywrightObservation observation) + { + LastActivityUtc = _clock.UtcNow; + LastObservation = observation; + CurrentUrl = observation?.CurrentUrl; + CurrentPageTitle = observation?.PageTitle; + IsAuthenticated = observation?.IsAuthenticated ?? false; + } + + internal async Task<IPage> GetOrCreatePageAsync(CancellationToken cancellationToken = default) + { + LastActivityUtc = _clock.UtcNow; + if (Page?.IsClosed == false) + { + return Page; + } + + var openPage = Context.Pages.FirstOrDefault(page => !page.IsClosed); + if (openPage is not null) + { + Page = openPage; + return openPage; + } + + Page = await Context.NewPageAsync().WaitAsync(cancellationToken); + return Page; + } + + public async ValueTask DisposeAsync() + { + Status = PlaywrightSessionStatus.Closed; + _cts.Cancel(); + _cts.Dispose(); + + try { await Page.CloseAsync(); } catch { /* best effort */ } + try { await Context.CloseAsync(); } catch { /* best effort */ } + try { await _browser?.CloseAsync(); } catch { /* best effort */ } + + _playwright.Dispose(); + } +} diff --git a/src/Modules/CrestApps.OrchardCore.AI.Playwright/Services/PlaywrightSessionManager.cs b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Services/PlaywrightSessionManager.cs new file mode 100644 index 000000000..b965b0bd9 --- /dev/null +++ b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Services/PlaywrightSessionManager.cs @@ -0,0 +1,356 @@ +using System.Collections.Concurrent; +using CrestApps.OrchardCore.AI.Playwright.Models; +using Microsoft.Extensions.Hosting; +using Microsoft.Extensions.Logging; +using Microsoft.Playwright; +using OrchardClock = OrchardCore.Modules.IClock; + +namespace CrestApps.OrchardCore.AI.Playwright.Services; + +/// <summary> +/// Singleton manager that owns all active Playwright browser sessions. +/// </summary> +public sealed class PlaywrightSessionManager : IPlaywrightSessionManager +{ + private readonly ConcurrentDictionary<string, PlaywrightSession> _sessions = new(StringComparer.OrdinalIgnoreCase); + private readonly OrchardClock _clock; + private readonly IPlaywrightObservationService _observationService; + private readonly IHostEnvironment _hostEnvironment; + private readonly ILogger<PlaywrightSessionManager> _logger; + + public PlaywrightSessionManager( + OrchardClock clock, + IPlaywrightObservationService observationService, + IHostEnvironment hostEnvironment, + ILogger<PlaywrightSessionManager> logger) + { + _clock = clock; + _observationService = observationService; + _hostEnvironment = hostEnvironment; + _logger = logger; + } + + /// <inheritdoc/> + public async Task<IPlaywrightSession> GetOrCreateAsync( + PlaywrightSessionRequest request, + CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(request); + ArgumentException.ThrowIfNullOrEmpty(request.ChatSessionId); + ArgumentException.ThrowIfNullOrEmpty(request.BaseUrl); + ArgumentException.ThrowIfNullOrEmpty(request.AdminBaseUrl); + + if (_sessions.TryGetValue(request.ChatSessionId, out var existing) && + existing.Status != PlaywrightSessionStatus.Closed) + { + existing.UpdateRequest(request); + return existing; + } + + _logger.LogInformation( + "Creating Playwright session '{ChatSessionId}' using the dedicated Playwright browser.", + request.ChatSessionId); + + var playwright = await Microsoft.Playwright.Playwright.CreateAsync(); + var (browser, context, page, actualMode) = await CreateBrowserArtifactsAsync(playwright, request, cancellationToken); + var session = new PlaywrightSession(request, _clock, playwright, browser, context, page) + { + BrowserMode = actualMode, + }; + + _sessions[request.ChatSessionId] = session; + + page = await session.GetOrCreatePageAsync(cancellationToken); + if (string.IsNullOrWhiteSpace(page.Url) || string.Equals(page.Url, "about:blank", StringComparison.OrdinalIgnoreCase)) + { + await page.GotoAsync(request.AdminBaseUrl, new PageGotoOptions + { + WaitUntil = WaitUntilState.DOMContentLoaded, + Timeout = 30_000, + }).WaitAsync(cancellationToken); + } + + await _observationService.CaptureAsync(session, cancellationToken); + await TryLoginIfNeededAsync(session, cancellationToken); + + return session; + } + + /// <inheritdoc/> + public IPlaywrightSession GetSession(string chatSessionId, string ownerId = null) + { + _sessions.TryGetValue(chatSessionId, out var session); + + if (session == null || session.Status == PlaywrightSessionStatus.Closed) + { + return null; + } + + if (!string.IsNullOrWhiteSpace(ownerId) && !string.Equals(session.OwnerId, ownerId, StringComparison.OrdinalIgnoreCase)) + { + return null; + } + + return session; + } + + public async Task EnsureAdminReadyAsync(IPlaywrightSession session, CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(session); + + if (session is not PlaywrightSession concreteSession) + { + return; + } + + var page = await concreteSession.GetOrCreatePageAsync(cancellationToken); + + await page.GotoAsync(concreteSession.AdminBaseUrl, new PageGotoOptions + { + WaitUntil = WaitUntilState.DOMContentLoaded, + Timeout = 30_000, + }).WaitAsync(cancellationToken); + + await _observationService.CaptureAsync(concreteSession, cancellationToken); + await TryLoginIfNeededAsync(concreteSession, cancellationToken); + } + + /// <inheritdoc/> + public void Stop(string chatSessionId, string ownerId = null) + { + if (GetSession(chatSessionId, ownerId) is PlaywrightSession session) + { + _logger.LogInformation("Stopping Playwright session '{ChatSessionId}'.", chatSessionId); + session.Stop(); + } + } + + /// <inheritdoc/> + public async Task CloseAsync(string chatSessionId, string ownerId = null) + { + if (GetSession(chatSessionId, ownerId) is not PlaywrightSession session) + { + return; + } + + if (_sessions.TryRemove(chatSessionId, out _)) + { + _logger.LogInformation("Closing Playwright session '{ChatSessionId}'.", chatSessionId); + await session.DisposeAsync(); + } + } + + public async Task CloseInactiveSessionsAsync(CancellationToken cancellationToken = default) + { + var utcNow = _clock.UtcNow; + var expiredSessions = _sessions.Values + .Where(session => session.Status != PlaywrightSessionStatus.Closed) + .Where(session => session.Status != PlaywrightSessionStatus.Running) + .Where(session => session.LastActivityUtc < utcNow - TimeSpan.FromMinutes(Math.Max(1, session.SessionInactivityTimeoutInMinutes))) + .ToList(); + + foreach (var session in expiredSessions) + { + cancellationToken.ThrowIfCancellationRequested(); + + _logger.LogDebug( + "Closing inactive Playwright session '{ChatSessionId}' after {TimeoutMinutes} minutes of inactivity.", + session.SessionId, + session.SessionInactivityTimeoutInMinutes); + + await CloseAsync(session.SessionId, session.OwnerId); + } + } + + /// <inheritdoc/> + public IReadOnlyDictionary<string, IPlaywrightSession> GetActiveSessions(string ownerId = null) + { + return _sessions + .Where(kvp => kvp.Value.Status != PlaywrightSessionStatus.Closed) + .Where(kvp => string.IsNullOrWhiteSpace(ownerId) + || string.Equals(kvp.Value.OwnerId, ownerId, StringComparison.OrdinalIgnoreCase)) + .ToDictionary(kvp => kvp.Key, kvp => (IPlaywrightSession)kvp.Value, StringComparer.OrdinalIgnoreCase); + } + + private async Task<(IBrowser Browser, IBrowserContext Context, IPage Page, PlaywrightBrowserMode ActualMode)> CreateBrowserArtifactsAsync( + IPlaywright playwright, + PlaywrightSessionRequest request, + CancellationToken cancellationToken) + { + var userDataDir = ResolvePersistentProfilePath(request); + Directory.CreateDirectory(userDataDir); + + var persistentContext = await playwright.Chromium.LaunchPersistentContextAsync(userDataDir, new BrowserTypeLaunchPersistentContextOptions + { + Headless = request.Headless, + SlowMo = request.Headless ? 0 : 200, + ViewportSize = new ViewportSize { Width = 1280, Height = 900 }, + }).WaitAsync(cancellationToken); + var persistentPage = await PrepareSingleWorkingPageAsync(persistentContext, cancellationToken); + + return (persistentContext.Browser, persistentContext, persistentPage, PlaywrightBrowserMode.PersistentContext); + } + + private string ResolvePersistentProfilePath(PlaywrightSessionRequest request) + { + if (!string.IsNullOrWhiteSpace(request.PersistentProfilePath)) + { + return request.PersistentProfilePath; + } + + var safeResourceId = string.IsNullOrWhiteSpace(request.ResourceItemId) ? request.ChatSessionId : request.ResourceItemId; + foreach (var invalid in Path.GetInvalidFileNameChars()) + { + safeResourceId = safeResourceId.Replace(invalid, '_'); + } + + return Path.Combine(_hostEnvironment.ContentRootPath, "App_Data", "Playwright", safeResourceId); + } + + private async Task TryLoginIfNeededAsync(PlaywrightSession session, CancellationToken cancellationToken) + { + var observation = session.LastObservation ?? await _observationService.CaptureAsync(session, cancellationToken); + if (!observation.IsLoginPage) + { + return; + } + + if (!session.Request.CanAttemptLogin) + { + _logger.LogInformation( + "Playwright session '{ChatSessionId}' reached the login page, but no saved login is configured.", + session.SessionId); + return; + } + + _logger.LogInformation( + "Playwright session '{ChatSessionId}' reached the login page. Attempting sign-in with the saved Playwright account.", + session.SessionId); + + var page = await session.GetOrCreatePageAsync(cancellationToken); + await FillFirstVisibleAsync(page, [ + "label:has-text('User name or email') + input", + "label:has-text('Username') + input", + "label:has-text('User name') + input", + "label:has-text('Email') + input", + "input[name='UserName']", + "input[name='Username']", + "input[name*='UserName']", + "input[name='Email']", + "input[name*='Email']", + "input[type='email']", + "#UserName" + ], session.Request.Username, cancellationToken); + await FillFirstVisibleAsync(page, [ + "label:has-text('Password') + input", + "input[name='Password']", + "input[name*='Password']", + "input[type='password']", + "#Password" + ], session.Request.Password, cancellationToken); + + if (!await TryClickFirstVisibleAsync(page, ["Log in", "Login", "Sign in", "Sign In"], cancellationToken)) + { + var submitLocator = page.Locator("button[type='submit'], input[type='submit']").First; + await submitLocator.ClickAsync(new LocatorClickOptions { Timeout = 10_000 }).WaitAsync(cancellationToken); + } + + try + { + await page.WaitForLoadStateAsync(LoadState.DOMContentLoaded, new PageWaitForLoadStateOptions + { + Timeout = 15_000, + }).WaitAsync(cancellationToken); + } + catch (TimeoutException) + { + // Validation errors can keep the page in place; observation below will surface them. + } + + await _observationService.CaptureAsync(session, cancellationToken); + + if (!session.IsAuthenticated) + { + _logger.LogWarning( + "Playwright session '{ChatSessionId}' could not sign in using the saved Playwright account.", + session.SessionId); + } + } + + private static async Task FillFirstVisibleAsync( + IPage page, + IReadOnlyList<string> selectors, + string value, + CancellationToken cancellationToken) + { + foreach (var selector in selectors) + { + var locator = page.Locator(selector).First; + if (await locator.CountAsync().WaitAsync(cancellationToken) == 0) + { + continue; + } + + if (!await locator.IsVisibleAsync().WaitAsync(cancellationToken)) + { + continue; + } + + await locator.FillAsync(value ?? string.Empty, new LocatorFillOptions + { + Timeout = 10_000, + }).WaitAsync(cancellationToken); + + var appliedValue = await locator.InputValueAsync().WaitAsync(cancellationToken); + if (string.Equals(appliedValue, value ?? string.Empty, StringComparison.Ordinal)) + { + return; + } + } + } + + private static async Task<bool> TryClickFirstVisibleAsync( + IPage page, + IReadOnlyList<string> candidateNames, + CancellationToken cancellationToken) + { + foreach (var candidateName in candidateNames) + { + var button = page.GetByRole(AriaRole.Button, new() { Name = candidateName, Exact = true }).First; + if (await button.CountAsync().WaitAsync(cancellationToken) > 0 + && await button.IsVisibleAsync().WaitAsync(cancellationToken)) + { + await button.ClickAsync(new LocatorClickOptions { Timeout = 10_000 }).WaitAsync(cancellationToken); + return true; + } + } + + return false; + } + + private static async Task<IPage> PrepareSingleWorkingPageAsync( + IBrowserContext context, + CancellationToken cancellationToken) + { + var openPages = context.Pages + .Where(page => !page.IsClosed) + .ToList(); + + var workingPage = openPages.FirstOrDefault() + ?? await context.NewPageAsync().WaitAsync(cancellationToken); + + foreach (var extraPage in openPages.Skip(1)) + { + try + { + await extraPage.CloseAsync().WaitAsync(cancellationToken); + } + catch + { + // Best effort. The working page is the only one we depend on. + } + } + + return workingPage; + } +} diff --git a/src/Modules/CrestApps.OrchardCore.AI.Playwright/Services/PlaywrightSessionRequestResolver.cs b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Services/PlaywrightSessionRequestResolver.cs new file mode 100644 index 000000000..875a604a8 --- /dev/null +++ b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Services/PlaywrightSessionRequestResolver.cs @@ -0,0 +1,147 @@ +using System.Security.Claims; +using CrestApps.OrchardCore.AI.Models; +using CrestApps.OrchardCore.AI.Playwright.Models; +using CrestApps.OrchardCore.AI.Playwright.Settings; +using Microsoft.AspNetCore.DataProtection; +using Microsoft.AspNetCore.Http; +using Microsoft.Extensions.Logging; +using Microsoft.Extensions.Options; +using OrchardCore.Admin; +using OrchardCore.Entities; + +namespace CrestApps.OrchardCore.AI.Playwright.Services; + +public sealed class PlaywrightSessionRequestResolver : IPlaywrightSessionRequestResolver +{ + private readonly IHttpContextAccessor _httpContextAccessor; + private readonly AdminOptions _adminOptions; + private readonly IDataProtectionProvider _dataProtectionProvider; + private readonly ILogger<PlaywrightSessionRequestResolver> _logger; + + public PlaywrightSessionRequestResolver( + IHttpContextAccessor httpContextAccessor, + IOptions<AdminOptions> adminOptions, + IDataProtectionProvider dataProtectionProvider, + ILogger<PlaywrightSessionRequestResolver> logger) + { + _httpContextAccessor = httpContextAccessor; + _adminOptions = adminOptions.Value; + _dataProtectionProvider = dataProtectionProvider; + _logger = logger; + } + + public PlaywrightSessionRequest Resolve(object resource, string chatSessionId) + { + if (string.IsNullOrWhiteSpace(chatSessionId) || resource is not Entity entity) + { + return null; + } + + var metadata = entity.As<PlaywrightSessionMetadata>() ?? new PlaywrightSessionMetadata(); + + if (resource is AIProfile profile) + { + var legacySettings = profile.GetSettings<PlaywrightProfileSettings>(); + metadata.Enabled = metadata.Enabled || legacySettings.Enabled; + } + + if (!metadata.Enabled) + { + return null; + } + + var httpContext = _httpContextAccessor.HttpContext; + if (httpContext is null) + { + _logger.LogWarning("Playwright session resolution failed because no HttpContext is available."); + return null; + } + + var baseUrl = NormalizeUrl( + metadata.BaseUrl, + $"{httpContext.Request.Scheme}://{httpContext.Request.Host}{httpContext.Request.PathBase}"); + + var adminBaseUrl = NormalizeUrl( + metadata.AdminBaseUrl, + CombineUrl(baseUrl, _adminOptions.AdminUrlPrefix)); + var username = metadata.Username?.Trim(); + var password = UnprotectPassword(metadata.ProtectedPassword); + var inactivityTimeoutInMinutes = resource is AIProfile profileResource + ? Math.Max(1, profileResource.GetSettings<AIProfileDataExtractionSettings>()?.SessionInactivityTimeoutInMinutes ?? PlaywrightConstants.DefaultSessionInactivityTimeoutInMinutes) + : PlaywrightConstants.DefaultSessionInactivityTimeoutInMinutes; + + return new PlaywrightSessionRequest + { + ChatSessionId = chatSessionId, + OwnerId = ResolveOwnerId(resource, httpContext.User), + ResourceItemId = GetResourceItemId(resource), + BaseUrl = baseUrl, + AdminBaseUrl = adminBaseUrl, + BrowserMode = PlaywrightBrowserMode.PersistentContext, + Username = username, + Password = password, + CanAttemptLogin = !string.IsNullOrWhiteSpace(username) && !string.IsNullOrWhiteSpace(password), + PersistentProfilePath = metadata.PersistentProfilePath?.Trim(), + Headless = metadata.Headless, + PublishByDefault = metadata.PublishByDefault, + SessionInactivityTimeoutInMinutes = inactivityTimeoutInMinutes, + }; + } + + private static string ResolveOwnerId(object resource, ClaimsPrincipal user) + { + return resource switch + { + AIProfile profile when !string.IsNullOrWhiteSpace(profile.OwnerId) => profile.OwnerId, + ChatInteraction interaction when !string.IsNullOrWhiteSpace(interaction.OwnerId) => interaction.OwnerId, + _ => user.FindFirstValue(ClaimTypes.NameIdentifier) ?? user.Identity?.Name ?? string.Empty, + }; + } + + private static string GetResourceItemId(object resource) + { + return resource switch + { + CrestApps.OrchardCore.Models.CatalogItem item => item.ItemId, + _ => null, + }; + } + + private static string NormalizeUrl(string configuredUrl, string fallbackUrl) + { + var value = string.IsNullOrWhiteSpace(configuredUrl) ? fallbackUrl : configuredUrl; + return value.Trim().TrimEnd('/'); + } + + private string UnprotectPassword(string protectedPassword) + { + if (string.IsNullOrWhiteSpace(protectedPassword)) + { + return null; + } + + try + { + var protector = _dataProtectionProvider.CreateProtector(PlaywrightConstants.ProtectorName); + return protector.Unprotect(protectedPassword); + } + catch (Exception ex) + { + _logger.LogWarning(ex, "Unable to unprotect the saved Playwright password."); + return null; + } + } + + internal static string CombineUrl(string baseUrl, string relativeSegment) + { + var trimmedBase = baseUrl?.TrimEnd('/') ?? string.Empty; + var trimmedSegment = relativeSegment?.Trim().Trim('/'); + + if (string.IsNullOrWhiteSpace(trimmedSegment)) + { + return trimmedBase; + } + + return $"{trimmedBase}/{trimmedSegment}"; + } +} diff --git a/src/Modules/CrestApps.OrchardCore.AI.Playwright/Services/PlaywrightSessionStatus.cs b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Services/PlaywrightSessionStatus.cs new file mode 100644 index 000000000..805f1e4be --- /dev/null +++ b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Services/PlaywrightSessionStatus.cs @@ -0,0 +1,16 @@ +namespace CrestApps.OrchardCore.AI.Playwright.Services; + +public enum PlaywrightSessionStatus +{ + /// <summary>Browser is open but no tool call is currently in flight.</summary> + Idle, + + /// <summary>A tool call is currently executing inside the browser.</summary> + Running, + + /// <summary>The user pressed Stop; the current operation was cancelled. Browser is still open.</summary> + Stopped, + + /// <summary>The session has been disposed and the browser window is closed.</summary> + Closed, +} diff --git a/src/Modules/CrestApps.OrchardCore.AI.Playwright/Services/PlaywrightToolRegistryProvider.cs b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Services/PlaywrightToolRegistryProvider.cs new file mode 100644 index 000000000..1a34735b2 --- /dev/null +++ b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Services/PlaywrightToolRegistryProvider.cs @@ -0,0 +1,41 @@ +using CrestApps.OrchardCore.AI.Core.Orchestration; +using CrestApps.OrchardCore.AI.Models; +using CrestApps.OrchardCore.AI.Playwright.Models; +using Microsoft.Extensions.AI; +using Microsoft.Extensions.DependencyInjection; + +namespace CrestApps.OrchardCore.AI.Playwright.Services; + +internal sealed class PlaywrightToolRegistryProvider : IToolRegistryProvider +{ + public Task<IReadOnlyList<ToolRegistryEntry>> GetToolsAsync( + AICompletionContext context, + CancellationToken cancellationToken = default) + { + if (context?.DisableTools == true || + context?.AdditionalProperties is null || + !context.AdditionalProperties.TryGetValue(PlaywrightConstants.CompletionContextKeys.SessionMetadata, out var metadataObject) || + metadataObject is not PlaywrightSessionMetadata { Enabled: true }) + { + return Task.FromResult<IReadOnlyList<ToolRegistryEntry>>([]); + } + + var entries = PlaywrightConstants.ToolSets.Deterministic + .Select(CreateEntry) + .ToList(); + + return Task.FromResult<IReadOnlyList<ToolRegistryEntry>>(entries); + } + + private static ToolRegistryEntry CreateEntry(string toolName) + { + return new ToolRegistryEntry + { + Id = $"playwright:{toolName}", + Name = toolName, + Description = PlaywrightConstants.GetToolDescription(toolName), + Source = ToolRegistryEntrySource.System, + CreateAsync = sp => ValueTask.FromResult(sp.GetKeyedService<AITool>(toolName)), + }; + } +} diff --git a/src/Modules/CrestApps.OrchardCore.AI.Playwright/Settings/PlaywrightProfileSettings.cs b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Settings/PlaywrightProfileSettings.cs new file mode 100644 index 000000000..75864bf25 --- /dev/null +++ b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Settings/PlaywrightProfileSettings.cs @@ -0,0 +1,11 @@ +namespace CrestApps.OrchardCore.AI.Playwright.Settings; + +/// <summary> +/// Legacy per-profile Playwright settings stored in +/// <see cref="CrestApps.OrchardCore.AI.Models.AIProfile.Settings"/>. +/// Only the enabled flag is still used. +/// </summary> +public sealed class PlaywrightProfileSettings +{ + public bool Enabled { get; set; } +} diff --git a/src/Modules/CrestApps.OrchardCore.AI.Playwright/Startup.cs b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Startup.cs new file mode 100644 index 000000000..52fcf3bf8 --- /dev/null +++ b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Startup.cs @@ -0,0 +1,78 @@ +using CrestApps.OrchardCore.AI; +using CrestApps.OrchardCore.AI.Core.Orchestration; +using CrestApps.OrchardCore.AI.Models; +using CrestApps.OrchardCore.AI.Playwright.BackgroundTasks; +using CrestApps.OrchardCore.AI.Playwright.Drivers; +using CrestApps.OrchardCore.AI.Playwright.Filters; +using CrestApps.OrchardCore.AI.Playwright.Handlers; +using CrestApps.OrchardCore.AI.Playwright.Services; +using CrestApps.OrchardCore.AI.Playwright.Tools; +using Microsoft.AspNetCore.Builder; +using Microsoft.AspNetCore.Mvc; +using Microsoft.AspNetCore.Routing; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.DependencyInjection.Extensions; +using OrchardCore.BackgroundTasks; +using OrchardCore.DisplayManagement.Handlers; +using OrchardCore.Environment.Shell; +using OrchardCore.Modules; + +namespace CrestApps.OrchardCore.AI.Playwright; + +[Feature(PlaywrightConstants.Feature.AdminWidget)] +public sealed class Startup : StartupBase +{ + public override void ConfigureServices(IServiceCollection services) + { + services.AddSingleton<IPlaywrightObservationService, PlaywrightObservationService>(); + services.AddSingleton<IPlaywrightActionVisualizer, PlaywrightActionVisualizer>(); + services.AddSingleton<IPlaywrightPageInspectionService, PlaywrightPageInspectionService>(); + services.AddSingleton<IPlaywrightSessionRequestResolver, PlaywrightSessionRequestResolver>(); + services.AddSingleton<IPlaywrightSessionManager, PlaywrightSessionManager>(); + services.AddSingleton<IOrchardAdminPlaywrightService, OrchardAdminPlaywrightService>(); + services.AddSingleton<IOrchardEvidenceService, OrchardEvidenceService>(); + services.AddSingleton<IBackgroundTask, PlaywrightSessionCleanupBackgroundTask>(); + + services.AddScoped<IFeatureEventHandler, PlaywrightFeatureEventHandler>(); + + services.TryAddEnumerable(ServiceDescriptor.Scoped<IToolRegistryProvider, PlaywrightToolRegistryProvider>()); + services.TryAddEnumerable(ServiceDescriptor.Scoped<IOrchestrationContextBuilderHandler, PlaywrightOrchestrationContextHandler>()); + services.TryAddEnumerable(ServiceDescriptor.Scoped<IChatInteractionSettingsHandler, PlaywrightChatInteractionSettingsHandler>()); + + services.AddDisplayDriver<AIProfile, PlaywrightProfileSettingsDisplayDriver>(); + services.AddDisplayDriver<ChatInteraction, ChatInteractionPlaywrightSettingsDisplayDriver>(); + + services.Configure<MvcOptions>(options => + { + options.Filters.Add<PlaywrightAdminWidgetFilter>(); + }); + + services.AddCoreAITool<CaptureStateTool>(PlaywrightConstants.ToolNames.CaptureState); + services.AddCoreAITool<OpenAdminHomeTool>(PlaywrightConstants.ToolNames.OpenAdminHome); + services.AddCoreAITool<OpenContentItemsTool>(PlaywrightConstants.ToolNames.OpenContentItems); + services.AddCoreAITool<ListContentItemsTool>(PlaywrightConstants.ToolNames.ListContentItems); + services.AddCoreAITool<OpenContentItemEditorTool>(PlaywrightConstants.ToolNames.OpenContentItemEditor); + services.AddCoreAITool<OpenNewContentItemTool>(PlaywrightConstants.ToolNames.OpenNewContentItem); + services.AddCoreAITool<OpenEditorTabTool>(PlaywrightConstants.ToolNames.OpenEditorTab); + services.AddCoreAITool<SetContentTitleTool>(PlaywrightConstants.ToolNames.SetContentTitle); + services.AddCoreAITool<SetFieldValueTool>(PlaywrightConstants.ToolNames.SetFieldValue); + services.AddCoreAITool<SetBodyFieldTool>(PlaywrightConstants.ToolNames.SetBodyField); + services.AddCoreAITool<SaveDraftTool>(PlaywrightConstants.ToolNames.SaveDraft); + services.AddCoreAITool<PublishContentTool>(PlaywrightConstants.ToolNames.PublishContent); + services.AddCoreAITool<PublishAndVerifyTool>(PlaywrightConstants.ToolNames.PublishAndVerify); + services.AddCoreAITool<GetPageContentTool>(PlaywrightConstants.ToolNames.InspectPageContent); + services.AddCoreAITool<FindElementTool>(PlaywrightConstants.ToolNames.FindElement); + services.AddCoreAITool<CheckElementExistsTool>(PlaywrightConstants.ToolNames.CheckElementExists); + services.AddCoreAITool<GetVisibleWidgetsTool>(PlaywrightConstants.ToolNames.GetVisibleWidgets); + services.AddCoreAITool<TakeScreenshotTool>(PlaywrightConstants.ToolNames.TakeScreenshot); + services.AddCoreAITool<DiagnoseOrchardActionTool>(PlaywrightConstants.ToolNames.DiagnoseOrchardAction); + } + + public override void Configure(IApplicationBuilder app, IEndpointRouteBuilder routes, IServiceProvider serviceProvider) + { + } +} + + + + diff --git a/src/Modules/CrestApps.OrchardCore.AI.Playwright/Tools/CaptureStateTool.cs b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Tools/CaptureStateTool.cs new file mode 100644 index 000000000..ba8bdd979 --- /dev/null +++ b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Tools/CaptureStateTool.cs @@ -0,0 +1,28 @@ +using System.Text.Json; +using CrestApps.OrchardCore.AI.Playwright.Services; +using Microsoft.Extensions.AI; +using Microsoft.Extensions.DependencyInjection; + +namespace CrestApps.OrchardCore.AI.Playwright.Tools; + +public sealed class CaptureStateTool : PlaywrightToolBase +{ + private static readonly JsonElement _schema = JsonSerializer.Deserialize<JsonElement>(""" + { + "type": "object", + "properties": {}, + "additionalProperties": false + } + """); + + public override string Name => PlaywrightConstants.ToolNames.CaptureState; + public override string Description => "Captures the current browser URL, title, heading, toast message, validation messages, and a short control summary."; + public override JsonElement JsonSchema => _schema; + + protected override ValueTask<object?> InvokeCoreAsync(AIFunctionArguments arguments, CancellationToken cancellationToken) + => ExecuteObservationStepAsync( + arguments, + cancellationToken, + "capture_state", + (session, token) => arguments.Services.GetRequiredService<IOrchardAdminPlaywrightService>().CaptureStateAsync(session, token)); +} diff --git a/src/Modules/CrestApps.OrchardCore.AI.Playwright/Tools/CheckElementExistsTool.cs b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Tools/CheckElementExistsTool.cs new file mode 100644 index 000000000..58e942b95 --- /dev/null +++ b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Tools/CheckElementExistsTool.cs @@ -0,0 +1,56 @@ +using System.Text.Json; +using CrestApps.OrchardCore.AI.Playwright.Services; +using Microsoft.Extensions.AI; +using Microsoft.Extensions.DependencyInjection; + +namespace CrestApps.OrchardCore.AI.Playwright.Tools; + +public sealed class CheckElementExistsTool : PlaywrightToolBase +{ + private static readonly JsonElement _schema = JsonSerializer.Deserialize<JsonElement>(""" + { + "type": "object", + "required": ["query"], + "properties": { + "query": { + "type": "string", + "description": "Visible text, widget name, label, or control name to check for on the current page." + }, + "maxMatches": { + "type": "integer", + "description": "Maximum number of matching elements to include in the response. Defaults to 3." + } + }, + "additionalProperties": false + } + """); + + public override string Name => PlaywrightConstants.ToolNames.CheckElementExists; + public override string Description => "Checks whether a requested control, widget, or text snippet is currently visible on the page."; + public override JsonElement JsonSchema => _schema; + + protected override ValueTask<object?> InvokeCoreAsync(AIFunctionArguments arguments, CancellationToken cancellationToken) + { + var query = arguments["query"]?.ToString(); + if (string.IsNullOrWhiteSpace(query)) + { + return new ValueTask<object?>("Parameter 'query' is required."); + } + + var maxMatches = 3; + if (arguments.TryGetValue("maxMatches", out var maxMatchesValue) + && maxMatchesValue is JsonElement jsonMaxMatches + && jsonMaxMatches.ValueKind == JsonValueKind.Number) + { + maxMatches = jsonMaxMatches.GetInt32(); + } + + return ExecuteSessionStepAsync(arguments, cancellationToken, async (session, token) => + { + var result = await arguments.Services.GetRequiredService<IPlaywrightPageInspectionService>() + .CheckElementExistsAsync(session, query, maxMatches, token); + + return Serialize(result); + }); + } +} diff --git a/src/Modules/CrestApps.OrchardCore.AI.Playwright/Tools/DiagnoseOrchardActionTool.cs b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Tools/DiagnoseOrchardActionTool.cs new file mode 100644 index 000000000..692088cf5 --- /dev/null +++ b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Tools/DiagnoseOrchardActionTool.cs @@ -0,0 +1,50 @@ +using System.Text.Json; +using CrestApps.OrchardCore.AI.Playwright.Services; +using Microsoft.Extensions.AI; +using Microsoft.Extensions.DependencyInjection; + +namespace CrestApps.OrchardCore.AI.Playwright.Tools; + +public sealed class DiagnoseOrchardActionTool : PlaywrightToolBase +{ + private static readonly JsonElement _schema = JsonSerializer.Deserialize<JsonElement>(""" + { + "type": "object", + "required": ["actionLabel"], + "properties": { + "actionLabel": { + "type": "string", + "description": "The OrchardCore admin action to locate. Examples: 'Edit', 'Publish Now', 'Save Draft', 'Preview', 'Delete', 'Clone', or any widget action label visible in the admin UI." + } + }, + "additionalProperties": false + } + """); + + public override string Name => PlaywrightConstants.ToolNames.DiagnoseOrchardAction; + + public override string Description => + "Attempts to find a named OrchardCore admin action (Edit, Publish Now, Save Draft, Preview, widget actions) " + + "using priority-ordered locator strategies. When the action is not found, captures a full-page screenshot, " + + "a container screenshot, and the raw page HTML so the missing action can be diagnosed."; + + public override JsonElement JsonSchema => _schema; + + protected override ValueTask<object?> InvokeCoreAsync(AIFunctionArguments arguments, CancellationToken cancellationToken) + { + var actionLabel = arguments["actionLabel"]?.ToString(); + if (string.IsNullOrWhiteSpace(actionLabel)) + { + return new ValueTask<object?>("Parameter 'actionLabel' is required."); + } + + return ExecuteSessionStepAsync(arguments, cancellationToken, async (session, token) => + { + var result = await arguments.Services + .GetRequiredService<IOrchardEvidenceService>() + .FindOrchardElementWithEvidenceAsync(session, actionLabel, token); + + return Serialize(result); + }); + } +} diff --git a/src/Modules/CrestApps.OrchardCore.AI.Playwright/Tools/FindElementTool.cs b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Tools/FindElementTool.cs new file mode 100644 index 000000000..77d85f404 --- /dev/null +++ b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Tools/FindElementTool.cs @@ -0,0 +1,56 @@ +using System.Text.Json; +using CrestApps.OrchardCore.AI.Playwright.Services; +using Microsoft.Extensions.AI; +using Microsoft.Extensions.DependencyInjection; + +namespace CrestApps.OrchardCore.AI.Playwright.Tools; + +public sealed class FindElementTool : PlaywrightToolBase +{ + private static readonly JsonElement _schema = JsonSerializer.Deserialize<JsonElement>(""" + { + "type": "object", + "required": ["query"], + "properties": { + "query": { + "type": "string", + "description": "Visible text, widget name, label, or control name to search for on the current page." + }, + "maxMatches": { + "type": "integer", + "description": "Maximum number of matches to return. Defaults to 5." + } + }, + "additionalProperties": false + } + """); + + public override string Name => PlaywrightConstants.ToolNames.FindElement; + public override string Description => "Finds visible page elements that match a requested widget name, label, or text snippet."; + public override JsonElement JsonSchema => _schema; + + protected override ValueTask<object?> InvokeCoreAsync(AIFunctionArguments arguments, CancellationToken cancellationToken) + { + var query = arguments["query"]?.ToString(); + if (string.IsNullOrWhiteSpace(query)) + { + return new ValueTask<object?>("Parameter 'query' is required."); + } + + var maxMatches = 5; + if (arguments.TryGetValue("maxMatches", out var maxMatchesValue) + && maxMatchesValue is JsonElement jsonMaxMatches + && jsonMaxMatches.ValueKind == JsonValueKind.Number) + { + maxMatches = jsonMaxMatches.GetInt32(); + } + + return ExecuteSessionStepAsync(arguments, cancellationToken, async (session, token) => + { + var result = await arguments.Services.GetRequiredService<IPlaywrightPageInspectionService>() + .FindElementsAsync(session, query, maxMatches, token); + + return Serialize(result); + }); + } +} diff --git a/src/Modules/CrestApps.OrchardCore.AI.Playwright/Tools/GetPageContentTool.cs b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Tools/GetPageContentTool.cs new file mode 100644 index 000000000..a9d62ac4e --- /dev/null +++ b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Tools/GetPageContentTool.cs @@ -0,0 +1,45 @@ +using System.Text.Json; +using CrestApps.OrchardCore.AI.Playwright.Services; +using Microsoft.Extensions.AI; +using Microsoft.Extensions.DependencyInjection; + +namespace CrestApps.OrchardCore.AI.Playwright.Tools; + +public sealed class GetPageContentTool : PlaywrightToolBase +{ + private static readonly JsonElement _schema = JsonSerializer.Deserialize<JsonElement>(""" + { + "type": "object", + "properties": { + "maxLength": { + "type": "integer", + "description": "Maximum number of characters to return from the visible page content. Defaults to 4000." + } + }, + "additionalProperties": false + } + """); + + public override string Name => PlaywrightConstants.ToolNames.InspectPageContent; + public override string Description => "Returns the visible content of the current page for grounded follow-up questions."; + public override JsonElement JsonSchema => _schema; + + protected override ValueTask<object?> InvokeCoreAsync(AIFunctionArguments arguments, CancellationToken cancellationToken) + { + var maxLength = 4000; + if (arguments.TryGetValue("maxLength", out var maxLengthValue) + && maxLengthValue is JsonElement jsonMaxLength + && jsonMaxLength.ValueKind == JsonValueKind.Number) + { + maxLength = jsonMaxLength.GetInt32(); + } + + return ExecuteSessionStepAsync(arguments, cancellationToken, async (session, token) => + { + var result = await arguments.Services.GetRequiredService<IPlaywrightPageInspectionService>() + .GetPageContentAsync(session, maxLength, token); + + return Serialize(result); + }); + } +} diff --git a/src/Modules/CrestApps.OrchardCore.AI.Playwright/Tools/GetVisibleWidgetsTool.cs b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Tools/GetVisibleWidgetsTool.cs new file mode 100644 index 000000000..6f0e5a5f4 --- /dev/null +++ b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Tools/GetVisibleWidgetsTool.cs @@ -0,0 +1,45 @@ +using System.Text.Json; +using CrestApps.OrchardCore.AI.Playwright.Services; +using Microsoft.Extensions.AI; +using Microsoft.Extensions.DependencyInjection; + +namespace CrestApps.OrchardCore.AI.Playwright.Tools; + +public sealed class GetVisibleWidgetsTool : PlaywrightToolBase +{ + private static readonly JsonElement _schema = JsonSerializer.Deserialize<JsonElement>(""" + { + "type": "object", + "properties": { + "maxItems": { + "type": "integer", + "description": "Maximum number of visible widget-like items to return. Defaults to 12." + } + }, + "additionalProperties": false + } + """); + + public override string Name => PlaywrightConstants.ToolNames.GetVisibleWidgets; + public override string Description => "Lists visible widget-like cards, headings, and editor sections on the current page."; + public override JsonElement JsonSchema => _schema; + + protected override ValueTask<object?> InvokeCoreAsync(AIFunctionArguments arguments, CancellationToken cancellationToken) + { + var maxItems = 12; + if (arguments.TryGetValue("maxItems", out var maxItemsValue) + && maxItemsValue is JsonElement jsonMaxItems + && jsonMaxItems.ValueKind == JsonValueKind.Number) + { + maxItems = jsonMaxItems.GetInt32(); + } + + return ExecuteSessionStepAsync(arguments, cancellationToken, async (session, token) => + { + var result = await arguments.Services.GetRequiredService<IPlaywrightPageInspectionService>() + .GetVisibleWidgetsAsync(session, maxItems, token); + + return Serialize(result); + }); + } +} diff --git a/src/Modules/CrestApps.OrchardCore.AI.Playwright/Tools/ListContentItemsTool.cs b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Tools/ListContentItemsTool.cs new file mode 100644 index 000000000..12f685d6f --- /dev/null +++ b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Tools/ListContentItemsTool.cs @@ -0,0 +1,45 @@ +using System.Text.Json; +using CrestApps.OrchardCore.AI.Playwright.Services; +using Microsoft.Extensions.AI; +using Microsoft.Extensions.DependencyInjection; + +namespace CrestApps.OrchardCore.AI.Playwright.Tools; + +public sealed class ListContentItemsTool : PlaywrightToolBase +{ + private static readonly JsonElement _schema = JsonSerializer.Deserialize<JsonElement>(""" + { + "type": "object", + "properties": { + "maxItems": { + "type": "integer", + "description": "Maximum number of visible content items to return. Defaults to 10." + } + }, + "additionalProperties": false + } + """); + + public override string Name => PlaywrightConstants.ToolNames.ListContentItems; + public override string Description => "Lists the visible Orchard content items from the current content items screen."; + public override JsonElement JsonSchema => _schema; + + protected override ValueTask<object?> InvokeCoreAsync(AIFunctionArguments arguments, CancellationToken cancellationToken) + { + var maxItems = 10; + if (arguments.TryGetValue("maxItems", out var maxItemsValue) + && maxItemsValue is JsonElement jsonMaxItems + && jsonMaxItems.ValueKind == JsonValueKind.Number) + { + maxItems = jsonMaxItems.GetInt32(); + } + + return ExecuteSessionStepAsync(arguments, cancellationToken, async (session, token) => + { + var result = await arguments.Services.GetRequiredService<IOrchardAdminPlaywrightService>() + .ListVisibleContentItemsAsync(session, maxItems, token); + + return Serialize(result); + }); + } +} diff --git a/src/Modules/CrestApps.OrchardCore.AI.Playwright/Tools/OpenAdminHomeTool.cs b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Tools/OpenAdminHomeTool.cs new file mode 100644 index 000000000..b02af9b96 --- /dev/null +++ b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Tools/OpenAdminHomeTool.cs @@ -0,0 +1,28 @@ +using System.Text.Json; +using CrestApps.OrchardCore.AI.Playwright.Services; +using Microsoft.Extensions.AI; +using Microsoft.Extensions.DependencyInjection; + +namespace CrestApps.OrchardCore.AI.Playwright.Tools; + +public sealed class OpenAdminHomeTool : PlaywrightToolBase +{ + private static readonly JsonElement _schema = JsonSerializer.Deserialize<JsonElement>(""" + { + "type": "object", + "properties": {}, + "additionalProperties": false + } + """); + + public override string Name => PlaywrightConstants.ToolNames.OpenAdminHome; + public override string Description => "Navigates to the Orchard admin home for the current tenant and validates whether the browser session is authenticated."; + public override JsonElement JsonSchema => _schema; + + protected override ValueTask<object?> InvokeCoreAsync(AIFunctionArguments arguments, CancellationToken cancellationToken) + => ExecuteObservationStepAsync( + arguments, + cancellationToken, + "open_admin_home", + (session, token) => arguments.Services.GetRequiredService<IOrchardAdminPlaywrightService>().OpenAdminHomeAsync(session, token)); +} diff --git a/src/Modules/CrestApps.OrchardCore.AI.Playwright/Tools/OpenContentItemEditorTool.cs b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Tools/OpenContentItemEditorTool.cs new file mode 100644 index 000000000..e42bb5385 --- /dev/null +++ b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Tools/OpenContentItemEditorTool.cs @@ -0,0 +1,44 @@ +using System.Text.Json; +using CrestApps.OrchardCore.AI.Playwright.Services; +using Microsoft.Extensions.AI; +using Microsoft.Extensions.DependencyInjection; + +namespace CrestApps.OrchardCore.AI.Playwright.Tools; + +public sealed class OpenContentItemEditorTool : PlaywrightToolBase +{ + private static readonly JsonElement _schema = JsonSerializer.Deserialize<JsonElement>(""" + { + "type": "object", + "required": ["title"], + "properties": { + "title": { + "type": "string", + "description": "Title of the content item to open for editing. The tool uses the current content list first and can use fuzzy matching when needed." + } + }, + "additionalProperties": false + } + """); + + public override string Name => PlaywrightConstants.ToolNames.OpenContentItemEditor; + public override string Description => "Opens the editor for an existing Orchard content item by title using the current content list first and row-scoped edit behavior when available."; + public override JsonElement JsonSchema => _schema; + + protected override ValueTask<object?> InvokeCoreAsync(AIFunctionArguments arguments, CancellationToken cancellationToken) + { + var title = arguments["title"]?.ToString(); + if (string.IsNullOrWhiteSpace(title)) + { + return new ValueTask<object?>("Parameter 'title' is required."); + } + + return ExecuteSessionStepAsync(arguments, cancellationToken, async (session, token) => + { + var result = await arguments.Services.GetRequiredService<IOrchardAdminPlaywrightService>() + .OpenContentItemEditorAsync(session, title, token); + + return Serialize(result); + }); + } +} diff --git a/src/Modules/CrestApps.OrchardCore.AI.Playwright/Tools/OpenContentItemsTool.cs b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Tools/OpenContentItemsTool.cs new file mode 100644 index 000000000..3895648aa --- /dev/null +++ b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Tools/OpenContentItemsTool.cs @@ -0,0 +1,28 @@ +using System.Text.Json; +using CrestApps.OrchardCore.AI.Playwright.Services; +using Microsoft.Extensions.AI; +using Microsoft.Extensions.DependencyInjection; + +namespace CrestApps.OrchardCore.AI.Playwright.Tools; + +public sealed class OpenContentItemsTool : PlaywrightToolBase +{ + private static readonly JsonElement _schema = JsonSerializer.Deserialize<JsonElement>(""" + { + "type": "object", + "properties": {}, + "additionalProperties": false + } + """); + + public override string Name => PlaywrightConstants.ToolNames.OpenContentItems; + public override string Description => "Opens the Orchard content items list using the current tenant's admin shell."; + public override JsonElement JsonSchema => _schema; + + protected override ValueTask<object?> InvokeCoreAsync(AIFunctionArguments arguments, CancellationToken cancellationToken) + => ExecuteObservationStepAsync( + arguments, + cancellationToken, + "open_content_items", + (session, token) => arguments.Services.GetRequiredService<IOrchardAdminPlaywrightService>().OpenContentItemsAsync(session, token)); +} diff --git a/src/Modules/CrestApps.OrchardCore.AI.Playwright/Tools/OpenEditorTabTool.cs b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Tools/OpenEditorTabTool.cs new file mode 100644 index 000000000..926211e3d --- /dev/null +++ b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Tools/OpenEditorTabTool.cs @@ -0,0 +1,51 @@ +using System.Text.Json; +using CrestApps.OrchardCore.AI.Playwright.Services; +using Microsoft.Extensions.AI; +using Microsoft.Extensions.DependencyInjection; + +namespace CrestApps.OrchardCore.AI.Playwright.Tools; + +public sealed class OpenEditorTabTool : PlaywrightToolBase +{ + private static readonly JsonElement _schema = JsonSerializer.Deserialize<JsonElement>(""" + { + "type": "object", + "required": ["tabName"], + "properties": { + "tabName": { + "type": "string", + "description": "Visible Orchard editor tab or top-level editor section name to open." + }, + "exact": { + "type": "boolean", + "description": "Whether to require an exact tab-name match." + } + }, + "additionalProperties": false + } + """); + + public override string Name => PlaywrightConstants.ToolNames.OpenEditorTab; + public override string Description => "Opens a named Orchard editor tab or top-level editor section on the current content editor."; + public override JsonElement JsonSchema => _schema; + + protected override ValueTask<object?> InvokeCoreAsync(AIFunctionArguments arguments, CancellationToken cancellationToken) + { + var tabName = arguments["tabName"]?.ToString(); + var exactMatch = arguments.TryGetValue("exact", out var exactValue) && exactValue is bool exact && exact; + + if (string.IsNullOrWhiteSpace(tabName)) + { + return new ValueTask<object?>("Parameter 'tabName' is required."); + } + + return ExecuteSessionStepAsync(arguments, cancellationToken, async (session, token) => + { + var result = await arguments.Services.GetRequiredService<IOrchardAdminPlaywrightService>() + .OpenEditorTabAsync(session, tabName, exactMatch, token); + + return Serialize(result); + }); + } +} + diff --git a/src/Modules/CrestApps.OrchardCore.AI.Playwright/Tools/OpenNewContentItemTool.cs b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Tools/OpenNewContentItemTool.cs new file mode 100644 index 000000000..717ee4734 --- /dev/null +++ b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Tools/OpenNewContentItemTool.cs @@ -0,0 +1,42 @@ +using System.Text.Json; +using CrestApps.OrchardCore.AI.Playwright.Services; +using Microsoft.Extensions.AI; +using Microsoft.Extensions.DependencyInjection; + +namespace CrestApps.OrchardCore.AI.Playwright.Tools; + +public sealed class OpenNewContentItemTool : PlaywrightToolBase +{ + private static readonly JsonElement _schema = JsonSerializer.Deserialize<JsonElement>(""" + { + "type": "object", + "required": ["contentType"], + "properties": { + "contentType": { + "type": "string", + "description": "The Orchard content type to create, for example SitePage." + } + }, + "additionalProperties": false + } + """); + + public override string Name => PlaywrightConstants.ToolNames.OpenNewContentItem; + public override string Description => "Starts the Orchard create-content flow for a specific content type and waits for the editor to load."; + public override JsonElement JsonSchema => _schema; + + protected override ValueTask<object?> InvokeCoreAsync(AIFunctionArguments arguments, CancellationToken cancellationToken) + { + var contentType = arguments["contentType"]?.ToString(); + if (string.IsNullOrWhiteSpace(contentType)) + { + return new ValueTask<object?>("Parameter 'contentType' is required."); + } + + return ExecuteObservationStepAsync( + arguments, + cancellationToken, + $"open_new_content_item:{contentType}", + (session, token) => arguments.Services.GetRequiredService<IOrchardAdminPlaywrightService>().OpenNewContentItemAsync(session, contentType, token)); + } +} diff --git a/src/Modules/CrestApps.OrchardCore.AI.Playwright/Tools/PlaywrightToolBase.cs b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Tools/PlaywrightToolBase.cs new file mode 100644 index 000000000..03e5ca191 --- /dev/null +++ b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Tools/PlaywrightToolBase.cs @@ -0,0 +1,155 @@ +using System.Text.Json; +using CrestApps.OrchardCore.AI; +using CrestApps.OrchardCore.AI.Models; +using CrestApps.OrchardCore.AI.Playwright.Models; +using CrestApps.OrchardCore.AI.Playwright.Services; +using Microsoft.AspNetCore.Http; +using Microsoft.Extensions.AI; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Logging; + +namespace CrestApps.OrchardCore.AI.Playwright.Tools; + +/// <summary> +/// Shared scaffolding for all Playwright browser tools. +/// Resolves or creates the session and wires the stop token. +/// </summary> +public abstract class PlaywrightToolBase : AIFunction +{ + public override IReadOnlyDictionary<string, object> AdditionalProperties { get; } = new Dictionary<string, object> + { + ["Strict"] = false, + }; + + protected async Task<IPlaywrightSession> ResolveSessionAsync( + AIFunctionArguments arguments, + CancellationToken cancellationToken) + { + var logger = arguments.Services.GetRequiredService<ILogger<PlaywrightToolBase>>(); + var sessionManager = arguments.Services.GetRequiredService<IPlaywrightSessionManager>(); + var requestResolver = arguments.Services.GetRequiredService<IPlaywrightSessionRequestResolver>(); + + var chatSessionId = ResolveChatSessionId(arguments); + if (string.IsNullOrWhiteSpace(chatSessionId)) + { + logger.LogWarning("Playwright tool invoked but no chat session ID could be resolved."); + return null; + } + + var resource = AIInvocationScope.Current?.ToolExecutionContext?.Resource; + var request = requestResolver.Resolve(resource, chatSessionId); + if (request is null) + { + logger.LogWarning("Playwright tool invoked but no Playwright-enabled resource metadata was found."); + return null; + } + + return await sessionManager.GetOrCreateAsync(request, cancellationToken); + } + + protected async ValueTask<object?> ExecuteSessionStepAsync( + AIFunctionArguments arguments, + CancellationToken cancellationToken, + Func<IPlaywrightSession, CancellationToken, Task<object?>> action) + { + var logger = arguments.Services.GetRequiredService<ILogger<PlaywrightToolBase>>(); + var session = await ResolveSessionAsync(arguments, cancellationToken); + if (session is null) + { + return "Playwright session is not available. Enable Playwright on the current profile or interaction first."; + } + + session.MarkRunning(); + using var linkedCts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken, session.StopToken); + + try + { + return await action(session, linkedCts.Token); + } + catch (OperationCanceledException) + { + return "Playwright operation was stopped by the user."; + } + catch (Exception ex) + { + logger.LogWarning( + ex, + "Playwright tool step failed for session '{ChatSessionId}'.", + session.SessionId); + + return Serialize(new + { + error = ex.Message, + browserMode = "DedicatedBrowser", + isAuthenticated = session.IsAuthenticated, + observation = session.LastObservation, + }); + } + finally + { + session.MarkIdle(); + } + } + + protected async ValueTask<object?> ExecuteObservationStepAsync( + AIFunctionArguments arguments, + CancellationToken cancellationToken, + string intent, + Func<IPlaywrightSession, CancellationToken, Task<PlaywrightObservation>> action) + { + return await ExecuteSessionStepAsync(arguments, cancellationToken, async (session, token) => + { + var observation = await action(session, token); + return Serialize(new + { + intent, + browserMode = "DedicatedBrowser", + isAuthenticated = session.IsAuthenticated, + observation, + }); + }); + } + + protected static string ResolveChatSessionId(AIFunctionArguments arguments) + { + if (AIInvocationScope.Current?.Items.TryGetValue(nameof(AIChatSession), out var sessionObject) == true + && sessionObject is AIChatSession chatSession) + { + return chatSession.SessionId; + } + + switch (AIInvocationScope.Current?.ToolExecutionContext?.Resource) + { + case ChatInteraction interaction when !string.IsNullOrWhiteSpace(interaction.ItemId): + return interaction.ItemId; + case CrestApps.OrchardCore.Models.CatalogItem item when !string.IsNullOrWhiteSpace(item.ItemId): + return item.ItemId; + } + + var httpContext = arguments.Services.GetRequiredService<IHttpContextAccessor>().HttpContext; + return httpContext?.User?.Identity?.Name; + } + + protected static string ResolveAbsoluteUrl(IPlaywrightSession session, string url) + { + if (string.IsNullOrWhiteSpace(url)) + { + return url; + } + + if (Uri.TryCreate(url, UriKind.Absolute, out var absolute)) + { + return absolute.ToString(); + } + + if (url.StartsWith('/')) + { + return PlaywrightSessionRequestResolver.CombineUrl(session.BaseUrl, url.TrimStart('/')); + } + + return PlaywrightSessionRequestResolver.CombineUrl(session.BaseUrl, url); + } + + protected static string Serialize(object value) + => JsonSerializer.Serialize(value); +} diff --git a/src/Modules/CrestApps.OrchardCore.AI.Playwright/Tools/PublishAndVerifyTool.cs b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Tools/PublishAndVerifyTool.cs new file mode 100644 index 000000000..56fdca265 --- /dev/null +++ b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Tools/PublishAndVerifyTool.cs @@ -0,0 +1,39 @@ +using System.Text.Json; +using CrestApps.OrchardCore.AI.Playwright.Services; +using Microsoft.Extensions.AI; +using Microsoft.Extensions.DependencyInjection; + +namespace CrestApps.OrchardCore.AI.Playwright.Tools; + +public sealed class PublishAndVerifyTool : PlaywrightToolBase +{ + private static readonly JsonElement _schema = JsonSerializer.Deserialize<JsonElement>(""" + { + "type": "object", + "properties": { + "expectedStatus": { + "type": "string", + "description": "Expected Orchard status after publish. Defaults to Published." + } + }, + "additionalProperties": false + } + """); + + public override string Name => PlaywrightConstants.ToolNames.PublishAndVerify; + public override string Description => "Publishes the current Orchard content item and returns Orchard-specific verification signals."; + public override JsonElement JsonSchema => _schema; + + protected override ValueTask<object?> InvokeCoreAsync(AIFunctionArguments arguments, CancellationToken cancellationToken) + { + var expectedStatus = arguments["expectedStatus"]?.ToString() ?? "Published"; + + return ExecuteSessionStepAsync(arguments, cancellationToken, async (session, token) => + { + var result = await arguments.Services.GetRequiredService<IOrchardAdminPlaywrightService>() + .PublishAndVerifyAsync(session, expectedStatus, token); + + return Serialize(result); + }); + } +} diff --git a/src/Modules/CrestApps.OrchardCore.AI.Playwright/Tools/PublishContentTool.cs b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Tools/PublishContentTool.cs new file mode 100644 index 000000000..b0ad611f0 --- /dev/null +++ b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Tools/PublishContentTool.cs @@ -0,0 +1,28 @@ +using System.Text.Json; +using CrestApps.OrchardCore.AI.Playwright.Services; +using Microsoft.Extensions.AI; +using Microsoft.Extensions.DependencyInjection; + +namespace CrestApps.OrchardCore.AI.Playwright.Tools; + +public sealed class PublishContentTool : PlaywrightToolBase +{ + private static readonly JsonElement _schema = JsonSerializer.Deserialize<JsonElement>(""" + { + "type": "object", + "properties": {}, + "additionalProperties": false + } + """); + + public override string Name => PlaywrightConstants.ToolNames.PublishContent; + public override string Description => "Publishes the current Orchard content item and captures the resulting page state."; + public override JsonElement JsonSchema => _schema; + + protected override ValueTask<object?> InvokeCoreAsync(AIFunctionArguments arguments, CancellationToken cancellationToken) + => ExecuteObservationStepAsync( + arguments, + cancellationToken, + "publish_content", + (session, token) => arguments.Services.GetRequiredService<IOrchardAdminPlaywrightService>().PublishContentAsync(session, token)); +} diff --git a/src/Modules/CrestApps.OrchardCore.AI.Playwright/Tools/SaveDraftTool.cs b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Tools/SaveDraftTool.cs new file mode 100644 index 000000000..b85909523 --- /dev/null +++ b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Tools/SaveDraftTool.cs @@ -0,0 +1,28 @@ +using System.Text.Json; +using CrestApps.OrchardCore.AI.Playwright.Services; +using Microsoft.Extensions.AI; +using Microsoft.Extensions.DependencyInjection; + +namespace CrestApps.OrchardCore.AI.Playwright.Tools; + +public sealed class SaveDraftTool : PlaywrightToolBase +{ + private static readonly JsonElement _schema = JsonSerializer.Deserialize<JsonElement>(""" + { + "type": "object", + "properties": {}, + "additionalProperties": false + } + """); + + public override string Name => PlaywrightConstants.ToolNames.SaveDraft; + public override string Description => "Saves the current Orchard content editor as a draft and captures the resulting page state."; + public override JsonElement JsonSchema => _schema; + + protected override ValueTask<object?> InvokeCoreAsync(AIFunctionArguments arguments, CancellationToken cancellationToken) + => ExecuteObservationStepAsync( + arguments, + cancellationToken, + "save_draft", + (session, token) => arguments.Services.GetRequiredService<IOrchardAdminPlaywrightService>().SaveDraftAsync(session, token)); +} diff --git a/src/Modules/CrestApps.OrchardCore.AI.Playwright/Tools/SetBodyFieldTool.cs b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Tools/SetBodyFieldTool.cs new file mode 100644 index 000000000..5eb010531 --- /dev/null +++ b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Tools/SetBodyFieldTool.cs @@ -0,0 +1,63 @@ +using System.Text.Json; +using CrestApps.OrchardCore.AI.Playwright.Services; +using Microsoft.Extensions.AI; +using Microsoft.Extensions.DependencyInjection; + +namespace CrestApps.OrchardCore.AI.Playwright.Tools; + +public sealed class SetBodyFieldTool : PlaywrightToolBase +{ + private static readonly JsonElement _schema = JsonSerializer.Deserialize<JsonElement>(""" + { + "type": "object", + "required": ["label", "value"], + "properties": { + "label": { + "type": "string", + "description": "Visible Orchard body-like field label to target, such as HtmlBody or Body." + }, + "value": { + "type": "string", + "description": "Value to write into the body field." + }, + "mode": { + "type": "string", + "description": "Whether to append to existing content or replace it.", + "enum": ["append", "replace"] + }, + "exact": { + "type": "boolean", + "description": "Whether to require an exact label match." + } + }, + "additionalProperties": false + } + """); + + public override string Name => PlaywrightConstants.ToolNames.SetBodyField; + public override string Description => "Sets a body-like Orchard field such as HtmlBody using append or replace behavior and rich-editor detection."; + public override JsonElement JsonSchema => _schema; + + protected override ValueTask<object?> InvokeCoreAsync(AIFunctionArguments arguments, CancellationToken cancellationToken) + { + var label = arguments["label"]?.ToString(); + var value = arguments["value"]?.ToString() ?? string.Empty; + var mode = arguments["mode"]?.ToString() ?? "append"; + var exactMatch = arguments.TryGetValue("exact", out var exactValue) && exactValue is bool exact && exact; + var append = !mode.Equals("replace", StringComparison.OrdinalIgnoreCase); + + if (string.IsNullOrWhiteSpace(label)) + { + return new ValueTask<object?>("Parameter 'label' is required."); + } + + return ExecuteSessionStepAsync(arguments, cancellationToken, async (session, token) => + { + var result = await arguments.Services.GetRequiredService<IOrchardAdminPlaywrightService>() + .SetBodyFieldAsync(session, label, value, mode, exactMatch, token); + + return Serialize(result); + }); + } +} + diff --git a/src/Modules/CrestApps.OrchardCore.AI.Playwright/Tools/SetContentTitleTool.cs b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Tools/SetContentTitleTool.cs new file mode 100644 index 000000000..100bf33ab --- /dev/null +++ b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Tools/SetContentTitleTool.cs @@ -0,0 +1,42 @@ +using System.Text.Json; +using CrestApps.OrchardCore.AI.Playwright.Services; +using Microsoft.Extensions.AI; +using Microsoft.Extensions.DependencyInjection; + +namespace CrestApps.OrchardCore.AI.Playwright.Tools; + +public sealed class SetContentTitleTool : PlaywrightToolBase +{ + private static readonly JsonElement _schema = JsonSerializer.Deserialize<JsonElement>(""" + { + "type": "object", + "required": ["title"], + "properties": { + "title": { + "type": "string", + "description": "The exact title value to set in the Orchard editor." + } + }, + "additionalProperties": false + } + """); + + public override string Name => PlaywrightConstants.ToolNames.SetContentTitle; + public override string Description => "Sets the Orchard content Title field using label-first fallback-safe locators. This edits the field only and does not save or publish."; + public override JsonElement JsonSchema => _schema; + + protected override ValueTask<object?> InvokeCoreAsync(AIFunctionArguments arguments, CancellationToken cancellationToken) + { + var title = arguments["title"]?.ToString(); + if (string.IsNullOrWhiteSpace(title)) + { + return new ValueTask<object?>("Parameter 'title' is required."); + } + + return ExecuteObservationStepAsync( + arguments, + cancellationToken, + $"set_content_title:{title}", + (session, token) => arguments.Services.GetRequiredService<IOrchardAdminPlaywrightService>().SetContentTitleAsync(session, title, token)); + } +} diff --git a/src/Modules/CrestApps.OrchardCore.AI.Playwright/Tools/SetFieldValueTool.cs b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Tools/SetFieldValueTool.cs new file mode 100644 index 000000000..fd8f12bf0 --- /dev/null +++ b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Tools/SetFieldValueTool.cs @@ -0,0 +1,62 @@ +using System.Text.Json; +using CrestApps.OrchardCore.AI.Playwright.Services; +using Microsoft.Extensions.AI; +using Microsoft.Extensions.DependencyInjection; + +namespace CrestApps.OrchardCore.AI.Playwright.Tools; + +public sealed class SetFieldValueTool : PlaywrightToolBase +{ + private static readonly JsonElement _schema = JsonSerializer.Deserialize<JsonElement>(""" + { + "type": "object", + "required": ["label", "value"], + "properties": { + "label": { + "type": "string", + "description": "Visible Orchard field label to target." + }, + "value": { + "type": "string", + "description": "Value to write into the field." + }, + "fieldType": { + "type": "string", + "description": "Typed Orchard field strategy. Supported values: auto, text, textarea, select, checkbox.", + "enum": ["auto", "text", "textarea", "select", "checkbox"] + }, + "exact": { + "type": "boolean", + "description": "Whether to require an exact label match." + } + }, + "additionalProperties": false + } + """); + + public override string Name => PlaywrightConstants.ToolNames.SetFieldValue; + public override string Description => "Sets an Orchard field by label using a typed field strategy instead of a generic fill action."; + public override JsonElement JsonSchema => _schema; + + protected override ValueTask<object?> InvokeCoreAsync(AIFunctionArguments arguments, CancellationToken cancellationToken) + { + var label = arguments["label"]?.ToString(); + var value = arguments["value"]?.ToString() ?? string.Empty; + var fieldType = arguments["fieldType"]?.ToString() ?? "auto"; + var exactMatch = arguments.TryGetValue("exact", out var exactValue) && exactValue is bool exact && exact; + + if (string.IsNullOrWhiteSpace(label)) + { + return new ValueTask<object?>("Parameter 'label' is required."); + } + + return ExecuteSessionStepAsync(arguments, cancellationToken, async (session, token) => + { + var result = await arguments.Services.GetRequiredService<IOrchardAdminPlaywrightService>() + .SetFieldValueAsync(session, label, value, fieldType, exactMatch, token); + + return Serialize(result); + }); + } +} + diff --git a/src/Modules/CrestApps.OrchardCore.AI.Playwright/Tools/TakeScreenshotTool.cs b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Tools/TakeScreenshotTool.cs new file mode 100644 index 000000000..d133b70d0 --- /dev/null +++ b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Tools/TakeScreenshotTool.cs @@ -0,0 +1,48 @@ +using System.Text.Json; +using CrestApps.OrchardCore.AI.Playwright.Services; +using Microsoft.Extensions.AI; +using Microsoft.Extensions.DependencyInjection; + +namespace CrestApps.OrchardCore.AI.Playwright.Tools; + +public sealed class TakeScreenshotTool : PlaywrightToolBase +{ + private static readonly JsonElement _schema = JsonSerializer.Deserialize<JsonElement>(""" + { + "type": "object", + "properties": { + "fullPage": { + "type": "boolean", + "description": "When true, captures the full page instead of only the current viewport. Defaults to false." + } + }, + "additionalProperties": false + } + """); + + public override string Name => PlaywrightConstants.ToolNames.TakeScreenshot; + public override string Description => "Captures a screenshot of the current page and returns the saved file path."; + public override JsonElement JsonSchema => _schema; + + protected override ValueTask<object?> InvokeCoreAsync(AIFunctionArguments arguments, CancellationToken cancellationToken) + { + var fullPage = false; + if (arguments.TryGetValue("fullPage", out var fullPageValue)) + { + fullPage = fullPageValue switch + { + bool isFullPage => isFullPage, + JsonElement jsonFullPage when jsonFullPage.ValueKind is JsonValueKind.True or JsonValueKind.False => jsonFullPage.GetBoolean(), + _ => false, + }; + } + + return ExecuteSessionStepAsync(arguments, cancellationToken, async (session, token) => + { + var result = await arguments.Services.GetRequiredService<IPlaywrightPageInspectionService>() + .TakeScreenshotAsync(session, fullPage, token); + + return Serialize(result); + }); + } +} diff --git a/src/Modules/CrestApps.OrchardCore.AI.Playwright/ViewModels/ChatInteractionPlaywrightSettingsViewModel.cs b/src/Modules/CrestApps.OrchardCore.AI.Playwright/ViewModels/ChatInteractionPlaywrightSettingsViewModel.cs new file mode 100644 index 000000000..8b66f661b --- /dev/null +++ b/src/Modules/CrestApps.OrchardCore.AI.Playwright/ViewModels/ChatInteractionPlaywrightSettingsViewModel.cs @@ -0,0 +1,29 @@ +using System.ComponentModel.DataAnnotations; + +namespace CrestApps.OrchardCore.AI.Playwright.ViewModels; + +public class ChatInteractionPlaywrightSettingsViewModel +{ + public bool PlaywrightEnabled { get; set; } + + [MaxLength(256)] + public string PlaywrightUsername { get; set; } + + [DataType(DataType.Password)] + public string PlaywrightPassword { get; set; } + + public bool HasSavedPassword { get; set; } + + [MaxLength(1024)] + public string PlaywrightBaseUrl { get; set; } + + [MaxLength(1024)] + public string PlaywrightAdminBaseUrl { get; set; } + + [MaxLength(2048)] + public string PlaywrightPersistentProfilePath { get; set; } + + public bool PlaywrightHeadless { get; set; } + + public bool PlaywrightPublishByDefault { get; set; } +} diff --git a/src/Modules/CrestApps.OrchardCore.AI.Playwright/ViewModels/PlaywrightProfileSettingsViewModel.cs b/src/Modules/CrestApps.OrchardCore.AI.Playwright/ViewModels/PlaywrightProfileSettingsViewModel.cs new file mode 100644 index 000000000..774a25d39 --- /dev/null +++ b/src/Modules/CrestApps.OrchardCore.AI.Playwright/ViewModels/PlaywrightProfileSettingsViewModel.cs @@ -0,0 +1,29 @@ +using System.ComponentModel.DataAnnotations; + +namespace CrestApps.OrchardCore.AI.Playwright.ViewModels; + +public class PlaywrightProfileSettingsViewModel +{ + public bool Enabled { get; set; } + + [MaxLength(256)] + public string Username { get; set; } + + [DataType(DataType.Password)] + public string Password { get; set; } + + public bool HasSavedPassword { get; set; } + + [MaxLength(1024)] + public string BaseUrl { get; set; } + + [MaxLength(1024)] + public string AdminBaseUrl { get; set; } + + [MaxLength(2048)] + public string PersistentProfilePath { get; set; } + + public bool Headless { get; set; } + + public bool PublishByDefault { get; set; } +} diff --git a/src/Modules/CrestApps.OrchardCore.AI.Playwright/Views/ChatInteractionPlaywrightSettings.Edit.cshtml b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Views/ChatInteractionPlaywrightSettings.Edit.cshtml new file mode 100644 index 000000000..859827c8d --- /dev/null +++ b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Views/ChatInteractionPlaywrightSettings.Edit.cshtml @@ -0,0 +1,91 @@ +@model CrestApps.OrchardCore.AI.Playwright.ViewModels.ChatInteractionPlaywrightSettingsViewModel +@using Microsoft.Extensions.Localization +@inject IStringLocalizer<CrestApps.OrchardCore.AI.Playwright.Drivers.ChatInteractionPlaywrightSettingsDisplayDriver> S + +<div class="mb-3"> + <h5>@S["Playwright Browser Automation"]</h5> + <p class="text-muted mb-3">@S["When enabled, this interaction can open its own browser window and carry out Orchard admin steps in that browser."]</p> + + <div class="mb-3"> + <div class="form-check"> + <input asp-for="PlaywrightEnabled" class="form-check-input" type="checkbox" /> + <label asp-for="PlaywrightEnabled" class="form-check-label">@S["Let this interaction open a browser and do the work"]</label> + </div> + </div> + + <div id="interaction-playwright-config" class="@(Model.PlaywrightEnabled ? "" : "d-none")"> + <div class="row"> + <div class="col-md-6 mb-3"> + <label asp-for="PlaywrightUsername" class="form-label">@S["Login username"]</label> + <input asp-for="PlaywrightUsername" class="form-control" autocomplete="username" /> + <span class="hint">@S["Optional. Used only if the browser reaches the Orchard login page."]</span> + </div> + + <div class="col-md-6 mb-3"> + <label asp-for="PlaywrightPassword" class="form-label">@S["Login password"]</label> + <input asp-for="PlaywrightPassword" class="form-control" type="password" autocomplete="current-password" /> + <span class="hint"> + @(Model.HasSavedPassword + ? S["Leave blank to keep the saved password."] + : S["Optional. Used only if the browser reaches the Orchard login page."]) + </span> + </div> + </div> + + <div class="mb-3"> + <div class="form-check"> + <input asp-for="PlaywrightPublishByDefault" class="form-check-input" type="checkbox" /> + <label asp-for="PlaywrightPublishByDefault" class="form-check-label">@S["Publish by default"]</label> + </div> + </div> + + <details class="mb-3"> + <summary class="fw-semibold">@S["Advanced options"]</summary> + + <div class="mt-3"> + <div class="mb-3"> + <label asp-for="PlaywrightPersistentProfilePath" class="form-label">@S["Browser profile folder"]</label> + <input asp-for="PlaywrightPersistentProfilePath" class="form-control" placeholder="@S["Leave blank to let the module manage it for you"]" /> + </div> + + <div class="mb-3"> + <label asp-for="PlaywrightBaseUrl" class="form-label">@S["Site URL override"]</label> + <input asp-for="PlaywrightBaseUrl" class="form-control" placeholder="@S["Leave blank to use the current tenant URL"]" /> + </div> + + <div class="mb-3"> + <label asp-for="PlaywrightAdminBaseUrl" class="form-label">@S["Admin URL override"]</label> + <input asp-for="PlaywrightAdminBaseUrl" class="form-control" placeholder="@S["Leave blank to derive it automatically"]" /> + </div> + + <div class="mb-3"> + <div class="form-check"> + <input asp-for="PlaywrightHeadless" class="form-check-input" type="checkbox" /> + <label asp-for="PlaywrightHeadless" class="form-check-label">@S["Run without showing the browser window"]</label> + </div> + </div> + </div> + </details> + </div> +</div> + +<script at="Foot"> + (function () { + var enabled = document.getElementById('@Html.IdFor(m => m.PlaywrightEnabled)'); + var block = document.getElementById('interaction-playwright-config'); + + function update() { + if (!enabled || !block) { + return; + } + + block.classList.toggle('d-none', !enabled.checked); + } + + if (enabled) { + enabled.addEventListener('change', update); + } + + update(); + })(); +</script> diff --git a/src/Modules/CrestApps.OrchardCore.AI.Playwright/Views/PlaywrightAdminWidget.cshtml b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Views/PlaywrightAdminWidget.cshtml new file mode 100644 index 000000000..44bf620bd --- /dev/null +++ b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Views/PlaywrightAdminWidget.cshtml @@ -0,0 +1,192 @@ +@{ + var apiBase = Context.Request.PathBase.ToString().TrimEnd('/') + "/api/playwright"; +} +<div id="pw-status-bar" + aria-live="polite" + style="display:none; position:fixed; bottom:0; left:0; right:0; z-index:10000; + background:#1f2937; color:#e5e7eb; font-family:monospace; font-size:13px; + padding:6px 14px; border-top:2px solid #2563eb; + align-items:center; gap:10px; flex-wrap:wrap;"> + + <span style="font-weight:600; color:#93c5fd;">Playwright</span> + + <span id="pw-status-label" style="color:#86efac;">Idle</span> + + <span id="pw-page-label" + style="color:#bfdbfe; max-width:260px; overflow:hidden; text-overflow:ellipsis; white-space:nowrap;"></span> + + <span id="pw-url-label" + style="color:#93c5fd; max-width:420px; overflow:hidden; text-overflow:ellipsis; white-space:nowrap;"></span> + + <span style="flex:1"></span> + + <button id="pw-stop-btn" + type="button" + onclick="pwStopSession()" + style="background:#fca5a5; color:#111827; border:none; border-radius:4px; + padding:3px 10px; cursor:pointer; font-size:12px; font-weight:600; display:none;"> + Stop + </button> + + <button id="pw-close-btn" + type="button" + onclick="pwCloseSession()" + style="background:#fdba74; color:#111827; border:none; border-radius:4px; + padding:3px 10px; cursor:pointer; font-size:12px; font-weight:600;"> + Close Browser + </button> +</div> + +<script> +(function () { + var storageKey = 'playwright-active-session-id'; + var bar = document.getElementById('pw-status-bar'); + var statusLabel = document.getElementById('pw-status-label'); + var pageLabel = document.getElementById('pw-page-label'); + var urlLabel = document.getElementById('pw-url-label'); + var stopBtn = document.getElementById('pw-stop-btn'); + var pollTimer = null; + var activeSid = null; + var API = '@Html.Raw(apiBase)'; + + function storeSessionId(sessionId) { + activeSid = sessionId || null; + + if (activeSid) { + localStorage.setItem(storageKey, activeSid); + } else { + localStorage.removeItem(storageKey); + } + } + + function getActiveSessionId() { + return activeSid + || localStorage.getItem(storageKey) + || sessionStorage.getItem('ai-chat-session-id'); + } + + function startPolling() { + if (pollTimer) { + return; + } + + pollTimer = setInterval(poll, 2000); + poll(); + } + + function updateBar(data) { + bar.style.display = 'flex'; + + var status = (data.status || '').toLowerCase(); + var labels = { + idle: 'Idle', + running: 'Running', + stopped: 'Stopped', + closed: 'Closed' + }; + + statusLabel.textContent = labels[status] || data.status || 'Unknown'; + statusLabel.style.color = status === 'running' + ? '#fde68a' + : (status === 'stopped' ? '#fca5a5' : '#86efac'); + + pageLabel.textContent = data.currentPageTitle || ''; + urlLabel.textContent = data.currentUrl || ''; + stopBtn.style.display = status === 'running' ? 'inline-block' : 'none'; + } + + function hideBar() { + bar.style.display = 'none'; + } + + async function poll() { + var sid = getActiveSessionId(); + + if (!sid) { + try { + var activeResp = await fetch(API + '/active', { credentials: 'same-origin' }); + if (!activeResp.ok) { + return; + } + + var activeSessions = await activeResp.json(); + if (activeSessions && activeSessions.length > 0) { + storeSessionId(activeSessions[0].chatSessionId); + updateBar(activeSessions[0]); + } else { + hideBar(); + } + } catch (_) { + } + + return; + } + + try { + var response = await fetch(API + '/' + encodeURIComponent(sid) + '/status', { credentials: 'same-origin' }); + if (!response.ok) { + return; + } + + var data = await response.json(); + if (data.isActive) { + storeSessionId(sid); + updateBar(data); + } else { + storeSessionId(null); + hideBar(); + } + } catch (_) { + } + } + + window.pwStopSession = async function () { + var sid = getActiveSessionId(); + if (!sid) { + return; + } + + try { + var response = await fetch(API + '/' + encodeURIComponent(sid) + '/stop', { + method: 'POST', + credentials: 'same-origin' + }); + var data = await response.json(); + updateBar(data); + } catch (_) { + } + }; + + window.pwCloseSession = async function () { + var sid = getActiveSessionId(); + if (!sid) { + return; + } + + try { + await fetch(API + '/' + encodeURIComponent(sid) + '/close', { + method: 'POST', + credentials: 'same-origin' + }); + } catch (_) { + } + + storeSessionId(null); + hideBar(); + }; + + document.addEventListener('initializingSessionOpenAIChat', function (event) { + if (event.detail && event.detail.sessionId) { + storeSessionId(event.detail.sessionId); + } + }); + + document.addEventListener('initializingChatInteraction', function (event) { + if (event.detail && event.detail.itemId) { + storeSessionId(event.detail.itemId); + } + }); + + startPolling(); +})(); +</script> diff --git a/src/Modules/CrestApps.OrchardCore.AI.Playwright/Views/PlaywrightProfileSettings.Edit.cshtml b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Views/PlaywrightProfileSettings.Edit.cshtml new file mode 100644 index 000000000..7689cf486 --- /dev/null +++ b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Views/PlaywrightProfileSettings.Edit.cshtml @@ -0,0 +1,94 @@ +@model CrestApps.OrchardCore.AI.Playwright.ViewModels.PlaywrightProfileSettingsViewModel +@using Microsoft.Extensions.Localization +@inject IStringLocalizer<CrestApps.OrchardCore.AI.Playwright.Drivers.PlaywrightProfileSettingsDisplayDriver> S + +<fieldset class="mb-3"> + <legend class="h6">@S["Playwright Browser Automation"]</legend> + + <div class="mb-3"> + <div class="form-check"> + <input asp-for="Enabled" class="form-check-input" type="checkbox" /> + <label asp-for="Enabled" class="form-check-label">@S["Let this AI profile open a browser and do the work"]</label> + </div> + <span class="hint">@S["When enabled, this profile can open its own browser window and carry out Orchard admin steps for you. The browser keeps its own session between runs so it can stay signed in. Playwright tools are added automatically."]</span> + </div> + + <div id="playwright-config-block" class="@(Model.Enabled ? "" : "d-none")"> + <div class="row"> + <div class="col-md-6 mb-3"> + <label asp-for="Username" class="form-label">@S["Login username"]</label> + <input asp-for="Username" class="form-control" autocomplete="username" /> + <span class="hint">@S["Optional. Used only if the browser reaches the Orchard login page."]</span> + </div> + + <div class="col-md-6 mb-3"> + <label asp-for="Password" class="form-label">@S["Login password"]</label> + <input asp-for="Password" class="form-control" type="password" autocomplete="current-password" /> + <span class="hint"> + @(Model.HasSavedPassword + ? S["Leave blank to keep the saved password."] + : S["Optional. Used only if the browser reaches the Orchard login page."]) + </span> + </div> + </div> + + <div class="mb-3"> + <div class="form-check"> + <input asp-for="PublishByDefault" class="form-check-input" type="checkbox" /> + <label asp-for="PublishByDefault" class="form-check-label">@S["Publish by default"]</label> + </div> + <span class="hint">@S["When unchecked, the AI prefers saving drafts unless you explicitly ask it to publish."]</span> + </div> + + <details class="mb-3"> + <summary class="fw-semibold">@S["Advanced options"]</summary> + + <div class="mt-3"> + <div class="mb-3"> + <label asp-for="PersistentProfilePath" class="form-label">@S["Browser profile folder"]</label> + <input asp-for="PersistentProfilePath" class="form-control" placeholder="@S["Leave blank to let the module manage it for you"]" /> + <span class="hint">@S["Optional. Most users should leave this empty."]</span> + </div> + + <div class="mb-3"> + <label asp-for="BaseUrl" class="form-label">@S["Site URL override"]</label> + <input asp-for="BaseUrl" class="form-control" placeholder="@S["Leave blank to use the current tenant URL"]" /> + </div> + + <div class="mb-3"> + <label asp-for="AdminBaseUrl" class="form-label">@S["Admin URL override"]</label> + <input asp-for="AdminBaseUrl" class="form-control" placeholder="@S["Leave blank to derive it automatically"]" /> + </div> + + <div class="mb-3"> + <div class="form-check"> + <input asp-for="Headless" class="form-check-input" type="checkbox" /> + <label asp-for="Headless" class="form-check-label">@S["Run without showing the browser window"]</label> + </div> + <span class="hint">@S["Leave this off while you are testing so you can watch what the AI is doing."]</span> + </div> + </div> + </details> + </div> +</fieldset> + +<script at="Foot"> + (function () { + var enabled = document.getElementById('@Html.IdFor(m => m.Enabled)'); + var block = document.getElementById('playwright-config-block'); + + function update() { + if (!enabled || !block) { + return; + } + + block.classList.toggle('d-none', !enabled.checked); + } + + if (enabled) { + enabled.addEventListener('change', update); + } + + update(); + })(); +</script> diff --git a/src/Modules/CrestApps.OrchardCore.AI.Playwright/Views/_ViewImports.cshtml b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Views/_ViewImports.cshtml new file mode 100644 index 000000000..5c09b6928 --- /dev/null +++ b/src/Modules/CrestApps.OrchardCore.AI.Playwright/Views/_ViewImports.cshtml @@ -0,0 +1,9 @@ +@inherits OrchardCore.DisplayManagement.Razor.RazorPage<TModel> + +@using OrchardCore.DisplayManagement +@using OrchardCore.DisplayManagement.Views + +@addTagHelper *, Microsoft.AspNetCore.Mvc.TagHelpers +@addTagHelper *, OrchardCore.DisplayManagement +@addTagHelper *, OrchardCore.ResourceManagement +@addTagHelper *, OrchardCore.Contents.TagHelpers diff --git a/src/Targets/CrestApps.OrchardCore.Cms.Core.Targets/CrestApps.OrchardCore.Cms.Core.Targets.csproj b/src/Targets/CrestApps.OrchardCore.Cms.Core.Targets/CrestApps.OrchardCore.Cms.Core.Targets.csproj index 881972147..3b685e2e3 100644 --- a/src/Targets/CrestApps.OrchardCore.Cms.Core.Targets/CrestApps.OrchardCore.Cms.Core.Targets.csproj +++ b/src/Targets/CrestApps.OrchardCore.Cms.Core.Targets/CrestApps.OrchardCore.Cms.Core.Targets.csproj @@ -37,6 +37,7 @@ <ProjectReference Include="..\..\Modules\CrestApps.OrchardCore.AI.Chat\CrestApps.OrchardCore.AI.Chat.csproj" PrivateAssets="none" /> <ProjectReference Include="..\..\Modules\CrestApps.OrchardCore.AI.Chat.Copilot\CrestApps.OrchardCore.AI.Chat.Copilot.csproj" PrivateAssets="none" /> <ProjectReference Include="..\..\Modules\CrestApps.OrchardCore.AI.Chat.Interactions\CrestApps.OrchardCore.AI.Chat.Interactions.csproj" PrivateAssets="none" /> + <ProjectReference Include="..\..\Modules\CrestApps.OrchardCore.AI.Playwright\CrestApps.OrchardCore.AI.Playwright.csproj" PrivateAssets="none" /> <ProjectReference Include="..\..\Modules\CrestApps.OrchardCore.AI.DataSources\CrestApps.OrchardCore.AI.DataSources.csproj" PrivateAssets="none" /> <ProjectReference Include="..\..\Modules\CrestApps.OrchardCore.AI.DataSources.AzureAI\CrestApps.OrchardCore.AI.DataSources.AzureAI.csproj" PrivateAssets="none" /> <ProjectReference Include="..\..\Modules\CrestApps.OrchardCore.AI.DataSources.Elasticsearch\CrestApps.OrchardCore.AI.DataSources.Elasticsearch.csproj" PrivateAssets="none" />