Skip to content

Conversation

Guido1Alessandro1Trevisan

1) This script resolves the previous issue where the workflow could not detect or pierce the Shadow DOM.

Shadow DOM Detection
I updated content.ts to handle all edge cases. On page load, the script monkey-patches the browser’s attachShadow method so that any element attempting to create a closed shadow root instead creates an open one. It also registers event listeners inside these roots. Additionally, it wraps customElements.define to trace future custom element constructors, scans the existing DOM for open roots, and observes the document for newly added shadow DOMs.
This enables the recorder to capture clicks, inputs, keypresses, scrolls, and navigation events as if they occurred in the regular DOM.

Shadow DOM Piercing
Piercing the Shadow DOM works similarly. We can’t rely on content.ts during workflow execution because the browser runs without the extension (as intended). Instead, I inject a nearly identical script each time a workflow is launched. This script monkey-patches all shadow roots from closed to open, enabling access by Playwright.

2) Pytests:

I've added one test for the recorder component

  • This test to verify that the recorder detects Shadow DOM elements properly.

I've added two tests for the workflow component

  • One for open Shadow DOM components that are immediately detectable.
  • One for closed Shadow DOM components that are to be monkey-patched on page load to become open.

Playwright cannot pierce closed Shadow DOMs directly (reference), so this approach is likely the only viable solution.

These tests are a work in progress, but I’m opening this PR to request early feedback so I can finalize the implementation. I believe this update is critical to improving the robustness of workflow-use.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant