Navigating Nested Iframes using observe

[ambarc]

(using experimental: true to enable iframes). We're testing on an ERP with a bunch of nested iframes. Stagehand version 2.4.1.

Issue: Stagehand's observe({ iframes: true }) returns 0 nested elements on complex iframe structures despite 14+ frames containing interactive content. We've seen this solved elsewhere by preserving natural DOM hierarchy. Stagehand's intent might be to get builders to rerun recursions to get more depth, but that wouldn't feel intuitive.

Starting this issue to understand whether what I'm seeing is expected behavior.

The structure of the page we're looking at, trying to target the search element.

https://app.example.com/main (Frame 0) [MAIN]
├── 📄 Main page content
├── 🖼️ iframe#navigation → https://app.example.com/navigation (Frame 1) [SAME-ORIGIN]
│   ├── 🔍 input#searchBar [INTERACTIVE] ← Target element
│   ├── 🔘 button#submitBtn [INTERACTIVE]  
│   └── 📝 Various nav links [INTERACTIVE]
├── 🖼️ iframe#workspace → https://app.example.com/workspace (Frame 2) [SAME-ORIGIN]
│   ├── 📊 Data grid with buttons [INTERACTIVE]
│   ├── ➕ Add buttons [INTERACTIVE]
│   └── 🖼️ nested iframe → https://app.example.com/reports (Frame 3) [SAME-ORIGIN]
│       └── 📈 Report controls [INTERACTIVE]
└── 🖼️ iframe#status → https://app.example.com/status (Frame 4) [SAME-ORIGIN]
    ├── 📋 select#departmentSelect [INTERACTIVE] 
    └── ℹ️ Status indicators [INTERACTIVE]

🚫 PROBLEM: observe({ iframes: true }) → 0 elements
✅ REALITY: 15+ interactive elements exist across frames
🔧 CAUSE: Element resolution breaks during tree merging

experimental: true required for iframe traversal
observe({ iframes: false }): 4 elements, not supported (generic iframe placeholders)
observe({ iframes: true }): 4 elements (despite 14 nested frames with actions and content)

This might be where the issue is? I haven't gone deep but I can if the maintainers think I'm on the right path.

// In observeHandler.ts - the iframe metadata gets lost
const { combinedTree, combinedXpathMap, discoveredIframes } = await (iframes
 ? getAccessibilityTreeWithFrames(...).then(({ combinedTree, combinedXpathMap }) => ({
     combinedTree,           // ✅ Combined tree exists
     combinedXpathMap,       // ✅ XPath mappings exist 
     discoveredIframes: []   // ❌ Always empty - breaks element resolution
   }))
 : // non-iframe path works correctly

Sample code - I can't share the ERP we were testing against.

const TEST_URLS = {
 MAIN_FRAME: 'https://app.example.com/main',
 NAV_FRAME: 'https://app.example.com/navigation',
 CONTENT_FRAME: 'https://app.example.com/workspace',
 STATUS_FRAME: 'https://app.example.com/status'
};


const TEST_ELEMENTS = {
 SEARCH_INPUT: 'searchInput',
 SUBMIT_BUTTON: 'submitBtn',
 DROPDOWN: 'departmentSelect'
};


// Test iframe observation
console.log('Testing iframe observation...');


const basicObs = await page.observe({ iframes: false });
console.log(`Basic: ${basicObs.length} elements`); // 4 unsupported


const iframeObs = await page.observe({ iframes: true }); 
console.log(`Iframe: ${iframeObs.length} elements`); // 4 surface-level iframes instead of more nested actions.


const frames = await page.frames();
console.log(`Total frames: ${frames.length}`);

I would guess that the fix (if we're aligned) involves observeHandler.ts building deeper trees, preserving frame boundaries, and potentially allowing code to target intra-iframe elements. We're working off a hand-rolled solution that builds DOMs this way and it'd be great to be able to use stagehand's fuller feature set. Thanks for the good work on this project!

[seanmcguire12]

hey there! observe should return elements regardless of the nesting level (it should be recursing through them all).
here is an example snippet:

async function example(stagehand: Stagehand) {
  const page = stagehand.page;
  await page.goto(
    "https://browserbase.github.io/stagehand-eval-sites/sites/nested-iframes/",
  );

  const observation = await page.observe({
    iframes: true,
  });
  console.log(observation);
}

(async () => {
  const stagehand = new Stagehand({
    env: "LOCAL",
    experimental: true,
  });
  await stagehand.init();
  await example(stagehand);
})();

which yields the following xpaths (which span across iframes):

[
  {
    description: 'Textbox for entering Username',
    method: 'fill',
    arguments: [ '' ],
    selector: 'xpath=/html[1]/body[1]/iframe[1]/html[1]/body[1]/iframe[1]/html[1]/body[1]/iframe[1]/html[1]/body[1]/form[1]/label[1]/input[1]'
  },
  {
    description: 'Textbox for entering Password',
    method: 'fill',
    arguments: [ '' ],
    selector: 'xpath=/html[1]/body[1]/iframe[1]/html[1]/body[1]/iframe[1]/html[1]/body[1]/iframe[1]/html[1]/body[1]/form[1]/label[2]/input[1]'
  },
  {
    description: 'Button to Log In',
    method: 'click',
    arguments: [],
    selector: 'xpath=/html[1]/body[1]/iframe[1]/html[1]/body[1]/iframe[1]/html[1]/body[1]/iframe[1]/html[1]/body[1]/form[1]/button[1]'
  }
]

if I'm understanding correctly it sounds like there is a bug here. if you've seen this behaviour on publicly available sites, please send along the URL! it would be very helpful for debugging.

[ambarc]

Hey @seanmcguire12 - it's good to know that observe should (rechecking)

1. Recurse through all available iframe elements, included nested iframes.
2. Should manage framesets similarly.
3. Should manage shadow nodes similarly (I'm adding this to my checklist).

Is there an iframe-heavy page you like to test on? I'm happy to try out what I'm observing against it. Else I'll see if I can reproduce what I'm talking about. Thanks for the response.

[ambarc]

Hi @seanmcguire12, I took another pass on stagehand 2.4.1.

Stagehand observe() iframe Failure on observe.

I constructed a few examples that distill the erroneous behavior. stagehand.page.observe({ iframes: true }) fails to detect elements inside iframe contexts. I think it's due to missing script injection on certain kinds of iframes. The latter is a simplified version of what we're seeing against an ERP we're working with.

A basic iframe setup like this seems to mostly work.

An iframe setup like this with separate iframe contexts (as an example), fails to recognize internal elements.

Failure Pattern:
Main frame: ✅ Works (script injected)
Other iframe contexts: ❌ Fail (window.getScrollableElementXpaths undefined)
Impact: observe() returns only iframe containers, zero interactive elements from within iframes. Renders Stagehand unusable on iframe-heavy applications.

With some logging I can get
Error: TypeError: window.getScrollableElementXpaths is not a function during accessibility tree building in stagehand/lib/a11y/utils.ts:1164 when processing iframe scrollable elements.

export async function findScrollableElementIds(
  stagehandPage: StagehandPage,
  targetFrame?: Frame,
): Promise<Set<number>> {
  // JS runs inside the right browsing context
  const xpaths: string[] = targetFrame
    ? await targetFrame.evaluate(() => window.getScrollableElementXpaths())
    : await stagehandPage.page.evaluate(() =>
        window.getScrollableElementXpaths(),
      );

The result is that stagehand can't "see" through the second example. Let me know if I'm doing something wrong.

I think it's managing to iterate through DOM construction OK, and then throwing errors on appending (?) an a11y tree.

I'd add that we're seeing <frameset> and a bunch of shadow DOM nodes for isolation. I haven't focused on those during testing here but wouldn't be surprised if they also show issues.

I'm excited to see stagehand work against more pages. I like the interface you've constructed as compared to some of the other tools out there. Hope this helps.

[seanmcguire12]

Hey @ambarc ! Thanks for the sample website and the detailed write up here. I just tested on the sample website, and stagehand is able to act, extract, and observe on the elements inside each of the iframes.

I did find a bug with scrolling though: stagehand is unable to scroll on elements inside of iframes.

The scrolling issue is a two part issue:

1. All of the scrolling related functions in performPlaywrightMethod operate on the page level. There is a clear fix for this which I am working on now.

2. When an iframe is same-process (all of the ones in the sample website), the scrollable elements are not denoted as such when we build the accessibility tree (the context that gets sent to the LLM).

I'm still investigating the cause of the second issue, but stay tuned for a fix!

[ambarc]

I'll update and try again. The ERP is still being problematic and maybe I'm not replicating the representative issues well (or hey this could be done!).

@seanmcguire12 one thought - is it possible that some of the accessibility tree functions are causing crashes in iframes?

I'm seeing

Error: TypeError: window.getScrollableElementXpaths is not a function during accessibility tree building in stagehand/lib/a11y/utils.ts:1164 when processing iframe scrollable elements.

while testing against the ERP we're working with, and i'm correlating that to not being able to see the elements I'm expecting to.

Edit 2 -

Nuked everything, installed fresh, and tried against:

1. My test breaking link, things do work better than I reported, so I'm investigating why.
2. The ERP, which failed to find interactive elements, and threw this error back (same as above):

   category: "observation"
    error: "Error: frame.evaluate: TypeError: window.getScrollableElementXpaths is not a function\n    at eval (eval at evaluate (:313:29), <anonymous>:1:14)\n    at UtilityScript.evaluate (<anonymous>:315:16)\n    at UtilityScript.<anonymous> (<anonymous>:1:44)"

My guess is that ERP frames can block script injection, which we're using and cleaning up in Stagehand. Creating the tree sans script injection is likely going to lead to a more stable outcome across legacy sites.

[ambarc]

Taking one more step, here's a rough diff that guards against the function not existing, and running against the ERP with this guard does let observe see a bunch of the frames that it previously couldn't. I'm sure there are other considerations (e.g. should we enforce building that a11y tree, is there a way to make script injection always succeed...) - not going into those. I think this path either needs a better injection guard or an allowance for injection failure.

--- a/lib/a11y/utils.ts
+++ b/lib/a11y/utils.ts
@@ -1159,9 +1159,25 @@ export async function findScrollableElementIds(
 ): Promise<Set<number>> {
-  // JS runs inside the right browsing context
-  const xpaths: string[] = targetFrame
-    ? await targetFrame.evaluate(() => window.getScrollableElementXpaths())
-    : await stagehandPage.page.evaluate(() =>
-        window.getScrollableElementXpaths(),
-      );
+  // Helper function to ensure getScrollableElementXpaths is available
+  const ensureScrollableFunction = async (frame: Frame | Page) => {
+    const hasFunction = await frame.evaluate(() => typeof window.getScrollableElementXpaths === 'function');
+    if (!hasFunction) {
+      console.warn('getScrollableElementXpaths not found, attempting re-injection...');
+      await frame.evaluate(() => {
+        // Re-inject fallback implementation
+        window.getScrollableElementXpaths = async function(topN) {
+          // Simplified scrollable element detection + xpath generation
+        };
+      });
+    }
+  };
+
+  let xpaths: string[] = [];
+  try {
+    if (targetFrame) {
+      await ensureScrollableFunction(targetFrame);
+      xpaths = await targetFrame.evaluate(() => window.getScrollableElementXpaths());
+    } else {
+      await ensureScrollableFunction(stagehandPage.page);
+      xpaths = await stagehandPage.page.evaluate(() => window.getScrollableElementXpaths());
+    }
+  } catch (error) {
+    console.warn('Error calling getScrollableElementXpaths:', error.message);
+    return new Set<number>();
+  }

[seanmcguire12]

Hey @ambarc, we recently merged some changes for supporting shadow doms (#954) & nested iframes. Would you be able to install from alpha and let me know if you have any success? Just make sure you are setting experimental: true in your stagehand constructor. To use the alpha version, you can put "@browserbasehq/stagehand": "alpha" in your package.json

[ambarc]

Hi @seanmcguire12 i'll give it a shot. I did figure out how to isolate a specific breaking case related to framesets (see the first section in this page now).