[shadow-root content] Javascript generated content missing in HTML but displaying in screenshot

[NiklasRosenstein]

Hi! I'm trying out changedetection for the first time. I want to get notified of changes to this page: https://jfrog.com/help/r/jfrog-release-information/artifactory-release-notes

They use Javascript to generate the page content, so I'm using a selenium/standalone-chrome:4 container as shown in the docker-compose.yml. There's also a "accept cookies" dialog that seemed to be the only content extracted at first (despite the actual content I want displaying in the background).

Text extracted

    Loading application...
Your web browser must have JavaScript enabled in order for this application to display correctly.
                  This site uses cookies to improve your browsing experience, analyze site usage, and to assist in marketing efforts. By selecting "Accept All Cookies", you agree to the storing of cookies on your device. You can learn more about how we use cookies and related technologies here: Privacy Notice and Cookie Policy
                  Accept All Cookies
                  Reject All
                  Cookies Settings

          Privacy Preference Center

                * Your Privacy

                * Strictly Necessary Cookies

                * Performance Cookies

                * Functional Cookies

                * Targeting Cookies

              Your Privacy

              When you visit any website, it may store or retrieve information on your browser, mostly in the form of cookies. This information might be about you, your preferences or your device and is mostly used to make the site work as you expect it to. The information does not usually directly identify you, but it can give you a more personalized web experience. Because we respect your right to privacy, you can choose not to allow some types of cookies. Click on the different category headings to find out more and change our default settings. However, blocking some types of cookies may impact your experience of the site and the services we are able to offer.
              More information

                Strictly Necessary Cookies

                    Always Active

              These are technologies that the digital services need in order to function, and that enable you to move around and use the digital services. Without these essential technologies, the digital services will not perform as smoothly for you as we would like them to, and we may not be able to provide the digital services or certain features you request. Examples of where these cookies are used include: to determine when you are signed in; to determine when your account has been inactive; and for other troubleshooting and security purposes.

                Performance Cookies

                  Performance Cookies

              These cookies allow us to count visits and traffic sources so we can measure and improve the performance of our site. They help us to know which pages are the most and least popular and see how visitors move around the site. All information these cookies collect is aggregated and therefore anonymous. If you do not allow these cookies we will not know when you have visited our site, and will not be able to monitor its performance.

                Functional Cookies

                  Functional Cookies

              These cookies enable the website to provide enhanced functionality and personalisation. They may be set by us or by third party providers whose services we have added to our pages. If you do not allow these cookies then some or all of these services may not function properly.

                Targeting Cookies

                  Targeting Cookies

              These cookies may be set through our site by our advertising partners. They may be used by those companies to build a profile of your interests and show you relevant adverts on other sites. They do not store directly personal information, but are based on uniquely identifying your browser and internet device. If you do not allow these cookies, you will experience less targeted advertising.

            Cookie List

                  Consent Leg.Interest
                    checkbox label label
                    checkbox label label
                    checkbox label label
        Clear
                checkbox label label
            Apply Cancel
          Confirm My Choices
            Reject All Allow All
  Skip to main content
        Search the Help Center
        Guides
          * All Guides
          * JFrog Release Notes
          * Xray Release Notes
          * JFrog REST APIs
          * Xray REST APIs
          * IDE, CLI, Frogbot, CI and SDK
          * Getting Started with JFrog
          * Artifactory User Guide
          * Advanced Security User Guide
          * Connect User Guide
          * Curation User Guide
          * Distribution User Guide
          * Qwak User Guide
          * My JFrog User Guide
          * Pipelines User Guide
          * JFrog Partner Integrations Guide
          * Installation & Setup Guide
          * Platform Administration Guide
        Sign In
        Menu
          Menu

My thinking was that maybe something causes the cookie pop-up to appear like the "primary content", so I've added the following Javascript to run before taking the screenshot:

setTimeout(() => {
  document.getElementById("onetrust-accept-btn-handler").click();
}, 10 * 1000);

But unfortunately that just removes the cookie pop-up text from the extract and still does not show the content in <div id="center">.

Text extracted

    Loading application...
Your web browser must have JavaScript enabled in order for this application to display correctly.
  Skip to main content
        Search the Help Center
        Guides
          * All Guides
          * JFrog Release Notes
          * Xray Release Notes
          * JFrog REST APIs
          * Xray REST APIs
          * IDE, CLI, Frogbot, CI and SDK
          * Getting Started with JFrog
          * Artifactory User Guide
          * Advanced Security User Guide
          * Connect User Guide
          * Curation User Guide
          * Distribution User Guide
          * Qwak User Guide
          * My JFrog User Guide
          * Pipelines User Guide
          * JFrog Partner Integrations Guide
          * Installation & Setup Guide
          * Platform Administration Guide
        Sign In
        Menu
          Menu

As you can see in the screenshots that are attached to the snapshot, the content I want is there. However when I download the latest HTML snapshot it's not there.

1727089350.html.zip

I've also tried to use the following XPath to no avail (it errors that it can't find the element).

xpath://div[@id="center"]

What am I missing?

[dgtlmoon]

It is really not necessary to click anything if you are scraping the page! the content is already there, you do not need to remove that cookie popup!

second, the content is injected into the html element "shadow root" so it not selectable

try this xpath //*[@id="FT-application-content"]/div[2]/div/div[2]/div[1]/ft-internal-reader-context but im not sure it will scrape..

we dont (yet) really support content stuffed into the shadow-root

[dgtlmoon]

The difference between Shadow DOM and standard DOM arises from how web components are designed to encapsulate and isolate content. The Shadow DOM was introduced as part of the Web Components specification, aimed at providing better componentization, encapsulation, and modularization of web development. Here’s a breakdown of why the Shadow DOM behaves differently and what makes it unique:

1. Encapsulation

• Standard DOM: All elements are part of a single DOM tree. Any JavaScript, CSS, or HTML can freely interact and affect any other element in the page unless restricted.
• Shadow DOM: Shadow DOM creates an isolated or "shadow" subtree inside a shadow host element. Elements in the shadow DOM are invisible and inaccessible from the main DOM (also called light DOM) unless explicitly accessed. CSS styles from the light DOM do not affect elements inside the shadow DOM, and vice versa.

Why: Encapsulation allows developers to build components where their internal structure and styles are hidden from the global DOM and won't interfere with other elements on the page. It’s particularly useful for reusable web components.

• Example:

  <div>I'm in the regular DOM</div>
  <my-component></my-component>

  If <my-component> has a shadow DOM, its internal structure (e.g., <p>I'm in the shadow DOM</p>) will be isolated from the rest of the page and won't interact with the div outside of it.

2. Shadow Root

• Standard DOM: All elements are in one flat tree, and the entire page can be queried using the usual selectors (document.querySelector, etc.).
• Shadow DOM: The shadow DOM is hidden behind a shadow root. Elements inside a shadow root are not accessible using regular DOM APIs unless you explicitly query the shadow root. Each shadow root is essentially its own DOM tree, isolated from the main tree.

Why: This isolation means that even though a shadow host exists in the main DOM, its children (inside the shadow root) aren't exposed unless you manually penetrate the shadow DOM.

3. Style Isolation

• Standard DOM: Styles are global. CSS from one part of the page can affect elements elsewhere unless you use specific CSS strategies like scoped styles.
• Shadow DOM: Styles inside a shadow DOM are isolated. CSS rules defined within a shadow DOM apply only to elements inside that shadow root. External styles from the light DOM won't penetrate the shadow DOM, and shadow DOM styles won't "leak out."

Why: This guarantees that a component can be styled internally without worrying about conflicts from external styles or accidentally affecting the rest of the page.

4. Slotting and Light DOM

• Standard DOM: The entire DOM is controlled directly by the page developer.
• Shadow DOM: Shadow DOM components can still interact with the light DOM via slots. Slots allow light DOM content (content in the main page) to be injected into specific places within the shadow DOM, but the encapsulation remains.

Why: Slots allow developers to provide a way for users of the component to pass content into the shadow DOM component while maintaining isolation for the internal structure.

Example:

<my-component>
  <p>Light DOM content</p>
</my-component>

If my-component has a shadow DOM with a slot like this:

<slot></slot>

The <p> from the light DOM will appear inside the shadow DOM at the slot location, but the internal structure and styles remain isolated.

5. JavaScript Access

• Standard DOM: All elements are accessible via regular JavaScript DOM APIs. You can query any element using document.querySelector() or similar methods.
• Shadow DOM: Shadow DOM requires special handling. You need to explicitly access a shadow root to interact with its contents via JavaScript. You can't use regular DOM methods to query inside a shadow DOM from the light DOM.

Why: This allows for encapsulation at the JavaScript level as well. The idea is that the component's internal structure and behavior should not be accidentally modified or interfered with by other scripts.

Example (trying to access shadow content):

const shadowHost = document.querySelector('my-component');
const shadowRoot = shadowHost.shadowRoot;
const shadowElement = shadowRoot.querySelector('p');

You must explicitly access shadowRoot to get to the content inside my-component.

6. Scoped Events

• Standard DOM: Events bubble up the DOM tree from children to parent, potentially triggering listeners all the way up to the root.
• Shadow DOM: Events originating inside a shadow DOM are encapsulated, meaning they don’t bubble up to the light DOM unless they are specifically designed to do so.

Why: This allows components to handle their internal events without interfering with the rest of the application, making them more predictable and modular.

Why Shadow DOM Was Introduced

The Shadow DOM was introduced to address several problems in traditional web development, especially with large-scale applications:

1. CSS Scoping Issues: As web apps grow, managing global CSS becomes harder, and styles can inadvertently affect elements they weren’t intended for.
2. Component Isolation: Modular, reusable web components should have their internal implementation hidden from the rest of the app, allowing easier reuse and maintenance.
3. Maintainability: Encapsulation improves the maintainability of web apps by keeping component logic and styles isolated, which minimizes unintended side effects.

Can You "Flatten" a Shadow DOM to the Standard DOM?

No, not directly. Since the whole point of the shadow DOM is encapsulation, its elements can't be automatically exposed as if they were in the standard DOM without breaking encapsulation. However, you can manually extract information from the shadow DOM (like we did in the recursive scraping example), but this is still a read-only operation.

If you really wanted to "flatten" it, you could theoretically pull all content from inside shadow DOMs, clone it, and insert it into the light DOM, but that would lose the benefits of encapsulation and could cause style conflicts or other issues. Here's a basic conceptual example of such an approach:

const shadowHost = document.querySelector('shadow-host-selector');
const shadowRoot = shadowHost.shadowRoot;

// Clone shadow DOM content and append it to the main DOM
const clonedContent = shadowRoot.innerHTML;
document.body.innerHTML += clonedContent;

This would insert the contents of the shadow DOM into the main DOM, but it would break the encapsulation, and things like styles and scoped behavior would no longer work correctly.

Conclusion

The Shadow DOM exists to allow developers to create encapsulated, reusable web components that can work in isolation. While it may seem "different" or complicated at first, it solves many real-world issues in web development, such as style leakage, maintainability, and component interoperability. If you need to access shadow DOM elements, you can, but understanding why it behaves differently helps to appreciate the design decisions behind web components.