Upgrade AsyncChromiumLoader to provide more detailed information

[mayurdb]

Checked

• I searched existing ideas and did not find a similar one
• I added a very descriptive title
• I've clearly described the feature request and motivation for it

Feature request

Currently, the AsyncChromiumLoader returns just the static content of the page. We should add a mechanism with which one can get a more comprehensive information lilke:

1. All the web-links on the page
2. Dynamically loaded content (upon scroll for example)

Motivation

We are trying to add LLMs based web-scrapped which is internally using the langchain-community AsynchronousApi
https://github.com/VinciGit00/Scrapegraph-ai

Proposal (If applicable)

No response

[mayurdb]

@Noah0115 @baskaryan ^ can you please comment on this?

[Emasoft]

This is very important. You cannot do webscraping without the links. I would also add that it should return the metadata of each link, otherwise the links would be meaningless.

• extract both relative and absolute path from the href
• text of the child textnode (url label)
• title attribute
• alt attribute
• text content of the textnode sibling immediately before the a element node
• the img/svg src in the child node (if present)
• alt attribute of child img/svg (if present)
• src attribute of child img/svg (if present)
• srcset attribute of child img/svg (if present)

All those info are easily extracted with BeautifulSoap.