RecursiveUrlLoader document metadata contains the content type of the response.

[coolbeevip]

Checked

• I searched existing ideas and did not find a similar one
• I added a very descriptive title
• I've clearly described the feature request and motivation for it

Feature request

Many web resources exist, such as images, css, js, fonts, etc. All of these are included in the response header file. If the document's metadata contains response header data, it will be more beneficial for developers to choose which documents to use.

Motivation

I want to ignore documents with content-type = text/css...

Proposal (If applicable)

Include response headers or content-type in document metadata.

metadata = {
"source": URL,
"content-type": "text/html; charset=utf-8"
}

[coolbeevip]

Only small changes are needed

point 1

langchain/libs/community/langchain_community/document_loaders/recursive_url_loader.py

Line 29 in c909ae0

def _metadata_extractor(raw_html: str, url: str) -> dict:

def _metadata_extractor(raw_html: str, url: str, content_type: str) -> dict:
    """Extract metadata from raw html using BeautifulSoup."""
    metadata = {"source": url, "content_type": content_type} 

point 2

langchain/libs/community/langchain_community/document_loaders/recursive_url_loader.py

Line 187 in c909ae0

metadata=self.metadata_extractor(response.text, url),

metadata=self.metadata_extractor(response.text, url, response.headers["Content-Type"]),

point 3

langchain/libs/community/langchain_community/document_loaders/recursive_url_loader.py

Line 273 in c909ae0

metadata=self.metadata_extractor(text, url),

metadata=self.metadata_extractor(text, url, response.headers["Content-Type"]),

[coolbeevip]

related #20875