Is there any need of splitting Recursive Loader

[deepak-hl]

all_urls = [url['url'] for url in urls if url.get('key') == 'True']
        all_urls.extend([url['url'] for url in urls if url.get('key') == 'False'])
        loader = RecursiveUrlLoader(all_urls[0], max_depth=6, prevent_outside=True)
        texts = load_and_split_documents(loader, chunk_size=3000, chunk_overlap=100)
        doc_texts_counters = texts

Is there any need of splitting into chunks in case of recursive url loader, as its comes out to be html content?

[dosubot[bot]]

Yes, it is necessary to split the content into chunks when using a RecursiveUrlLoader, especially since the content is HTML. Text splitters break large documents into smaller chunks, which is useful for indexing and passing data into a model, as large chunks are harder to search and may not fit in a model's context window ^[1].

To continue talking to Dosu, mention @dosu.

[deepak-hl]

@dosu
how can i fetch the content from recursive url loader instead of html content

[dosubot[bot]]

To fetch non-HTML content using a RecursiveUrlLoader, you can customize the extractor function to handle different content types. Here's an example:

import requests
from langchain_community.document_loaders import RecursiveUrlLoader

def custom_extractor(content: str) -> str:
    # Implement logic to handle non-HTML content
    # For example, if the content is JSON, you might parse it here
    return content

loader = RecursiveUrlLoader(
    "https://example.com/api/data",  # URL pointing to non-HTML content
    extractor=custom_extractor,
)

docs = loader.load()
print(docs[0].page_content)

This setup allows you to define how the content should be processed, enabling you to fetch and handle non-HTML content types ^[1].

To continue talking to Dosu, mention @dosu.