Not able to crawl pdf from RecursiveUrlLoader

[deepak-hl]

I am not able to crawl website which contains pdf, using recursive url loader

class MultiUrlRecursiveLoader(RecursiveUrlLoader):
    def __init__(
        self,
        urls: List[str],
        max_depth: Optional[int] = 2,
        use_async: Optional[bool] = None,
        extractor: Optional[Callable[[str], str]] = None,
        metadata_extractor: Optional[Callable[[str, str, requests.Response], dict]] = None,
        exclude_dirs: Optional[Sequence[str]] = (),
        timeout: Optional[int] = 10,
        prevent_outside: bool = True,
        link_regex: Union[str, re.Pattern, None] = None,
        headers: Optional[dict] = None,
        check_response_status: bool = False,
        continue_on_failure: bool = True,
        # base_url: Optional[str] = None,
        # autoset_encoding: bool = True,
        # encoding: Optional[str] = None,
    ) -> None:
        self.urls = urls
        super().__init__(
            url=urls[0],  # Pass the first URL to the parent class
            max_depth=max_depth,
            use_async=use_async,
            extractor=extractor,
            metadata_extractor=metadata_extractor,
            exclude_dirs=exclude_dirs,
            timeout=timeout,
            prevent_outside=prevent_outside,
            link_regex=link_regex,
            headers=headers,
            check_response_status=check_response_status,
            continue_on_failure=continue_on_failure,
            # base_url=base_url,
            # autoset_encoding=autoset_encoding,
            # encoding=encoding,
        )

    def lazy_load(self) -> Iterator[Document]:
        visited: Set[str] = set()
        results = []
        for url in self.urls:
            if self.use_async:
                results.extend(asyncio.run(self._async_get_child_links_recursive(url, visited)))
            else:
                results.extend(self._get_child_links_recursive(url, visited))
        return iter(results or [])
        
                loader = MultiUrlRecursiveLoader(
            urls=all_urls, max_depth=250
        )
        texts = load_and_split_documents(loader, chunk_size=2000, chunk_overlap=50)
        print(texts)
        doc_texts_counters = texts

below is the type which i am not able to handle :-

Document(metadata={'source': 'https://lResume.pdf', 'content_type': 'application/pdf', 'data_source_id': '344'}, page_content='�\x1b�qZe�z�^��b3\x19�>��\x0c��lQ\x0cf��(G�\��]O:�?N\x07\\x10\x04\x0e\x0e��\x03�:�A�I6��d�\x01X��I���ʰ�\z�3��rY]\x0e���\x0c��'��z��h�\x16���D�\rZ��T�c��D�L��P!|=��?�0֓��\x04�\x1e�P\x1dpppppp�=��=:Q֓��C�\x08\x14\x03N��\x16\x05,\x16�\x05m�\x19�U\x19�t=I���&�M��*v��l�\x18\x14��c�\x1c�z�\U0006df7a\x1e�:�\x05!���89\x10\x1d�\x03��$�\r\x06�k�Z��\x07Ӳ5�����:m\x1e���q��^�ם��zý?�U��\x19�6�֪�ї-\x19L:��kE\x15=��\x1a]O�8�C$LW����c\x1c�W8�U�G\x0frt\x04pppppp�=8\x18�(_B���>\x02�т�#�\x15l\x19F��f�YpZ�\x08\x042\x026k�ݢ8�:����-\x196��n�e؎����[{\x0f�/�\x01\x17D�����@��\x10���o�5&\x13��&���\x03�\x1b؟�����~���3)~w��w��N�ϟ�ߟ�-�j�\x1a�\x0e��n\x0892�v�b�g��6�1}\x02���N�BX��߽s�;\x0e�+\x1c�*��\x079\x7f"pppppp�=�\x19\x1b��\x00�3�>�\x0cs�5�f�\x00��dϠ\x1bN��Y\x11gđ�p٬n��\x0f��es8�\x0c�=�;��d��+�\x03�\x0b~�\x05����qr�hJ\x16�~?\x1aӘ��1�\rFg\x0f�������?\x10��Bfk(3/\x14�\x02\x1e_0\x18�}����p��\x0e���0�\x1d\x0e�C�\x1a���ӑqL�5���\x0f��Q�\t88���\n��J��AU4\x05888888�\x1e��\x1e�(B�?��#p(�\x0cG��\x01N��;\x1dN;N�2s���.��k�y�\x06����z�.���qf��G�\��]\r5��ׁ.(\x02\x0e\x0e��\x03�3s���\x7f;��X �h1��=\x00��u���P(+��ʲ�"����̬Pf +�(k\x7f*\x0e�F���2\x18܆���v[l����8�O��$�!\x1f"a�j\x06\x0e�\x1f�0*$\x7f��у�t&pppppp�=���?Q���/\x0f�#pY�tz�\x02���v�]n\'�\r��|o����9�>�Q�t\x05}�N����t�zܞ#>��]\r5��ׁ.(\x05\x0e\x0e��\x03����dJ�v\\k���d5��=\x00o��+K���rB��\x1c�=\'X������rr�r�����f�t�9�\x1ei����\x1c��uL�5�\x1d��\x0f��l[����8�G���q����\x07\x1c\x1c\x1c\x1c\x1c\x1c}\x0f�o�N�/!��C�\x08<6����z��y�^�\u05cdӪPQ��0��\x0b��������\x03~�/3����|�#\x7f.����\x1e�_�\x03.�\x04\x0e\x0e��\x03UK\x0b�l\x06�\x12"��\x06!�ͬ�z\x00>�B\x03*�s�#��<�3/\��\x1f��\tF�����4���͚����c�x�ͩdf�gfz��z\x12M;xȇHب�\n\x1c\x1c?�a<\x12�%x���Z\n\x1c\x1c\x1c\x1c\x1c\x1c}\x0f�o�\x1481p��\x07�\x1e�v�;���\x04_ ���e��8���+\t������\x15\x0e�m��h8�\r��!_Q�\x1f8�\x1d���\x03�\x0b�\x05U���qr���rl��ߎ\x1b\x1c\x0eȶ8,�p\x0f l�р�Ņ�%yY�%\x19��꒒hIa4��dp��T�a\x7f��\x08�m���<\x1c\n�\x1d\x1e[8T\x1c\n��i�F���\x1er�ϲ�����q\x18_�Ā�hAS\x7f\x15pppppp�=�2���\x01�g4}\x04AW�\x17�C\x10�8Áp0\x1c��-w���H0+ǟI\x7f�\x11\r�r�\x03Yaw0;\�\x15�:�\x1dߟdd��ׁ.�\x07\x0e\x0e��\x03M7W�͖�����|��f��\x01d�ٚLcEiieQE��WYX_Y\x19�,�/��?���TBٸ��\x18=j\x1fH�t���ъhV��k4���C��l�����8�/ K��hA�t3pppppp�=\x142>Q���/\x0f�#��FBY��(d�z�#�Y�\x11�V\x15�\x07�\r���-\x08\x07\x0b\n2\�Y%\x05��ܜ̬��A�ٹG|�\x0c8�p��ׁ.h\x02\x0e\x0e��\x03\x13�\x05�\x1d"�m�̄b�����\x01���\x7f�\x01��ˊ��3Cե#�����\x17�WW��ޟJ4\x1f7o^�ӕ���#3�μ�����1���&���')

[dosubot[bot]]

I found a similar unsolved discussion regarding an error while using the RecursiveUrlLoader ^[1].

The RecursiveUrlLoader class does have built-in error handling and logging mechanisms for unsupported content types like PDFs. It uses a try-except block to handle exceptions during the loading process. If an exception occurs and the continue_on_failure attribute is set to True, it logs a warning message and continues processing other URLs ^[2].

To handle PDF content, you might need to implement a custom extractor function that can process PDF files. This function can be passed to the extractor parameter of the RecursiveUrlLoader. You may also need to ensure that any libraries required for PDF processing (such as PyPDF2 or pdfminer) are installed and used within your extractor function.

To continue talking to Dosu, mention @dosu.

[deepak-hl]

@dosu
how can we add pdf extractor in recursive url loader?

[dosubot[bot]]

To add a PDF extractor to the RecursiveUrlLoader, you can define a custom extractor function that processes PDF content and pass it to the extractor parameter when creating an instance of RecursiveUrlLoader. You can use a library like PyPDF2 to extract text from PDF files. Here's an example of how you might set this up:

from langchain_community.document_loaders import RecursiveUrlLoader
import PyPDF2

def pdf_extractor(pdf_content: bytes) -> str:
    # Assuming pdf_content is the raw bytes of the PDF file
    reader = PyPDF2.PdfReader(pdf_content)
    text = ""
    for page in reader.pages:
        text += page.extract_text()
    return text

loader = RecursiveUrlLoader(
    "https://example.com/",
    extractor=pdf_extractor
)
docs = loader.load()
print(docs[0].page_content[:200])

This code snippet demonstrates how to set up a PDF extractor. You will need to adapt it to handle the actual content type and structure of the URLs you are working with ^[1].

To continue talking to Dosu, mention @dosu.

[deepak-hl]

@dosu
can we pass other extractor too like for xml content with pdf extarctor?

[dosubot[bot]]

Yes, you can pass multiple extractors for different content types, such as XML and PDF, by creating a custom extractor function that handles each content type based on the Content-Type header. Here's an example of how you can set this up:

from langchain_community.document_loaders import RecursiveUrlLoader
import PyPDF2
from bs4 import BeautifulSoup
import requests

def custom_extractor(content: bytes, content_type: str) -> str:
    if "application/pdf" in content_type:
        reader = PyPDF2.PdfReader(content)
        text = ""
        for page in reader.pages:
            text += page.extract_text()
        return text
    elif "application/xml" in content_type or "text/xml" in content_type:
        soup = BeautifulSoup(content, "xml")
        return soup.get_text()
    else:
        # Default to returning raw content or handle other types
        return content.decode('utf-8', errors='ignore')

def extractor_with_content_type(response: requests.Response) -> str:
    content_type = response.headers.get("Content-Type", "")
    return custom_extractor(response.content, content_type)

loader = RecursiveUrlLoader(
    "https://example.com/",
    extractor=extractor_with_content_type
)
docs = loader.load()
print(docs[0].page_content[:200])

In this example, the custom_extractor function checks the Content-Type and applies the appropriate extraction logic for PDFs and XMLs. You can extend this function to handle additional content types as needed.

To continue talking to Dosu, mention @dosu.