feat: add SSL verification option to recursive link fetching (#46)

In the current implementation of the method _get_child_links_recursive,
the requests.get call doesn't accept verify as a parameter. This does
not allow users to disable SSL certificate verification when needed.
Please consider exposing the verify parameter as a configurable argument
to the method, defaulting to True for safety, but allowing users to
override it when necessary.


https://github.com/langchain-ai/langchain-community/blob/bc87773064735e649cfd798185502e156d5e948a/libs/community/langchain_community/document_loaders/recursive_url_loader.py#L376-L377

---------

Co-authored-by: Mason Daugherty <[email protected]>

Author: rajasblack

libs/community/langchain_community/document_loaders/recursive_url_loader.py

@@ -408,7 +408,11 @@ def _get_child_links_recursive(
         visited.add(url)
         try:
             response = requests.get(
-                url, timeout=self.timeout, headers=self.headers, proxies=self.proxies
+                url,
+                timeout=self.timeout,
+                headers=self.headers,
+                proxies=self.proxies,
+                verify=self.ssl,
             )
 
             if self.encoding is not None:
@@ -488,7 +492,7 @@ async def _async_get_child_links_recursive(
         )
         visited.add(url)
         try:
-            async with session.get(url) as response:
+            async with session.get(url, ssl=self.ssl) as response:
                 text = await response.text()
                 if self.check_response_status and 400 <= response.status <= 599:
                     raise ValueError(f"Received HTTP status {response.status}")