Skip to content

Commit 94c0b74

Browse files
chigwellErick Friis
authored and
shinxi
committed
docs: add langchain-pull-md Markdown loader (#29024)
- [x] **PR title**: "docs: add langchain-pull-md Markdown loader" - [x] **PR message**: - **Description:** This PR introduces the `langchain-pull-md` package to the LangChain community. It includes a new document loader that utilizes the pull.md service to convert URLs into Markdown format, particularly useful for handling web pages rendered with JavaScript frameworks like React, Angular, or Vue.js. This loader helps in efficient and reliable Markdown conversion directly from URLs without local rendering, reducing server load. - **Issue:** NA - **Dependencies:** requests >=2.25.1 - **Twitter handle:** https://x.com/eugeneevstafev?s=21 - [x] **Add tests and docs**: 1. Added unit tests to verify URL checking and conversion functionalities. 2. Created a comprehensive example notebook detailing the usage of the new loader. - [x] **Lint and test**: - Completed local testing using `make format`, `make lint`, and `make test` commands as per the LangChain contribution guidelines. **Related Links:** - [Package Repository](https://github.com/chigwell/langchain-pull-md) - [PyPI Package](https://pypi.org/project/langchain-pull-md/) --------- Co-authored-by: Erick Friis <[email protected]>
1 parent 889c681 commit 94c0b74

File tree

3 files changed

+186
-0
lines changed

3 files changed

+186
-0
lines changed

‎docs/docs/integrations/document_loaders/pull_md.ipynb‎

Lines changed: 140 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,140 @@
1+
{
2+
"cells": [
3+
{
4+
"cell_type": "raw",
5+
"metadata": {},
6+
"source": [
7+
"---\n",
8+
"sidebar_label: PullMdLoader\n",
9+
"---"
10+
]
11+
},
12+
{
13+
"cell_type": "markdown",
14+
"metadata": {},
15+
"source": [
16+
"# PullMdLoader\n",
17+
"\n",
18+
"Loader for converting URLs into Markdown using the pull.md service.\n",
19+
"\n",
20+
"This package implements a [document loader](/docs/concepts/document_loaders/) for web content. Unlike traditional web scrapers, PullMdLoader can handle web pages built with dynamic JavaScript frameworks like React, Angular, or Vue.js, converting them into Markdown without local rendering.\n",
21+
"\n",
22+
"## Overview\n",
23+
"### Integration details\n",
24+
"\n",
25+
"| Class | Package | Local | Serializable | JS Support |\n",
26+
"| :--- | :--- | :---: | :---: | :---: |\n",
27+
"| PullMdLoader | langchain-pull-md | ✅ | ✅ | ❌ |\n"
28+
]
29+
},
30+
{
31+
"cell_type": "markdown",
32+
"metadata": {},
33+
"source": [
34+
"## Setup\n",
35+
"\n",
36+
"### Installation\n",
37+
"\n",
38+
"```bash\n",
39+
"pip install langchain-pull-md\n",
40+
"```"
41+
]
42+
},
43+
{
44+
"cell_type": "markdown",
45+
"metadata": {},
46+
"source": [
47+
"### Initialization"
48+
]
49+
},
50+
{
51+
"cell_type": "code",
52+
"execution_count": 10,
53+
"metadata": {},
54+
"outputs": [],
55+
"source": [
56+
"from langchain_pull_md.markdown_loader import PullMdLoader\n",
57+
"\n",
58+
"# Instantiate the loader with a URL\n",
59+
"loader = PullMdLoader(url=\"https://example.com\")"
60+
]
61+
},
62+
{
63+
"cell_type": "markdown",
64+
"metadata": {},
65+
"source": [
66+
"### Load"
67+
]
68+
},
69+
{
70+
"cell_type": "code",
71+
"execution_count": 11,
72+
"metadata": {},
73+
"outputs": [],
74+
"source": [
75+
"documents = loader.load()"
76+
]
77+
},
78+
{
79+
"cell_type": "code",
80+
"execution_count": 12,
81+
"metadata": {},
82+
"outputs": [
83+
{
84+
"data": {
85+
"text/plain": [
86+
"{'source': 'https://example.com',\n",
87+
" 'page_content': '# Example Domain\\nThis domain is used for illustrative examples in documents. You may use this domain in literature without prior coordination or asking for permission.'}"
88+
]
89+
},
90+
"execution_count": 12,
91+
"metadata": {},
92+
"output_type": "execute_result"
93+
}
94+
],
95+
"source": [
96+
"documents[0].metadata"
97+
]
98+
},
99+
{
100+
"cell_type": "markdown",
101+
"metadata": {},
102+
"source": [
103+
"## Lazy Load\n",
104+
"\n",
105+
"No lazy loading is implemented. `PullMdLoader` performs a real-time conversion of the provided URL into Markdown format whenever the `load` method is called."
106+
]
107+
},
108+
{
109+
"cell_type": "markdown",
110+
"metadata": {},
111+
"source": [
112+
"## API reference:\n",
113+
"\n",
114+
"- [GitHub](https://github.com/chigwell/langchain-pull-md)\n",
115+
"- [PyPi](https://pypi.org/project/langchain-pull-md/)"
116+
]
117+
}
118+
],
119+
"metadata": {
120+
"kernelspec": {
121+
"display_name": "Python 3 (ipykernel)",
122+
"language": "python",
123+
"name": "python3"
124+
},
125+
"language_info": {
126+
"codemirror_mode": {
127+
"name": "ipython",
128+
"version": 3
129+
},
130+
"file_extension": ".py",
131+
"mimetype": "text/x-python",
132+
"name": "python",
133+
"nbconvert_exporter": "python",
134+
"pygments_lexer": "ipython3",
135+
"version": "3.10.12"
136+
}
137+
},
138+
"nbformat": 4,
139+
"nbformat_minor": 5
140+
}

‎docs/docs/integrations/providers/pull-md.mdx‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
# PullMd Loader
2+
3+
>[PullMd](https://pull.md/) is a service that converts web pages into Markdown format. The `langchain-pull-md` package utilizes this service to convert URLs, especially those rendered with JavaScript frameworks like React, Angular, or Vue.js, into Markdown without the need for local rendering.
4+
5+
## Installation and Setup
6+
7+
To get started with `langchain-pull-md`, you need to install the package via pip:
8+
9+
```bash
10+
pip install langchain-pull-md
11+
```
12+
13+
See the [usage example](/docs/integrations/document_loaders/pull_md) for detailed integration and usage instructions.
14+
15+
## Document Loader
16+
17+
The `PullMdLoader` class in `langchain-pull-md` provides an easy way to convert URLs to Markdown. It's particularly useful for loading content from modern web applications for use within LangChain's processing capabilities.
18+
19+
```python
20+
from langchain_pull_md import PullMdLoader
21+
22+
# Initialize the loader with a URL of a JavaScript-rendered webpage
23+
loader = PullMdLoader(url='https://example.com')
24+
25+
# Load the content as a Document
26+
documents = loader.load()
27+
28+
# Access the Markdown content
29+
for document in documents:
30+
print(document.page_content)
31+
```
32+
33+
This loader supports any URL and is particularly adept at handling sites built with dynamic JavaScript, making it a versatile tool for markdown extraction in data processing workflows.
34+
35+
## API Reference
36+
37+
For a comprehensive guide to all available functions and their parameters, visit the [API reference](https://github.com/chigwell/langchain-pull-md).
38+
39+
## Additional Resources
40+
41+
- [GitHub Repository](https://github.com/chigwell/langchain-pull-md)
42+
- [PyPi Package](https://pypi.org/project/langchain-pull-md/)

‎libs/packages.yml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -320,4 +320,8 @@ packages:
320320
- name: langchain-dappier
321321
path: .
322322
repo: DappierAI/langchain-dappier
323+
downloads: 0
324+
- name: langchain-pull-md
325+
path: .
326+
repo: chigwell/langchain-pull-md
323327
downloads: 0

0 commit comments

Comments
 (0)