feat: release a new version

Author: coder-hxl

CHANGELOG.md

@@ -1,3 +1,29 @@
+# [v10.1.0](https://github.com/coder-hxl/x-crawl/compare/v10.0.2..v10.1.0) (2025-04-06)
+
+### 🚀 Features
+
+- Added ollama
+- Change the openai model type to string
+
+### ⛓️ Dependencies
+
+- puppeteer from 22.13.1 to 24.6.0
+- openai from 4.52.7 to 4.91.1
+- upgrade non-major dependencies to the latest version
+
+---
+
+### 🚀 特征
+
+- 新增 ollama
+- openai 模型类型改为 string
+
+### ⛓️ 依赖关系
+
+- puppeteer 从 22.13.1 升至 24.6.0
+- openai 从 4.52.7 升至 4.91.1
+- 非主要依赖项升级最新版本
+
 # [v10.0.2](https://github.com/coder-hxl/x-crawl/compare/v10.0.1..v10.0.2) (2024-07-21)
 
 ### 🚀 Features
@@ -14,7 +40,7 @@
 
 ### 🚀 特征
 
-- OpenAIChatModel 类型新增 'gpt-4o' | 'gpt-4o-2024-05-13'  | 'gpt-4-turbo'  | 'gpt-4-turbo-2024-04-09' ，与 openai 保持同步。
+- OpenAIChatModel 类型新增 'gpt-4o' | 'gpt-4o-2024-05-13' | 'gpt-4-turbo' | 'gpt-4-turbo-2024-04-09' ，与 openai 保持同步。
 
 ### ⛓️ 依赖关系
 

README.md

@@ -7,13 +7,13 @@ x-crawl is a flexible Node.js AI-assisted crawler library. Flexible usage and po
 It consists of two parts:
 
 - Crawler: It consists of a crawler API and various functions that can work normally even without relying on AI.
-- AI: Currently based on the large AI model provided by OpenAI, AI simplifies many tedious operations.
+- AI: Integrate ollama and openai, AI simplifies many tedious operations.
 
 > If you find x-crawl helpful, or you like x-crawl, you can give [x-crawl repository](https://github.com/coder-hxl/x-crawl) a like on GitHub A star. Your support is the driving force for our continuous improvement! thank you for your support!
 
 ## Features
 
-- **🤖 AI Assistance** - Powerful AI assistance function makes crawler work more efficient, intelligent and convenient.
+- **🤖 AI Assistance** - Integrate ollama and openai, powerful AI assistance function makes crawler work more efficient, intelligent and convenient.
 - **🖋️ Flexible writing** - A single crawling API is suitable for multiple configurations, and each configuration method has its own advantages.
 - **⚙️Multiple uses** - Supports crawling dynamic pages, static pages, interface data and file data.
 - **⚒️ Control page** - Crawling dynamic pages supports automated operations, keyboard input, event operations, etc.
@@ -56,28 +56,30 @@ const crawlOpenAIApp = createCrawlOpenAI({
 })
 
 // crawlPage is used to crawl pages
-crawlApp.crawlPage('https://www.example.cn/s/select_homes').then(async (res) => {
-  const { page, browser } = res.data
+crawlApp
+  .crawlPage('https://www.example.cn/s/select_homes')
+  .then(async (res) => {
+    const { page, browser } = res.data
 
-  // Wait for the element to appear on the page and get the HTML
-  const targetSelector = '[data-tracking-id="TOP_REVIEWED_LISTINGS"]'
-  await page.waitForSelector(targetSelector)
-  const highlyHTML = await page.$eval(targetSelector, (el) => el.innerHTML)
+    // Wait for the element to appear on the page and get the HTML
+    const targetSelector = '[data-tracking-id="TOP_REVIEWED_LISTINGS"]'
+    await page.waitForSelector(targetSelector)
+    const highlyHTML = await page.$eval(targetSelector, (el) => el.innerHTML)
 
-  // Let AI obtain image links and remove duplicates
-  const srcResult = await crawlOpenAIApp.parseElements(
-    highlyHTML,
-    `Get the image link, don't source it inside, and de-duplicate it`
-  )
+    // Let AI obtain image links and remove duplicates
+    const srcResult = await crawlOpenAIApp.parseElements(
+      highlyHTML,
+      `Get the image link, don't source it inside, and de-duplicate it`
+    )
 
-  browser.close()
+    browser.close()
 
-  // crawlFile is used to crawl file resources
-  crawlApp.crawlFile({
-    targets: srcResult.elements.map((item) => item.src),
-    storeDirs: './upload'
+    // crawlFile is used to crawl file resources
+    crawlApp.crawlFile({
+      targets: srcResult.elements.map((item) => item.src),
+      storeDirs: './upload'
+    })
   })
-})
 ```
 
 > [!TIP]

package.json

@@ -1,7 +1,7 @@
 {
   "private": true,
   "name": "x-crawl",
-  "version": "10.0.2",
+  "version": "10.1.0",
   "author": "coderHXL",
   "description": "x-crawl is a flexible Node.js AI-assisted crawler library.",
   "license": "MIT",
@@ -68,4 +68,4 @@
     "fingerprint",
     "multifunction"
   ]
-}
+}

publish/README.md

@@ -7,13 +7,13 @@ x-crawl is a flexible Node.js AI-assisted crawler library. Flexible usage and po
 It consists of two parts:
 
 - Crawler: It consists of a crawler API and various functions that can work normally even without relying on AI.
-- AI: Currently based on the large AI model provided by OpenAI, AI simplifies many tedious operations.
+- AI: Integrate ollama and openai, AI simplifies many tedious operations.
 
 > If you find x-crawl helpful, or you like x-crawl, you can give [x-crawl repository](https://github.com/coder-hxl/x-crawl) a like on GitHub A star. Your support is the driving force for our continuous improvement! thank you for your support!
 
 ## Features
 
-- **🤖 AI Assistance** - Powerful AI assistance function makes crawler work more efficient, intelligent and convenient.
+- **🤖 AI Assistance** - Integrate ollama and openai, powerful AI assistance function makes crawler work more efficient, intelligent and convenient.
 - **🖋️ Flexible writing** - A single crawling API is suitable for multiple configurations, and each configuration method has its own advantages.
 - **⚙️Multiple uses** - Supports crawling dynamic pages, static pages, interface data and file data.
 - **⚒️ Control page** - Crawling dynamic pages supports automated operations, keyboard input, event operations, etc.
@@ -56,28 +56,30 @@ const crawlOpenAIApp = createCrawlOpenAI({
 })
 
 // crawlPage is used to crawl pages
-crawlApp.crawlPage('https://www.example.cn/s/select_homes').then(async (res) => {
-  const { page, browser } = res.data
-
-  // Wait for the element to appear on the page and get the HTML
-  const targetSelector = '[data-tracking-id="TOP_REVIEWED_LISTINGS"]'
-  await page.waitForSelector(targetSelector)
-  const highlyHTML = await page.$eval(targetSelector, (el) => el.innerHTML)
-
-  // Let the AI get the image link and de-duplicate it (the more detailed the description, the better)
-  const srcResult = await crawlOpenAIApp.parseElements(
-    highlyHTML,
-    `Get the image link, don't source it inside, and de-duplicate it`
-  )
-
-  browser.close()
-
-  // crawlFile is used to crawl file resources
-  crawlApp.crawlFile({
-    targets: srcResult.elements.map((item) => item.src),
-    storeDirs: './upload'
+crawlApp
+  .crawlPage('https://www.example.cn/s/select_homes')
+  .then(async (res) => {
+    const { page, browser } = res.data
+
+    // Wait for the element to appear on the page and get the HTML
+    const targetSelector = '[data-tracking-id="TOP_REVIEWED_LISTINGS"]'
+    await page.waitForSelector(targetSelector)
+    const highlyHTML = await page.$eval(targetSelector, (el) => el.innerHTML)
+
+    // Let the AI get the image link and de-duplicate it (the more detailed the description, the better)
+    const srcResult = await crawlOpenAIApp.parseElements(
+      highlyHTML,
+      `Get the image link, don't source it inside, and de-duplicate it`
+    )
+
+    browser.close()
+
+    // crawlFile is used to crawl file resources
+    crawlApp.crawlFile({
+      targets: srcResult.elements.map((item) => item.src),
+      storeDirs: './upload'
+    })
   })
-})
 ```
 
 **You can even send the whole HTML to the AI to help us operate, because the website content is more complex you also need to describe the location to get more accurately, and will consume a lot of Tokens.**

publish/package.json

@@ -1,6 +1,6 @@
 {
   "name": "x-crawl",
-  "version": "10.0.2",
+  "version": "10.1.0",
   "author": "coderHXL",
   "description": "x-crawl is a flexible Node.js AI-assisted crawler library.",
   "license": "MIT",
@@ -41,8 +41,9 @@
   "dependencies": {
     "chalk": "5.4.1",
     "https-proxy-agent": "^7.0.6",
+    "ollama": "^0.5.14",
     "openai": "^4.91.1",
     "ora": "^8.2.0",
     "puppeteer": "24.6.0"
   }
-}
+}