You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
You can try it in [demo app](https://scraper.nepochataya.pp.ua/) ([source](https://github.com/s0ph1e/web-scraper))
16
+
You can try it in [demo app](https://scraper.nepochataya.pp.ua/) ([source](https://github.com/website-scraper/web-scraper))
17
17
18
-
**Note:** by default dynamic websites (where content is loaded by js) may be saved not correctly because `website-scraper` doesn't execute js, it only parses http responses for html and css files. If you need to download dynamic website take a look on [website-scraper-phantom](https://github.com/s0ph1e/node-website-scraper-phantom).
18
+
**Note:** by default dynamic websites (where content is loaded by js) may be saved not correctly because `website-scraper` doesn't execute js, it only parses http responses for html and css files. If you need to download dynamic website take a look on [website-scraper-phantom](https://github.com/website-scraper/node-website-scraper-phantom).
*[onResourceError](#onresourceerror) - callback called when resource's downloading is failed
65
65
*[updateMissingSources](#updatemissingsources) - update url for missing sources with absolute url
66
66
67
-
Default options you can find in [lib/config/defaults.js](https://github.com/s0ph1e/node-website-scraper/blob/master/lib/config/defaults.js) or get them using `scrape.defaults`.
67
+
Default options you can find in [lib/config/defaults.js](https://github.com/website-scraper/node-website-scraper/blob/master/lib/config/defaults.js) or get them using `scrape.defaults`.
68
68
69
69
#### urls
70
70
Array of objects which contain urls to download and filenames for them. **_Required_**.
@@ -215,10 +215,10 @@ scrape({
215
215
}
216
216
}).then(console.log).catch(console.log);
217
217
```
218
-
Scrape function resolves with array of [Resource](https://github.com/s0ph1e/node-website-scraper/blob/master/lib/resource.js) objects which contain `metadata` property from `httpResponseHandler`.
218
+
Scrape function resolves with array of [Resource](https://github.com/website-scraper/node-website-scraper/blob/master/lib/resource.js) objects which contain `metadata` property from `httpResponseHandler`.
219
219
220
220
#### resourceSaver
221
-
Class which saves [Resources](https://github.com/s0ph1e/node-website-scraper/blob/master/lib/resource.js), should have methods `saveResource` and `errorCleanup` which return Promises. Use it to save files where you need: to dropbox, amazon S3, existing directory, etc. By default all files are saved in local file system to new directory passed in `directory` option (see [lib/resource-saver/index.js](https://github.com/s0ph1e/node-website-scraper/blob/master/lib/resource-saver/index.js)).
221
+
Class which saves [Resources](https://github.com/website-scraper/node-website-scraper/blob/master/lib/resource.js), should have methods `saveResource` and `errorCleanup` which return Promises. Use it to save files where you need: to dropbox, amazon S3, existing directory, etc. By default all files are saved in local file system to new directory passed in `directory` option (see [lib/resource-saver/index.js](https://github.com/website-scraper/node-website-scraper/blob/master/lib/resource-saver/index.js)).
222
222
```javascript
223
223
scrape({
224
224
urls: ['http://example.com/'],
@@ -231,7 +231,7 @@ scrape({
231
231
```
232
232
233
233
#### onResourceSaved
234
-
Function called each time when resource is saved to file system. Callback is called with [Resource](https://github.com/s0ph1e/node-website-scraper/blob/master/lib/resource.js) object. Defaults to `null` - no callback will be called.
234
+
Function called each time when resource is saved to file system. Callback is called with [Resource](https://github.com/website-scraper/node-website-scraper/blob/master/lib/resource.js) object. Defaults to `null` - no callback will be called.
235
235
```javascript
236
236
scrape({
237
237
urls: ['http://example.com/'],
@@ -243,7 +243,7 @@ scrape({
243
243
```
244
244
245
245
#### onResourceError
246
-
Function called each time when resource's downloading/handling/saving to fs was failed. Callback is called with - [Resource](https://github.com/s0ph1e/node-website-scraper/blob/master/lib/resource.js) object and `Error` object. Defaults to `null` - no callback will be called.
246
+
Function called each time when resource's downloading/handling/saving to fs was failed. Callback is called with - [Resource](https://github.com/website-scraper/node-website-scraper/blob/master/lib/resource.js) object and `Error` object. Defaults to `null` - no callback will be called.
247
247
```javascript
248
248
scrape({
249
249
urls: ['http://example.com/'],
@@ -281,7 +281,7 @@ scrape({
281
281
## callback
282
282
Callback function, optional, includes following parameters:
283
283
-`error`: if error - `Error` object, if success - `null`
284
-
-`result`: if error - `null`, if success - array of [Resource](https://github.com/s0ph1e/node-website-scraper/blob/master/lib/resource.js) objects containing:
284
+
-`result`: if error - `null`, if success - array of [Resource](https://github.com/website-scraper/node-website-scraper/blob/master/lib/resource.js) objects containing:
285
285
-`url`: url of loaded page
286
286
-`filename`: filename where page was saved (relative to `directory`)
0 commit comments