Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions scrapegraph-js/package.json
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
{
"name": "scrapegraph-sdk",
"name": "scrapegraph-js",
"author": "ScrapeGraphAI",
"version": "0.0.1",
"description": "Scrape and extract structured data from a webpage using ScrapeGraph AI.",
Expand All @@ -23,7 +23,8 @@
"module": "index.js",
"type": "module",
"dependencies": {
"axios": "^1.6.0"
"axios": "^1.6.0",
"zod": "^3.23.8"
},
"devDependencies": {
"dotenv": "^16.4.5"
Expand Down
223 changes: 129 additions & 94 deletions scrapegraph-js/readme.md
Original file line number Diff line number Diff line change
@@ -1,150 +1,185 @@
# ScrapeGraph JS SDK
# 🌐 ScrapeGraph JavaScript SDK

A JavaScript SDK for interacting with the ScrapeGraph AI API. This SDK provides easy-to-use functions for web scraping, managing credits, and submitting feedback.
[![npm version](https://badge.fury.io/js/scrapegraph-js.svg)](https://badge.fury.io/js/scrapegraph-js)
[![License](https://img.shields.io/badge/License-MIT-blue.svg)](https://opensource.org/licenses/MIT)
[![Build Status](https://github.com/ScrapeGraphAI/scrapegraph-sdk/actions/workflows/ci.yml/badge.svg)](https://github.com/ScrapeGraphAI/scrapegraph-sdk/actions)
[![Documentation Status](https://img.shields.io/badge/docs-latest-brightgreen.svg)](https://docs.scrapegraphai.com)

## Installation
Official JavaScript/TypeScript SDK for the ScrapeGraph AI API - Smart web scraping powered by AI.

Install the package using npm:
## 🚀 Features

```bash
npm install scrapegraph-js
```
- ✨ Smart web scraping with AI
- 🔄 Fully asynchronous design
- 🔍 Detailed error handling
- ⚡ Automatic retries and logging
- 🔐 Secure API authentication

## Usage
## 📦 Installation

> [!WARNING]
> Remember not to write API keys directly in the code; instead, store them securely in `.env` files.
Install the package using npm or yarn:

First, import the required functions:
```bash
# Using npm
npm i scrapegraph-js

```javascript
import { smartScraper, getSmartScraperRequest, getCredits, sendFeedback } from 'scrapegraph-sdk';
# Using yarn
yarn add scrapegraph-js
```

### Scraping Websites

#### Basic scraping
## 🔧 Quick Start

> **Note**: Store your API keys securely in environment variables. Use `.env` files and libraries like `dotenv` to load them into your app.

### Basic Example

```javascript
import { smartScraper } from 'scrapegraph-sdk';
import { smartScraper } from 'scrapegraph-js';
import 'dotenv/config';

const apiKey = process.env.SGAI_APIKEY;
const url = 'https://scrapegraphai.com';
// Initialize variables
const apiKey = process.env.SGAI_APIKEY; // Set your API key as an environment variable
const websiteUrl = 'https://example.com';
const prompt = 'What does the company do?';

try {
const response = await smartScraper(apiKey, url, prompt);
console.log(response);
} catch (error) {
console.error(error);
}
(async () => {
try {
const response = await smartScraper(apiKey, websiteUrl, prompt);
console.log(response.result);
} catch (error) {
console.error('Error:', error);
}
})();
```

#### Scraping with custom output schema
## 🎯 Examples

```javascript
import { smartScraper } from 'scrapegraph-sdk';
### Scraping Websites

const apiKey = 'your_api_key';
const url = 'https://scrapegraphai.com';
const prompt = 'What does the company do?';
const schema = //TODO

try {
const response = await smartScraper(apiKey, url, prompt, schema);
console.log(response);
} catch (error) {
console.error(error);
}
#### Basic Scraping

```javascript
import { smartScraper } from 'scrapegraph-js';

const apiKey = 'your-api-key';
const url = 'https://example.com';
const prompt = 'Extract the main heading and description.';

(async () => {
try {
const response = await smartScraper(apiKey, url, prompt);
console.log(response.result);
} catch (error) {
console.error('Error:', error);
}
})();
```

### Checking Credits
#### Scraping with Custom Output Schema

```javascript
import { getCredist } from 'scrapegraph-sdk';
//TODO
```

const apiKey = 'your_api_key';
### Checking API Credits

try {
const myCredit = await getCredits(apiKey);
console.log(myCredit)
} catch (error) {
console.error(error)
}
```javascript
import { getCredits } from 'scrapegraph-js';

const apiKey = 'your-api-key';

(async () => {
try {
const credits = await getCredits(apiKey);
console.log('Available credits:', credits);
} catch (error) {
console.error('Error fetching credits:', error);
}
})();
```

### Submitting Feedback

```javascript
import { sendFeedback } from 'scrapegraph-sdk';
import { sendFeedback } from 'scrapegraph-js';

const apiKey = 'your_api_key';
const apiKey = 'your-api-key';
const requestId = '16a63a80-c87f-4cde-b005-e6c3ecda278b';
const rating = 5;
const feedbackMessage = 'This is a test feedback message.';

try {
const feedback_response = await sendFeedback(apiKey, requestId, rating, feedbackMessage);
console.log(feedback_response);
} catch (error) {
console.error(error)
}
const feedbackText = 'This is a test feedback message.';

(async () => {
try {
const response = await sendFeedback(apiKey, requestId, rating, feedbackText);
console.log('Feedback response:', response);
} catch (error) {
console.error('Error sending feedback:', error);
}
})();
```

## API Reference

### scrape(apiKey, url[, options])
## 📚 Documentation

Scrapes a website and returns the extracted data.
For detailed documentation, visit [docs.scrapegraphai.com](https://docs.scrapegraphai.com)

Parameters:
- `apiKey` (string): Your ScrapeGraph AI API key
- `url` (string): The URL to scrape
- `options` (object, optional):
- `elements` (array): Specific elements to extract
- `wait_for` (string): CSS selector to wait for before scraping
- `javascript` (boolean): Enable JavaScript rendering
## 🛠️ Development

### credits(apiKey)
### Setup

Retrieves your current credit balance.
1. Clone the repository:
```bash
git clone https://github.com/ScrapeGraphAI/scrapegraph-sdk.git
cd scrapegraph-sdk/scrapegraph-js
```

Parameters:
- `apiKey` (string): Your ScrapeGraph AI API key
2. Install dependencies:
```bash
npm install
```

### feedback(apiKey, requestId, rating, feedbackText)
3. Run linting and testing:
```bash
npm run lint
npm test
```

Submits feedback for a scraping request.
### Running Tests

Parameters:
- `apiKey` (string): Your ScrapeGraph AI API key
- `requestId` (string): The request ID from the scrape response
- `rating` (number): Rating score
- `feedbackText` (string) (optional): Feedback message
```bash
# Run all tests
npm test

## Error Handling
# Run tests with coverage
npm run test:coverage
```

All functions return javascript `Error` object with imformation. In case of errors, the response will include error details:
## 📝 License

// TODO error list
This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.

```javascript
{
"statusCode": 400,
"title": "HTTP error occurred"
"details": "Error details",

}
```
## 🤝 Contributing

## License
Contributions are welcome! Please feel free to submit a Pull Request. For major changes, please open an issue first to discuss what you would like to change.

MIT
1. Fork the repository
2. Create your feature branch (`git checkout -b feature/AmazingFeature`)
3. Commit your changes (`git commit -m 'Add some AmazingFeature'`)
4. Push to the branch (`git push origin feature/AmazingFeature`)
5. Open a Pull Request

## Support
## 🔗 Links

For support, please visit [ScrapeGraph AI Documentation](https://sgai-api.onrender.com/docs).
- [Website](https://scrapegraphai.com)
- [Documentation](https://scrapegraphai.com/documentation)
- [GitHub](https://github.com/ScrapeGraphAI/scrapegraph-sdk)

## 💬 Support

- 📧 Email: [email protected]
- 💻 GitHub Issues: [Create an issue](https://github.com/ScrapeGraphAI/scrapegraph-sdk/issues)
- 🌟 Feature Requests: [Request a feature](https://github.com/ScrapeGraphAI/scrapegraph-sdk/issues/new)

---

Made with ❤️ by [ScrapeGraph AI](https://scrapegraphai.com)
7 changes: 3 additions & 4 deletions scrapegraph-py/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -155,10 +155,9 @@ Contributions are welcome! Please feel free to submit a Pull Request. For major

## 🔗 Links

- [Website](https://scrapegraphai.com)
- [Documentation](https://docs.scrapegraphai.com)
- [API Reference](https://docs.scrapegraphai.com/api)
- [GitHub](https://github.com/ScrapeGraphAI/scrapegraph-sdk)
- [Website](https://scrapegraphai.com)
- [Documentation](https://scrapegraphai.com/documentation)
- [GitHub](https://github.com/ScrapeGraphAI/scrapegraph-sdk)

## 💬 Support

Expand Down
Loading