Merge branch 'pre/beta' into feat/parallel-node-execution

Author: PeriniM

CHANGELOG.md

@@ -1,3 +1,65 @@
+## [0.11.0-beta.7](https://github.com/VinciGit00/Scrapegraph-ai/compare/v0.11.0-beta.6...v0.11.0-beta.7) (2024-05-13)
+
+
+### Bug Fixes
+
+* bug for claude ([d0167de](https://github.com/VinciGit00/Scrapegraph-ai/commit/d0167dee71779a3c1e1e042e17a41134b93b3c78))
+
+
+### Docs
+
+* **refactor:** changed example ([c7ec114](https://github.com/VinciGit00/Scrapegraph-ai/commit/c7ec114274da64f0b61cee80afe908a36ad26b78))
+
+## [0.11.0-beta.6](https://github.com/VinciGit00/Scrapegraph-ai/compare/v0.11.0-beta.5...v0.11.0-beta.6) (2024-05-13)
+
+
+### Bug Fixes
+
+* **fetch-node:** removed isSoup from default ([0c15947](https://github.com/VinciGit00/Scrapegraph-ai/commit/0c1594737f878ed5672f4c889fdf9b4e0d7ec49a))
+
+## [0.11.0-beta.5](https://github.com/VinciGit00/Scrapegraph-ai/compare/v0.11.0-beta.4...v0.11.0-beta.5) (2024-05-13)
+
+
+### Features
+
+* **webdriver-backend:** add dynamic import scripts from module and file ([db2234b](https://github.com/VinciGit00/Scrapegraph-ai/commit/db2234bf5d2f2589b080cd4136f33c4f4443bdfb))
+* **proxy-rotation:** add parse (IP address) or search (from broker) functionality for proxy rotation ([2170131](https://github.com/VinciGit00/Scrapegraph-ai/commit/217013181da06abe8d71d9db70e809ea4ebd8236))
+* added proxy rotation ([0c36a7e](https://github.com/VinciGit00/Scrapegraph-ai/commit/0c36a7ec1f32ee073d9e0f534a2cb97aba3d7a1f))
+* **safe-web-driver:** enchanced the original `AsyncChromiumLoader` web driver with proxy protection and flexible kwargs and backend ([768719c](https://github.com/VinciGit00/Scrapegraph-ai/commit/768719cce80953fa6cbe283e442420116c438f16))
+
+
+### Bug Fixes
+
+* **pytest:** add dependency for mocking testing functions ([2f4fd45](https://github.com/VinciGit00/Scrapegraph-ai/commit/2f4fd45700ebf1db0c429b5a6249386d1a111615))
+* **chromium-loader:** ensure it subclasses langchain's base loader ([b54d984](https://github.com/VinciGit00/Scrapegraph-ai/commit/b54d984c134c8cbc432fd111bb161d3d53cf4a85))
+* **proxy-rotation:** removed duplicated arg and passed the loader_kwarhs correctly to the node ([1e9a564](https://github.com/VinciGit00/Scrapegraph-ai/commit/1e9a56461632999c5dc09f5aa930c14c954025ad))
+* **proxy-rotation:** removed max_shape duplicate ([5d6d996](https://github.com/VinciGit00/Scrapegraph-ai/commit/5d6d996e8f6132101d4c3af835d74f0674baffa1))
+
+
+### Docs
+
+* **refactor:** added proxy-rotation usage and refactor readthedocs ([e256b75](https://github.com/VinciGit00/Scrapegraph-ai/commit/e256b758b2ada641f97b23b1cf6c6b0174563d8a))
+
+## [0.11.0-beta.4](https://github.com/VinciGit00/Scrapegraph-ai/compare/v0.11.0-beta.3...v0.11.0-beta.4) (2024-05-12)
+
+
+### Features
+
+* add new prompt info ([e2350ed](https://github.com/VinciGit00/Scrapegraph-ai/commit/e2350eda6249d8e121344d12c92645a3887a5b76))
+
+## [0.11.0-beta.3](https://github.com/VinciGit00/Scrapegraph-ai/compare/v0.11.0-beta.2...v0.11.0-beta.3) (2024-05-12)
+
+
+### Features
+
+* add support for deepseek-chat ([156b67b](https://github.com/VinciGit00/Scrapegraph-ai/commit/156b67b91e1798f67082123e2c0087d358a32d4d)), closes [#222](https://github.com/VinciGit00/Scrapegraph-ai/issues/222)
+
+
+### Docs
+
+* add diagram showing general structure/flow of the library ([13ae918](https://github.com/VinciGit00/Scrapegraph-ai/commit/13ae9180ac5e7ef11dad1a210cf8790e797397dd))
+* update overview diagram with more models ([b441b30](https://github.com/VinciGit00/Scrapegraph-ai/commit/b441b30a5c60dda105964f69bd4cef06825f5c74))
+
 ## [0.11.0-beta.2](https://github.com/VinciGit00/Scrapegraph-ai/compare/v0.11.0-beta.1...v0.11.0-beta.2) (2024-05-10)
 
 

docs/assets/project_overview_diagram.fig

@@ -30,4 +30,3 @@
 # https://www.sphinx-doc.org/en/master/usage/configuration.html#options-for-html-output
 
 html_theme = 'sphinx_rtd_theme'
-html_static_path = ['_static']

docs/assets/project_overview_diagram.png

@@ -1,7 +1,9 @@
 Examples
 ========
 
-Here some example of the different ways to scrape with ScrapegraphAI
+Let's suppose you want to scrape a website to get a list of projects with their descriptions.
+You can use the `SmartScraperGraph` class to do that.
+The following examples show how to use the `SmartScraperGraph` class with OpenAI models and local models.
 
 OpenAI models
 ^^^^^^^^^^^^^
@@ -78,7 +80,7 @@ After that, you can run the following code, using only your machine resources br
    # ************************************************
 
    smart_scraper_graph = SmartScraperGraph(
-      prompt="List me all the news with their description.",
+      prompt="List me all the projects with their description.",
       # also accepts a string with the already downloaded HTML code
       source="https://perinim.github.io/projects",
       config=graph_config
@@ -87,3 +89,4 @@ After that, you can run the following code, using only your machine resources br
    result = smart_scraper_graph.run()
    print(result)
 
+To find out how you can customize the `graph_config` dictionary, by using different LLM and adding new parameters, check the `Scrapers` section!

docs/assets/searchgraph.png

@@ -7,26 +7,35 @@ for this project.
 Prerequisites
 ^^^^^^^^^^^^^
 
-- `Python 3.8+ <https://www.python.org/downloads/>`_
-- `pip <https://pip.pypa.io/en/stable/getting-started/>`
-- `ollama <https://ollama.com/>` *optional for local models 
+- `Python >=3.9,<3.12 <https://www.python.org/downloads/>`_
+- `pip <https://pip.pypa.io/en/stable/getting-started/>`_
+- `Ollama <https://ollama.com/>`_ (optional for local models)
 
 
 Install the library
 ^^^^^^^^^^^^^^^^^^^^
 
+The library is available on PyPI, so it can be installed using the following command:
+
 .. code-block:: bash
 
    pip install scrapegraphai
 
+**Note:** It is higly recommended to install the library in a virtual environment (conda, venv, etc.)
+
+If your clone the repository, you can install the library using `poetry <https://python-poetry.org/docs/>`_:
+
+.. code-block:: bash
+
+   poetry install
+
 Additionally on Windows when using WSL
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
+If you are using Windows Subsystem for Linux (WSL) and you are facing issues with the installation of the library, you might need to install the following packages:
+
 .. code-block:: bash
 
    sudo apt-get -y install libnss3 libnspr4 libgbm1 libasound2
 
-As simple as that! You are now ready to scrape gnamgnamgnam 👿👿👿
-
-
 

docs/assets/smartscrapergraph.png

@@ -3,12 +3,6 @@
    You can adapt this file completely to your liking, but it should at least
    contain the root `toctree` directive.
 
-Welcome to scrapegraphai-ai's documentation!
-=======================================
-
-Here you will find all the information you need to get started.
-The following sections will guide you through the installation process and the usage of the library.
-
 .. toctree::
    :maxdepth: 2
    :caption: Introduction
@@ -22,6 +16,19 @@ The following sections will guide you through the installation process and the u
 
    getting_started/installation
    getting_started/examples
+
+.. toctree::
+   :maxdepth: 2
+   :caption: Scrapers
+
+   scrapers/graphs
+   scrapers/llm
+   scrapers/graph_config
+
+.. toctree::
+   :maxdepth: 2
+   :caption: Modules
+
    modules/modules
 
 Indices and tables