Merge pull request #129 from VinciGit00/main

reallignement

Author: VinciGit00

CHANGELOG.md

@@ -1,17 +1,75 @@
-## [0.5.0-beta.8](https://github.com/VinciGit00/Scrapegraph-ai/compare/v0.5.0-beta.7...v0.5.0-beta.8) (2024-05-02)
+## [0.6.0](https://github.com/VinciGit00/Scrapegraph-ai/compare/v0.5.2...v0.6.0) (2024-05-02)
 
 
 ### Features
 
+* added node and graph for CSV scraping ([4d542a8](https://github.com/VinciGit00/Scrapegraph-ai/commit/4d542a88f7d949a5ba360dcd880716c8110a5d14))
 * Allow end users to pass model instances for llm and embedding model ([b86aac2](https://github.com/VinciGit00/Scrapegraph-ai/commit/b86aac2188887642564a34d13d55d0fcff220ec1))
+* modified node name ([02d1af0](https://github.com/VinciGit00/Scrapegraph-ai/commit/02d1af006cb89bf860ee4f1186f582e2049a8e3d))
+
+
+### CI
+
+* **release:** 0.5.0-beta.7 [skip ci] ([40b2a34](https://github.com/VinciGit00/Scrapegraph-ai/commit/40b2a346d57865ca21915ecaa658096c52a2cc6b))
+* **release:** 0.5.0-beta.8 [skip ci] ([c11331a](https://github.com/VinciGit00/Scrapegraph-ai/commit/c11331a26ac325dfcf489272442ceeed13225a39))
+
+## [0.5.2](https://github.com/VinciGit00/Scrapegraph-ai/compare/v0.5.1...v0.5.2) (2024-05-02)
+
+
+### Bug Fixes
 
-## [0.5.0-beta.7](https://github.com/VinciGit00/Scrapegraph-ai/compare/v0.5.0-beta.6...v0.5.0-beta.7) (2024-05-01)
+* bug on script_creator_graph.py ([4a3bc37](https://github.com/VinciGit00/Scrapegraph-ai/commit/4a3bc37f2fbb24953edd68f28234ff14302ac120))
+
+## [0.5.1](https://github.com/VinciGit00/Scrapegraph-ai/compare/v0.5.0...v0.5.1) (2024-05-02)
+
+
+### Bug Fixes
+
+* examples and graphs ([5cf4e4f](https://github.com/VinciGit00/Scrapegraph-ai/commit/5cf4e4f92f024041c44211aebd2e3bdf73351a00))
+
+
+### Docs
+
+* added venv suggestion ([ba2b24b](https://github.com/VinciGit00/Scrapegraph-ai/commit/ba2b24b4cd82d63f9235051eb0e95519c51fd639))
+* base and fetch node ([e981796](https://github.com/VinciGit00/Scrapegraph-ai/commit/e9817963c8e98e35662cc5a140b0348792d25307))
+* change contributing.md with new ci/cd workflow ([3e91a46](https://github.com/VinciGit00/Scrapegraph-ai/commit/3e91a46522ab1f6b2f733efd234b06df4687c695))
+* fixed basegraph docstring ([29427c2](https://github.com/VinciGit00/Scrapegraph-ai/commit/29427c233485816967c4ecd6c1951351be9b27ce))
+* graphs and helpers docstrings ([0631985](https://github.com/VinciGit00/Scrapegraph-ai/commit/0631985e6156bd21ec5317faff9e345c8aa7f88b))
+* refactor examples ([c11fc28](https://github.com/VinciGit00/Scrapegraph-ai/commit/c11fc288963e1a2818e451279a3bf53eb33e22be))
+* refactor models docstrings ([18c20eb](https://github.com/VinciGit00/Scrapegraph-ai/commit/18c20eb03de183a0311be5ffe21f53ec4edf1b87))
+* refactor nodes docstrings ([1409797](https://github.com/VinciGit00/Scrapegraph-ai/commit/140979747598210674131befadd786800c9fb5ec))
+* update utils docstrings ([cf038b3](https://github.com/VinciGit00/Scrapegraph-ai/commit/cf038b33eaae42f65d7d9c782b5729092b272dd0))
+
+## [0.5.0](https://github.com/VinciGit00/Scrapegraph-ai/compare/v0.4.1...v0.5.0) (2024-04-30)
 
 
 ### Features
 
-* added node and graph for CSV scraping ([4d542a8](https://github.com/VinciGit00/Scrapegraph-ai/commit/4d542a88f7d949a5ba360dcd880716c8110a5d14))
-* modified node name ([02d1af0](https://github.com/VinciGit00/Scrapegraph-ai/commit/02d1af006cb89bf860ee4f1186f582e2049a8e3d))
+* add cluade integration ([e0ffc83](https://github.com/VinciGit00/Scrapegraph-ai/commit/e0ffc838b06c0f024026a275fc7f7b4243ad5cf9))
+* add co-author ([719a353](https://github.com/VinciGit00/Scrapegraph-ai/commit/719a353410992cc96f46ec984a5d3ec372e71ad2))
+* **fetch:** added playwright support ([42ab0aa](https://github.com/VinciGit00/Scrapegraph-ai/commit/42ab0aa1d275b5798ab6fc9feea575fe59b6e767))
+* added verbose flag to suppress print statements ([2dd7817](https://github.com/VinciGit00/Scrapegraph-ai/commit/2dd7817cfb37cfbeb7e65b3a24655ab238f48026))
+* base groq + requirements + toml update with groq ([7dd5b1a](https://github.com/VinciGit00/Scrapegraph-ai/commit/7dd5b1a03327750ffa5b2fb647eda6359edd1fc2))
+* **refactor:** changed variable names ([8fba7e5](https://github.com/VinciGit00/Scrapegraph-ai/commit/8fba7e5490f916b325588443bba3fff5c0733c17))
+* **llm:** implemented groq model ([dbbf10f](https://github.com/VinciGit00/Scrapegraph-ai/commit/dbbf10fc77b34d99d64c6cd7f74524b6d8e57fa5))
+* updated requirements.txt ([d368725](https://github.com/VinciGit00/Scrapegraph-ai/commit/d36872518a6d234eba5f8b7ddca7da93797874b2))
+
+
+### Bug Fixes
+
+* script generator and add new benchmarks ([e3d0194](https://github.com/VinciGit00/Scrapegraph-ai/commit/e3d0194dc93b20dc254fc48bba11559bf8a3a185))
+
+
+### CI
+
+* **release:** 0.4.0-beta.3 [skip ci] ([d13321b](https://github.com/VinciGit00/Scrapegraph-ai/commit/d13321b2f86d98e2a3a0c563172ca0dd29cdf5fb))
+* **release:** 0.5.0-beta.1 [skip ci] ([450291f](https://github.com/VinciGit00/Scrapegraph-ai/commit/450291f52e48cd35b2b8cc50ff66f5336326fa25))
+* **release:** 0.5.0-beta.2 [skip ci] ([ff7d12f](https://github.com/VinciGit00/Scrapegraph-ai/commit/ff7d12f1389d8eed87e9f6b2fc8b099767a904a9))
+* **release:** 0.5.0-beta.3 [skip ci] ([7e81f7c](https://github.com/VinciGit00/Scrapegraph-ai/commit/7e81f7c03f79c43219743be52affabbaf0d66387))
+* **release:** 0.5.0-beta.4 [skip ci] ([14e56f6](https://github.com/VinciGit00/Scrapegraph-ai/commit/14e56f6ab1711a08e749edbda860d349db491dae))
+* **release:** 0.5.0-beta.5 [skip ci] ([5ac97e2](https://github.com/VinciGit00/Scrapegraph-ai/commit/5ac97e2fb321be40c9787fbf8cb53fa62cf0ce06))
+* **release:** 0.5.0-beta.6 [skip ci] ([9356124](https://github.com/VinciGit00/Scrapegraph-ai/commit/9356124ce39568e88f7d2965181579c4ff0a5752))
+
 
 ## [0.5.0-beta.6](https://github.com/VinciGit00/Scrapegraph-ai/compare/v0.5.0-beta.5...v0.5.0-beta.6) (2024-04-30)
 

CONTRIBUTING.md

@@ -15,22 +15,31 @@ Thank you for your interest in contributing to **ScrapeGraphAI**! We welcome con
 
 To get started with contributing, follow these steps:
 
-1. Fork the repository on GitHub.
+1. Fork the repository on GitHub **(FROM pre/beta branch)**.
 2. Clone your forked repository to your local machine.
-3. Install the necessary dependencies.
+3. Install the necessary dependencies from requirements.txt or via pyproject.toml as you prefere :).
 4. Make your changes or additions.
 5. Test your changes thoroughly.
 6. Commit your changes with descriptive commit messages.
 7. Push your changes to your forked repository.
-8. Submit a pull request to the main repository.
+8. Submit a pull request to the pre/beta branch.
+
+N.B All the pull request to the main branch will be rejected!
 
 ## Contributing Guidelines
 
 Please adhere to the following guidelines when contributing to ScrapeGraphAI:
 
 - Follow the code style and formatting guidelines specified in the [Code Style](#code-style) section.
-- Make sure your changes are well-documented and include any necessary updates to the project's documentation.
-- Write clear and concise commit messages that describe the purpose of your changes.
+- Make sure your changes are well-documented and include any necessary updates to the project's documentation and requirements if needed.
+- Write clear and concise commit messages that describe the purpose of your changes and the last commit before the pull request has to follow the following format:
+  - `feat: Add new feature`
+  - `fix: Correct issue with existing feature`
+  - `docs: Update documentation`
+  - `style: Improve formatting and style`
+  - `refactor: Restructure code`
+  - `test: Add or update tests`
+  - `perf: Improve performance`
 - Be respectful and considerate towards other contributors and maintainers.
 
 ## Code Style
@@ -42,6 +51,7 @@ Please make sure to format your code accordingly before submitting a pull reques
 - [Style Guide for Python Code](https://www.python.org/dev/peps/pep-0008/)
 - [Google Python Style Guide](https://google.github.io/styleguide/pyguide.html)
 - [The Hitchhiker's Guide to Python](https://docs.python-guide.org/writing/style/)
+- [Pylint style of code for the documentation](https://pylint.pycqa.org/en/1.6.0/tutorial.html)
 
 ## Submitting a Pull Request
 
@@ -53,7 +63,7 @@ To submit your changes for review, please follow these steps:
 4. Select your forked repository and the branch containing your changes.
 5. Provide a descriptive title and detailed description for your pull request.
 6. Reviewers will provide feedback and discuss any necessary changes.
-7. Once your pull request is approved, it will be merged into the main repository.
+7. Once your pull request is approved, it will be merged into the pre/beta branch.
 
 ## Reporting Issues
 

README.md

@@ -27,6 +27,9 @@ you will also need to install Playwright for javascript-based scraping:
 ```bash
 playwright install
 ```
+
+**Note**: it is recommended to install the library in a virtual environment to avoid conflicts with other libraries 🐱
+
 ## 🔍 Demo
 Official streamlit demo:
 

examples/groq/.env.example

@@ -1 +1,2 @@
-GROQ_APIKEY= "your groq key"
+GROQ_APIKEY= "your groq key"
+OPENAI_APIKEY="your openai api key"

examples/groq/smart_scraper_groq.py renamed to examples/groq/smart_scraper_groq_ollama.py

@@ -0,0 +1,52 @@
+""" 
+Basic example of scraping pipeline using SmartScraper
+"""
+
+import os
+from dotenv import load_dotenv
+from scrapegraphai.graphs import SmartScraperGraph
+from scrapegraphai.utils import prettify_exec_info
+
+load_dotenv()
+
+
+# ************************************************
+# Define the configuration for the graph
+# ************************************************
+
+groq_key = os.getenv("GROQ_APIKEY")
+openai_key = os.getenv("OPENAI_APIKEY")
+
+graph_config = {
+    "llm": {
+        "model": "groq/gemma-7b-it",
+        "api_key": groq_key,
+        "temperature": 0
+    },
+    "embeddings": {
+        "api_key": openai_key,
+        "model": "gpt-3.5-turbo",
+    },
+    "headless": False
+}
+
+# ************************************************
+# Create the SmartScraperGraph instance and run it
+# ************************************************
+
+smart_scraper_graph = SmartScraperGraph(
+    prompt="List me all the projects with their description.",
+    # also accepts a string with the already downloaded HTML code
+    source="https://perinim.github.io/projects/",
+    config=graph_config
+)
+
+result = smart_scraper_graph.run()
+print(result)
+
+# ************************************************
+# Get graph execution info
+# ************************************************
+
+graph_exec_info = smart_scraper_graph.get_execution_info()
+print(prettify_exec_info(graph_exec_info))

examples/groq/smart_scraper_groq_openai.py

@@ -40,6 +40,7 @@
 fetch_node = FetchNode(
     input="url | local_dir",
     output=["doc"],
+    node_config={"headless": True, "verbose": True}
 )
 parse_node = ParseNode(
     input="doc",

examples/mixed_models/search_graph_mixed.py renamed to examples/mixed_models/search_graph_groq_ollama.py

@@ -53,11 +53,3 @@
 
 graph_exec_info = smart_scraper_graph.get_execution_info()
 print(prettify_exec_info(graph_exec_info))
-
-
-# ************************************************
-# Get graph execution info
-# ************************************************
-
-graph_exec_info = smart_scraper_graph.get_execution_info()
-print(prettify_exec_info(graph_exec_info))

examples/openai/custom_graph_openai.py

@@ -20,7 +20,7 @@
         "api_key": openai_key,
         "model": "gpt-3.5-turbo",
     },
-    "library": "beautifoulsoup"
+    "library": "beautifulsoup"
 }
 
 # ************************************************