Skip to content

Commit 3cf7089

Browse files
VinciGit00VinciGit00
authored
Merge branch 'pre/beta' into temp
2 parents b4b90b3 + bf0a2f3 commit 3cf7089

39 files changed

+799
-254
lines changed

‎CHANGELOG.md‎

Lines changed: 21 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -13,46 +13,47 @@
1313
## [1.9.1](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.9.0...v1.9.1) (2024-07-12)
1414

1515

16+
1617
### Bug Fixes
1718

18-
* solve a burr integration ([881290b](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/881290b5066b39c505532656671fbf65f8fc312c))
19+
* add gpt o mini for azure ([77777c8](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/77777c898d1fad40f340b06c5b36d35b65409ea6))
1920

20-
## [1.9.0](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.8.0...v1.9.0) (2024-07-09)
21+
## [1.10.0-beta.2](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.10.0-beta.1...v1.10.0-beta.2) (2024-07-19)
2122

2223

2324
### Features
2425

25-
* add fireworks integration ([df0e310](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/df0e3108299071b849d7e055bd11d72764d24f08))
26-
* add integration for infos ([3bf5f57](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/3bf5f570a8f8e1b037a7ad3c9f583261a1536421))
27-
* add integrations for markdown files ([2804434](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/2804434a9ee12c52ae8956a88b1778a4dd3ec32f))
28-
* add vertexai integration ([119514b](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/119514bdfc2a16dfb8918b0c34ae7cc43a01384c))
29-
* improve md prompt recognition ([5fe694b](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/5fe694b6b4545a5091d16110318b992acfca4f58))
26+
* add gpt4o omni ([431edb7](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/431edb7bb2504f4c1335c3ae3ce2f91867fa7222))
3027

28+
## [1.10.0-beta.1](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.9.1...v1.10.0-beta.1) (2024-07-19)
3129

32-
### Bug Fixes
3330

34-
* add test ([3a537ee](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/3a537eec6fef1743924a9aa5cef0ba2f8d44bf11))
35-
* fix pyproject.toml ([7570bf8](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/7570bf8294e49bc54ec9e296aaadb763873390ca))
31+
### Features
3632

33+
* add searchngx integration ([5c92186](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/5c9218608140bf694fbfd96aa90276bc438bb475))
34+
* refactoring_to_md function ([602dd00](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/602dd00209ee1d72a1223fc4793759450921fcf9))
3735

38-
### chore
3936

40-
* **Docker:** fix port number ([afeb81f](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/afeb81f77a884799192d79dcac85666190fb1c9d))
41-
* **CI:** fix pylint workflow ([583c321](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/583c32106e827f50235d8fc69511652fd4b07a35))
42-
* **rye:** rebuild lockfiles ([27c2dd2](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/27c2dd23517a7e4b14fafd00320a8b81f73145dc))
37+
### Bug Fixes
4338

39+
* search link node ([cf3ab55](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/cf3ab5564ae5c415c63d1771b32ea68f5169ca82))
4440

45-
### Docs
4641

47-
* **roadmap:** fix urls ([14faba4](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/14faba4f00dd9f947f8dc5e0b51be49ea684179f))
48-
* **roadmap:** next steps ([3e644f4](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/3e644f498f05eb505fbd4e94b144c81567569aaa))
42+
### chore
43+
44+
* correct search engine name ([7ba2f6a](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/7ba2f6ae0b9d2e9336e973e1f57ab8355c739e27))
45+
* remove unused import ([fd1b7cb](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/fd1b7cb24a7c252277607abde35826e3c58e34ef))
46+
* remove unused workflow ([5c6dd8d](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/5c6dd8de4da08f09b5dd93c525d14b44778c9659))
47+
* **ci:** upgrade lockfiles ([c7b05a4](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/c7b05a4993df14d6ed4848121a3cd209571232f7))
48+
* upgrade tiktoken ([7314bc3](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/7314bc383068db590662bf7e512f799529308991))
4949

5050

5151
### CI
5252

53-
* **release:** 1.8.1-beta.1 [skip ci] ([8f9f96f](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/8f9f96f7e7ff41d2fff5bbbf18bf4fc85d4f98b3))
54-
* **release:** 1.9.0-beta.1 [skip ci] ([146432d](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/146432d476f775510441b062935adc47190141e2))
55-
* **release:** 1.9.0-beta.2 [skip ci] ([5cb5fbf](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/5cb5fbf5503eec9b34a6691eb993716cc9a821d6))
53+
* **release:** 1.9.0-beta.3 [skip ci] ([d3e63d9](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/d3e63d91be79f74e8a3fdb00e692d546c24cead5))
54+
* **release:** 1.9.0-beta.4 [skip ci] ([2fa04b5](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/2fa04b58159abf7af890ebc0768fe23d51bf177f))
55+
* **release:** 1.9.0-beta.5 [skip ci] ([bb62439](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/bb624399cfc3924825892dd48697fc298ad3b002))
56+
* **release:** 1.9.0-beta.6 [skip ci] ([54a69de](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/54a69de69e8077e02fd5584783ca62cc2e0ec5bb))
5657

5758
## [1.9.0-beta.2](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.9.0-beta.1...v1.9.0-beta.2) (2024-07-05)
5859

‎examples/anthropic/search_link_graph_haiku.py‎

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
"""
2+
Example of Search Graph
3+
"""
4+
import os
5+
from dotenv import load_dotenv
6+
from scrapegraphai.graphs import SearchGraph
7+
from scrapegraphai.utils import convert_to_csv, convert_to_json, prettify_exec_info
8+
from langchain_openai import AzureChatOpenAI
9+
from langchain_openai import AzureOpenAIEmbeddings
10+
11+
# ************************************************
12+
# Define the configuration for the graph
13+
# ************************************************
14+
15+
load_dotenv()
16+
17+
llm_model_instance = AzureChatOpenAI(
18+
openai_api_version=os.environ["AZURE_OPENAI_API_VERSION"],
19+
azure_deployment=os.environ["AZURE_OPENAI_CHAT_DEPLOYMENT_NAME"]
20+
)
21+
22+
embedder_model_instance = AzureOpenAIEmbeddings(
23+
azure_deployment=os.environ["AZURE_OPENAI_EMBEDDINGS_DEPLOYMENT_NAME"],
24+
openai_api_version=os.environ["AZURE_OPENAI_API_VERSION"],
25+
)
26+
27+
# ************************************************
28+
# Create the SmartScraperGraph instance and run it
29+
# ************************************************
30+
31+
graph_config = {
32+
"llm": {"model_instance": llm_model_instance},
33+
"embeddings": {"model_instance": embedder_model_instance}
34+
}
35+
36+
# ************************************************
37+
# Create the SearchGraph instance and run it
38+
# ************************************************
39+
40+
search_graph = SearchGraph(
41+
prompt="List me the best escursions near Trento",
42+
config=graph_config
43+
)
44+
45+
result = search_graph.run()
46+
print(result)
47+
48+
# ************************************************
49+
# Get graph execution info
50+
# ************************************************
51+
52+
graph_exec_info = search_graph.get_execution_info()
53+
print(prettify_exec_info(graph_exec_info))
54+
55+
# Save to json and csv
56+
convert_to_csv(result, "result")
57+
convert_to_json(result, "result")

‎examples/azure/search_link_graph_azure.py‎

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
"""
2+
Example of Search Graph
3+
"""
4+
import os
5+
from dotenv import load_dotenv
6+
from scrapegraphai.graphs import SearchGraph
7+
from scrapegraphai.utils import convert_to_csv, convert_to_json, prettify_exec_info
8+
9+
# ************************************************
10+
# Define the configuration for the graph
11+
# ************************************************
12+
13+
load_dotenv()
14+
15+
groq_key = os.getenv("GROQ_APIKEY")
16+
17+
graph_config = {
18+
"llm": {
19+
"model": "groq/gemma-7b-it",
20+
"api_key": groq_key,
21+
"temperature": 0
22+
},
23+
"embeddings": {
24+
"model": "ollama/nomic-embed-text",
25+
"temperature": 0,
26+
# "base_url": "http://localhost:11434", # set ollama URL arbitrarily
27+
},
28+
"headless": False
29+
}
30+
31+
# ************************************************
32+
# Create the SearchGraph instance and run it
33+
# ************************************************
34+
35+
search_graph = SearchGraph(
36+
prompt="List me the best escursions near Trento",
37+
config=graph_config
38+
)
39+
40+
result = search_graph.run()
41+
print(result)
42+
43+
# ************************************************
44+
# Get graph execution info
45+
# ************************************************
46+
47+
graph_exec_info = search_graph.get_execution_info()
48+
print(prettify_exec_info(graph_exec_info))
49+
50+
# Save to json and csv
51+
convert_to_csv(result, "result")
52+
convert_to_json(result, "result")

‎examples/bedrock/search_link_graph_bedrock.py‎

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
"""
2+
Example of Search Graph
3+
"""
4+
import os
5+
from dotenv import load_dotenv
6+
from scrapegraphai.graphs import SearchGraph
7+
from scrapegraphai.utils import convert_to_csv, convert_to_json, prettify_exec_info
8+
9+
# ************************************************
10+
# Define the configuration for the graph
11+
# ************************************************
12+
13+
graph_config = {
14+
"llm": {
15+
"client": "client_name",
16+
"model": "bedrock/anthropic.claude-3-sonnet-20240229-v1:0",
17+
"temperature": 0.0
18+
},
19+
"embeddings": {
20+
"model": "bedrock/cohere.embed-multilingual-v3"
21+
}
22+
}
23+
24+
# ************************************************
25+
# Create the SearchGraph instance and run it
26+
# ************************************************
27+
28+
search_graph = SearchGraph(
29+
prompt="List me the best escursions near Trento",
30+
config=graph_config
31+
)
32+
33+
result = search_graph.run()
34+
print(result)
35+
36+
# ************************************************
37+
# Get graph execution info
38+
# ************************************************
39+
40+
graph_exec_info = search_graph.get_execution_info()
41+
print(prettify_exec_info(graph_exec_info))
42+
43+
# Save to json and csv
44+
convert_to_csv(result, "result")
45+
convert_to_json(result, "result")

‎examples/deepseek/search_link_graph_deepseek.py‎

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
"""
2+
Example of Search Graph
3+
"""
4+
import os
5+
from dotenv import load_dotenv
6+
from scrapegraphai.graphs import SearchGraph
7+
from scrapegraphai.utils import convert_to_csv, convert_to_json, prettify_exec_info
8+
9+
# ************************************************
10+
# Define the configuration for the graph
11+
# ************************************************
12+
13+
load_dotenv()
14+
15+
deepseek_key = os.getenv("DEEPSEEK_APIKEY")
16+
17+
graph_config = {
18+
"llm": {
19+
"model": "deepseek-chat",
20+
"openai_api_key": deepseek_key,
21+
"openai_api_base": 'https://api.deepseek.com/v1',
22+
},
23+
"embeddings": {
24+
"model": "ollama/nomic-embed-text",
25+
"temperature": 0,
26+
# "base_url": "http://localhost:11434", # set ollama URL arbitrarily
27+
},
28+
"verbose": True,
29+
}
30+
31+
# ************************************************
32+
# Create the SearchGraph instance and run it
33+
# ************************************************
34+
35+
search_graph = SearchGraph(
36+
prompt="List me the best escursions near Trento",
37+
config=graph_config
38+
)
39+
40+
result = search_graph.run()
41+
print(result)
42+
43+
# ************************************************
44+
# Get graph execution info
45+
# ************************************************
46+
47+
graph_exec_info = search_graph.get_execution_info()
48+
print(prettify_exec_info(graph_exec_info))
49+
50+
# Save to json and csv
51+
convert_to_csv(result, "result")
52+
convert_to_json(result, "result")

‎examples/ernie/search_graph_ernie.py‎

Lines changed: 10 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -12,15 +12,18 @@
1212
# Define the configuration for the graph
1313
# ************************************************
1414

15-
openai_key = os.getenv("OPENAI_APIKEY")
16-
1715
graph_config = {
1816
"llm": {
19-
"api_key": openai_key,
20-
"model": "gpt-3.5-turbo",
21-
},
22-
"max_results": 2,
23-
"verbose": True,
17+
"model": "ernie-bot-turbo",
18+
"ernie_client_id": "<ernie_client_id>",
19+
"ernie_client_secret": "<ernie_client_secret>",
20+
"temperature": 0.1
21+
},
22+
"embeddings": {
23+
"model": "ollama/nomic-embed-text",
24+
"temperature": 0,
25+
"base_url": "http://localhost:11434"},
26+
"library": "beautifulsoup"
2427
}
2528

2629
# ************************************************

‎examples/ernie/search_link_graph_ernie.py‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
"""
2+
Example of Search Graph
3+
"""
4+
from scrapegraphai.graphs import SearchGraph
5+
from scrapegraphai.utils import convert_to_csv, convert_to_json, prettify_exec_info
6+
7+
# ************************************************
8+
# Define the configuration for the graph
9+
# ************************************************
10+
11+
graph_config = {
12+
"llm": {
13+
"model": "ernie-bot-turbo",
14+
"ernie_client_id": "<ernie_client_id>",
15+
"ernie_client_secret": "<ernie_client_secret>",
16+
"temperature": 0.1
17+
},
18+
"embeddings": {
19+
"model": "ollama/nomic-embed-text",
20+
"temperature": 0,
21+
"base_url": "http://localhost:11434"},
22+
"library": "beautifulsoup"
23+
}
24+
25+
# ************************************************
26+
# Create the SearchGraph instance and run it
27+
# ************************************************
28+
29+
search_graph = SearchGraph(
30+
prompt="List me the best escursions near Trento",
31+
config=graph_config
32+
)
33+
34+
result = search_graph.run()
35+
print(result)
36+
37+
# ************************************************
38+
# Get graph execution info
39+
# ************************************************
40+
41+
graph_exec_info = search_graph.get_execution_info()
42+
print(prettify_exec_info(graph_exec_info))
43+
44+
# Save to json and csv
45+
convert_to_csv(result, "result")
46+
convert_to_json(result, "result")

0 commit comments

Comments
 (0)