![\"Open](\"https://colab.research.google.com/assets/colab-badge.svg\"){kind=icon}
\n",
+ "\n",
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "jEkuKbcRrPcK"
+ },
+ "source": [
+ "## 🕷️ Research Agent with `scrapegraph`, `langgraph`, `tavily` and `MySQL`"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ ""
+ ],
+ "metadata": {
+ "id": "-GmCGaVI3kcT"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "IzsyDXEWwPVt"
+ },
+ "source": [
+ "### 🔧 Install `dependencies`"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "metadata": {
+ "id": "os_vm0MkIxr9"
+ },
+ "outputs": [],
+ "source": [
+ "%%capture\n",
+ "!pip install langgraph langchain-scrapegraph langchain-openai \"langchain-community>=0.2.11\" tavily-python"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "apBsL-L2KzM7"
+ },
+ "source": [
+ "### 🔑 Import `ScrapeGraph`, `Tavily` and `OpenAI` API keys"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "ol9gQbAFkh9b"
+ },
+ "source": [
+ "You can find the Scrapegraph API key [here](https://dashboard.scrapegraphai.com/)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "sffqFG2EJ8bI",
+ "outputId": "132bdfb9-45c8-4c26-8ac2-d13593ba392a"
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Scrapegraph API key:\n",
+ "··········\n",
+ "Tavily API key:\n",
+ "··········\n",
+ "OpenAI API key:\n",
+ "··········\n"
+ ]
+ }
+ ],
+ "source": [
+ "import getpass\n",
+ "import os\n",
+ "\n",
+ "if not os.environ.get(\"SGAI_API_KEY\"):\n",
+ " os.environ[\"SGAI_API_KEY\"] = getpass.getpass(\"Scrapegraph API key:\\n\")\n",
+ "\n",
+ "if not os.environ.get(\"TAVILY_API_KEY\"):\n",
+ " os.environ[\"TAVILY_API_KEY\"] = getpass.getpass(\"Tavily API key:\\n\")\n",
+ "\n",
+ "if not os.environ.get(\"OPENAI_API_KEY\"):\n",
+ " os.environ[\"OPENAI_API_KEY\"] = getpass.getpass(\"OpenAI API key:\\n\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "jnqMB2-xVYQ7"
+ },
+ "source": [
+ "### 📝 Defining an `Output Schema` for Webpage Content Extraction\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "VZvxbjfXvbgd"
+ },
+ "source": [
+ "If you already know what you want to extract from a webpage, you can **define an output schema** using **Pydantic**. This schema acts as a \"blueprint\" that tells the AI how to structure the response.\n",
+ "\n",
+ "| \n", + " | title | \n", + "link | \n", + "description | \n", + "
|---|---|---|---|
| 0 | \n", + "CES 2025 day 3: the 11 best gadgets we've seen... | \n", + "https://www.techradar.com/tech-events/ces-2025... | \n", + "A roundup of the top gadgets showcased on the ... | \n", + "
| 1 | \n", + "CES 2025 proves AI is everywhere, unstoppable,... | \n", + "https://www.techradar.com/computing/artificial... | \n", + "An exploration of the pervasive presence of AI... | \n", + "
| 2 | \n", + "Forget tri-folding phones - this pocketable fo... | \n", + "https://www.techradar.com/televisions/projecto... | \n", + "A review of a new portable folding projector t... | \n", + "
| 3 | \n", + "HDMI 2.2 officially revealed at CES 2025: here... | \n", + "https://www.techradar.com/tech/hdmi-2-2-offici... | \n", + "Details on the announcement of HDMI 2.2 and it... | \n", + "
| 4 | \n", + "This awesome E Ink poster frame looks like the... | \n", + "https://www.techradar.com/home/smart-home/this... | \n", + "A look at a new E Ink poster frame that repres... | \n", + "
| 5 | \n", + "I saw Hisense's RGB mini-LED TV, and this vivi... | \n", + "https://www.techradar.com/televisions/i-saw-hi... | \n", + "An impression of Hisense's new RGB mini-LED TV... | \n", + "
| 6 | \n", + "I saw Samsung's new The Frame Pro mini-LED TV,... | \n", + "https://www.techradar.com/televisions/i-saw-sa... | \n", + "Samsung's The Frame Pro mini-LED TV has been s... | \n", + "
| 7 | \n", + "This robot vacuum can also bring you a sandwic... | \n", + "https://www.techradar.com/tech-events/this-rob... | \n", + "The SwitchBot K20+ Pro is highlighted as a sta... | \n", + "
| 8 | \n", + "Nvidia unveils new GeForce RTX 5090, RTX 5080,... | \n", + "https://www.techradar.com/computing/gpu/nvidia... | \n", + "Nvidia has revealed its latest RTX 5000 series... | \n", + "
| 9 | \n", + "I heard LG's new speakers made with will.i.am,... | \n", + "https://www.techradar.com/audio/wireless-bluet... | \n", + "LG's new range of Bluetooth speakers, develope... | \n", + "
| 10 | \n", + "The new Garmin Instinct 3 had me climbing a wa... | \n", + "https://www.techradar.com/health-fitness/the-n... | \n", + "The Garmin Instinct 3 smartwatch, featuring a ... | \n", + "
| 11 | \n", + "CES 2025 day 1: featuring the 11 best gadgets ... | \n", + "https://www.techradar.com/tech-events/the-best... | \n", + "A roundup of the best gadgets from the first d... | \n", + "
| 12 | \n", + "CES 2025 day 1: the 11 best gadgets we’ve seen... | \n", + "https://www.techradar.com/tech-events/the-best... | \n", + "A roundup of the best gadgets showcased on the... | \n", + "
| 13 | \n", + "I tried the world's first interactive The Last... | \n", + "https://www.techradar.com/tech/i-tried-the-wor... | \n", + "An immersive experience based on the popular g... | \n", + "
| 14 | \n", + "This adorable little robot cat will stop you b... | \n", + "https://www.techradar.com/home/coffee-machines... | \n", + "A cute gadget from Yukai Engineering that cool... | \n", + "
| 15 | \n", + "Forget bird watching, bug watching could be th... | \n", + "https://www.techradar.com/home/smart-home/forg... | \n", + "New tech from Wonder that allows users to obse... | \n", + "
| 16 | \n", + "Philips Hue now works with the latest LG TVs, ... | \n", + "https://www.techradar.com/televisions/philips-... | \n", + "Exciting news about the Philips Hue Sync TV ap... | \n", + "
| 17 | \n", + "Lenovo's new Legion Go S gaming handheld might... | \n", + "https://www.techradar.com/computing/lenovos-ne... | \n", + "Lenovo introduces a gaming handheld that runs ... | \n", + "
| 18 | \n", + "JBL's new elite headphones include a touchscre... | \n", + "https://www.techradar.com/audio/wireless-headp... | \n", + "JBL has unveiled its flagship Tour One M3 head... | \n", + "
| 19 | \n", + "Technics' intriguing new flagship wireless ear... | \n", + "https://www.techradar.com/audio/earbuds-airpod... | \n", + "Technics has launched the EAH-AZ100 earbuds at... | \n", + "
| 20 | \n", + "Victrola's new turntable with Auracast and apt... | \n", + "https://www.techradar.com/audio/turntables/vic... | \n", + "Victrola has introduced the Wave turntable at ... | \n", + "
| 21 | \n", + "CES 2025 day 2: the 11 coolest gadgets we've s... | \n", + "https://www.techradar.com/tech-events/the-best... | \n", + "A roundup of standout gadgets from the second ... | \n", + "
| 22 | \n", + "Some of CES 2025's weirdest robot pets are abs... | \n", + "https://www.techradar.com/tech/some-of-ces-202... | \n", + "TechRadar highlights a significant increase in... | \n", + "
| 23 | \n", + "Suunto's new waterproof headphones can store 8... | \n", + "https://www.techradar.com/health-fitness/suunt... | \n", + "Suunto introduces waterproof headphones that u... | \n", + "
| 24 | \n", + "Lenovo unveils world's first rollable display ... | \n", + "https://www.techradar.com/computing/lenovo-unv... | \n", + "Lenovo confirms the launch of the ThinkBook Pl... | \n", + "
| 25 | \n", + "Samsung's massive 115-inch mini-LED TV is its ... | \n", + "https://www.techradar.com/televisions/samsungs... | \n", + "Samsung showcases its 115-inch QN90F TV, which... | \n", + "
| 26 | \n", + "This smart-home control button might look like... | \n", + "https://www.techradar.com/home/smart-home/this... | \n", + "The Flic Duo is a versatile smart button that ... | \n", + "
| 27 | \n", + "Hisense reveals a 163-inch micro-LED TV at CES... | \n", + "https://www.techradar.com/televisions/hisense-... | \n", + "Hisense introduces a 163-inch version of its T... | \n", + "
| 28 | \n", + "I witnessed Hisense's giant 163-inch micro-LED... | \n", + "https://www.techradar.com/televisions/i-witnes... | \n", + "A firsthand account of Hisense's impressive 16... | \n", + "
| 29 | \n", + "Delta Air Lines announces generative AI assist... | \n", + "No content available | \n", + "Delta Air Lines revealed a generative AI assis... | \n", + "
| 30 | \n", + "These new video doorbell and security camera u... | \n", + "https://www.techradar.com/home/smart-home/thes... | \n", + "The Swann Xtreem4K security camera utilizes AI... | \n", + "
| 31 | \n", + "The best wearable and fitness tech of CES 2025... | \n", + "https://www.techradar.com/health-fitness/the-b... | \n", + "A roundup of the top wearable and health tech ... | \n", + "
| 32 | \n", + "This awesome E Ink poster frame looks like the... | \n", + "https://www.techradar.com/home/smart-home/this... | \n", + "PocketBook's new E Ink digital art frames offe... | \n", + "
| 33 | \n", + "Smart-lighting brand Nanoleaf unveils LED Ligh... | \n", + "https://www.techradar.com/health-fitness/smart... | \n", + "Nanoleaf introduced an FDA-certified LED light... | \n", + "
| 34 | \n", + "The coolest smart home tech of CES 2025 | \n", + "No content available | \n", + "A guide to the most innovative smart home gadg... | \n", + "
| 35 | \n", + "I've been smart home hunting at CES 2025, and ... | \n", + "https://www.techradar.com/home/smart-home/ive-... | \n", + "This article highlights the latest multi-taski... | \n", + "
| 36 | \n", + "Goodbye, buttons - BMW's new Panoramic iDrive ... | \n", + "https://www.techradar.com/vehicle-tech/hybrid-... | \n", + "BMW unveiled its new iDrive system at CES 2025... | \n", + "
| 37 | \n", + "Levoit's new pet-parent air purifier will swit... | \n", + "https://www.techradar.com/home/small-appliance... | \n", + "The Levoit Pet Odor & Hair Air Purifier is des... | \n", + "
| 38 | \n", + "Nanoleaf's Ambilight alternative is now even e... | \n", + "https://www.techradar.com/home/smart-lights/na... | \n", + "Nanoleaf has released an updated version of it... | \n", + "
| 39 | \n", + "This coffee machine brews espresso using water... | \n", + "https://www.techradar.com/home/coffee-machines... | \n", + "The KaraPod espresso machine uses water conden... | \n", + "
| 40 | \n", + "This Batman-style location tracker brings sate... | \n", + "https://www.techradar.com/phones/phone-accesso... | \n", + "HMD's OffGrid device provides off-grid connect... | \n", + "
| 41 | \n", + "This might be the coolest iPhone accessory at ... | \n", + "https://www.techradar.com/phones/phone-accesso... | \n", + "Belkin's new Stage PowerGrip is a magnetic pow... | \n", + "
| 42 | \n", + "Eat your heart out Daleks - Dreame's newest ro... | \n", + "https://www.techradar.com/home/robot-vacuums/e... | \n", + "The Dreame X50 Ultra Complete can climb over o... | \n", + "
| 43 | \n", + "Narwal's new robovac will moonwalk your floors... | \n", + "https://www.techradar.com/home/robot-vacuums/n... | \n", + "The Narwal Flow features a unique cleaning met... | \n", + "
| 44 | \n", + "Denon's 22-speaker EV audio system promises he... | \n", + "https://www.techradar.com/audio/denons-22-spea... | \n", + "Denon's new audio system for electric vehicles... | \n", + "
| 45 | \n", + "Koorui unleashes the world's fastest monitor | \n", + "https://www.techradar.com/computing/gpu/how-hi... | \n", + "The Koorui G7 monitor features a 750Hz refresh... | \n", + "
| 46 | \n", + "A 750Hz gaming monitor has come out of nowhere... | \n", + "https://www.techradar.com/computing/monitors/a... | \n", + "Discussion on the emergence of a 750Hz gaming ... | \n", + "
| 47 | \n", + "HDMI 2.2 officially revealed at CES 2025: here... | \n", + "https://www.techradar.com/tech/hdmi-2-2-offici... | \n", + "Announcement of HDMI 2.2, which supports resol... | \n", + "
| 48 | \n", + "CES 2025 day 3: the 11 best gadgets we've seen... | \n", + "https://www.techradar.com/tech-events/ces-2025... | \n", + "A recap of the best gadgets showcased on Day 3... | \n", + "
| 49 | \n", + "TCL's ultra-affordable 60XE uses NXTpaper to p... | \n", + "https://www.techradar.com/phones/tcls-ultra-af... | \n", + "The TCL 60XE smartphone features NXTpaper tech... | \n", + "
| 50 | \n", + "Want to boost your iPhone storage by 2TB? SanD... | \n", + "https://www.techradar.com/phones/want-to-boost... | \n", + "SanDisk and ShiftCam introduced external SSDs ... | \n", + "
| 51 | \n", + "I tried Panasonic's new Z95B OLED TV, and it t... | \n", + "https://www.techradar.com/televisions/i-tried-... | \n", + "Panasonic's flagship Z95B OLED TV is highlight... | \n", + "
| 52 | \n", + "Our strangest CES 2025 finds | \n", + "https://www.tiktok.com/@techradar?lang=en | \n", + "A roundup of unusual gadgets from CES 2025, in... | \n", + "
| 53 | \n", + "The RGB micro-LED backlight wars begin | \n", + "https://www.techradar.com/televisions/samsung-... | \n", + "Samsung showcased an 8K RGB micro-LED TV at CE... | \n", + "
| 54 | \n", + "Delta Air Lines had the most impressive CES sh... | \n", + "https://www.techradar.com/tech-events/delta-ai... | \n", + "Delta Air Lines showcased an extravagant prese... | \n", + "
| 55 | \n", + "Good news for some Disney Plus subscribers | \n", + "https://www.techradar.com/streaming/own-a-sams... | \n", + "Disney Plus will soon be available in HDR10+ f... | \n", + "
| 56 | \n", + "It’s here - our top 25 gadgets of CES 2025 | \n", + "https://www.techradar.com/tech/best-of-ces-2025 | \n", + "TechRadar has compiled a list of the top 25 ga... | \n", + "
| 57 | \n", + "It’s time to meet the CES 2025 robots | \n", + "No content available | \n", + "CES 2025 featured advanced humanoid robots, in... | \n", + "
| 58 | \n", + "The 11 most exciting tech trends of 2025, acco... | \n", + "https://www.techradar.com/tech-events/the-11-m... | \n", + "This article discusses the most exciting techn... | \n", + "
| 59 | \n", + "LG's most stunning OLED TV of all time made a ... | \n", + "https://www.techradar.com/televisions/lgs-most... | \n", + "The article covers the unveiling of LG's lates... | \n", + "
\n",
+ " ![\"ScrapeGraph](\"https://raw.githubusercontent.com/VinciGit00/Scrapegraph-ai/main/docs/assets/api-banner.png\"\n",)
\n",
+ "