|
| 1 | +--- |
| 2 | +title: Make - AI crawling Actor integration |
| 3 | +description: Learn about AI Crawling scraper modules. |
| 4 | +sidebar_label: AI Crawling |
| 5 | +sidebar_position: 6 |
| 6 | +slug: /integrations/make/ai-crawling |
| 7 | +toc_max_heading_level: 4 |
| 8 | +--- |
| 9 | + |
| 10 | +## Apify Scraper for AI Crawling |
| 11 | + |
| 12 | +Apify Scraper for AI Crawling from [Apify](https://apify.com/) lets you extract text content from websites to feed AI models, LLM applications, vector databases, or Retrieval Augmented Generation(RAG) pipelines. It supports rich formatting using Markdown, cleans the HTML of irrelevant elements, downloads linked files, and integrates with AI ecosystems like LangChain, LlamaIndex, and other LLM frameworks. |
| 13 | + |
| 14 | +To use these modules, you need an [Apify account](https://console.apify.com) and an [API token](https://docs.apify.com/platform/integrations/api#api-token). You can find your token in the [Apify Console](https://console.apify.com/) under **Settings > Integrations**. After connecting, you can automate content extraction at scale and incorporate the results into your AI workflows. |
| 15 | + |
| 16 | +## Connect Apify Scraper for AI Crawling |
| 17 | + |
| 18 | +1. Create an account at [Apify](https://console.apify.com/). You can sign up using your email, Gmail, or GitHub account. |
| 19 | + |
| 20 | +  |
| 21 | + |
| 22 | +1. To connect your Apify account with Make, you need to get the Apify API token. In the Apify Console, navigate to **[Settings > API & Integrations](https://console.apify.com/settings/integrations)**. |
| 23 | + |
| 24 | +  |
| 25 | + |
| 26 | +1. Find your token under **Personal API tokens** section. You can also create a new API token with multiple customizable permissions by clicking on **+ Create a new token**. |
| 27 | +1. Click the **Copy** icon next to your API token to copy it to your clipboard. Then, return to your Make scenario interface. |
| 28 | + |
| 29 | +  |
| 30 | + |
| 31 | +1. In Make, click **Add** to open the **Create a connection** dialog of the chosen Apify Scraper module. |
| 32 | +1. In the **API token** field, paste the API token you copied from Apify. Provide a clear **Connection name**, and click **Save**. |
| 33 | + |
| 34 | +  |
| 35 | + |
| 36 | +Once connected, you can build workflows to automate website extraction and integrate results into your AI applications. |
| 37 | + |
| 38 | +## Apify Scraper for Website Content modules |
| 39 | + |
| 40 | +After connecting the app, you can use one of the two modules as native scrapers to extract website content. |
| 41 | + |
| 42 | +### Standard Settings Module |
| 43 | + |
| 44 | +The Standard Settings module is a streamlined component of the Website Content Crawler that allows you to quickly extract content from websites using optimized default settings. This module is perfect for extracting content from blogs, documentation sites, knowledge bases, or any text-rich website to feed into AI models. |
| 45 | + |
| 46 | +#### How it works |
| 47 | + |
| 48 | +The crawler starts with one or more **Start URLs** you provide, typically the top-level URL of a documentation site, blog, or knowledge base. It then: |
| 49 | + |
| 50 | +- Crawls these start URLs |
| 51 | +- Finds links to other pages on the site |
| 52 | +- Recursively crawls those pages as long as their URL is under the start URL |
| 53 | +- Respects URL patterns for inclusion/exclusion |
| 54 | +- Automatically skips duplicate pages with the same canonical URL |
| 55 | +- Provides various settings to customize crawling behavior (crawler type, max pages, depth, concurrency, etc.) |
| 56 | + |
| 57 | +Once a web page is loaded, the Actor processes its HTML to ensure quality content extraction: |
| 58 | + |
| 59 | +- Waits for dynamic content to load if using a headless browser |
| 60 | +- Can scroll to a certain height to ensure all page content is loaded |
| 61 | +- Can expand clickable elements to reveal hidden content |
| 62 | +- Removes DOM nodes matching specific CSS selectors (like navigation, headers, footers) |
| 63 | +- Optionally keeps only content matching specific CSS selectors |
| 64 | +- Removes cookie warnings using browser extensions |
| 65 | +- Transforms the page using the selected HTML transformer to extract the main content |
| 66 | + |
| 67 | +#### Output data |
| 68 | + |
| 69 | +For each crawled web page, you'll receive: |
| 70 | + |
| 71 | +- _Page metadata_: URL, title, description, canonical URL |
| 72 | +- _Cleaned text content_: The main article content with irrelevant elements removed |
| 73 | +- _Markdown formatting_: Structured content with headers, lists, links, and other formatting preserved |
| 74 | +- _Crawl information_: Loaded URL, referrer URL, timestamp, HTTP status |
| 75 | +- _Optional file downloads_: PDFs, DOCs, and other linked documents |
| 76 | + |
| 77 | +```json title="Sample output (shortened)" |
| 78 | +{ |
| 79 | + "url": "https://docs.apify.com/academy/web-scraping-for-beginners", |
| 80 | + "crawl": { |
| 81 | + "loadedUrl": "https://docs.apify.com/academy/web-scraping-for-beginners", |
| 82 | + "loadedTime": "2025-04-22T14:33:20.514Z", |
| 83 | + "referrerUrl": "https://docs.apify.com/academy", |
| 84 | + "depth": 1, |
| 85 | + "httpStatusCode": 200 |
| 86 | + }, |
| 87 | + "metadata": { |
| 88 | + "canonicalUrl": "https://docs.apify.com/academy/web-scraping-for-beginners", |
| 89 | + "title": "Web scraping for beginners | Apify Documentation", |
| 90 | + "description": "Learn the basics of web scraping with a step-by-step tutorial and practical exercises.", |
| 91 | + "languageCode": "en", |
| 92 | + "markdown": "# Web scraping for beginners\n\nWelcome to our comprehensive web scraping tutorial for beginners. This guide will take you through the fundamentals of extracting data from websites, with practical examples and exercises.\n\n## What is web scraping?\n\nWeb scraping is the process of extracting data from websites. It involves making HTTP requests to web servers, downloading HTML pages, and parsing them to extract the desired information.\n\n## Why learn web scraping?\n\n- **Data collection**: Gather information for research, analysis, or business intelligence\n- **Automation**: Save time by automating repetitive data collection tasks\n- **Integration**: Connect web data with your applications or databases\n- **Monitoring**: Track changes on websites automatically\n\n## Getting started\n\nTo begin web scraping, you'll need to understand the basics of HTML, CSS selectors, and HTTP. This tutorial will guide you through these concepts step by step.\n\n...", |
| 93 | + "text": "Web scraping for beginners\n\nWelcome to our comprehensive web scraping tutorial for beginners. This guide will take you through the fundamentals of extracting data from websites, with practical examples and exercises.\n\nWhat is web scraping?\n\nWeb scraping is the process of extracting data from websites. It involves making HTTP requests to web servers, downloading HTML pages, and parsing them to extract the desired information.\n\nWhy learn web scraping?\n\n- Data collection: Gather information for research, analysis, or business intelligence\n- Automation: Save time by automating repetitive data collection tasks\n- Integration: Connect web data with your applications or databases\n- Monitoring: Track changes on websites automatically\n\nGetting started\n\nTo begin web scraping, you'll need to understand the basics of HTML, CSS selectors, and HTTP. This tutorial will guide you through these concepts step by step.\n\n..." |
| 94 | + } |
| 95 | +} |
| 96 | +``` |
| 97 | + |
| 98 | +### Advanced Settings Module |
| 99 | + |
| 100 | +The Advanced Settings module provides complete control over the content extraction process, allowing you to fine-tune every aspect of the crawling and transformation pipeline. This module is ideal for complex websites, JavaScript-heavy applications, or when you need precise control over content extraction. |
| 101 | + |
| 102 | +#### Key features |
| 103 | + |
| 104 | +- _Multiple Crawler Options_: Choose between headless browsers (Playwright) or faster HTTP clients (Cheerio) |
| 105 | +- _Custom Content Selection_: Specify exactly which elements to keep or remove |
| 106 | +- _Advanced Navigation Control_: Set crawling depth, scope, and URL patterns |
| 107 | +- _Dynamic Content Handling_: Wait for JavaScript-rendered content to load |
| 108 | +- _Interactive Element Support_: Click expandable sections to reveal hidden content |
| 109 | +- _Multiple Output Formats_: Save content as Markdown, HTML, or plain text |
| 110 | +- _Proxy Configuration_: Use proxies to handle geo-restrictions or avoid IP blocks |
| 111 | +- _Content Transformation Options_: Multiple algorithms for optimal content extraction |
| 112 | + |
| 113 | +#### How it works |
| 114 | + |
| 115 | +The Advanced Settings module provides granular control over the entire crawling process: |
| 116 | + |
| 117 | +1. _Crawler Selection_: Choose from Playwright (Firefox/Chrome), or Cheerio based on website complexity |
| 118 | +2. _URL Management_: Define precise scoping with include/exclude URL patterns |
| 119 | +3. _DOM Manipulation_: Control which HTML elements to keep or remove |
| 120 | +4. _Content Transformation_: Apply specialized algorithms for content extraction |
| 121 | +5. _Output Formatting_: Select from multiple formats for AI model compatibility |
| 122 | + |
| 123 | +#### Configuration options |
| 124 | + |
| 125 | +Advanced Settings offers numerous configuration options, including: |
| 126 | + |
| 127 | +- _Crawler Type_: Select the rendering engine (browser or HTTP client) |
| 128 | +- _Content Extraction Algorithm_: Choose from multiple HTML transformers |
| 129 | +- _Element Selectors_: Specify which elements to keep, remove, or click |
| 130 | +- _URL Patterns_: Define URL inclusion/exclusion patterns with glob syntax |
| 131 | +- _Crawling Parameters_: Set concurrency, depth, timeouts, and retries |
| 132 | +- _Proxy Configuration_: Configure proxy settings for robust crawling |
| 133 | +- _Output Options_: Select content formats and storage options |
| 134 | + |
| 135 | +#### Output data |
| 136 | + |
| 137 | +In addition to the standard output fields, Advanced Settings provides: |
| 138 | + |
| 139 | +- _Multiple Format Options_: Content in Markdown, HTML, or plain text |
| 140 | +- _Debug Information_: Detailed extraction diagnostics and snapshots |
| 141 | +- _HTML Transformations_: Results from different content extraction algorithms |
| 142 | +- _File Storage Options_: Flexible storage for HTML, screenshots, or downloaded files |
| 143 | + |
| 144 | +Looking for more than just AI crawling? You can use other native Make apps powered by Apify: |
| 145 | + |
| 146 | +- [Instagram Data](platform/integrations/make/instagram) |
| 147 | +- [TikTok Data](platform/integration/make/tiktok) |
| 148 | +- [Google Maps Emails Data](platform/integrations/make/maps) |
| 149 | +- [YouTube Data](platform/integrations/make/youtube) |
| 150 | +- [Amazon](platform/integrations/make/amazon) |
| 151 | + |
| 152 | +And more! Because you can access any of our 4,500+ scrapers on Apify Store by using the [general Apify connections](https://www.make.com/en/integrations/apify). |
0 commit comments