Skip to content

Commit 7d7d83a

Browse files
authored
docs: add Make apps documentation (#1540)
add separate documents for ig,fb,amazon,tiktok & yt add screenshots reformat the docs to fit docs style adjust prose to fit the style guide remake make.com integration page into category
1 parent 45da62e commit 7d7d83a

32 files changed

+1748
-13
lines changed
Lines changed: 153 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,153 @@
1+
---
2+
title: Make - AI crawling Actor integration
3+
description: Learn about AI Crawling scraper modules.
4+
sidebar_label: AI Crawling
5+
sidebar_position: 6
6+
slug: /integrations/make/ai-crawling
7+
toc_max_heading_level: 4
8+
---
9+
10+
## Apify Scraper for AI Crawling
11+
12+
Apify Scraper for AI Crawling from [Apify](https://apify.com/) lets you extract text content from websites to feed AI models, LLM applications, vector databases, or Retrieval Augmented Generation(RAG) pipelines. It supports rich formatting using Markdown, cleans the HTML of irrelevant elements, downloads linked files, and integrates with AI ecosystems like LangChain, LlamaIndex, and other LLM frameworks.
13+
14+
To use these modules, you need an [Apify account](https://console.apify.com) and an [API token](https://docs.apify.com/platform/integrations/api#api-token). You can find your token in the [Apify Console](https://console.apify.com/) under **Settings > Integrations**. After connecting, you can automate content extraction at scale and incorporate the results into your AI workflows.
15+
16+
## Connect Apify Scraper for AI Crawling
17+
18+
1. Create an account at [Apify](https://console.apify.com/). You can sign up using your email, Gmail, or GitHub account.
19+
20+
![Sign up page](images/ai-crawling/wcc-signup.png)
21+
22+
1. To connect your Apify account to Make, you can use an OAuth connection (recommended) or an Apify API token. To get the Apify API token, navigate to **[Settings > API & Integrations](https://console.apify.com/settings/integrations)** in the Apify Console.
23+
24+
![Apify Console token for Make.png](images/Apify_Console_token_for_Make.png)
25+
26+
1. Find your token under **Personal API tokens** section. You can also create a new API token with multiple customizable permissions by clicking on **+ Create a new token**.
27+
1. Click the **Copy** icon next to your API token to copy it to your clipboard. Then, return to your Make scenario interface.
28+
29+
![Apify token on Make.png](images/Apify_token_on_Make.png)
30+
31+
1. In Make, click **Add** to open the **Create a connection** dialog of the chosen Apify Scraper module.
32+
1. In the **API token** field, paste the API token you copied from Apify. Provide a clear **Connection name**, and click **Save**.
33+
34+
![Make API token](images/ai-crawling/image%201.png)
35+
36+
Once connected, you can build workflows to automate website extraction and integrate results into your AI applications.
37+
38+
## Apify Scraper for Website Content modules
39+
40+
After connecting the app, you can use one of the two modules as native scrapers to extract website content.
41+
42+
### Standard Settings Module
43+
44+
The Standard Settings module is a streamlined component of the Website Content Crawler that allows you to quickly extract content from websites using optimized default settings. This module is perfect for extracting content from blogs, documentation sites, knowledge bases, or any text-rich website to feed into AI models.
45+
46+
#### How it works
47+
48+
The crawler starts with one or more **Start URLs** you provide, typically the top-level URL of a documentation site, blog, or knowledge base. It then:
49+
50+
- Crawls these start URLs
51+
- Finds links to other pages on the site
52+
- Recursively crawls those pages as long as their URL is under the start URL
53+
- Respects URL patterns for inclusion/exclusion
54+
- Automatically skips duplicate pages with the same canonical URL
55+
- Provides various settings to customize crawling behavior (crawler type, max pages, depth, concurrency, etc.)
56+
57+
Once a web page is loaded, the Actor processes its HTML to ensure quality content extraction:
58+
59+
- Waits for dynamic content to load if using a headless browser
60+
- Can scroll to a certain height to ensure all page content is loaded
61+
- Can expand clickable elements to reveal hidden content
62+
- Removes DOM nodes matching specific CSS selectors (like navigation, headers, footers)
63+
- Optionally keeps only content matching specific CSS selectors
64+
- Removes cookie warnings using browser extensions
65+
- Transforms the page using the selected HTML transformer to extract the main content
66+
67+
#### Output data
68+
69+
For each crawled web page, you'll receive:
70+
71+
- _Page metadata_: URL, title, description, canonical URL
72+
- _Cleaned text content_: The main article content with irrelevant elements removed
73+
- _Markdown formatting_: Structured content with headers, lists, links, and other formatting preserved
74+
- _Crawl information_: Loaded URL, referrer URL, timestamp, HTTP status
75+
- _Optional file downloads_: PDFs, DOCs, and other linked documents
76+
77+
```json title="Sample output (shortened)"
78+
{
79+
"url": "https://docs.apify.com/academy/web-scraping-for-beginners",
80+
"crawl": {
81+
"loadedUrl": "https://docs.apify.com/academy/web-scraping-for-beginners",
82+
"loadedTime": "2025-04-22T14:33:20.514Z",
83+
"referrerUrl": "https://docs.apify.com/academy",
84+
"depth": 1,
85+
"httpStatusCode": 200
86+
},
87+
"metadata": {
88+
"canonicalUrl": "https://docs.apify.com/academy/web-scraping-for-beginners",
89+
"title": "Web scraping for beginners | Apify Documentation",
90+
"description": "Learn the basics of web scraping with a step-by-step tutorial and practical exercises.",
91+
"languageCode": "en",
92+
"markdown": "# Web scraping for beginners\n\nWelcome to our comprehensive web scraping tutorial for beginners. This guide will take you through the fundamentals of extracting data from websites, with practical examples and exercises.\n\n## What is web scraping?\n\nWeb scraping is the process of extracting data from websites. It involves making HTTP requests to web servers, downloading HTML pages, and parsing them to extract the desired information.\n\n## Why learn web scraping?\n\n- **Data collection**: Gather information for research, analysis, or business intelligence\n- **Automation**: Save time by automating repetitive data collection tasks\n- **Integration**: Connect web data with your applications or databases\n- **Monitoring**: Track changes on websites automatically\n\n## Getting started\n\nTo begin web scraping, you'll need to understand the basics of HTML, CSS selectors, and HTTP. This tutorial will guide you through these concepts step by step.\n\n...",
93+
"text": "Web scraping for beginners\n\nWelcome to our comprehensive web scraping tutorial for beginners. This guide will take you through the fundamentals of extracting data from websites, with practical examples and exercises.\n\nWhat is web scraping?\n\nWeb scraping is the process of extracting data from websites. It involves making HTTP requests to web servers, downloading HTML pages, and parsing them to extract the desired information.\n\nWhy learn web scraping?\n\n- Data collection: Gather information for research, analysis, or business intelligence\n- Automation: Save time by automating repetitive data collection tasks\n- Integration: Connect web data with your applications or databases\n- Monitoring: Track changes on websites automatically\n\nGetting started\n\nTo begin web scraping, you'll need to understand the basics of HTML, CSS selectors, and HTTP. This tutorial will guide you through these concepts step by step.\n\n..."
94+
}
95+
}
96+
```
97+
98+
### Advanced Settings Module
99+
100+
The Advanced Settings module provides complete control over the content extraction process, allowing you to fine-tune every aspect of the crawling and transformation pipeline. This module is ideal for complex websites, JavaScript-heavy applications, or when you need precise control over content extraction.
101+
102+
#### Key features
103+
104+
- _Multiple Crawler Options_: Choose between headless browsers (Playwright) or faster HTTP clients (Cheerio)
105+
- _Custom Content Selection_: Specify exactly which elements to keep or remove
106+
- _Advanced Navigation Control_: Set crawling depth, scope, and URL patterns
107+
- _Dynamic Content Handling_: Wait for JavaScript-rendered content to load
108+
- _Interactive Element Support_: Click expandable sections to reveal hidden content
109+
- _Multiple Output Formats_: Save content as Markdown, HTML, or plain text
110+
- _Proxy Configuration_: Use proxies to handle geo-restrictions or avoid IP blocks
111+
- _Content Transformation Options_: Multiple algorithms for optimal content extraction
112+
113+
#### How it works
114+
115+
The Advanced Settings module provides granular control over the entire crawling process:
116+
117+
1. _Crawler Selection_: Choose from Playwright (Firefox/Chrome), or Cheerio based on website complexity
118+
2. _URL Management_: Define precise scoping with include/exclude URL patterns
119+
3. _DOM Manipulation_: Control which HTML elements to keep or remove
120+
4. _Content Transformation_: Apply specialized algorithms for content extraction
121+
5. _Output Formatting_: Select from multiple formats for AI model compatibility
122+
123+
#### Configuration options
124+
125+
Advanced Settings offers numerous configuration options, including:
126+
127+
- _Crawler Type_: Select the rendering engine (browser or HTTP client)
128+
- _Content Extraction Algorithm_: Choose from multiple HTML transformers
129+
- _Element Selectors_: Specify which elements to keep, remove, or click
130+
- _URL Patterns_: Define URL inclusion/exclusion patterns with glob syntax
131+
- _Crawling Parameters_: Set concurrency, depth, timeouts, and retries
132+
- _Proxy Configuration_: Configure proxy settings for robust crawling
133+
- _Output Options_: Select content formats and storage options
134+
135+
#### Output data
136+
137+
In addition to the standard output fields, Advanced Settings provides:
138+
139+
- _Multiple Format Options_: Content in Markdown, HTML, or plain text
140+
- _Debug Information_: Detailed extraction diagnostics and snapshots
141+
- _HTML Transformations_: Results from different content extraction algorithms
142+
- _File Storage Options_: Flexible storage for HTML, screenshots, or downloaded files
143+
144+
Looking for more than just AI crawling? You can use other native Make apps powered by Apify:
145+
146+
- [Instagram Data](/platform/integrations/make/instagram)
147+
- [TikTok Data](/platform/integrations/make/tiktok)
148+
- [Google Search](/platform/integrations/make/search)
149+
- [Google Maps Emails Data](/platform/integrations/make/maps)
150+
- [YouTube Data](/platform/integrations/make/youtube)
151+
- [Amazon](/platform/integrations/make/amazon)
152+
153+
And more! Because you can access any of our 4,500+ scrapers on Apify Store by using the [general Apify connections](https://www.make.com/en/integrations/apify).

0 commit comments

Comments
 (0)