Skip to content

Commit 1008549

Browse files
mstephen19mstephen19
authored
Merge pull request #450 from apify/new-content-migrations
docs: migrations
2 parents e5a5a65 + dd398a3 commit 1008549

File tree

2 files changed

+263
-0
lines changed

2 files changed

+263
-0
lines changed

‎content/academy/apify_platform/running_a_web_server.md

Lines changed: 229 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,229 @@
1+
---
2+
title: Running a web server on the Apify platform
3+
description: A web server running in an actor can act as a communication channel with the outside world. Learn how to easily set one up with Node.js.
4+
menuWeight: 5.4
5+
paths:
6+
- apify-platform/running-a-web-server
7+
---
8+
9+
# Running a web server on the Apify platform
10+
11+
Sometimes, an actor needs a channel for communication with other systems (or humans). This channel might be used to receive commands, to provide info about progress, or both. To implement this, we will run a HTTP web server inside the actor that will provide:
12+
13+
- An API to receive commands.
14+
- An HTML page displaying output data.
15+
16+
Running a web server in an actor is a piece of cake! Each actor run is available at a unique URL (container URL) which always takes the form `https://CONTAINER-KEY.runs.apify.net`. This URL is available in the [**actor run** object](https://docs.apify.com/api/v2#/reference/actor-runs/run-object-and-its-storages/get-run) returned by the Apify API, as well as in the Apify console.
17+
18+
If you start a web server on the port defined by the **APIFY_CONTAINER_PORT** environment variable (the default value is **4321**), the container URL becomes available and gets displayed in the **Live View** tab in the actor run console.
19+
20+
For more details, see [the documentation](https://docs.apify.com/actor/run#container-web-server).
21+
22+
## [](#building-the-actor) Building the actor
23+
24+
Let's try to build the following actor:
25+
26+
- The actor will provide an API to receive URLs to be processed.
27+
- For each URL, the actor will create a screenshot.
28+
- The screenshot will be stored in the key-value store.
29+
- The actor will provide a web page displaying thumbnails linked to screenshots and a HTML form to submit new URLs.
30+
31+
To achieve this we will use the following technologies:
32+
33+
- [Express.js](https://expressjs.com) framework to create the server
34+
- [Puppeteer](https://pptr.dev) to grab screenshots.
35+
- The [Apify SDK](https://sdk.apify.com) to access Apify storages to store the screenshots.
36+
37+
Our server needs two paths:
38+
39+
- `/` - Index path will display a page form to submit a new URL and the thumbnails of processed URLs.
40+
- `/add-url` - Will provide an API to add new URLs using a HTTP POST request.
41+
42+
First, we'll import `express` and create an Express.js app. Then, we'll add some middleware that will allow us to receive form submissions.
43+
44+
```JavaScript
45+
import Apify from 'apify';
46+
import express from 'express';
47+
48+
const app = express()
49+
50+
app.use(express.json());
51+
app.use(express.urlencoded({ extended: true }));
52+
```
53+
54+
Now we need to read the following environment variables:
55+
56+
- **APIFY_CONTAINER_PORT** contains a port number where we must start the server.
57+
- **APIFY_CONTAINER_URL** contains a URL under which we can access the container.
58+
- **APIFY_DEFAULT_KEY_VALUE_STORE_ID** is simply the ID of the default key-value store of this actor where we can store screenshots.
59+
60+
```JavaScript
61+
const {
62+
APIFY_CONTAINER_PORT,
63+
APIFY_CONTAINER_URL,
64+
APIFY_DEFAULT_KEY_VALUE_STORE_ID,
65+
} = process.env;
66+
```
67+
68+
Next, we'll create an array of the processed URLs where the **n**th URL has its screenshot stored under the key **n**.jpg in the key-value store.
69+
70+
```JavaScript
71+
const processedUrls = [];
72+
```
73+
74+
After that, the index route is ready to be defined.
75+
76+
```JavaScript
77+
app.get('/', (req, res) => {
78+
let listItems = '';
79+
80+
// For each of the processed
81+
processedUrls.forEach((url, index) => {
82+
const imageUrl = `https://api.apify.com/v2/key-value-stores/${APIFY_DEFAULT_KEY_VALUE_STORE_ID}/records/${index}.jpg`;
83+
84+
// Display the screenshots below the form
85+
listItems += `<li>
86+
<a href="${imageUrl}" target="_blank">
87+
<img src="${imageUrl}" width="300px" />
88+
<br />
89+
${url}
90+
</a>
91+
</li>`;
92+
});
93+
94+
const pageHtml = `<html>
95+
<head><title>Example</title></head>
96+
<body>
97+
<form method="POST" action="${APIFY_CONTAINER_URL}/add-url">
98+
URL: <input type="text" name="url" placeholder="http://example.com" />
99+
<input type="submit" value="Add" />
100+
<hr />
101+
<ul>${listItems}</ul>
102+
</form>
103+
</body>
104+
</html>`;
105+
106+
res.send(pageHtml);
107+
});
108+
```
109+
110+
And then the a second path that receives the new URL submitted using the HTML form; after the URL is processed, it redirects the user back to the root path.
111+
112+
```JavaScript
113+
app.post('/add-url', async (req, res) => {
114+
const { url } = req.body;
115+
console.log(`Got new URL: ${url}`);
116+
117+
// Start chrome browser and open new page ...
118+
const browser = await Apify.launchPuppeteer();
119+
const page = await browser.newPage();
120+
121+
// ... go to our URL and grab a screenshot ...
122+
await page.goto(url);
123+
const screenshot = await page.screenshot({ type: 'jpeg' });
124+
125+
// ... close browser ...
126+
await page.close();
127+
await browser.close();
128+
129+
// ... save screenshot to key-value store and add URL to processedUrls.
130+
await Apify.setValue(`${processedUrls.length}.jpg`, screenshot, { contentType: 'image/jpeg' });
131+
processedUrls.push(url);
132+
133+
res.redirect('/');
134+
});
135+
```
136+
137+
And finally we need to start the web server.
138+
139+
```JavaScript
140+
// Start the web server!
141+
app.listen(APIFY_CONTAINER_PORT, () => {
142+
console.log(`Application is listening at URL ${APIFY_CONTAINER_URL}.`);
143+
});
144+
```
145+
146+
### [](#final-code) Final code
147+
148+
```JavaScript
149+
import Apify from 'apify';
150+
import express from 'express';
151+
152+
const app = express()
153+
154+
app.use(express.json());
155+
app.use(express.urlencoded({ extended: true }));
156+
157+
const {
158+
APIFY_CONTAINER_PORT,
159+
APIFY_CONTAINER_URL,
160+
APIFY_DEFAULT_KEY_VALUE_STORE_ID,
161+
} = process.env;
162+
163+
const processedUrls = [];
164+
165+
app.get('/', (req, res) => {
166+
let listItems = '';
167+
168+
// For each of the processed
169+
processedUrls.forEach((url, index) => {
170+
const imageUrl = `https://api.apify.com/v2/key-value-stores/${APIFY_DEFAULT_KEY_VALUE_STORE_ID}/records/${index}.jpg`;
171+
172+
// Display the screenshots below the form
173+
listItems += `<li>
174+
<a href="${imageUrl}" target="_blank">
175+
<img src="${imageUrl}" width="300px" />
176+
<br />
177+
${url}
178+
</a>
179+
</li>`;
180+
});
181+
182+
const pageHtml = `<html>
183+
<head><title>Example</title></head>
184+
<body>
185+
<form method="POST" action="${APIFY_CONTAINER_URL}/add-url">
186+
URL: <input type="text" name="url" placeholder="http://example.com" />
187+
<input type="submit" value="Add" />
188+
<hr />
189+
<ul>${listItems}</ul>
190+
</form>
191+
</body>
192+
</html>`;
193+
194+
res.send(pageHtml);
195+
});
196+
197+
app.post('/add-url', async (req, res) => {
198+
const { url } = req.body;
199+
console.log(`Got new URL: ${url}`);
200+
201+
// Start chrome browser and open new page ...
202+
const browser = await Apify.launchPuppeteer();
203+
const page = await browser.newPage();
204+
205+
// ... go to our URL and grab a screenshot ...
206+
await page.goto(url);
207+
const screenshot = await page.screenshot({ type: 'jpeg' });
208+
209+
// ... close browser ...
210+
await page.close();
211+
await browser.close();
212+
213+
// ... save screenshot to key-value store and add URL to processedUrls.
214+
await Apify.setValue(`${processedUrls.length}.jpg`, screenshot, { contentType: 'image/jpeg' });
215+
processedUrls.push(url);
216+
217+
res.redirect('/');
218+
});
219+
220+
app.listen(APIFY_CONTAINER_PORT, () => {
221+
console.log(`Application is listening at URL ${APIFY_CONTAINER_URL}.`);
222+
});
223+
```
224+
225+
When we deploy and run this actor on the Apify platform, then we can open the **Live View** tab in the actor console to submit the URL to your actor through the form. After the URL is successfully submitted, it appears in the actor log.
226+
227+
With that we're done! And our application works like a charm :)
228+
229+
The complete code of this actor is available [here](https://www.apify.com/apify/example-web-server). You can run it there or copy it to your account.

‎content/academy/choosing_the_right_scraper.md

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
---
2+
title: How to choose the right scraper for the job
3+
description: Understand how to choose the best scraper for your use-case by understanding some basic concepts.
4+
menuWeight: 20
5+
category: tutorials
6+
paths:
7+
- choosing-the-right-scraper
8+
---
9+
10+
# [](#choosing-the-right-scraper) Choosing the right scraper for the job
11+
12+
There are two main ways you can proceed with building your crawler:
13+
14+
1. Using plain HTTP requests.
15+
2. Using an automated browser.
16+
17+
We will briefly go through the pros and cons of both, and also will cover the basic steps on how to determine which one should you go with.
18+
19+
## [](#performance) Performance
20+
21+
First, let's discuss performance. Plain HTTP request-based scraping will **always** be faster than browser-based scraping. When using plain requests, the page's HTML is not rendered, no JavaScript is executed, no images are loaded, etc. Also, there's no memory used by the browser, and there are no CPU-hungry operations.
22+
23+
If it were only a question of performance, you'd of course use request-based scraping every time; however, it's unfortunately not that simple.
24+
25+
## [](#dynamic-pages) Dynamic pages & blocking
26+
27+
Some websites do not load any data without a browser, as they need to execute some scripts to show it (these are known as [dynamic pages]({{@link dealing_with_dynamic_pages.md}})). Another problem is blocking. If the website is collecting a [browser fingerprint]({{@link anti_scraping/techniques/fingerprinting.md}}), it is very easy for it to distinguish between a real user and a bot (crawler) and block access.
28+
29+
## [](#making-the-choice) Making the choice
30+
31+
When choosing which scraper to use, we would suggest first checking whether the website works without JavaScript or not. Probably the easiest way to do so is to use the [Quick Javascript Switcher]({{@link tools/quick_javascript_switcher.md}}) extension for Chrome. If JavaScript is not needed, or you've spotted some XHR requests in the **Network** tab with the data you need, you probably won't need to use an automated browser browser. You can then check what data is received in response using [Postman]({{@link tools/postman.md}}) or [Insomnia]({{@link tools/insomnia.md}}) or try to sending a few requests programmatically. If the data is there and you're not blocked straight away, a request-based scraper is probably the way to go.
32+
33+
It also depends of course on whether you need to fill in some data (like a username and password) or select a location (such as entering zip code manually). Tasks where interacting with the page is absolutely necessary cannot be done using plain HTTP scraping, and require headless browsers. In some cases, you might also decide to use a browser-based solution in order to better blend in with the rest of the "regular" traffic coming from real users.
34+

0 commit comments

Comments
 (0)