Skip to content

Commit afdfee2

Browse files
mstephen19mstephen19
committed
feat(running-a-web-server)
1 parent 3df9bde commit afdfee2

File tree

1 file changed

+229
-0
lines changed

1 file changed

+229
-0
lines changed

‎content/academy/apify_platform/running_a_web_server.md

Lines changed: 229 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,229 @@
1+
---
2+
title: Running a web server on the Apify platform
3+
description: A web server running in an actor can act as a communication channel with the outside world. Learn how to easily set one up with Node.js.
4+
menuWeight: 5.4
5+
paths:
6+
- apify-platform/running-a-web-server
7+
---
8+
9+
# Running a web server on the Apify platform
10+
11+
Sometimes, an actor needs a channel for communication with other systems (or humans). This channel might be used to receive commands, to provide info about progress, or both. To implement this, we will run a HTTP web server inside the actor that will provide:
12+
13+
- An API to receive commands.
14+
- An HTML page displaying output data.
15+
16+
Running a web server in an actor is a piece of cake! Each actor run is available at a unique URL (container URL) which always takes the form `https://CONTAINER-KEY.runs.apify.net`. This URL is available in the [**actor run** object](https://docs.apify.com/api/v2#/reference/actor-runs/run-object-and-its-storages/get-run) returned by the Apify API, as well as in the Apify console.
17+
18+
If you start a web server on the port defined by the **APIFY_CONTAINER_PORT** environment variable (the default value is **4321**), the container URL becomes available and gets displayed in the **Live View** tab in the actor run console.
19+
20+
For more details, see [the documentation](https://docs.apify.com/actor/run#container-web-server).
21+
22+
## [](#building-the-actor) Building the actor
23+
24+
Let's try to build the following actor:
25+
26+
- The actor will provide an API to receive URLs to be processed.
27+
- For each URL, the actor will create a screenshot.
28+
- The screenshot will be stored in the key-value store.
29+
- The actor will provide a web page displaying thumbnails linked to screenshots and a HTML form to submit new URLs.
30+
31+
To achieve this we will use the following technologies:
32+
33+
- [Express.js](https://expressjs.com) framework to create the server
34+
- [Puppeteer](https://pptr.dev) to grab screenshots.
35+
- The [Apify SDK](https://sdk.apify.com) to access Apify storages to store the screenshots.
36+
37+
Our server needs two paths:
38+
39+
- `/` - Index path will display a page form to submit a new URL and the thumbnails of processed URLs.
40+
- `/add-url` - Will provide an API to add new URLs using a HTTP POST request.
41+
42+
First, we'll import `express` and create an Express.js app. Then, we'll add some middleware that will allow us to receive form submissions.
43+
44+
```JavaScript
45+
import Apify from 'apify';
46+
import express from 'express';
47+
48+
const app = express()
49+
50+
app.use(express.json());
51+
app.use(express.urlencoded({ extended: true }));
52+
```
53+
54+
Now we need to read the following environment variables:
55+
56+
- **APIFY_CONTAINER_PORT** contains a port number where we must start the server.
57+
- **APIFY_CONTAINER_URL** contains a URL under which we can access the container.
58+
- **APIFY_DEFAULT_KEY_VALUE_STORE_ID** is simply the ID of the default key-value store of this actor where we can store screenshots.
59+
60+
```JavaScript
61+
const {
62+
APIFY_CONTAINER_PORT,
63+
APIFY_CONTAINER_URL,
64+
APIFY_DEFAULT_KEY_VALUE_STORE_ID,
65+
} = process.env;
66+
```
67+
68+
Next, we'll create an array of the processed URLs where the **n**th URL has its screenshot stored under the key **n**.jpg in the key-value store.
69+
70+
```JavaScript
71+
const processedUrls = [];
72+
```
73+
74+
After that, the index route is ready to be defined.
75+
76+
```JavaScript
77+
app.get('/', (req, res) => {
78+
let listItems = '';
79+
80+
// For each of the processed
81+
processedUrls.forEach((url, index) => {
82+
const imageUrl = `https://api.apify.com/v2/key-value-stores/${APIFY_DEFAULT_KEY_VALUE_STORE_ID}/records/${index}.jpg`;
83+
84+
// Display the screenshots below the form
85+
listItems += `<li>
86+
<a href="${imageUrl}" target="_blank">
87+
<img src="${imageUrl}" width="300px" />
88+
<br />
89+
${url}
90+
</a>
91+
</li>`;
92+
});
93+
94+
const pageHtml = `<html>
95+
<head><title>Example</title></head>
96+
<body>
97+
<form method="POST" action="${APIFY_CONTAINER_URL}/add-url">
98+
URL: <input type="text" name="url" placeholder="http://example.com" />
99+
<input type="submit" value="Add" />
100+
<hr />
101+
<ul>${listItems}</ul>
102+
</form>
103+
</body>
104+
</html>`;
105+
106+
res.send(pageHtml);
107+
});
108+
```
109+
110+
And then the a second path that receives the new URL submitted using the HTML form; after the URL is processed, it redirects the user back to the root path.
111+
112+
```JavaScript
113+
app.post('/add-url', async (req, res) => {
114+
const { url } = req.body;
115+
console.log(`Got new URL: ${url}`);
116+
117+
// Start chrome browser and open new page ...
118+
const browser = await Apify.launchPuppeteer();
119+
const page = await browser.newPage();
120+
121+
// ... go to our URL and grab a screenshot ...
122+
await page.goto(url);
123+
const screenshot = await page.screenshot({ type: 'jpeg' });
124+
125+
// ... close browser ...
126+
await page.close();
127+
await browser.close();
128+
129+
// ... save screenshot to key-value store and add URL to processedUrls.
130+
await Apify.setValue(`${processedUrls.length}.jpg`, screenshot, { contentType: 'image/jpeg' });
131+
processedUrls.push(url);
132+
133+
res.redirect('/');
134+
});
135+
```
136+
137+
And finally we need to start the web server.
138+
139+
```JavaScript
140+
// Start the web server!
141+
app.listen(APIFY_CONTAINER_PORT, () => {
142+
console.log(`Application is listening at URL ${APIFY_CONTAINER_URL}.`);
143+
});
144+
```
145+
146+
### [](#final-code) Final code
147+
148+
```JavaScript
149+
import Apify from 'apify';
150+
import express from 'express';
151+
152+
const app = express()
153+
154+
app.use(express.json());
155+
app.use(express.urlencoded({ extended: true }));
156+
157+
const {
158+
APIFY_CONTAINER_PORT,
159+
APIFY_CONTAINER_URL,
160+
APIFY_DEFAULT_KEY_VALUE_STORE_ID,
161+
} = process.env;
162+
163+
const processedUrls = [];
164+
165+
app.get('/', (req, res) => {
166+
let listItems = '';
167+
168+
// For each of the processed
169+
processedUrls.forEach((url, index) => {
170+
const imageUrl = `https://api.apify.com/v2/key-value-stores/${APIFY_DEFAULT_KEY_VALUE_STORE_ID}/records/${index}.jpg`;
171+
172+
// Display the screenshots below the form
173+
listItems += `<li>
174+
<a href="${imageUrl}" target="_blank">
175+
<img src="${imageUrl}" width="300px" />
176+
<br />
177+
${url}
178+
</a>
179+
</li>`;
180+
});
181+
182+
const pageHtml = `<html>
183+
<head><title>Example</title></head>
184+
<body>
185+
<form method="POST" action="${APIFY_CONTAINER_URL}/add-url">
186+
URL: <input type="text" name="url" placeholder="http://example.com" />
187+
<input type="submit" value="Add" />
188+
<hr />
189+
<ul>${listItems}</ul>
190+
</form>
191+
</body>
192+
</html>`;
193+
194+
res.send(pageHtml);
195+
});
196+
197+
app.post('/add-url', async (req, res) => {
198+
const { url } = req.body;
199+
console.log(`Got new URL: ${url}`);
200+
201+
// Start chrome browser and open new page ...
202+
const browser = await Apify.launchPuppeteer();
203+
const page = await browser.newPage();
204+
205+
// ... go to our URL and grab a screenshot ...
206+
await page.goto(url);
207+
const screenshot = await page.screenshot({ type: 'jpeg' });
208+
209+
// ... close browser ...
210+
await page.close();
211+
await browser.close();
212+
213+
// ... save screenshot to key-value store and add URL to processedUrls.
214+
await Apify.setValue(`${processedUrls.length}.jpg`, screenshot, { contentType: 'image/jpeg' });
215+
processedUrls.push(url);
216+
217+
res.redirect('/');
218+
});
219+
220+
app.listen(APIFY_CONTAINER_PORT, () => {
221+
console.log(`Application is listening at URL ${APIFY_CONTAINER_URL}.`);
222+
});
223+
```
224+
225+
When we deploy and run this actor on the Apify platform, then we can open the **Live View** tab in the actor console to submit the URL to your actor through the form. After the URL is successfully submitted, it appears in the actor log.
226+
227+
With that we're done! And our application works like a charm :)
228+
229+
The complete code of this actor is available [here](https://www.apify.com/apify/example-web-server). You can run it there or copy it to your account.

0 commit comments

Comments
 (0)