Skip to content

Commit bc6fbb5

Browse files
fnesvedaTC-MO
andauthored
feat: Improve Apify Proxy Usage docs (#935)
Every now and then, we have a free user that can't use Apify Proxy from inside of the Actors because they're trying to connect to `proxy.apify.com`, not internally through `$APIFY_PROXY_HOSTNAME`, which fails since it's considered an external connection, which is banned on the Free plan. This improves the Apify Proxy connection instructions to hopefully better explain that when connecting to Apify Proxy from inside an Actor, they should either use the helpers in the `Actor` class in the SDK, or use the `APIFY_PROXY_HOSTNAME`, `APIFY_PROXY_PORT` and `APIFY_PROXY_PASSWORD` environment variables. Closes #740 --------- Co-authored-by: Michał Olender <[email protected]>
1 parent 23a5b98 commit bc6fbb5

File tree

1 file changed

+48
-11
lines changed

1 file changed

+48
-11
lines changed

sources/platform/proxy/usage.md

Lines changed: 48 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -13,23 +13,60 @@ slug: /proxy/usage
1313

1414
## Connection settings
1515

16-
To connect to the Apify Proxy, you use the [HTTP proxy protocol](https://en.wikipedia.org/wiki/Proxy_server#Web_proxy_servers). This means that you need to configure your HTTP client to use the proxy server at `proxy.apify.com:8000` and provide it with your Apify Proxy password and the other parameters described below.
16+
To connect to Apify Proxy, you use the [HTTP proxy protocol](https://en.wikipedia.org/wiki/Proxy_server#Web_proxy_servers). This means that you need to configure your HTTP client to use the proxy server at the Apify Proxy hostname and provide it with your Apify Proxy password and the other parameters described below.
1717

1818
The full connection string has the following format:
1919

2020
```text
21-
http://<username>:<password>@proxy.apify.com:8000
21+
http://<username>:<password>@<hostname>:<port>
2222
```
2323

24+
:::caution
25+
All usage of Apify Proxy with your password is charged towards your account. Do not share the password with untrusted parties or use it from insecure networks, as **the password is sent unencrypted** due to the HTTP protocol's [limitations](https://www.guru99.com/difference-http-vs-https.html).
26+
:::
27+
28+
### External connection
29+
30+
If you want to connect to Apify Proxy from outside of the Apify Platform, you need to have a paid Apify plan (to prevent abuse).
31+
If you need to test Apify Proxy before you subscribe, please [contact our support](https://apify.com/contact).
32+
2433
| Parameter | Value / explanation |
2534
|---------------------|---------------------|
26-
| Proxy type | `HTTP` |
2735
| Hostname | `proxy.apify.com`, alternatively you can use static IP addresses `18.208.102.16`, `35.171.134.41`. |
2836
| Port | `8000` |
2937
| Username | Specifies the proxy parameters such as groups, [session](#sessions) and location. See [username parameters](#username-parameters) below for details. <br/>**Note**: this is not your Apify username.|
30-
| Password | Proxy password. Your password is displayed on the [Proxy](https://console.apify.com/proxy/groups) page in Apify Console. In Apify [Actors](../actors/index.mdx), it is passed as the `APIFY_PROXY_PASSWORD` environment variable. See the [environment variables docs](../actors/development/programming_interface/environment_variables.md) for more details. |
38+
| Password | Apify Proxy password. Your password is displayed on the [Proxy](https://console.apify.com/proxy/groups) page in Apify Console. <br/>**Note**: this is not your Apify account password. |
39+
40+
:::caution
41+
If you use these connection parameters for connecting to Apify Proxy from your Actors running on the Apify Platform, the connection will still be considered external, it will not work on the Free plan, and on paid plans you will be charged for external data transfer. Please use the connection parameters from the [Connection from Actors](#connection-from-actors) section when using Apify Proxy from Actors.
42+
:::
43+
44+
Example connection string for external connections:
45+
46+
```text
47+
http://auto:[email protected]:8000
48+
```
49+
50+
### Connection from Actors
51+
52+
If you want to connect to Apify Proxy from Actors running on the Apify Platform, the recommended way is to use built-in proxy configuration tools in the [Apify SDK JavaScript](/sdk/js/docs/guides/proxy-management) or [Apify SDK Python](/sdk/python/docs/concepts/proxy-management)
53+
54+
If you don't want to use these helpers, and want to connect to Apify Proxy manually, you can find the right configuration values in [environment variables](../actors/development/programming_interface/environment_variables.md) provided to the Actor.
55+
By using this configuration, you ensure that you connect to Apify Proxy directly through the Apify infrastructure, bypassing any external connection via the Internet, thereby improving the connection speed, and ensuring you don't pay for external data transfer.
56+
57+
| Parameter | Source / explanation |
58+
|---------------------|---------------------|
59+
| Hostname | `APIFY_PROXY_HOSTNAME` environment variable |
60+
| Port | `APIFY_PROXY_PORT` environment variable |
61+
| Username | Specifies the proxy parameters such as groups, [session](#sessions) and location. See [username parameters](#username-parameters) below for details. <br/>**Note**: this is not your Apify username.|
62+
| Password | `APIFY_PROXY_PASSWORD` environment variable |
3163

32-
> **WARNING:** All usage of Apify Proxy with your password is charged towards your account. Do not share the password with untrusted parties or use it from insecure networks – **the password is sent unencrypted** due to the HTTP protocol's [limitations](https://www.guru99.com/difference-http-vs-https.html).
64+
Example connection string creation:
65+
66+
```js
67+
const { APIFY_PROXY_HOSTNAME, APIFY_PROXY_PORT, APIFY_PROXY_PASSWORD } = process.env;
68+
const connectionString = `http://auto:${APIFY_PROXY_PASSWORD}@${APIFY_PROXY_HOSTNAME}:${APIFY_PROXY_PORT}`;
69+
```
3370

3471
### Username parameters
3572

@@ -67,7 +104,7 @@ The table below describes the available parameters.
67104
<td>Optional</td>
68105
<td>
69106
<p>If specified to <code>session-new_job_123</code>, for example, all proxied requests with the same session identifier are routed through the same IP address. If not specified, each proxied request is assigned a randomly picked least used IP address.</p>
70-
<p>The session string can only contain numbers (0-9), letters (a-z or A-Z), dot (.), underscore (_), a tilde (~). The maximum length is 50 characters.</p>
107+
<p>The session string can only contain numbers (09), letters (a-z or A-Z), dot (.), underscore (_), a tilde (~). The maximum length is 50 characters.</p>
71108
<p>Session management may work differently for residential and SERP proxies. Check relevant documentations for more details.</p>
72109
</td>
73110
</tr>
@@ -103,8 +140,8 @@ Web scrapers can rotate the IP addresses they use to access websites. They assig
103140

104141
Depending on whether you use a [browser](https://apify.com/apify/web-scraper) or [HTTP requests](https://apify.com/apify/cheerio-scraper) for your scraping jobs, IP address rotation works differently.
105142

106-
* Browsera different IP address is used for each browser.
107-
* HTTP requesta different IP address is used for each request.
143+
* Browsera different IP address is used for each browser.
144+
* HTTP requesta different IP address is used for each request.
108145

109146
Use [sessions](#sessions) to controll how you rotate and [persist](#session-persistence) IP addresses. See our guide [Anti-scraping techniques](/academy/anti-scraping/techniques) to learn more about IP address rotation and our findings on how blocking works.
110147

@@ -134,12 +171,12 @@ To test that your requests are proxied and IP addresses are being [rotated](/aca
134171

135172
### A different approach to `502 Bad Gateway`
136173

137-
There are times when the `502` status code is not comprehensive enough. Therefore, we have modified our server with `590-599` codes instead to provide more insight:
174+
Sometimes when the `502` status code is not comprehensive enough. Therefore, we have modified our server with `590-599` codes instead to provide more insight:
138175

139176
* `590 Non Successful`: upstream responded with non-200 status code.
140177
* `591 RESERVED`: *this status code is reserved for further use.*
141-
* `592 Status Code Out Of Range`: upstream responded with status code different than 100-999.
142-
* `593 Not Found`: DNS lookup failed - [`EAI_NODATA`](https://github.com/libuv/libuv/blob/cdbba74d7a756587a696fb3545051f9a525b85ac/include/uv.h#L82) or [`EAI_NONAME`](https://github.com/libuv/libuv/blob/cdbba74d7a756587a696fb3545051f9a525b85ac/include/uv.h#L83).
178+
* `592 Status Code Out Of Range`: upstream responded with status code different than 100999.
179+
* `593 Not Found`: DNS lookup failed, indicating either [`EAI_NODATA`](https://github.com/libuv/libuv/blob/cdbba74d7a756587a696fb3545051f9a525b85ac/include/uv.h#L82) or [`EAI_NONAME`](https://github.com/libuv/libuv/blob/cdbba74d7a756587a696fb3545051f9a525b85ac/include/uv.h#L83).
143180
* `594 Connection Refused`: upstream refused connection.
144181
* `595 Connection Reset`: connection reset due to loss of connection or timeout.
145182
* `596 Broken Pipe`: trying to write on a closed socket.

0 commit comments

Comments
 (0)