You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Every now and then, we have a free user that can't use Apify Proxy from
inside of the Actors because they're trying to connect to
`proxy.apify.com`, not internally through `$APIFY_PROXY_HOSTNAME`, which
fails since it's considered an external connection, which is banned on
the Free plan.
This improves the Apify Proxy connection instructions to hopefully
better explain that when connecting to Apify Proxy from inside an Actor,
they should either use the helpers in the `Actor` class in the SDK, or
use the `APIFY_PROXY_HOSTNAME`, `APIFY_PROXY_PORT` and
`APIFY_PROXY_PASSWORD` environment variables.
Closes#740
---------
Co-authored-by: Michał Olender <[email protected]>
Copy file name to clipboardExpand all lines: sources/platform/proxy/usage.md
+48-11Lines changed: 48 additions & 11 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -13,23 +13,60 @@ slug: /proxy/usage
13
13
14
14
## Connection settings
15
15
16
-
To connect to the Apify Proxy, you use the [HTTP proxy protocol](https://en.wikipedia.org/wiki/Proxy_server#Web_proxy_servers). This means that you need to configure your HTTP client to use the proxy server at `proxy.apify.com:8000` and provide it with your Apify Proxy password and the other parameters described below.
16
+
To connect to Apify Proxy, you use the [HTTP proxy protocol](https://en.wikipedia.org/wiki/Proxy_server#Web_proxy_servers). This means that you need to configure your HTTP client to use the proxy server at the Apify Proxy hostname and provide it with your Apify Proxy password and the other parameters described below.
17
17
18
18
The full connection string has the following format:
19
19
20
20
```text
21
-
http://<username>:<password>@proxy.apify.com:8000
21
+
http://<username>:<password>@<hostname>:<port>
22
22
```
23
23
24
+
:::caution
25
+
All usage of Apify Proxy with your password is charged towards your account. Do not share the password with untrusted parties or use it from insecure networks, as **the password is sent unencrypted** due to the HTTP protocol's [limitations](https://www.guru99.com/difference-http-vs-https.html).
26
+
:::
27
+
28
+
### External connection
29
+
30
+
If you want to connect to Apify Proxy from outside of the Apify Platform, you need to have a paid Apify plan (to prevent abuse).
31
+
If you need to test Apify Proxy before you subscribe, please [contact our support](https://apify.com/contact).
32
+
24
33
| Parameter | Value / explanation |
25
34
|---------------------|---------------------|
26
-
| Proxy type |`HTTP`|
27
35
| Hostname |`proxy.apify.com`, alternatively you can use static IP addresses `18.208.102.16`, `35.171.134.41`. |
28
36
| Port |`8000`|
29
37
| Username | Specifies the proxy parameters such as groups, [session](#sessions) and location. See [username parameters](#username-parameters) below for details. <br/>**Note**: this is not your Apify username.|
30
-
| Password | Proxy password. Your password is displayed on the [Proxy](https://console.apify.com/proxy/groups) page in Apify Console. In Apify [Actors](../actors/index.mdx), it is passed as the `APIFY_PROXY_PASSWORD` environment variable. See the [environment variables docs](../actors/development/programming_interface/environment_variables.md) for more details. |
38
+
| Password | Apify Proxy password. Your password is displayed on the [Proxy](https://console.apify.com/proxy/groups) page in Apify Console. <br/>**Note**: this is not your Apify account password. |
39
+
40
+
:::caution
41
+
If you use these connection parameters for connecting to Apify Proxy from your Actors running on the Apify Platform, the connection will still be considered external, it will not work on the Free plan, and on paid plans you will be charged for external data transfer. Please use the connection parameters from the [Connection from Actors](#connection-from-actors) section when using Apify Proxy from Actors.
42
+
:::
43
+
44
+
Example connection string for external connections:
If you want to connect to Apify Proxy from Actors running on the Apify Platform, the recommended way is to use built-in proxy configuration tools in the [Apify SDK JavaScript](/sdk/js/docs/guides/proxy-management) or [Apify SDK Python](/sdk/python/docs/concepts/proxy-management)
53
+
54
+
If you don't want to use these helpers, and want to connect to Apify Proxy manually, you can find the right configuration values in [environment variables](../actors/development/programming_interface/environment_variables.md) provided to the Actor.
55
+
By using this configuration, you ensure that you connect to Apify Proxy directly through the Apify infrastructure, bypassing any external connection via the Internet, thereby improving the connection speed, and ensuring you don't pay for external data transfer.
| Username | Specifies the proxy parameters such as groups, [session](#sessions) and location. See [username parameters](#username-parameters) below for details. <br/>**Note**: this is not your Apify username.|
> **WARNING:** All usage of Apify Proxy with your password is charged towards your account. Do not share the password with untrusted parties or use it from insecure networks – **the password is sent unencrypted** due to the HTTP protocol's [limitations](https://www.guru99.com/difference-http-vs-https.html).
@@ -67,7 +104,7 @@ The table below describes the available parameters.
67
104
<td>Optional</td>
68
105
<td>
69
106
<p>If specified to <code>session-new_job_123</code>, for example, all proxied requests with the same session identifier are routed through the same IP address. If not specified, each proxied request is assigned a randomly picked least used IP address.</p>
70
-
<p>The session string can only contain numbers (0-9), letters (a-z or A-Z), dot (.), underscore (_), a tilde (~). The maximum length is 50 characters.</p>
107
+
<p>The session string can only contain numbers (0–9), letters (a-z or A-Z), dot (.), underscore (_), a tilde (~). The maximum length is 50 characters.</p>
71
108
<p>Session management may work differently for residential and SERP proxies. Check relevant documentations for more details.</p>
72
109
</td>
73
110
</tr>
@@ -103,8 +140,8 @@ Web scrapers can rotate the IP addresses they use to access websites. They assig
103
140
104
141
Depending on whether you use a [browser](https://apify.com/apify/web-scraper) or [HTTP requests](https://apify.com/apify/cheerio-scraper) for your scraping jobs, IP address rotation works differently.
105
142
106
-
* Browser – a different IP address is used for each browser.
107
-
* HTTP request – a different IP address is used for each request.
143
+
* Browser—a different IP address is used for each browser.
144
+
* HTTP request—a different IP address is used for each request.
108
145
109
146
Use [sessions](#sessions) to controll how you rotate and [persist](#session-persistence) IP addresses. See our guide [Anti-scraping techniques](/academy/anti-scraping/techniques) to learn more about IP address rotation and our findings on how blocking works.
110
147
@@ -134,12 +171,12 @@ To test that your requests are proxied and IP addresses are being [rotated](/aca
134
171
135
172
### A different approach to `502 Bad Gateway`
136
173
137
-
There are times when the `502` status code is not comprehensive enough. Therefore, we have modified our server with `590-599` codes instead to provide more insight:
174
+
Sometimes when the `502` status code is not comprehensive enough. Therefore, we have modified our server with `590-599` codes instead to provide more insight:
138
175
139
176
*`590 Non Successful`: upstream responded with non-200 status code.
140
177
*`591 RESERVED`: *this status code is reserved for further use.*
141
-
*`592 Status Code Out Of Range`: upstream responded with status code different than 100-999.
142
-
*`593 Not Found`: DNS lookup failed -[`EAI_NODATA`](https://github.com/libuv/libuv/blob/cdbba74d7a756587a696fb3545051f9a525b85ac/include/uv.h#L82) or [`EAI_NONAME`](https://github.com/libuv/libuv/blob/cdbba74d7a756587a696fb3545051f9a525b85ac/include/uv.h#L83).
178
+
*`592 Status Code Out Of Range`: upstream responded with status code different than 100–999.
179
+
*`593 Not Found`: DNS lookup failed, indicating either[`EAI_NODATA`](https://github.com/libuv/libuv/blob/cdbba74d7a756587a696fb3545051f9a525b85ac/include/uv.h#L82) or [`EAI_NONAME`](https://github.com/libuv/libuv/blob/cdbba74d7a756587a696fb3545051f9a525b85ac/include/uv.h#L83).
0 commit comments