You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/index.rst
+12-2Lines changed: 12 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -55,8 +55,19 @@ How to use it
55
55
56
56
57
57
With the middleware, the usage of crawlera is automatic, every request will go through crawlera without nothing to worry about.
58
+
If you want to *disable* crawlera on a specific Request, you can do so by updating `meta` with `dont_proxy=True`::
58
59
59
-
Remember that you are now making request to Crawlera, and the Crawlera service will be the one actually making the requests to the different sites.
60
+
61
+
scrapy.Request(
62
+
'http://example.com',
63
+
meta={
64
+
'dont_proxy': True,
65
+
...
66
+
},
67
+
)
68
+
69
+
70
+
Remember that you are now making requests to Crawlera, and the Crawlera service will be the one actually making the requests to the different sites.
60
71
61
72
If you need to specify special `Crawlera Headers <https://doc.scrapinghub.com/crawlera.html#request-headers>`_, just apply them as normal `Scrapy Headers <https://doc.scrapy.org/en/latest/topics/request-response.html#scrapy.http.Request.headers>`_.
62
73
@@ -78,7 +89,6 @@ requests with `DEFAULT_REQUEST_HEADERS <http://doc.scrapy.org/en/1.0/topics/sett
78
89
but keep sending ``X-Crawlera-*`` headers in your requests, those will be removed from the
79
90
request headers.
80
91
81
-
82
92
This Middleware also adds some configurable Scrapy Settings, check :ref:`the complete list here <settings>`.
0 commit comments