Skip to content

Latest commit

 

History

History
80 lines (58 loc) · 2.23 KB

automap.rst

File metadata and controls

80 lines (58 loc) · 2.23 KB

Automatic request parameters

To send a Scrapy request through Zyte API letting Zyte API request parameters be automatically chosen based on the parameters of that Scrapy request, set the :reqmeta:`zyte_api_automap` key in :attr:`Request.meta <scrapy.http.Request.meta>` to True.

For example:

import scrapy


class SampleQuotesSpider(scrapy.Spider):
    name = "sample_quotes"

    async def start(self):
        yield scrapy.Request(
            url="https://quotes.toscrape.com/",
            meta={
                "zyte_api_automap": True,
            },
        )

    def parse(self, response):
        print(response.text)
        # "<html>…</html>"

In :ref:`transparent mode <transparent>`, :reqmeta:`zyte_api_automap` is True by default.

See :ref:`request` to learn how exactly request parameters are mapped when using automatic request parameters.

Changing parameters

You may set :reqmeta:`zyte_api_automap` in :attr:`Request.meta <scrapy.http.Request.meta>` to a :class:`dict` of Zyte API parameters to add, modify, or remove (by setting to None) automatic request parameters. This also works in :ref:`transparent mode <transparent>`.

Enabling :http:`request:browserHtml`, :http:`request:screenshot`, or an automatic extraction property, unsets :http:`request:httpResponseBody` and :http:`request:httpResponseHeaders`, and makes Request.headers become :http:`request:requestHeaders` instead of :http:`request:customHttpRequestHeaders`. For example, the following Scrapy request:

Request(
    url="https://quotes.toscrape.com",
    headers={"Referer": "https://example.com/"},
    meta={"zyte_api_automap": {"browserHtml": True}},
)

Results in a request to the Zyte API data extraction endpoint with the following parameters:

{
    "browserHtml": true,
    "experimental": {
        "responseCookies": true
    },
    "requestHeaders": {"referer": "https://example.com/"},
    "url": "https://quotes.toscrape.com"
}

See also: :ref:`request-unsupported`.