Handle request list and proxy configuration inputs in a user-friendly way

[janbuchar]

• https://docs.apify.com/platform/actors/development/actor-definition/input-schema/specification/v1#array - request list editor
• https://docs.apify.com/platform/actors/development/actor-definition/input-schema/specification/v1#object - proxy editor

An actor.json like this:

{
    "actorSpecification": 1,
    "name": "example-actor",
    "version": "0.0",
    "buildTag": "latest",
    "environmentVariables": {},
    "input": {
        "properties": {
            "startUrls": {
                "title": "Start URLs",
                "type": "array",
                "description": "URLs to start with",
                "prefill": [
                    {
                        "url": "https://apify.com"
                    }
                ],
                "editor": "requestListSources"
            },
            "proxyConfiguration": {
                "title": "Proxy configuration",
                "type": "object",
                "description": "Select proxies to be used by your crawler.",
                "prefill": {
                    "useApifyProxy": true
                },
                "editor": "proxy"
            }
        }
    }
}

will have an input editor like this:

and calling Actor.get_input() in the actor will result into something like this:

{
    "proxyConfiguration": {
        "useApifyProxy": True,
        "apifyProxyGroups": ["BUYPROXIES63811", "SHADER"],
    },
    "startUrls": [
        {"url": "https://google.com", "method": "GET"},
        {"url": "https://salas.hu", "method": "GET"},
        {"url": "https://trivago.hu", "method": "GET"},
        {"requestsFromUrl": "https://lorem.se/ipsum.txt", "method": "GET"},
    ],
}

It would be awesome if we could implement some helpers to simplify converting these structures into crawlee.storage.RequestList or apify.ProxyConfiguration.

Since it is pretty convenient to have the input structure described with a Pydantic model (so you can do Input.model_validate(await Actor.get_input()), I would love to explore the possibility of adding some Pydantic models that you could use in your Input models and then pass them to ProxyConfiguration or RequestList constructors (or some factory methods).

[Pijukatel]

Pydantic seems natural for handling inputs, on the other hand it seems like it was already intentionally removed: #74

I do not have enough historical context. Is the reasoning behind not using Pydantic here still valid?

[vdusek]

Pydantic seems natural for handling inputs, on the other hand it seems like it was already intentionally removed: #74

I do not have enough historical context. Is the reasoning behind not using Pydantic here still valid?

We put it back. Pydantic is now used in many places throughout Crawlee, and so in the SDK as well. See #64 (comment) for more details.