[feature] ability to set useragent header or rotate it?

[kocoten1992]

Version and OS
v0.45.3 on linux/docker

Is your feature request related to a problem? Please describe.
I've been block by cloudflare for a site, I used to crawl a specific page (only one page) every 20 minutes, now I've set it to 45 minutes, but the situation doesn't get improved, if I try to made the request manually (via curl with changed useragent) - it still work, so I think the ability to set useragent is really valuable.

Describe the solution you'd like
Rotate some predefined useragents header? Or maybe just some latest browser agent header?

Describe the use-case and give concrete real-world examples
Attach any HTML/JSON, give links to sites, screenshots etc, we are not mind readers

    Please enable cookies.

      Sorry, you have been blocked

Skip to main content

          Why have I been blocked?
          What's New

          What can I do to resolve this?
            2023/11/13

          You can email the site owner to let them know you were blocked. Please include what you were doing when this page came up and the Cloudflare Ray ID found at the bottom of this page.
            Event Cover Photos

      Cloudflare Ray ID: 8268e7dfb9e68b82 • Your IP: Click to reveal .... • Performance & security by Cloudflare

[dgtlmoon]

you can already set user agent if under 'chrome' selection

but hmm... maybe for you is better to use proxy like brightdata that automatically rotates the IP, i dont think its really browser header related, more like it knows your IP has too many attempts

[dgtlmoon]

the best is auto-rotating residential proxies

[jgupta]

#1930 should help with this.