How to Combine Playwright and HTTP Requests in Crawlee?

[kwdiwt]

I'm working on a web scraping project that requires both browser automation and direct HTTP requests. Could the maintainers suggest best practices for mixing Playwright-based page handling with regular HTTP requests in Crawlee?

[kwdiwt]

How to Switch from Playwright Login to HTTP Crawler After Cookie Acquisition？

[janbuchar]

Hello @kwdiwt and thank you for your interest in Crawlee! If you don't need to use a BrowserPool, I suggest that you just perform the login using playwright without any Crawlee wrappers, retrieve the cookies and use them to construct your HttpCrawler (or any of its subclasses - CheerioCrawler etc.):

crawler = HttpCrawler({
  // ...
  sessionPoolOptions: {
    sessionOptions: {
      cookieJar: {
        "yourCookie": "value"
      } // this can be a toughcookie.CookieJar instance as well
    }
  }
})

await crawler.run()

If you need to perform the login for each new session (perhaps to avoid getting blocked), you can use the createSessionFunction option (https://crawlee.dev/api/core/interface/SessionPoolOptions#createSessionFunction) instead of sessionOptions.

[kwdiwt]

Can PlaywrightCrawler and CheerioCrawler Coexist with Shared Context(cookies/session storage/header...) ? Has expamle?

[janbuchar]

Sorry for the late response! Currently, the session pool cannot be easily shared between multiple crawlers. You could customize the createSessionFunction for both crawlers so that it delegates to a shared session pool, I believe. Sadly I have no example to demonstrate that.

[kwdiwt]

OK,THX