-
-
Notifications
You must be signed in to change notification settings - Fork 148
Implement curl_cffi #1845
Copy link
Copy link
Open
Description
Problem
Some court websites (e.g. Massachusetts Superior Court / socialaw.com) block requests from standard HTTP clients, returning 403 responses. These sites use TLS fingerprinting or
similar bot-detection techniques that requests cannot bypass.
Solution
Add curl_cffi as a dependency to enable browser TLS fingerprint impersonation. Scrapers that need it can set self.impersonate = True (or a specific browser string like "safari") to route requests through curl_cffi instead of requests.
This will allow us to continue scraping opinions from nh, masssuperct and lactapp_3
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels
Type
Projects
Status
Backlog