-
-
Notifications
You must be signed in to change notification settings - Fork 0
Description
Bug Description
The Webshare.list_proxies() method ignores the page parameter in search_params and always fetches ALL pages from the API, causing extremely long execution times (35-70 minutes) instead of fetching just the specified page.
Expected Behavior
When search_params={"page": 1, "page_size": 25} is provided, the method should fetch only page 1 with 25 proxies and return immediately (~1 second).
Actual Behavior
The method loops through ALL pages (2,151 pages for 215,084 proxies) regardless of the page parameter in search_params, taking 35-70 minutes to complete.
Steps to Reproduce
on
from proxyproviders import Webshare
proxy_provider = Webshare(
api_key="your-api-key",
search_params={"mode": "backbone", "page": 1, "page_size": 25}
)
This should fetch only page 1, but instead fetches all 2,151 pages
proxies = proxy_provider.list_proxies(force_refresh=True)## Root Cause
In providers/webshare.py, the _fetch_proxies() method has a while True loop that:
-
Starts with
page = 0and incrementspage += 1in each iteration -
Overrides the
pageparameter fromsearch_paramswith the loop counter -
Continues looping until
data.get("next") is None(all pages fetched)
hon
def _fetch_proxies(self) -> List[Proxy]:
all_proxies = []
page = 0while True: # ← Always loops through all pages
page += 1 # ← Overrides search_params["page"]
default_params = {
"mode": "direct",
"page": page, # ← Ignores search_params["page"]
"page_size": 100,
}
params = {**default_params, **self.search_params}
# ... fetch page ...
if data.get("next") is None:
break # ← Only stops when no more pages## Workaround
Direct API call works correctly:
import requests
response = requests.get(
"https://proxy.webshare.io/api/v2/proxy/list",
headers={"Authorization": f"Token {api_key}"},
params={"mode": "backbone", "page": 1, "page_size": 25}
)
proxies = response.json()["results"] # Returns only page 1## Environment
Impact
- High: Makes the library unusable for fetching a specific page
- Performance: 35-70 minutes vs expected 1 second
- User experience: Appears to hang/freeze
Please fix this problem