Skip to content

Webshare.list_proxies() ignores page parameter in search_params and fetches all pages instead of specified page #3

@hahust191806

Description

@hahust191806

Bug Description

The Webshare.list_proxies() method ignores the page parameter in search_params and always fetches ALL pages from the API, causing extremely long execution times (35-70 minutes) instead of fetching just the specified page.

Expected Behavior

When search_params={"page": 1, "page_size": 25} is provided, the method should fetch only page 1 with 25 proxies and return immediately (~1 second).

Actual Behavior

The method loops through ALL pages (2,151 pages for 215,084 proxies) regardless of the page parameter in search_params, taking 35-70 minutes to complete.

Steps to Reproduce

on
from proxyproviders import Webshare

proxy_provider = Webshare(
api_key="your-api-key",
search_params={"mode": "backbone", "page": 1, "page_size": 25}
)

This should fetch only page 1, but instead fetches all 2,151 pages

proxies = proxy_provider.list_proxies(force_refresh=True)## Root Cause

In providers/webshare.py, the _fetch_proxies() method has a while True loop that:

  1. Starts with page = 0 and increments page += 1 in each iteration

  2. Overrides the page parameter from search_params with the loop counter

  3. Continues looping until data.get("next") is None (all pages fetched)
    hon
    def _fetch_proxies(self) -> List[Proxy]:
    all_proxies = []
    page = 0

    while True: # ← Always loops through all pages
    page += 1 # ← Overrides search_params["page"]
    default_params = {
    "mode": "direct",
    "page": page, # ← Ignores search_params["page"]
    "page_size": 100,
    }
    params = {**default_params, **self.search_params}
    # ... fetch page ...
    if data.get("next") is None:
    break # ← Only stops when no more pages## Workaround

Direct API call works correctly:
import requests

response = requests.get(
"https://proxy.webshare.io/api/v2/proxy/list",
headers={"Authorization": f"Token {api_key}"},
params={"mode": "backbone", "page": 1, "page_size": 25}
)
proxies = response.json()["results"] # Returns only page 1## Environment

Impact

  • High: Makes the library unusable for fetching a specific page
  • Performance: 35-70 minutes vs expected 1 second
  • User experience: Appears to hang/freeze

Please fix this problem

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions