bump docs

Author: davidteather

README.md

@@ -9,7 +9,6 @@ Unified python interface for proxy provider APIs, supports BrightData, WebShare,
 - [Getting Started](#getting-started)
   - [How to Support The Project](#how-to-support-the-project)
   - [Installing](#installing)
-  - [Common Issues](#common-issues)
 - [Quick Start Guide](#quick-start-guide)
   - [Examples](https://github.com/davidteather/proxyproviders/tree/main/examples)
 
@@ -39,5 +38,112 @@ python -m pip install proxyproviders
 
 ## Quick Start Guide
 
+Here's a quick bit of code to get you started! There's also a few [examples in the folder](https://github.com/davidteather/proxyproviders/tree/main/examples). 
 
+**Note:** If you want to learn how to web scrape websites check my [free and open-source course for learning everything web scraping](https://github.com/davidteather/everything-web-scraping)
+
+### Choose a Proxy Provider
+
+If you already haven't, choose which proxy provider to use. You can find a [list in the documentation](https://davidteather.github.io/proxyproviders/#proxyproviders-supported-providers). After choosing, look at the documentation around the specific provider you've choosen. Where needed, we've laid out steps to get api keys in the documentation. These steps will vary slightly by provider. For this example I'll be using the [Webshare Provider](https://davidteather.github.io/proxyproviders/#proxyproviders.providers.webshare.Webshare) because it's both easy to setup and they give you 10 free data center proxies to test out.
+
+You can create an account on webshare [here](https://www.webshare.io/?referral_code=3x5812idzzzp) (affiliate link), then head into the [API tab](https://dashboard.webshare.io/userapi/keys) on the side and generate a new token. Keep this API token safe and don't post it publically.
+
+### Basic Example
+
+After you can list out your proxies with
+```py
+from proxyproviders import Webshare
+
+proxy_provider = Webshare(api_key="your-api-key")
+
+proxies = proxy_provider.list_proxies()
+
+print(proxies)
+```
+
+Each provider has their own custom options, the `Webshare` class lets you specify url params according to their [api spec](https://apidocs.webshare.io/proxy-list/list#parameters), here's an example which will only return proxies that are based in the US.
+
+```py
+proxy_provider = Webshare(api_key="your-api-key", params={"country_code_in": "US"})
+```
+
+### Using ProxyConfig
+
+For any shared logic across all types of proxy providers, we use the `ProxyConfig` data class to configure them. The full docs for [ProxyConfig are here](https://davidteather.github.io/proxyproviders/#proxyproviders.proxy_provider.ProxyConfig). In this example we will configure it to use a shorter `refresh_interval` than default.
+
+```py
+from proxyproviders import Webshare, ProxyConfig
+import time
+
+config = ProxyConfig(refresh_interval=3)
+ws = Webshare(api_key="your-api-token", config=config)
+
+proxies = ws.list_proxies() # calls API 
+
+ws.list_proxies() # cached
+
+time.sleep(5)
+ws.list_proxies() # calls API since it's more than 3s later
+```
+
+### Function Using Generic ProxyProvider
+
+Since all proxy providers implement the same interface, we can make a function that allows us to easily swap out and utilize different providers. This is the main appeal of having a unified interface. It allows other modules to be provider agnostic, like my [TikTokAPI](https://github.com/davidteather/TikTok-Api) package.
+
+```py
+from proxyproviders import Webshare, ProxyProvider, ProxyConfig
+
+def some_function(provider: ProxyProvider):
+    proxies = provider.list_proxies()
+    print(proxies)
+
+webshare = Webshare(api_key="your_api_key")
+brightdata = BrightData(api_key="your_api_key")
+
+some_function(webshare)
+some_function(brightdata)
+```
+
+### Making Your Own Proxy Provider
+
+Here's a skeleton of how you can make your very own `ProxyProvider` class. You'll need to implemenet all the required functions of the `ProxyProvider` which may be more than what's here at the time of writing.
+
+If you do find yourself making one of these, consider contributing it back to the repository so everyone can use them :D
+
+```py
+from proxyproviders import ProxyProvider, ProxyConfig, Proxy
+from typing import List, Optional
+
+class MyProxyProvider(ProxyProvider):
+    def __init__(self, config: Optional[ProxyConfig] = None):
+        super().__init__(config=config)
+
+    def _fetch_proxies(self):
+        proxies: List[Proxy] = []
+
+        for i in range(10):
+            # TODO: your real proxy fetching logic
+
+            # There are required fields on the Proxy class, be sure that these are filled out properly
+            # especially if you're using it with another library.
+            proxy = Proxy(
+                id=str(i),
+                username="username",
+                password="password",
+                proxy_address="192.168.1.1",
+                port=80,
+            )
+
+            proxies.append(proxy)
+
+        return proxies
+
+def some_function(provider: ProxyProvider):
+    proxies = provider.list_proxies()
+    for proxy in proxies:
+        print(proxy)
+
+provider = MyProxyProvider()
+some_function(provider) # calls the function with the provider
+```
 

proxyproviders/models/proxy.py

@@ -21,14 +21,14 @@ class Proxy:
     port: int
     """The port number through which the proxy connection is established"""
 
-    country_code: Optional[str]
+    country_code: Optional[str] = None
     """The country code where the proxy is located, e.g., 'US', 'FR'. Optional"""
 
-    city_name: Optional[str]
+    city_name: Optional[str] = None
     """The city name where the proxy is located, e.g., 'New York', 'Paris'. Optional"""
 
-    created_at: Optional[datetime]
+    created_at: Optional[datetime] = None
     """The timestamp when the proxy was created. Optional"""
 
-    protocols: List[str]
+    protocols: List[str] = None
     """A list of connection protocols supported by the proxy, e.g., ['http', 'https']"""

proxyproviders/providers/webshare.py

@@ -28,7 +28,7 @@ class Webshare(ProxyProvider):
         proxy_provider = Webshare(api_key="your-api-key", params={"country_code_in": "US"})
 
         # Fetch proxies
-        proxies = proxy_provider.fetch_proxies()
+        proxies = proxy_provider.list_proxies()
 
         # With config
         from proxyproviders import ProxyConfig
@@ -37,7 +37,7 @@ class Webshare(ProxyProvider):
         proxy_provider = Webshare(api_key="your-api-key", params={"country_code_in": "US"}, config=config)
 
         # Fetch proxies
-        proxies = proxy_provider.fetch_proxies()
+        proxies = proxy_provider.list_proxies()
     """
     _BASE_URL = "https://proxy.webshare.io/api/v2"
     _PROTOCOLS = ["http", "https"]
@@ -80,7 +80,6 @@ def _fetch_proxies(self) -> List[Proxy]:
             if data.get("next") is None: # no more pages
                 break
 
-        self._set_proxies(all_proxies)
         return all_proxies
 
     def _convert_to_proxy(self, data: Dict) -> Proxy:

proxyproviders/proxy_provider.py

@@ -49,7 +49,8 @@ def list_proxies(self, force_refresh: bool = False) -> List[Proxy]:
         :param force_refresh: If True, fetches new proxies even if the list is not considered stale.
         """
         if force_refresh or self.should_refresh() or not self._proxies:
-            self._fetch_proxies()
+            proxies = self._fetch_proxies()
+            self._set_proxies(proxies)
 
         with self._lock:
             return list(self._proxies) if self._proxies else []
@@ -59,7 +60,7 @@ def list_proxies(self, force_refresh: bool = False) -> List[Proxy]:
     #
     @abstractmethod
     def _fetch_proxies(self) -> List[Proxy]:
-        """Fetch proxies from the provider and calls super()._set_proxies().
+        """Fetch proxies from the provider implementation.
         This is not meant to be called directly, as it will always pull from the API even if not stale.
         """
         pass

tests/test_proxy_provider.py

@@ -38,7 +38,6 @@ def test_force_refresh():
     class TestProxyProvider(MockProxyProvider):
         def _fetch_proxies(self) -> List[Proxy]:
             proxies = [1, 2, 3]
-            self._set_proxies(proxies)
             return proxies
 
     provider = TestProxyProvider()
@@ -62,12 +61,6 @@ def test_refresh_interval():
     # When refresh_interval is 0, it should never refresh automatically
     provider = MockProxyProvider(config=ProxyConfig(refresh_interval=0))
     provider._fetch_proxies = lambda: [1, 2, 3]
-    
-    # No proxies initially, so should be an empty list
-    assert provider.list_proxies() == []
-    
-    # Proxies should not refresh even after calling list_proxies
-    assert provider.list_proxies() == []
 
     # Simulate a fetch and set proxies to check refresh
     provider._set_proxies([1, 2, 3])