feat: add rolling deployment support for zero-downtime proxy recycling

- Add RollingDeploymentManager to coordinate proxy recycling across providers
- Implement minimum availability constraints during age-based recycling
- Add batch size limits to control concurrent recycling operations
- Update all providers (DigitalOcean, AWS, GCP, Hetzner, Vultr) with rolling logic
- Add configuration via environment variables (ROLLING_DEPLOYMENT, ROLLING_MIN_AVAILABLE, ROLLING_BATCH_SIZE)
- Implement automatic adjustment when min_available >= min_scaling to prevent deadlock
- Add comprehensive API endpoints for monitoring and controlling rolling deployments
- Include detailed documentation and best practices guide
- Add extensive unit tests with 100% coverage of rolling deployment logic

Author: claffin

README.md

@@ -46,6 +46,7 @@
   - [Web Interface](#web-interface)
   - [API Documentation](#api-documentation)
   - [Programmatic Usage](#programmatic-usage)
+- [Rolling Deployments](#rolling-deployments)
 - [Multi-Account Provider Support](#multi-account-provider-support)
 - [API Examples](#cloudproxy-api-examples)
 - [Roadmap](#roadmap)
@@ -85,6 +86,7 @@ CloudProxy exposes an API and modern UI for managing your proxy infrastructure.
 * Multi-provider support
 * Multiple accounts per provider
 * Automatic proxy rotation
+* **Rolling deployments** - Zero-downtime proxy recycling
 * Health monitoring
 * Fixed proxy pool management (maintains target count)
 
@@ -293,6 +295,47 @@ my_request = requests.get("https://api.ipify.org", proxies=proxies)
 
 For more detailed examples of using CloudProxy as a Python package, see the [Python Package Usage Guide](docs/python-package-usage.md).
 
+## Rolling Deployments
+
+CloudProxy supports rolling deployments to ensure zero-downtime proxy recycling. This feature maintains a minimum number of healthy proxies during age-based recycling operations.
+
+### Configuration
+
+Enable rolling deployments with these environment variables:
+
+```bash
+# Enable rolling deployments
+ROLLING_DEPLOYMENT=True
+
+# Minimum proxies to keep available during recycling
+ROLLING_MIN_AVAILABLE=3  
+
+# Maximum proxies to recycle simultaneously
+ROLLING_BATCH_SIZE=2
+```
+
+### How It Works
+
+When proxies reach their age limit:
+1. The system checks if recycling would violate minimum availability
+2. Proxies are recycled in batches to maintain service continuity
+3. New proxies are created as old ones are removed
+4. The process continues until all aged proxies are replaced
+
+### Monitoring
+
+Check rolling deployment status via the API:
+
+```bash
+# Get overall status
+curl http://localhost:8000/rolling
+
+# Get provider-specific status
+curl http://localhost:8000/rolling/digitalocean
+```
+
+For detailed documentation, see the [Rolling Deployments Guide](docs/rolling-deployments.md).
+
 ## Multi-Account Provider Support
 
 CloudProxy now supports multiple accounts per provider, allowing you to:

cloudproxy/main.py

@@ -19,6 +19,7 @@
 
 from cloudproxy.providers import settings
 from cloudproxy.providers.settings import delete_queue, restart_queue
+from cloudproxy.providers.rolling import rolling_manager
 
 sys.path.append(os.path.dirname(os.path.dirname(os.path.dirname(__file__))))
 
@@ -726,6 +727,162 @@ def configure_instance(
         config=ProviderInstance(**instance_config)
     )
 
+# Rolling Deployment Models
+class RollingDeploymentConfig(BaseModel):
+    enabled: bool = Field(description="Whether rolling deployment is enabled")
+    min_available: int = Field(ge=0, description="Minimum number of proxies to keep available during recycling")
+    batch_size: int = Field(ge=1, description="Maximum number of proxies to recycle simultaneously")
+
+class RollingDeploymentStatus(BaseModel):
+    healthy: int = Field(description="Number of healthy proxies")
+    pending: int = Field(description="Number of pending proxies")
+    pending_recycle: int = Field(description="Number of proxies pending recycling")
+    recycling: int = Field(description="Number of proxies currently being recycled")
+    last_update: str = Field(description="Last update timestamp")
+    healthy_ips: List[str] = Field(description="List of healthy proxy IPs")
+    pending_recycle_ips: List[str] = Field(description="List of IPs pending recycling")
+    recycling_ips: List[str] = Field(description="List of IPs currently being recycled")
+
+class RollingDeploymentResponse(BaseModel):
+    metadata: Metadata = Field(default_factory=Metadata)
+    message: str
+    config: RollingDeploymentConfig
+    status: Dict[str, RollingDeploymentStatus] = Field(description="Status by provider/instance")
+
+@app.get("/rolling", tags=["Rolling Deployment"], response_model=RollingDeploymentResponse)
+def get_rolling_deployment_status():
+    """
+    Get the current rolling deployment configuration and status.
+    
+    Returns:
+        RollingDeploymentResponse: Current rolling deployment configuration and status
+    """
+    config = RollingDeploymentConfig(
+        enabled=settings.config["rolling_deployment"]["enabled"],
+        min_available=settings.config["rolling_deployment"]["min_available"],
+        batch_size=settings.config["rolling_deployment"]["batch_size"]
+    )
+    
+    raw_status = rolling_manager.get_recycling_status()
+    status = {}
+    for key, data in raw_status.items():
+        status[key] = RollingDeploymentStatus(**data)
+    
+    return RollingDeploymentResponse(
+        message="Rolling deployment status retrieved successfully",
+        config=config,
+        status=status
+    )
+
+@app.patch("/rolling", tags=["Rolling Deployment"], response_model=RollingDeploymentResponse)
+def update_rolling_deployment_config(update: RollingDeploymentConfig):
+    """
+    Update the rolling deployment configuration.
+    
+    Args:
+        update: New rolling deployment configuration
+        
+    Returns:
+        RollingDeploymentResponse: Updated configuration and current status
+    """
+    # Update configuration
+    settings.config["rolling_deployment"]["enabled"] = update.enabled
+    settings.config["rolling_deployment"]["min_available"] = update.min_available
+    settings.config["rolling_deployment"]["batch_size"] = update.batch_size
+    
+    # Get current status
+    raw_status = rolling_manager.get_recycling_status()
+    status = {}
+    for key, data in raw_status.items():
+        status[key] = RollingDeploymentStatus(**data)
+    
+    return RollingDeploymentResponse(
+        message="Rolling deployment configuration updated successfully",
+        config=update,
+        status=status
+    )
+
+@app.get("/rolling/{provider}", tags=["Rolling Deployment"], response_model=RollingDeploymentResponse)
+def get_provider_rolling_status(provider: str):
+    """
+    Get rolling deployment status for a specific provider.
+    
+    Args:
+        provider: The name of the provider
+        
+    Returns:
+        RollingDeploymentResponse: Rolling deployment status for the provider
+        
+    Raises:
+        HTTPException: If the provider is not found
+    """
+    if provider not in settings.config["providers"]:
+        raise HTTPException(
+            status_code=404,
+            detail=f"Provider '{provider}' not found"
+        )
+    
+    config = RollingDeploymentConfig(
+        enabled=settings.config["rolling_deployment"]["enabled"],
+        min_available=settings.config["rolling_deployment"]["min_available"],
+        batch_size=settings.config["rolling_deployment"]["batch_size"]
+    )
+    
+    raw_status = rolling_manager.get_recycling_status(provider=provider)
+    status = {}
+    for key, data in raw_status.items():
+        status[key] = RollingDeploymentStatus(**data)
+    
+    return RollingDeploymentResponse(
+        message=f"Rolling deployment status for '{provider}' retrieved successfully",
+        config=config,
+        status=status
+    )
+
+@app.get("/rolling/{provider}/{instance}", tags=["Rolling Deployment"], response_model=RollingDeploymentResponse)
+def get_instance_rolling_status(provider: str, instance: str):
+    """
+    Get rolling deployment status for a specific provider instance.
+    
+    Args:
+        provider: The name of the provider
+        instance: The name of the instance
+        
+    Returns:
+        RollingDeploymentResponse: Rolling deployment status for the instance
+        
+    Raises:
+        HTTPException: If the provider or instance is not found
+    """
+    if provider not in settings.config["providers"]:
+        raise HTTPException(
+            status_code=404,
+            detail=f"Provider '{provider}' not found"
+        )
+    
+    if instance not in settings.config["providers"][provider]["instances"]:
+        raise HTTPException(
+            status_code=404,
+            detail=f"Provider '{provider}' instance '{instance}' not found"
+        )
+    
+    config = RollingDeploymentConfig(
+        enabled=settings.config["rolling_deployment"]["enabled"],
+        min_available=settings.config["rolling_deployment"]["min_available"],
+        batch_size=settings.config["rolling_deployment"]["batch_size"]
+    )
+    
+    raw_status = rolling_manager.get_recycling_status(provider=provider, instance=instance)
+    status = {}
+    for key, data in raw_status.items():
+        status[key] = RollingDeploymentStatus(**data)
+    
+    return RollingDeploymentResponse(
+        message=f"Rolling deployment status for '{provider}/{instance}' retrieved successfully",
+        config=config,
+        status=status
+    )
+
 if __name__ == "__main__":
     main()
 

cloudproxy/providers/aws/main.py

@@ -12,6 +12,7 @@
     start_proxy,
 )
 from cloudproxy.providers.settings import delete_queue, restart_queue, config
+from cloudproxy.providers.rolling import rolling_manager
 
 
 def aws_deployment(min_scaling, instance_config=None):
@@ -58,18 +59,28 @@ def aws_check_alive(instance_config=None):
     """
     if instance_config is None:
         instance_config = config["providers"]["aws"]["instances"]["default"]
+    
+    # Get instance name for rolling deployment tracking
+    instance_name = next(
+        (name for name, inst in config["providers"]["aws"]["instances"].items() 
+         if inst == instance_config), 
+        "default"
+    )
 
     ip_ready = []
+    pending_ips = []
+    instances_to_recycle = []
+    
+    # First pass: identify healthy and pending instances
     for instance in list_instances(instance_config):
         try:
             elapsed = datetime.datetime.now(
                 datetime.timezone.utc
             ) - instance["Instances"][0]["LaunchTime"]
+            
             if config["age_limit"] > 0 and elapsed > datetime.timedelta(seconds=config["age_limit"]):
-                delete_proxy(instance["Instances"][0]["InstanceId"], instance_config)
-                logger.info(
-                    f"Recycling AWS {instance_config.get('display_name', 'default')} instance, reached age limit -> " + instance["Instances"][0]["PublicIpAddress"]
-                )
+                # Queue for potential recycling
+                instances_to_recycle.append((instance, elapsed))
             elif instance["Instances"][0]["State"]["Name"] == "stopped":
                 logger.info(
                     f"Waking up: AWS {instance_config.get('display_name', 'default')} -> Instance " + instance["Instances"][0]["InstanceId"]
@@ -87,7 +98,9 @@ def aws_check_alive(instance_config=None):
                 logger.info(
                     f"Pending: AWS {instance_config.get('display_name', 'default')} -> " + instance["Instances"][0]["PublicIpAddress"]
                 )
-            # Must be "pending" if none of the above, check if alive or not.
+                if "PublicIpAddress" in instance["Instances"][0]:
+                    pending_ips.append(instance["Instances"][0]["PublicIpAddress"])
+            # Must be "running" if none of the above, check if alive or not.
             elif check_alive(instance["Instances"][0]["PublicIpAddress"]):
                 logger.info(
                     f"Alive: AWS {instance_config.get('display_name', 'default')} -> " + instance["Instances"][0]["PublicIpAddress"]
@@ -104,8 +117,57 @@ def aws_check_alive(instance_config=None):
                     logger.info(
                         f"Waiting: AWS {instance_config.get('display_name', 'default')} -> " + instance["Instances"][0]["PublicIpAddress"]
                     )
+                    if "PublicIpAddress" in instance["Instances"][0]:
+                        pending_ips.append(instance["Instances"][0]["PublicIpAddress"])
         except (TypeError, KeyError):
             logger.info(f"Pending: AWS {instance_config.get('display_name', 'default')} -> allocating ip")
+    
+    # Update rolling manager with current proxy health status
+    rolling_manager.update_proxy_health("aws", instance_name, ip_ready, pending_ips)
+    
+    # Handle rolling deployments for age-limited instances
+    if instances_to_recycle and config["rolling_deployment"]["enabled"]:
+        rolling_config = config["rolling_deployment"]
+        
+        for inst, elapsed in instances_to_recycle:
+            if "PublicIpAddress" in inst["Instances"][0]:
+                instance_ip = inst["Instances"][0]["PublicIpAddress"]
+                
+                # Check if we can recycle this instance according to rolling deployment rules
+                if rolling_manager.can_recycle_proxy(
+                    provider="aws",
+                    instance=instance_name,
+                    proxy_ip=instance_ip,
+                    total_healthy=len(ip_ready),
+                    min_available=rolling_config["min_available"],
+                    batch_size=rolling_config["batch_size"],
+                    rolling_enabled=True,
+                    min_scaling=instance_config["scaling"]["min_scaling"]
+                ):
+                    # Mark as recycling and delete
+                    rolling_manager.mark_proxy_recycling("aws", instance_name, instance_ip)
+                    delete_proxy(inst["Instances"][0]["InstanceId"], instance_config)
+                    rolling_manager.mark_proxy_recycled("aws", instance_name, instance_ip)
+                    logger.info(
+                        f"Rolling deployment: Recycled AWS {instance_config.get('display_name', 'default')} instance (age limit) -> {instance_ip}"
+                    )
+                else:
+                    logger.info(
+                        f"Rolling deployment: Deferred recycling AWS {instance_config.get('display_name', 'default')} instance -> {instance_ip}"
+                    )
+    elif instances_to_recycle and not config["rolling_deployment"]["enabled"]:
+        # Standard non-rolling recycling
+        for inst, elapsed in instances_to_recycle:
+            delete_proxy(inst["Instances"][0]["InstanceId"], instance_config)
+            if "PublicIpAddress" in inst["Instances"][0]:
+                logger.info(
+                    f"Recycling AWS {instance_config.get('display_name', 'default')} instance, reached age limit -> " + inst["Instances"][0]["PublicIpAddress"]
+                )
+            else:
+                logger.info(
+                    f"Recycling AWS {instance_config.get('display_name', 'default')} instance, reached age limit -> " + inst["Instances"][0]["InstanceId"]
+                )
+    
     return ip_ready