docs(fortuna): add comprehensive multi-replica setup documentation

- Add Multiple Replica Setup section to README.md with modulo assignment explanation
- Add replica_config examples to config.sample.yaml for 2, 3, and 5 replica setups
- Include deployment considerations, failover behavior, and wallet separation requirements
- Add validation for backup_delay_seconds > 0 to prevent race conditions

Co-Authored-By: Tejas Badadare <[email protected]>

Author: devin-ai-integration[bot]

apps/fortuna/README.md

@@ -40,6 +40,60 @@ Please add the changed files in the `.sqlx` folder to your git commit.
 The Fortuna binary has a command-line interface to perform useful operations on the contract, such as
 registering a new randomness provider, or drawing a random value. To see the available commands, simply run `cargo run`.
 
+## Multiple Replica Setup
+
+Fortuna supports running multiple replica instances for high availability and reliability. This prevents service interruption if one instance goes down and distributes the workload across multiple instances.
+
+### How Replica Assignment Works
+
+- Each replica is assigned a unique `replica_id` (0, 1, 2, etc.)
+- Requests are distributed using modulo assignment: `sequence_number % total_replicas`
+- Each replica primarily handles requests assigned to its ID
+- After a configurable delay, replicas will process requests from other replicas as backup (failover)
+
+### Example Configurations
+
+**Two Replica Setup (Blue/Green):**
+```yaml
+# Replica 0 (Blue) - handles even sequence numbers (0, 2, 4, ...)
+keeper:
+  replica_config:
+    replica_id: 0
+    total_replicas: 2
+    backup_delay_seconds: 30
+
+# Replica 1 (Green) - handles odd sequence numbers (1, 3, 5, ...)  
+keeper:
+  replica_config:
+    replica_id: 1
+    total_replicas: 2
+    backup_delay_seconds: 30
+```
+
+**Three Replica Setup:**
+```yaml
+# Replica 0 - handles sequence numbers 0, 3, 6, 9, ...
+keeper:
+  replica_config:
+    replica_id: 0
+    total_replicas: 3
+    backup_delay_seconds: 45
+```
+
+### Deployment Considerations
+
+1. **Separate Wallets**: Each replica MUST use a different private key to avoid nonce conflicts
+2. **Backup Delay**: Set `backup_delay_seconds` long enough to allow primary replica to process requests, but short enough for acceptable failover time (recommended: 30-60 seconds)
+3. **Monitoring**: Monitor each replica's processing metrics to ensure proper load distribution
+4. **Gas Management**: Each replica needs sufficient ETH balance for gas fees
+
+### Failover Behavior
+
+- Primary replica processes requests immediately
+- Backup replicas wait for `backup_delay_seconds` before checking if request is still unfulfilled
+- If request is already fulfilled during the delay, backup replica skips processing
+- This prevents duplicate transactions and wasted gas while ensuring reliability
+
 ## Local Development
 
 To start an instance of the webserver for local testing, you first need to perform a few setup steps:

apps/fortuna/config.sample.yaml

@@ -88,3 +88,31 @@ keeper:
     value: 0xabcd
     # For production, you can store the private key in a file.
     # file: keeper-key.txt
+# Multi-replica configuration documentation
+
+  # Optional: Multi-replica configuration for high availability and load distribution
+  # Uncomment and configure for production deployments with multiple Fortuna instances
+  # replica_config:
+  #   replica_id: 0              # Unique identifier for this replica (0, 1, 2, ...)
+  #   total_replicas: 2          # Total number of replica instances running
+  #   backup_delay_seconds: 30   # Seconds to wait before processing other replicas' requests
+  #
+  # Example configurations:
+  # 
+  # Two-replica setup (Blue/Green):
+  # - Replica 0: handles even sequence numbers (0, 2, 4, ...)
+  # - Replica 1: handles odd sequence numbers (1, 3, 5, ...)
+  #
+  # Three-replica setup:
+  # - Replica 0: handles sequence numbers 0, 3, 6, 9, ...
+  # - Replica 1: handles sequence numbers 1, 4, 7, 10, ...
+  # - Replica 2: handles sequence numbers 2, 5, 8, 11, ...
+  #
+  # Five-replica setup:
+  # - Replica 0: handles sequence numbers 0, 5, 10, 15, ...
+  # - Replica 1: handles sequence numbers 1, 6, 11, 16, ...
+  # - Replica 2: handles sequence numbers 2, 7, 12, 17, ...
+  # - Replica 3: handles sequence numbers 3, 8, 13, 18, ...
+  # - Replica 4: handles sequence numbers 4, 9, 14, 19, ...
+  #
+  # IMPORTANT: Each replica MUST use a different private_key to avoid nonce conflicts!

apps/fortuna/src/config.rs

@@ -101,6 +101,9 @@ impl Config {
             if replica_config.replica_id >= replica_config.total_replicas {
                 return Err(anyhow!("Keeper replica configuration is invalid. replica_id must be less than total_replicas."));
             }
+            if replica_config.backup_delay_seconds == 0 {
+                return Err(anyhow!("Keeper replica configuration is invalid. backup_delay_seconds must be greater than 0 to prevent race conditions."));
+            }
         }
 
         Ok(config)