Backport: Adjust service chain ownership when reassigning services between workers (#5435)

examples/controller/Cargo.toml

@@ -10,6 +10,7 @@ bcs.workspace = true
 futures.workspace = true
 linera-sdk.workspace = true
 log.workspace = true
+serde = { workspace = true, features = ["derive"] }
 serde_json = { workspace = true }
 
 [target.'cfg(not(target_arch = "wasm32"))'.dev-dependencies]

examples/controller/README.md

@@ -92,26 +92,77 @@ sequenceDiagram
     participant W2 as Worker 2
     A->>C: ExecuteControllerCommand<br/>{UpdateService}
     C->>C: Update services registry
-    C->>W1: Message::Start{service_id}
-    C->>W2: Message::Start{service_id}
+    C->>W1: Start{service_id,<br/>owners_to_remove: {},<br/>start_height: None}
+    C->>W2: Start{service_id,<br/>owners_to_remove: {},<br/>start_height: None}
     W1->>W1: Add to local_services
     W2->>W2: Add to local_services
 ```
 
+### Service Handoff (Reassignment)
+
+When a service is moved from one worker to another, all three chain types
+participate in a two-phase handoff protocol. The new worker's owners are added
+to the service chain first, then the old worker's owners are removed, ensuring
+there is no gap in ownership.
+
+```mermaid
+sequenceDiagram
+    participant A as Admin
+    participant C as Controller Chain
+    participant WA as Old Worker Chain
+    participant S as Service Chain
+    participant WB as New Worker Chain
+
+    A->>C: ExecuteControllerCommand<br/>{UpdateService{service, [WB]}}
+    C->>C: Record pending handoff
+    C->>WA: Stop{service, new_owners: [WB]}
+
+    note over WA,S: Phase 1: Add new owners
+    WA->>S: AddOwners{service, [WB]}
+    S->>S: Add WB as chain owner
+    S->>WA: OwnersAdded{service, block_height}
+
+    WA->>WA: Remove from local_services
+    WA->>C: HandoffStarted{service, block_height}
+
+    C->>C: Resolve pending handoff
+    C->>WB: Start{service,<br/>owners_to_remove: [WA],<br/>start_height: block_height}
+    WB->>WB: Add to local_pending_services
+
+    note over WB,S: Phase 2: Remove old owners
+    WB->>WB: StartLocalService at start_height
+    WB->>S: RemoveOwners{[WA]}
+    S->>S: Remove WA as chain owner
+    WB->>WB: Move to local_services
+```
+
 ### Service Removal
 
+When a service is removed, Stop messages are sent to all workers. Each worker
+initiates ownership cleanup through the service chain before removing the
+service locally, following the same handoff protocol but with empty new owners.
+
 ```mermaid
 sequenceDiagram
     participant A as Admin
     participant C as Controller Chain
     participant W1 as Worker 1
     participant W2 as Worker 2
+    participant S as Service Chain
     A->>C: ExecuteControllerCommand<br/>{RemoveService}
     C->>C: Remove from services registry
-    C->>W1: Message::Stop{service_id}
-    C->>W2: Message::Stop{service_id}
+    C->>W1: Stop{service_id, new_owners: {}}
+    C->>W2: Stop{service_id, new_owners: {}}
+
+    W1->>S: AddOwners{service_id, {}}
+    S->>W1: OwnersAdded{service_id, block_height}
     W1->>W1: Remove from local_services
+    W1->>C: HandoffStarted{service_id, block_height}
+
+    W2->>S: AddOwners{service_id, {}}
+    S->>W2: OwnersAdded{service_id, block_height}
     W2->>W2: Remove from local_services
+    W2->>C: HandoffStarted{service_id, block_height}
 ```
 
 ## Operations
@@ -156,7 +207,11 @@ Messages are sent between chains to coordinate state:
 | `ExecuteWorkerCommand` | Worker -> Controller | Register/deregister worker |
 | `ExecuteControllerCommand` | Any -> Controller | Admin commands |
 | `Reset` | Controller -> Worker | Clear worker state |
-| `Start { service_id }` | Controller -> Worker | Start a service |
-| `Stop { service_id }` | Controller -> Worker | Stop a service |
+| `Start { service_id, owners_to_remove, start_height }` | Controller -> Worker | Start a service, optionally with handoff info |
+| `Stop { service_id, new_owners }` | Controller -> Worker | Stop a service, initiating ownership handoff via the service chain |
 | `FollowChain { chain_id }` | Controller -> Worker | Follow a chain |
 | `ForgetChain { chain_id }` | Controller -> Worker | Stop following a chain |
+| `AddOwners { service_id, new_owners }` | Worker -> Service Chain | Add new owners to a service chain during handoff |
+| `RemoveOwners { owners_to_remove }` | Worker -> Service Chain | Remove old owners from a service chain after handoff |
+| `OwnersAdded { service_id, added_at }` | Service Chain -> Worker | Confirm new owners were added at a given block height |
+| `HandoffStarted { service_id, target_block_height }` | Worker -> Controller | Notify controller that handoff phase 1 is complete |