add defined behaviors to the ServiceCIDR API and behaviors

Author: aojea

keps/sig-network/1880-multiple-service-cidrs/README.md

@@ -135,6 +135,8 @@ Implement a new allocation logic for Services IPs that:
   services.spec.IPFamily and services.spec.IPFamilyPolicy, a simple webhook or an admission plugin
   can set this fields to the desired default, so the allocation logic doesn't have to handle it.
 - Removing the apiserver flags that define the service IP CIDRs, though that may be possible in the future.
+- Any admin or cluster wide process related to Services, like automating the default Service CIDR range, though,
+this KEP will implment the behaviours and primitives necessaries to perform those kind of operations automatically.
 
 ## Proposal
 
@@ -162,6 +164,8 @@ different and collide with other APIs, like Gateway APIs, we are adding the foll
 
  The new allocator logic can be used by other APIs, like Gateway API.
 
+The new well defined behaviors and objects implemented will allow future developments to perform admin and cluster wide operations on the Service ranges.
+
 ### User Stories
 
 #### Story 1
@@ -279,12 +283,7 @@ and Services.
 In order to be completely backwards compatible, the bootstrap process will remain the same, the
 difference is that instead of creating a bitmap based on the flags, it will create a new
 ServiceCIDR object from the flags (flags configuration removal is out of scope of this KEP)
- ...
-
-```
-<<[UNRESOLVED bootstrap>>
-Option 1:
-... with a special well-known name `kubernetes`.
+with a special well-known name `kubernetes`.
 
 The new bootstrap process will be:
 
@@ -309,68 +308,25 @@ All the apiservers will be synchronized on the ServiceCIDR and default Service c
 Changes on the configuration imply manual removal of the ServiceCIDR and default Service, automatically
 the rest of the apiservers will race and the winner will set the configuration of the cluster.
 
-Pros:
-- Simple to implement
-- Align with current behavior of kubernetes.default, though this can be a Con as well, since this
-the existing behavior was unexpected
-Cons:
-- Requires manual intervention
-
-Option 2:
-... with a special label `networking.kubernetes.io/service-cidr-from-flags` set to `"true"`.
-
-It now has to handle the possibility of multiple ServiceCIDR with the special label, and
-also updating the configuration, per example, from single-stack to dual-stack.
-
-The new bootstrap process will be:
+This behavior align with current behavior of kubernetes.default, that it makes it consistent and easier to think
+about, allowing future developments to use it to implement more complex operations at the admin cluster level.
 
-```
-at startup:
- read_flags
- if invalid flags
-  exit
- run default-service-ip-range controller
- run kubernetes.default service loop (it uses the first ip from the subnet defined in the flags)
- run service-repair loop (reconcile services, ipaddresses)
- run apiserver
-
-controller:
- watch all ServiceCIDR objects labelled from-flags
-  ignore being-deleted ranges
- wait for first sync
-
-controller on_event:
- if no default range matching exactly my flags
-  log
-  create a ServiceCIDR from my flags
-   generateName: "from-flags-"
-   from-flags label: "true"
- else if multiple
-  log
-  if multiple ranges match exactly my flags (or a single-family subset of)
-    log
-    delete all subsets, leaving the largest set that exactly matches on at least on family
-  endif
- endif
- if kubernetes.default does not exist
-  create it
-```
+#### The special "default" ServiceCIDR
 
-Pros:
-- Automatically handles conflicts, no admin operation required
-Cons:
-- Complex to implement
+The `kubernetes.default` Service is expected to be covered by a valid range. Each apiserver 
+ensure that a ServiceCIDR object exists to cover its own flag-defined ranges. If someone were to force-delete
+the ServiceCIDR covering `kubernetes.default` it would be treated the same as before, any apiserver will try
+to recreate the Service from its configured default Service CIDR flag-defined range.
 
-<<[/UNRESOLVED]>>
-```
-
-#### The special "default" ServiceCIDR
+This well-known an establish behavior can allow administrators to replace the `kubernetes.default` by a
+series of operations:
+1. Initial state: 2 kube-apiservers with default ServiceCIDR 10.0.0.0/24
+2. Apiservers will create the `kubernetes.default` Service with ClusterIP 10.0.0.1.
+3. Upgrade kube-apiservers and replace the service-cidr flag to 192.168.0.0/24
+4. Delete the ServiceCIDRs objects and the `kubernetes.default` Service.
+5. The kube-apiserver will recreate the `kubernetes.default` with IP 192.168.0.1.
 
-The `kubernetes.default` Service is expected to be covered by a valid range. Each apiserver will
-ensure that a ServiceCIDR object exists to cover its own flag-defined ranges, so this should
-be true in normal cases. If someone were to force-delete the ServiceCIDR covering `kubernetes.default` it
-would be treated the same as any Service in the repair loop, which will generate warnings about
-orphaned Service IPs.
+Note this can also be used to switch the IP family of the cluster.
 
 #### Service IP Allocation
 
@@ -481,7 +437,8 @@ deletion can take some time, one allocator can successfully allocate an IP addre
 inside of the old Service CIDR).
 
 There is one controller that will periodically check that the 1-on-1 relation between IPAddresses and Services is
-correct, and will start sending events to warn the user that it has to fix/recreate the corresponding Service.
+correct, and will start sending events to warn the user that it has to fix/recreate the corresponding Service,
+keeping the same behavior that exists today.
 
 #### API