correct with latest termination decisions, add pseudocode

Signed-off-by: Matthias Bertschy <[email protected]>

Author: matthyx

keps/sig-node/753-sidecar-containers/README.md

@@ -872,22 +872,88 @@ Here is the proposed approach:
 1. Sidecar containers that have a `PreStop` hook will be notified when the Pod has begun terminating
    by executing the `PreStop` hook. This happens at the same time as regular containers, and begins
    the Pod's termination grace period countdown.
-2. Sidecar containers enter the `Terminated` state and are no longer restarted if they fail.
-3. Once the last primary container terminates, the last started sidecar container is notified by
+2. Once the last primary container terminates, the last started sidecar container is notified by
    sending a `SIGTERM` signal.
-4. The next sidecar (in reverse order) is notified by sending a `SIGTERM` signal after the previous
+3. The next sidecar (in reverse order) is notified by sending a `SIGTERM` signal after the previous
    sidecar container terminates.
-5. This continues until all sidecar containers have terminated, or the Pod's termination grace period
-   expires. In the latter case, all containers are terminated with minimum grace period and the Pod
-   will be terminated after that.
+4. This continues until all sidecar containers have terminated, or the Pod's termination grace period
+   expires.
+5. In the latter case, all remaining containers are notified by a `SIGTERM`, followed by a fixed
+   grace period of 2s and finally terminated.
+6. The Pod will be terminated after that.
+
+Pseudocode for the above:
+
+```
+func terminatePod() {
+  // notify all sidecar containers with preStop hook, asynchronously
+  for sidecar in sidecarContainers {
+    if sidecar has preStop hook {
+      go execute preStop hook // async
+    }
+  }
+  // notify all containers with preStop hook and then SIGTERM, asynchronously
+  for container in containers {
+    if container has preStop hook {
+      go func(container) { // async
+        execute preStop hook
+        send SIGTERM
+      }
+    }
+  }
+  for {
+    switch {
+      case grace period expired:
+        for anyContainer in sidecarContainers + containers {
+          if anyContainer is running {
+            send SIGTERM
+          }
+        }
+        sleep 2s
+        for anyContainer in sidecarContainer + containers {
+          if anyContainer is running {
+            send SIGKILL
+          }
+        }
+        return
+      case all containers are terminated:
+        // sidecars are terminated in reverse order
+        for sidecar in reverse(sidecarContainers) {
+          // sidecar is already terminating, let it finish
+          if sidecar is terminating {
+            break
+          }
+          // next sidecar to terminate
+          else if sidecar is running {
+            send SIGTERM
+            break
+          }
+        }
+        sleep 1s
+      case all sidecarContainers are terminated:
+        return
+      default:
+        sleep 1s
+    }
+  }
+}
+```
 
 It is worth noting that, like with regular containers, `PreStop` hook must complete before the `SIGTERM`
 signal to stop the sidecar container can be sent. Therefore, ordering and graceful termination of sidecars
 can only be guaranteed if the `PreStop` hook completes within the Pod's termination grace period.
 
+Sidecars continue to be restarted until they enter the `Terminated` state which they are notified
+by a `SIGTERM` signal. This is to ensure that sidecars that fail are restarted until the TGPS expires.
+
+We might postpone running the `livenessProbe` for restarted sidecar containers during termination
+until GA, depending on the implementation complexity.
+
 If we compare this to the initial proposal, the following behaviors are preserved:
 - Sidecars should not begin termination until all primary containers have
   terminated.
+  - Implicit in this is that sidecars should continue to be restarted until all
+    primary containers have terminated.
 - Sidecars should terminate serially and in reverse order. I.e. the first
   sidecar to initialize should be the last sidecar to terminate.
 
@@ -898,12 +964,6 @@ The additional benefits of this approach comparing to initial proposal:
   same time as before and can utilize as much of the graceful termination period as they need. The Pod graceful
   termination period semantic also stay unchanged.
 
-However, the following behaviors is not preserved:
-- Sidecars should continue to be restarted until all primary containers have terminated.
-
-We think this is a reasonable tradeoff and consistent with the best effort nature of keeping 
-sidecar containers running during the Pod's lifetime.
-
 ### Other
 
 This behavior needs to be adjusted: