don't sleep for infinity but 2 hours instead (#435)

# Summary

This pull request primarily updates the sleep durations for various
BusyBox sidecar containers and test scripts across multiple Kubernetes
fixtures and test files, changing them from indefinite or long-running
sleeps to a fixed 7200 seconds (2 hours). Additionally, it includes a
minor improvement to the cluster cleaner script to prevent failures when
querying deleted resources, and bumps the image version for the cluster
cleaner.

**Kubernetes fixtures and pod specs:**
* Changed BusyBox container `args` from `"infinity"` or long sleep
values to `"7200"` seconds in multiple test fixture files
(`mongodb-multi-central-sts-override.yaml`,
`mongodb-multi-sts-override.yaml`, `om_ops_manager_pod_spec.yaml`,
`replica-set-custom-podspec.yaml`,
`sharded-cluster-custom-podspec.yaml`, `standalone-custom-podspec.yaml`)
to avoid issues with indefinite sleep and improve test reliability.


**Test deployment templates:**
* Changed the keepalive container command in
`mongodb-enterprise-tests.yaml` to use `sleep 7200` instead of `sleep
3600` for consistency with other changes.

**Cluster cleaner maintenance:**
* Improved error handling in `clean-failed-namespaces.sh` by suppressing
errors when querying for deleted resources, ensuring the script
continues deletion without failing.
* Bumped `IMAGE_VERSION` in the cluster cleaner `Makefile` from `0.14`
to `0.15` to reflect these updates.
## Proof of Work

- green ci 
- successful master run 

## Checklist

- [x] Have you linked a jira ticket and/or is the ticket in the title?
- [x] Have you checked whether your jira ticket required DOCSP changes?
- [x] Have you added changelog file?
    - use `skip-changelog` label if not needed
- refer to [Changelog files and Release
Notes](https://github.com/mongodb/mongodb-kubernetes/blob/master/CONTRIBUTING.md#changelog-files-and-release-notes)
section in CONTRIBUTING.md for more details

Author: nammn

docker/cluster-cleaner/Makefile

@@ -1,4 +1,4 @@
-IMAGE_VERSION=0.14
+IMAGE_VERSION=0.15
 
 .PHONY: all
 all: build push install

docker/cluster-cleaner/scripts/clean-failed-namespaces.sh

@@ -8,7 +8,8 @@ delete_resources_safely() {
     kubectl delete "${resource_type}" --all -n "${namespace}" --wait=true --timeout=10s || true
 
     # Check if any resources are still stuck
-    resources=$(kubectl get "$resource_type" -n "${namespace}" --no-headers -o custom-columns=":metadata.name")
+    # Let's not fail here and continue deletion
+    resources=$(kubectl get "$resource_type" -n "${namespace}" --no-headers -o custom-columns=":metadata.name" 2>/dev/null || true)
 
     for resource in ${resources}; do
         echo "${resource_type}/${resource} is still present, force deleting..."

docker/mongodb-kubernetes-tests/tests/multicluster/fixtures/mongodb-multi-central-sts-override.yaml

@@ -12,13 +12,13 @@ spec:
     spec:
       template:
         spec:
-          # FIXME workaround for sleep infinity hanging
+          # FIXME workaround for sleep 7200 hanging
           shareProcessNamespace: true
           containers:
           - name: sidecar1
             image: busybox
             command: ["sleep"]
-            args: [ "infinity" ]
+            args: [ "7200" ]
   opsManager:
     configMapRef:
       name: my-project

docker/mongodb-kubernetes-tests/tests/multicluster/fixtures/mongodb-multi-sts-override.yaml

@@ -28,7 +28,7 @@ spec:
               - name: sidecar1
                 image: busybox
                 command: ["sleep"]
-                args: [ "infinity" ]
+                args: [ "7200" ]
           volumeClaimTemplates:
           - metadata:
               name: data
@@ -44,7 +44,7 @@ spec:
               - name: sidecar2
                 image: busybox
                 command: ["sleep"]
-                args: [ "infinity" ]
+                args: [ "7200" ]
           volumeClaimTemplates:
           - metadata:
               name: data
@@ -60,7 +60,7 @@ spec:
               - name: sidecar3
                 image: busybox
                 command: ["sleep"]
-                args: [ "infinity" ]
+                args: [ "7200" ]
           volumeClaimTemplates:
           - metadata:
               name: data

docker/mongodb-kubernetes-tests/tests/opsmanager/fixtures/om_ops_manager_pod_spec.yaml

@@ -66,7 +66,7 @@ spec:
             - name: appdb-sidecar
               image: busybox
               command: ["sleep"]
-              args: ["infinity"]
+              args: ["7200"]
               resources:
                 limits:
                   cpu: "1"

docker/mongodb-kubernetes-tests/tests/opsmanager/withMonitoredAppDB/om_ops_manager_pod_spec.py

@@ -96,7 +96,7 @@ def test_appdb_pod_template_containers(self, ops_manager: MongoDBOpsManager):
         appdb_sidecar_container = containers_by_name["appdb-sidecar"]
         assert appdb_sidecar_container.image == "busybox"
         assert appdb_sidecar_container.command == ["sleep"]
-        assert appdb_sidecar_container.args == ["infinity"]
+        assert appdb_sidecar_container.args == ["7200"]
 
     def test_appdb_persistence(self, ops_manager: MongoDBOpsManager, namespace: str):
         # appdb pod volume claim template

docker/mongodb-kubernetes-tests/tests/replicaset/fixtures/replica-set-custom-podspec.yaml

@@ -25,7 +25,7 @@ spec:
           - name: side-car
             image: busybox:latest
             command: ["/bin/sh"]
-            args: ["-c", "echo ok > /somewhere/busybox_file && sleep 86400"]
+            args: ["-c", "echo ok > /somewhere/busybox_file && sleep 7200"]
             volumeMounts:
               - mountPath: /somewhere
                 name: test-volume

docker/mongodb-kubernetes-tests/tests/replicaset/replica_set_custom_podspec.py

@@ -139,7 +139,7 @@ def test_stateful_set_spec_updated(replica_set, namespace):
                 }
             ],
             "command": ["/bin/sh"],
-            "args": ["-c", "echo ok > /somewhere/busybox_file && sleep 86400"],
+            "args": ["-c", "echo ok > /somewhere/busybox_file && sleep 7200"],
         },
     ]
 

docker/mongodb-kubernetes-tests/tests/shardedcluster/fixtures/sharded-cluster-custom-podspec.yaml

@@ -43,7 +43,7 @@ spec:
           - name: sharded-cluster-sidecar
             image: busybox
             command: ["sleep"]
-            args: [ "infinity" ]
+            args: [ "7200" ]
             resources:
               limits:
                 cpu: "1"
@@ -63,7 +63,7 @@ spec:
             - name: sharded-cluster-sidecar-override
               image: busybox
               command: ["sleep"]
-              args: [ "infinity" ]
+              args: [ "7200" ]
               resources:
                 limits:
                   cpu: "1"

docker/mongodb-kubernetes-tests/tests/standalone/fixtures/standalone-custom-podspec.yaml

@@ -21,7 +21,7 @@ spec:
           - name: standalone-sidecar
             image: busybox
             command: ["sleep"]
-            args: [ "infinity" ]
+            args: [ "7200" ]
         terminationGracePeriodSeconds: 10
         affinity:
           podAntiAffinity: