CLOUDP-298173 - add telemetry and sending to segment (#4042)

## Ticket
jira: https://jira.mongodb.org/browse/CLOUDP-298172

## Description
- Following the TD:
REDACTED
- mms part: 10gen/mms#120058

## Patch
- fully passing
[patch](https://evergreen.mongodb.com/version/67bc44f4f98ec40007296f99?redirect_spruce_users=true)

## Proof - Manual Run - send to cloud-dev
- source new context: `manual_telemetry_multi_cluster`
- running locally against cloud-dev
```
2025-02-17T11:31:28.678+0100	DEBUG	telemetry/client.go:85	request:
POST https://cloud-dev.mongodb.com/api/private/unauth/telemetry/events
response:
HTTP/1.1 202 Accepted
Vary: Accept-Encoding
Server: mdbws
X-Xgen-Up-Proto: HTTP/2
Date: Mon, 17 Feb 2025 10:31:28 GMT
X-Permitted-Cross-Domain-Policies: none
X-Mongodb-Service-Version: gitHash=9aa461ae9a1f8c3cd18c3457914b99306449028b; versionString=master
X-Frame-Options: DENY
Strict-Transport-Security: max-age=31536000; includeSubdomains;
Referrer-Policy: strict-origin-when-cross-origin
X-Content-Type-Options: nosniff
Content-Length: 0

	{"module": "Telemetry"}
2025-02-17T11:31:28.782+0100	INFO	memberwatch/memberwatch.go:92	Cluster kind-e2e-cluster-2 reported healthy
2025-02-17T11:31:28.812+0100	DEBUG	telemetry/collector.go:114	Collecting Clusters telemetry	{"module": "Telemetry"}
```

![image](https://github.com/user-attachments/assets/701265e9-79c4-4726-bc88-3d720f23a4b4)

## Proof - Automated Run - send to cloud-dev
- running against cloud-dev
-
[Test](https://spruce.mongodb.com/task/ops_manager_kubernetes_e2e_operator_race_ubi_with_telemetry_e2e_om_reconcile_race_patch_f316028ac429a42806039a834ce78906b5e2c1f1_67b347574f066300076b1be0_25_02_17_14_27_36/files?execution=0&sortBy=STATUS&sortDir=ASC)
- [Configmap with
Data](REDACTED)
-
[segment](https://app.segment.com/mongodb/sources/cloud_dev_backend/schema/events)
  - search for "Kubernetes Operator Usage Snapshot"

## Proof - Manual Run - Sending against MMS which is not deployed yet
client logs just print them as debug

```
2025-02-17T16:09:44.680+0100	DEBUG	telemetry/collector.go:128	Collecting operator events!	{"module": "Telemetry"}
2025-02-17T16:09:45.713+0100	DEBUG	telemetry/client.go:85	request:
POST https://cloud.mongodb.com/api/private/unauth/telemetry/events
response:
HTTP/1.1 400 Bad Request
Date: Mon, 17 Feb 2025 15:09:45 GMT
Referrer-Policy: strict-origin-when-cross-origin
X-Content-Type-Options: nosniff
X-Mongodb-Service-Version: gitHash=5b762cbda0932584a68a48029bd99606d717b4d1; versionString=v20250205
X-Frame-Options: DENY
Content-Length: 305
X-Envoy-Upstream-Service-Time: 4
Server: mdbws
Strict-Transport-Security: max-age=31536000; includeSubdomains;
X-Permitted-Cross-Domain-Policies: none
Content-Type: application/json
```

## Proof - Automated Run - not sending to MMS (default)
- this is the default case for all tests except for one test and variant
as defined in the
[TD](REDACTED)
- `lastSendTimestamp<Type>` will default to the `initialValue`
```
- apiVersion: v1
  data:
    lastSendPayloadClusters: '[{"timestamp":"2025-02-19T13:13:00.993772296Z","source":"Clusters","properties":{"kubernetesAPIVersion":"v1.30.4","kubernetesClusterID":"9e086ee1-c482-4695-a232-9822fa041e7c","kubernetesFlavour":"Test
      Cluster"}},{"timestamp":"2025-02-19T13:13:00.993775761Z","source":"Clusters","properties":{"kubernetesAPIVersion":"v1.30.4","kubernetesClusterID":"6cb8a37d-5e90-463a-b734-111f11794eca","kubernetesFlavour":"Unknown"}},{"timestamp":"2025-02-19T13:13:00.993779379Z","source":"Clusters","properties":{"kubernetesAPIVersion":"v1.30.4","kubernetesClusterID":"53aa9828-413c-4e44-ac67-81f1380e4dcd","kubernetesFlavour":"Unknown"}},{"timestamp":"2025-02-19T13:13:00.99378209Z","source":"Clusters","properties":{"kubernetesAPIVersion":"v1.30.4","kubernetesClusterID":"832aeec0-8568-4d6d-9040-a499ab287e76","kubernetesFlavour":"Unknown"}}]'
    lastSendPayloadDeployments: "null"
    lastSendPayloadOperators: '[{"timestamp":"2025-02-19T13:13:00.976844943Z","source":"Operators","properties":{"kubernetesClusterID":"9e086ee1-c482-4695-a232-9822fa041e7c","kubernetesClusterIDs":["53aa9828-413c-4e44-ac67-81f1380e4dcd","6cb8a37d-5e90-463a-b734-111f11794eca","832aeec0-8568-4d6d-9040-a499ab287e76"],"operatorID":"c2204fa1-85d2-4b05-82ee-6986b5047c46","operatorType":"MEKO","operatorVersion":"67b5d275ae3943000721076d"}}]'
    lastSendTimestampClusters: initialValue
    lastSendTimestampDeployments: initialValue
    lastSendTimestampOperators: initialValue
    operatorUUID: c2204fa1-85d2-4b05-82ee-6986b5047c46
  kind: ConfigMap
```

Author: nammn

.evergreen-tasks.yml

@@ -1129,7 +1129,7 @@ tasks:
       - func: e2e_test
 
   # this test is run, with an operator with race enabled
-  - name: e2e_om_reconcile_race
+  - name: e2e_om_reconcile_race_with_telemetry
     tags: [ "patch-run" ]
     commands:
       - func: e2e_test

.evergreen.yml

@@ -720,13 +720,14 @@ task_groups:
       - e2e_operator_upgrade_appdb_tls
     <<: *teardown_group
 
-  # e2e_operator_race_task_group includes the tests for testing the operator with race detector enabled
-  - name: e2e_operator_race_task_group
+  # e2e_operator_race_with_telemetry_task_group includes the tests for testing the operator with race detector enabled
+  # additionally, it sends telemetry to cloud-dev; more here: https://wiki.corp.mongodb.com/display/MMS/Telemetry
+  - name: e2e_operator_race_with_telemetry_task_group
     max_hosts: -1
     <<: *setup_group_multi_cluster
     <<: *setup_and_teardown_task
     tasks:
-      - e2e_om_reconcile_race
+      - e2e_om_reconcile_race_with_telemetry
     <<: *teardown_group
 
   # e2e_operator_task_group includes the tests for the specific Operator configuration/behavior. They may deal with
@@ -1253,14 +1254,14 @@ buildvariants:
       - name: e2e_static_ops_manager_kind_only_task_group
       - name: e2e_static_ops_manager_kind_6_0_only_task_group
 
-  - name: e2e_operator_race_ubi
-    display_name: e2e_operator_race_ubi
+  - name: e2e_operator_race_ubi_with_telemetry
+    display_name: e2e_operator_race_ubi_with_telemetry
     tags: [ "e2e_test_suite" ]
     run_on:
       - ubuntu1804-xlarge
     <<: *base_om7_dependency
     tasks:
-      - name: e2e_operator_race_task_group
+      - name: e2e_operator_race_with_telemetry_task_group
 
   - name: e2e_smoke
     display_name: e2e_smoke

.gitignore

@@ -8,6 +8,7 @@ docker/mongodb-enterprise-operator/content/mongodb-enterprise-operator
 docker/mongodb-enterprise-database/content/mongodb-mms-automation-agent-version.properties
 docker/mongodb-enterprise-database/content/readinessprobe
 docker/mongodb-enterprise-ops-manager/scripts/mmsconfiguration
+docker/mongodb-enterprise-tests/public
 
 my-*
 .vscode

api/v1/mdb/zz_generated.deepcopy.go

@@ -42,16 +42,20 @@ spec:
               value: prod
             - name: MDB_DEFAULT_ARCHITECTURE
               value: non-static
-            - name: WATCH_NAMESPACE
+            - name: NAMESPACE
               valueFrom:
                 fieldRef:
                   fieldPath: metadata.namespace
-            - name: NAMESPACE
+            - name: WATCH_NAMESPACE
               valueFrom:
                 fieldRef:
                   fieldPath: metadata.namespace
             - name: MANAGED_SECURITY_CONTEXT
               value: 'true'
+            - name: MDB_OPERATOR_TELEMETRY_SEND_FREQUENCY
+              value: "168h"
+            - name: MDB_OPERATOR_TELEMETRY_COLLECTION_FREQUENCY
+              value: "1h"
             - name: CLUSTER_CLIENT_TIMEOUT
               value: "10"
             - name: IMAGE_PULL_POLICY

api/v1/om/zz_generated.deepcopy.go

@@ -1,5 +1,50 @@
 ---
 # Source: enterprise-operator/templates/operator-roles.yaml
+---
+# Additional ClusterRole for clusterVersionDetection
+kind: ClusterRole
+apiVersion: rbac.authorization.k8s.io/v1
+metadata:
+  name: mongodb-enterprise-operator-cluster-telemetry
+rules:
+
+  # Non-resource URL permissions
+  - nonResourceURLs:
+      - "/version"
+    verbs:
+      - get
+  # Cluster-scoped resource permissions
+  - apiGroups:
+      - ''
+    resources:
+      - namespaces
+    resourceNames:
+      - kube-system
+    verbs:
+      - get
+  - apiGroups:
+      - ''
+    resources:
+      - nodes
+    verbs:
+      - list
+---
+# Source: enterprise-operator/templates/operator-roles.yaml
+# ClusterRoleBinding for clusterVersionDetection
+kind: ClusterRoleBinding
+apiVersion: rbac.authorization.k8s.io/v1
+metadata:
+  name: mongodb-enterprise-operator-mongodb-cluster-telemetry-binding
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: mongodb-enterprise-operator-cluster-telemetry
+subjects:
+  - kind: ServiceAccount
+    name: mongodb-enterprise-operator
+    namespace: mongodb
+---
+# Source: enterprise-operator/templates/operator-roles.yaml
 kind: Role
 apiVersion: rbac.authorization.k8s.io/v1
 metadata:

api/v1/status/zz_generated.deepcopy.go

@@ -125,11 +125,11 @@ def process_run_and_check(args, **kwargs):
     except subprocess.CalledProcessError as exc:
         if exc.stdout is not None:
             stdout = exc.stdout.decode("utf-8")
-            logger.info(stdout)
+            logger.error(f"stdout: {stdout}")
         if exc.stderr is not None:
             stderr = exc.stderr.decode("utf-8")
-            logger.info(stderr)
-        logger.info(exc.output)
+            logger.error(f"stderr: {stderr}")
+        logger.error(f"output: {exc.output}")
         raise