refactor(observability): remove duplicated shared files; give each component its own namespace

Problem 1: the composable split had copy-pasted namespace.yaml,
helm-repositories.yaml, and datasources/ into each component
subdirectory while leaving the originals at the root. That duplicated
~200 lines and caused the root kustomize build to fail on conflicting
resources.

- Deleted the root-level namespace.yaml, helm-repositories.yaml, and
  datasources/ directory.
- Kept a single copy of the datasources under grafana/datasources/ (where
  the Grafana instance lives).
- Pared each component's helm-repositories.yaml to just the repo that
  component actually consumes. Gave loki/tempo distinct HelmRepository
  names (loki-charts, tempo-charts) so the composed root kustomize build
  does not fail on duplicate source.toolkit.fluxcd.io resources.
- Root components/observability/kustomization.yaml now references the six
  component subdirectories only.
- Dropped prometheus-crds/namespace.yaml entirely — the kustomization
  only installs cluster-scoped CRDs so no namespace is needed.

Problem 2: everything still deployed to telemetry-system, which defeats
the point of per-component composition. Each component now has its own
namespace so kubectl delete ns <x> cleanly uninstalls it:

- victoria-metrics-system (was telemetry-system)
- grafana-system
- loki-system
- tempo-system
- otel-collector-system

Cross-component references are now fully qualified service DNS names:

- Grafana datasources point at vmsingle/vmalertmanager in
  victoria-metrics-system, loki-system-loki in loki-system, and
  tempo-system-tempo in tempo-system.
- The OTel Collector's otlp, loki, and prometheusremotewrite exporters
  point at the new FQDNs.
- VMAlert's datasource, notifier, and remoteWrite URLs use the new
  victoria-metrics-system service names.
- VM defaultDashboards.grafanaOperator.allowCrossNamespaceImport is now
  true so dashboards created in victoria-metrics-system can target the
  Grafana CR in grafana-system.

Taskfile's per-component waits updated to reference the new namespaces
(vmagent/vmsingle in victoria-metrics-system, otel-collector-collector
DaemonSet in otel-collector-system).

README refreshed to document the subcomponent layout, namespaces, and
removal procedure.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Author: scotwells

Taskfile.yml

@@ -450,8 +450,8 @@ tasks:
       - echo "⏳ Waiting for Victoria Metrics Stack HelmRelease …"
       - kubectl -n flux-system wait helmrelease/vm --for=condition=Ready --timeout={{.WAIT_TIMEOUT}}
       - echo "⏳ Waiting for vmagent & vmsingle to be Ready …"
-      - kubectl wait --for=condition=Available deploy -l app.kubernetes.io/name=vmagent -n telemetry-system --timeout={{.WAIT_TIMEOUT}} || true
-      - kubectl wait --for=condition=Available deploy -l app.kubernetes.io/name=vmsingle -n telemetry-system --timeout={{.WAIT_TIMEOUT}} || true
+      - kubectl wait --for=condition=Available deploy -l app.kubernetes.io/name=vmagent -n victoria-metrics-system --timeout={{.WAIT_TIMEOUT}} || true
+      - kubectl wait --for=condition=Available deploy -l app.kubernetes.io/name=vmsingle -n victoria-metrics-system --timeout={{.WAIT_TIMEOUT}} || true
       - echo "✅ Victoria Metrics is ready"
 
   install-otel-collector:
@@ -480,7 +480,7 @@ tasks:
           sleep 10
         done
       - echo "⏳ Waiting for OTel Collector DaemonSet …"
-      - kubectl -n telemetry-system rollout status daemonset/otel-collector-collector --timeout={{.WAIT_TIMEOUT}}
+      - kubectl -n otel-collector-system rollout status daemonset/otel-collector-collector --timeout={{.WAIT_TIMEOUT}}
       - echo "✅ OpenTelemetry Collector is ready"
 
   install-grafana:

components/observability/README.md

@@ -4,39 +4,61 @@ Optional observability stack for test infrastructure, providing metrics, logs, a
 
 ## Overview
 
-This component is an **optional add-on** deployed after core infrastructure. It provides comprehensive telemetry capabilities using Grafana, Victoria Metrics, Loki, and Tempo.
+This component is an **optional add-on** deployed after core infrastructure. It provides comprehensive telemetry capabilities using Grafana, Victoria Metrics, Loki, Tempo, and the OpenTelemetry Collector.
 
-## Components
+The stack is split into composable subcomponents — each deploys into its own namespace so it can be installed, upgraded, or removed independently (`kubectl delete ns <component>-system` cleanly uninstalls).
 
-- **Grafana**: Visualization and dashboards
-- **Victoria Metrics**: Metrics collection and storage
-- **Loki**: Log aggregation and storage
-- **Tempo**: Distributed tracing storage
-- **Promtail**: Log collection agent
+## Subcomponents
+
+| Component         | Namespace                 | Purpose                               |
+| ----------------- | ------------------------- | ------------------------------------- |
+| `prometheus-crds` | (cluster-scoped)          | Prometheus Operator CRDs              |
+| `victoria-metrics`| `victoria-metrics-system` | Metrics collection and storage        |
+| `otel-collector`  | `otel-collector-system`   | OpenTelemetry DaemonSet collector     |
+| `loki`            | `loki-system`             | Log aggregation and storage           |
+| `tempo`           | `tempo-system`            | Distributed tracing storage           |
+| `grafana`         | `grafana-system`          | Visualization, dashboards, datasources|
 
 ## Access
 
 - **Grafana UI**: Available at NodePort 30000 (admin/datum123)
-- **Default Datasources**: Victoria Metrics (metrics), Loki (logs), Tempo (traces)
+- **Default Datasources**: Victoria Metrics (metrics), Loki (logs), Tempo (traces), Alertmanager
 
 ## Prerequisites
 
 Core test infrastructure must be running before deploying observability components.
 
 ## Deployment
 
-Deploy using the `install-observability` task target:
+Deploy the whole stack:
 
 ```bash
 task install-observability
 ```
 
-This deploys all components and configures datasources automatically.
+Or install components individually:
+
+```bash
+task install-prometheus-crds
+task install-victoria-metrics
+task install-otel-collector
+task install-loki
+task install-tempo
+task install-grafana
+```
 
 ## Removal
 
-To remove the observability stack:
+To remove an individual component:
+
+```bash
+kubectl delete namespace <component>-system
+```
+
+To remove the whole stack:
 
 ```bash
-kubectl delete namespace observability
+for ns in grafana-system tempo-system loki-system otel-collector-system victoria-metrics-system; do
+  kubectl delete namespace "$ns"
+done
 ```

components/observability/datasources/alertmanager-datasource.yaml

@@ -2,7 +2,7 @@ apiVersion: grafana.integreatly.org/v1beta1
 kind: GrafanaDatasource
 metadata:
   name: alertmanager
-  namespace: telemetry-system
+  namespace: grafana-system
 spec:
   instanceSelector:
     matchLabels:
@@ -11,7 +11,7 @@ spec:
     name: Alertmanager
     type: alertmanager
     access: proxy
-    url: http://vmalertmanager-telemetry-system-vm.telemetry-system.svc.cluster.local:9093
+    url: http://vmalertmanager-victoria-metrics-system-vm.victoria-metrics-system.svc.cluster.local:9093
     isDefault: true
     editable: true
     jsonData:

components/observability/datasources/kustomization.yaml

@@ -2,7 +2,7 @@ apiVersion: grafana.integreatly.org/v1beta1
 kind: GrafanaDatasource
 metadata:
   name: loki-datasource
-  namespace: telemetry-system
+  namespace: grafana-system
   labels:
     dashboards: "grafana"
 spec:
@@ -13,7 +13,7 @@ spec:
     name: Loki
     type: loki
     access: proxy
-    url: http://telemetry-system-loki.telemetry-system.svc.cluster.local:3100
+    url: http://loki-system-loki.loki-system.svc.cluster.local:3100
     isDefault: false
     editable: true
     jsonData:

components/observability/datasources/loki-datasource.yaml

@@ -2,7 +2,7 @@ apiVersion: grafana.integreatly.org/v1beta1
 kind: GrafanaDatasource
 metadata:
   name: tempo-datasource
-  namespace: telemetry-system
+  namespace: grafana-system
   labels:
     dashboards: "grafana"
 spec:
@@ -13,7 +13,7 @@ spec:
     name: Tempo
     type: tempo
     access: proxy
-    url: http://telemetry-system-tempo.telemetry-system.svc.cluster.local:3100
+    url: http://tempo-system-tempo.tempo-system.svc.cluster.local:3100
     uid: tempo-datasource
     isDefault: false
     editable: true