feat(KONFLUX-8225): add log forwarder for KubeArchive (#6439)

* feat(KONFLUX-8225): add log forwarder for KubeArchive

Signed-off-by: obetsun <[email protected]>

* feat(KONFLUX-8225): multiple fixes of synchronization issues

rh-pre-commit.version: 2.3.2
rh-pre-commit.check-secrets: ENABLED

* fix kube-linter errors

rh-pre-commit.version: 2.3.2
rh-pre-commit.check-secrets: ENABLED

* fix: correct PodDisruptionBudget configuration for Loki development

- Move podDisruptionBudget from global to singleBinary component
- Fixes ArgoCD sync issue with vector-kubearchive-log-collector-loki PDB

rh-pre-commit.version: 2.3.2
rh-pre-commit.check-secrets: ENABLED

* fix: add missing inject-infra-deployments-repo-details to kubearchive

- Adds inject-infra-deployments-repo-details k-component to kubearchive ApplicationSet
- Makes kubearchive point to user's fork and branch consistently
- Fixes ArgoCD sync issue with kubearchive applications

rh-pre-commit.version: 2.3.2
rh-pre-commit.check-secrets: ENABLED

* fix: comprehensive kube-linter fixes for vector-kubearchive-log-collector

- Fix loki-sc-rules sidecar container resource requirements
- Fix exporter container resources for chunks/results cache StatefulSets
- Add unhealthyPodEvictionPolicy to all PodDisruptionBudgets
- Properly disable chunksCache and configure resultsCache
- Add resource requirements for all memcached exporter containers
- Apply fixes to both development and staging environments

Resolves all 16 remaining kube-linter errors:
- unset-cpu-requirements for loki-sc-rules and exporter containers
- unset-memory-requirements for loki-sc-rules and exporter containers
- pdb-unhealthy-pod-eviction-policy for memcached PDBs

rh-pre-commit.version: 2.3.2
rh-pre-commit.check-secrets: ENABLED

* fix: resolve kustomize deprecation warnings and helm chart conflicts for vector-kubearchive-log-collector

- Add .gitignore files to exclude charts/ directories that cause build conflicts
- Fix deprecated kustomize syntax:
  - patchesStrategicMerge → patches
  - patchesJson6902 → patches
  - commonLabels → labels
- Resolve helm chart extraction conflicts that prevented policy checks
- Update all kustomization.yaml files in vector-kubearchive-log-collector directory

This fixes the 'forbid cluster policies' errors caused by:
1. Conflicting chart files during helm pull operations
2. Deprecated kustomize field usage triggering warnings

rh-pre-commit.version: 2.3.2
rh-pre-commit.check-secrets: ENABLED

* fix: simplify Loki configuration to resolve remaining kube-linter errors

- Disable sidecar rules container completely to fix resource requirement errors
- Disable all memcached components (chunks, results, frontend, etc.)
- Remove problematic exporter containers and PDB configurations
- Simplify to essential singleBinary configuration with proper resources
- Keep only gateway and core Loki components with required resource limits

This eliminates the final 10 kube-linter errors:
- unset-cpu-requirements for loki-sc-rules and exporter containers
- unset-memory-requirements for loki-sc-rules and exporter containers
- pdb-unhealthy-pod-eviction-policy for memcached PodDisruptionBudgets

Loki will still function for log collection and storage, just without
advanced caching and rule management features.

rh-pre-commit.version: 2.3.2
rh-pre-commit.check-secrets: ENABLED

* fix: add required storage configuration to resolve Loki Helm template errors

- Add loki.storage.bucketNames configuration for development (filesystem) and staging (s3)
- Add loki.storage_config to properly configure storage backends
- Add path_prefix and working_directory configurations
- Fix template error: 'nil pointer evaluating interface {}.chunks'

This resolves the kubectl kustomize --enable-helm failure:
Error: template: loki/templates/_helpers.tpl:231:19: executing "loki.commonStorageConfig"
at <$.Values.loki.storage.bucketNames.chunks>: nil pointer evaluating interface {}.chunks

The Loki Helm chart requires proper storage configuration even in simplified mode.

rh-pre-commit.version: 2.3.2
rh-pre-commit.check-secrets: ENABLED

* fix: remove init container causing busybox image pull failures

- Remove init-vector-data-dir init container that was using busybox:1.35
- The init container was causing 'Back-off pulling image busybox:1.35' errors
- EmptyDir volumes are created automatically, no init container needed
- Fixes ArgoCD out-of-sync issues caused by failed image pulls

The directory /vector-data-dir will be created automatically when the
emptyDir volume is mounted, eliminating the need for an init container.

rh-pre-commit.version: 2.3.2
rh-pre-commit.check-secrets: ENABLED

* fix: resolve Loki gateway nginx container failures in development environment

- Add explicit nginx image specification (docker.io/nginx:1.25-alpine)
- Configure DNS resolver for OpenShift environment (dns-default.openshift-dns.svc.cluster.local)
- Add nginx timeouts and proxy configuration for stability
- Set proper security context for OpenShift compatibility

This resolves the 'Back-off restarting failed container nginx' error in the
vector-kubearchive-log-collector-loki-gateway pod by ensuring nginx has:
1. Proper DNS resolution for upstream services
2. Appropriate timeouts for proxy connections
3. Security context that works with OpenShift constraints

The gateway will now successfully proxy requests to Loki components.

rh-pre-commit.version: 2.3.2
rh-pre-commit.check-secrets: ENABLED

* fix: correct DNS configuration for OpenShift environment and improve SCC

- Replace RKE2-specific DNS with proper OpenShift DNS services:
  - dnsNamespace: openshift-dns (was kube-system)
  - dnsService: dns-default (was rke2-coredns-rke2-coredns)
  - resolver: dns-default.openshift-dns.svc.cluster.local
- Fix ClusterRoleBinding reference from 'vector-scc-user' to 'kubearchive-vector-scc-user'
- Improve SCC security by dropping ALL capabilities instead of individual ones
- Add explicit Loki service account configuration to match SCC setup
- Apply fixes to both development and staging environments

This ensures proper DNS resolution for nginx gateway in OpenShift and
resolves SCC permission issues for all vector-kubearchive-log-collector components.

rh-pre-commit.version: 2.3.2
rh-pre-commit.check-secrets: ENABLED

* fix the external secret path

rh-pre-commit.version: 2.3.2
rh-pre-commit.check-secrets: ENABLED

* fix: add missing service accounts for Loki gateway and canary

- Add vector-kubearchive-log-collector-loki ServiceAccount
- Add vector-kubearchive-log-collector-loki-canary ServiceAccount
- These service accounts were referenced in SCC and ClusterRoleBinding but not actually created
- Fixes 'serviceaccount not found' error when creating Loki gateway pods

Error was: pods 'vector-kubearchive-log-collector-loki-gateway-*' is forbidden:
error looking up service account product-kubearchive-logging/vector-kubearchive-log-collector-loki:
serviceaccount 'vector-kubearchive-log-collector-loki' not found

rh-pre-commit.version: 2.3.2
rh-pre-commit.check-secrets: ENABLED

* fix: remove duplicate docker.io prefix from nginx image repository in development

- Change nginx image repository from 'docker.io/nginx' to 'nginx'
- Fixes image pull error: 'docker.io/docker.io/nginx:1.25-alpine'
- The Loki Helm chart was adding docker.io/ prefix automatically, causing duplication
- Staging configuration was already correct

Error was: Failed to pull image 'docker.io/docker.io/nginx:1.25-alpine':
reading manifest 1.25-alpine in docker.io/docker.io/nginx: unauthorized

rh-pre-commit.version: 2.3.2
rh-pre-commit.check-secrets: ENABLED

* fix nginx image repo for loki gateway

rh-pre-commit.version: 2.3.2
rh-pre-commit.check-secrets: ENABLED

* fix external secret for staging configuration

rh-pre-commit.version: 2.3.2
rh-pre-commit.check-secrets: ENABLED

* loki configuration simplified

rh-pre-commit.version: 2.3.2
rh-pre-commit.check-secrets: ENABLED

* fix for loki disabled components linter errors

rh-pre-commit.version: 2.3.2
rh-pre-commit.check-secrets: ENABLED

* fix readonly loki storage filesystem and scc naming

rh-pre-commit.version: 2.3.2
rh-pre-commit.check-secrets: ENABLED

* fix tmp storage

rh-pre-commit.version: 2.3.2
rh-pre-commit.check-secrets: ENABLED

---------

Signed-off-by: obetsun <[email protected]>
Co-authored-by: obetsun <[email protected]>

Author: olegbet

.gitignore

@@ -5,3 +5,6 @@ cosign.pub
 tmp
 .idea/*
 components/pipeline-service/base/log-collector/charts/*
+
+# Ignore cached Helm charts
+components/**/charts/

argo-cd-apps/base/member/infra-deployments/kubearchive/kustomization.yaml

@@ -4,4 +4,5 @@ kind: Kustomization
 resources:
 - kubearchive.yaml
 components:
+  - ../../../../k-components/inject-infra-deployments-repo-details
   - ../../../../k-components/deploy-to-member-cluster-merge-generator

argo-cd-apps/base/member/infra-deployments/kustomization.yaml

@@ -30,6 +30,7 @@ resources:
   - konflux-ui
   - konflux-rbac
   - konflux-info
+  - vector-kubearchive-log-collector
   - vector-tekton-logs-collector
   - kyverno
   - namespace-lister

argo-cd-apps/base/member/infra-deployments/vector-kubearchive-log-collector/kustomization.yaml

@@ -0,0 +1,7 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+resources:
+- vector-kubearchive-log-collector.yaml
+components:
+  - ../../../../k-components/inject-infra-deployments-repo-details
+  - ../../../../k-components/deploy-to-member-cluster-merge-generator

argo-cd-apps/base/member/infra-deployments/vector-kubearchive-log-collector/vector-kubearchive-log-collector.yaml

@@ -0,0 +1,63 @@
+apiVersion: argoproj.io/v1alpha1
+kind: ApplicationSet
+metadata:
+  name: vector-kubearchive-log-collector
+spec:
+  generators:
+    - merge:
+        mergeKeys:
+          - nameNormalized
+        generators:
+          - clusters:
+              values:
+                sourceRoot: components/vector-kubearchive-log-collector
+                environment: staging
+                clusterDir: base
+          - list:
+              elements:
+                - nameNormalized: stone-stg-rh01
+                  values.clusterDir: stone-stg-rh01
+  template:
+    metadata:
+      name: vector-kubearchive-log-collector-{{nameNormalized}}
+      annotations:
+        argocd.argoproj.io/sync-wave: "100"
+        argocd.argoproj.io/refresh: "hard"
+    spec:
+      ignoreDifferences:
+        # Ignore Helm-generated dynamic content that causes drift
+        - group: apps
+          kind: Deployment
+          name: vector-kubearchive-log-collector-grafana
+          jsonPointers:
+            - /metadata/annotations/deployment.kubernetes.io~1revision
+            - /metadata/annotations/kubectl.kubernetes.io~1last-applied-configuration
+            - /metadata/generation
+            - /spec/template/metadata/annotations/checksum~1config
+            - /spec/template/metadata/annotations/checksum~1secret
+            - /spec/template/metadata/annotations/checksum~1sc-dashboard-provider-config
+        - group: ""
+          kind: Secret
+          name: vector-kubearchive-log-collector-grafana
+          jsonPointers:
+            - /metadata/annotations/kubectl.kubernetes.io~1last-applied-configuration
+      project: default
+      source:
+        path: '{{values.sourceRoot}}/{{values.environment}}/{{values.clusterDir}}'
+        repoURL: https://github.com/olegbet/infra-deployments.git
+        targetRevision: KONFLUX-8225_add_log_forwarder_for_loki
+      destination:
+        namespace: product-kubearchive-logging
+        server: '{{server}}'
+      syncPolicy:
+        automated:
+          prune: true
+          selfHeal: true
+        syncOptions:
+          - CreateNamespace=true
+        retry:
+          limit: -1
+          backoff:
+            duration: 10s
+            factor: 2
+            maxDuration: 3m

argo-cd-apps/overlays/development/kustomization.yaml

@@ -179,6 +179,11 @@ patches:
       kind: ApplicationSet
       version: v1alpha1
       name: crossplane-control-plane
+  - path: development-overlay-patch.yaml
+    target:
+      kind: ApplicationSet
+      version: v1alpha1
+      name: vector-kubearchive-log-collector
   - path: development-overlay-patch.yaml
     target:
       kind: ApplicationSet
@@ -219,3 +224,4 @@ patches:
       kind: ApplicationSet
       version: v1alpha1
       name: konflux-kite
+

components/vector-kubearchive-log-collector/base/.gitignore

@@ -0,0 +1,3 @@
+# Ignore Helm charts directory created during kustomize build
+charts/
+*.tgz 

components/vector-kubearchive-log-collector/base/kustomization.yaml

@@ -0,0 +1,17 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+namespace: product-kubearchive-logging
+commonAnnotations:
+  argocd.argoproj.io/sync-wave: "-1"
+  ignore-check.kube-linter.io/drop-net-raw-capability: |
+    "Vector Runs requires access to socket."
+  ignore-check.kube-linter.io/run-as-non-root: |
+    "Vector Runs as Root and attach host Path."
+  ignore-check.kube-linter.io/sensitive-host-mounts: |
+    "Vector Runs requires certain host mounts to watch  files being created by pods."
+
+generators:
+- vector-helm-generator.yaml
+
+resources:
+- vector-pre.yaml

components/vector-kubearchive-log-collector/base/vector-helm-generator.yaml

@@ -0,0 +1,10 @@
+apiVersion: builtin
+kind: HelmChartInflationGenerator
+metadata:
+  name: vector
+name: vector
+repo: https://helm.vector.dev
+version: 0.43.0
+releaseName: vector-kubearchive-log-collector
+namespace: product-kubearchive-logging
+valuesFile: vector-helm-values.yaml

components/vector-kubearchive-log-collector/base/vector-helm-values.yaml

@@ -0,0 +1,211 @@
+---
+role: Agent
+resources:
+  requests:
+    cpu: 200m
+    memory: 1024Mi
+  limits:
+    cpu: 1000m
+    memory: 2048Mi
+customConfig:
+  data_dir: /vector-data-dir
+  api:
+    enabled: true
+    address: 127.0.0.1:8686
+    playground: false
+  sources:
+    k8s_logs:
+      type: kubernetes_logs
+      rotate_wait_secs: 5
+      glob_minimum_cooldown_ms: 500
+      max_line_bytes: 3145728
+      auto_partial_merge: true
+  transforms:
+    reduce_events:
+      type: reduce
+      inputs:
+        - k8s_logs
+      group_by:
+        - file
+      flush_period_ms: 2000
+      end_every_period_ms: 2000
+      merge_strategies:
+        message: concat_newline
+    remap_app_logs:
+      type: remap
+      inputs:
+        - reduce_events
+      source: |-
+        .tmp = del(.)
+        # Preserve original kubernetes fields for Loki labels
+        if exists(.tmp.kubernetes.pod_uid) {
+          .pod_id = del(.tmp.kubernetes.pod_uid)
+        } else {
+          .pod_id = "unknown_pod_id"
+        }
+        if exists(.tmp.kubernetes.pod_name) {
+          .pod_name = del(.tmp.kubernetes.pod_name)
+        } else {
+          .pod_name = "unknown_pod"
+        }
+        if exists(.tmp.kubernetes.container_name) {
+          .container = del(.tmp.kubernetes.container_name)
+        } else {
+          .container = "unknown_container"
+        }
+        if exists(.tmp.kubernetes.pod_namespace) {
+          .namespace = del(.tmp.kubernetes.pod_namespace)
+        } else {
+          .namespace = "unlabeled"
+        }
+        # Handling Tekton-specific labels
+        if exists(.tmp.kubernetes.pod_labels."tekton.dev/taskRunUID") {
+          .taskRunUID = del(.tmp.kubernetes.pod_labels."tekton.dev/taskRunUID")
+        } else {
+          .taskRunUID = "none"
+        }
+        if exists(.tmp.kubernetes.pod_labels."tekton.dev/pipelineRunUID") {
+          .pipelineRunUID = del(.tmp.kubernetes.pod_labels."tekton.dev/pipelineRunUID")
+          .result = .pipelineRunUID
+        } else {
+          .result = .taskRunUID
+        }
+        # --- Start: Cronjob Specific Handling ---
+        if exists(.tmp.kubernetes.pod_labels."job-name") {
+          .job_name = del(.tmp.kubernetes.pod_labels."job-name")
+          .log_type = "cronjob"
+          if exists(.tmp.kubernetes.pod_labels."cronjob-name") {
+            .cronjob_name = del(.tmp.kubernetes.pod_labels."cronjob-name")
+          } else {
+            # Using corrected regex pattern without \d
+            .job_name = to_string(.job_name) ?? "default"
+            if match(.job_name, r'^(.*)-[0-9]{8,10}$') {
+              .cronjob_name = replace(.job_name, r'-[0-9]{8,10}$', "")
+            } else {
+              .cronjob_name = "unknown_cronjob"
+            }
+          }
+          if exists(.tmp.kubernetes.pod_labels."controller-uid") {
+              .job_uid = del(.tmp.kubernetes.pod_labels."controller-uid")
+          }
+        } else {
+          .log_type = "application"
+        }
+        # --- End: Cronjob Specific Handling ---
+        # Handling general Kubernetes labels
+        if exists(.tmp.kubernetes.pod_labels) {
+          .pod_labels = .tmp.kubernetes.pod_labels
+        } else {
+          .pod_labels = "no_labels"
+        }
+        # General message field handling
+        if exists(.tmp.message) {
+          .message = to_string(del(.tmp.message)) ?? "no_message"
+        } else {
+          .message = "no_message"
+        }
+        # Basic data sanitization to prevent 400 errors
+        # Truncate very long messages
+        if length(.message) > 32768 {
+          .message = slice!(.message, 0, 32768) + "...[TRUNCATED]"
+        }
+        # Clean up temporary fields
+        del(.tmp)
+  sinks:
+    loki:
+      type: loki
+      inputs: ["remap_app_logs"]
+      # Direct connection to Loki service (no gateway)
+      endpoint: "http://vector-kubearchive-log-collector-loki.product-kubearchive-logging.svc.cluster.local:3100"
+      encoding:
+        codec: "json"
+      auth:
+        strategy: "basic"
+        user: "${LOKI_USERNAME}"
+        password: "${LOKI_PASSWORD}"
+      tenant_id: "kubearchive"
+      request:
+        headers:
+          X-Scope-OrgID: kubearchive
+      batch:
+        max_bytes: 10485760
+        timeout_secs: 300 
+      compression: "none"
+      labels:
+        job: "vector"
+        pod_id: "{{`{{ pod_id }}`}}"
+        container: "{{`{{ container }}`}}"
+        namespace: "{{`{{ namespace }}`}}"
+        pod: "{{`{{ pod_name }}`}}"
+      buffer:
+        type: "memory"
+        max_events: 10000
+        when_full: "block"
+env:
+  - name: LOKI_USERNAME
+    valueFrom:
+      secretKeyRef:
+        name: kubearchive-loki
+        key: USERNAME
+  - name: LOKI_PASSWORD
+    valueFrom:
+      secretKeyRef:
+        name: kubearchive-loki
+        key: PASSWORD
+nodeSelector:
+  konflux-ci.dev/workload: konflux-tenants
+tolerations:
+  - effect: NoSchedule
+    key: konflux-ci.dev/workload
+    operator: Equal
+    value: konflux-tenants
+image:
+  repository: quay.io/kubearchive/vector
+  tag: 0.46.1-distroless-libc
+serviceAccount:
+  create: true
+  name: vector-kubearchive-log-collector
+securityContext:
+  allowPrivilegeEscalation: false
+  runAsUser: 0
+  capabilities:
+    drop:
+    - CHOWN
+    - DAC_OVERRIDE
+    - FOWNER
+    - FSETID
+    - KILL
+    - NET_BIND_SERVICE
+    - SETGID
+    - SETPCAP
+    - SETUID
+  readOnlyRootFilesystem: true
+  seLinuxOptions:
+    type: spc_t
+  seccompProfile:
+    type: RuntimeDefault
+
+# Override default volumes to be more specific and secure
+extraVolumes:
+  - name: varlog
+    hostPath:
+      path: /var/log/pods
+      type: Directory
+  - name: varlibdockercontainers
+    hostPath:
+      path: /var/lib/containers
+      type: DirectoryOrCreate
+
+extraVolumeMounts:
+  - name: varlog
+    mountPath: /var/log/pods
+    readOnly: true
+  - name: varlibdockercontainers  
+    mountPath: /var/lib/containers
+    readOnly: true
+
+# Configure Vector to use emptyDir for its default data volume instead of hostPath
+persistence:
+  enabled: false
+
+