Merge branch 'master' into add-more-parameters-opensearch-output-plugin

Author: discostur

charts/fluent-operator/Chart.yaml

@@ -6,7 +6,7 @@ keywords:
   - fluent-bit
   - fluentd
   - operator
-version: 3.5.2
+version: 4.0.0-rc1
 # renovate: datasource=docker depName=ghcr.io/fluent/fluent-operator/fluent-operator
 appVersion: "3.5.0"
 icon: https://raw.githubusercontent.com/fluent/fluent-operator/master/docs/images/fluent-operator-icon.svg
@@ -19,6 +19,8 @@ maintainers:
   - name: marcofranssen
     email: [email protected]
     url: https://marcofranssen.nl
+  - name: joshuabaird
+    email: [email protected]
 dependencies:
   - name: fluent-bit-crds
     repository: "file://charts/fluent-bit-crds"

charts/fluent-operator/MIGRATION-v4.md

@@ -0,0 +1,147 @@
+# Migration Guide: Fluent Operator Helm Chart v3.x to v4.0
+
+## Overview
+
+v4.0 simplifies container runtime configuration by removing dynamic detection for the `docker` runtime via initContainers and adopting static, configuration-based paths. The `docker` runtime has not been used widely since Kubernetes v1.24 (2022) and modern Kubernetes distributions now use the `containerd` runtime.
+
+## Breaking Changes
+
+### 1. Default Container Runtime Changed
+
+**What Changed:**
+
+- Default `containerRuntime` changed from `docker` to `containerd`
+
+**Impact:**
+
+- Users who never explicitly set `containerRuntime` will now use containerd defaults
+- Log parser will change from `docker` to `cri` format
+- Systemd filter will target `containerd.service` instead of `docker.service`
+
+**Who Is Affected:**
+
+- Users still running Docker container runtime (uncommon - Docker support was removed from Kubernetes in v1.24, May 2022)
+- Users who relied on the default value without explicit configuration
+
+**Migration:**
+
+
+The containerRuntime now defaults to `containerd`. In `v3.x` the implicit default was `docker`. Use `containerRuntime: docker` to maintain `v3.x` behavior.
+
+```diff
++ containerRuntime: containerd
+```
+
+### 2. initContainers Removed
+
+**What Changed:**
+
+- Removed dynamic Docker root directory detection via initContainer
+- Removed `operator.initcontainer.*` configuration options
+- Container log paths are now statically configured
+
+**Impact:**
+
+- initContainer no longer runs before the operator starts
+- Removes dependency on third party outdated images for initContainers
+- Docker socket no longer mounted for path detection
+
+**Who Is Affected:**
+
+- Users who customized `operator.initcontainer.image` or `operator.initcontainer.resources`
+- Users with Docker installations using non-standard root directories
+
+**Migration:**
+
+The operator initContainer has been removed.
+
+```diff
+  operator:
+-   initcontainer:
+-     image:
+-       registry: docker.io
+-       repository: docker
+-       tag: "20.10"
+    resources:
+      limits:
+        cpu: 100m
+        memory: 64Mi
+```
+
+### 3. Log Path Configuration Removed
+
+**What Changed:**
+
+- Removed `operator.logPath.containerd` and `operator.logPath.crio`
+- Removed ability to configure custom log paths
+- Log paths are now determined automatically based on `containerRuntime`
+
+**Impact:**
+
+- Old `operator.logPath.*` configuration is ignored
+- Each container runtime uses its standard default path
+
+**Who Is Affected:**
+
+- Users who set custom paths via `operator.logPath.containerd` or `operator.logPath.crio`
+- Users with non-standard container log directory locations
+
+**Migration:**
+
+If you were using custom log paths, you must ensure your container runtime uses the standard default paths shown below, or adjust your container runtime configuration to use these standard paths.
+
+## Default Paths by Runtime
+
+v4.0 uses the following default paths based on the configured `containerRuntime`:
+
+| Container Runtime | Default Path |
+|-------------------|--------------|
+| `containerd` | `/var/log/containers` |
+| `crio` | `/var/log/containers` |
+| `docker` | `/var/lib/docker/containers` |
+
+## Migration Scenarios
+
+### Scenario 1: Using Containerd (Default) - No Changes Needed ✅
+
+```yaml
+# v3.x
+containerRuntime: containerd  # or not set
+# ... rest of config
+
+# v4.0 - No changes required!
+# The new defaults work out of the box
+```
+
+### Scenario 2: Using CRI-O - Minimal Changes
+
+```yaml
+# v3.x
+containerRuntime: crio
+# ... rest of config
+
+# v4.0 - Explicitly set runtime (same as before)
+containerRuntime: crio
+# Default path /var/log/containers works for most CRI-O installations
+```
+
+### Scenario 3: Using Docker
+
+```yaml
+# v3.x
+containerRuntime: docker
+# (relied on automatic detection)
+
+# v4.0 - Must explicitly set runtime
+containerRuntime: docker
+# Uses default path: /var/lib/docker/containers
+# This works for standard Docker installations
+# If your Docker uses a custom root directory, you must reconfigure Docker
+# to use the standard path
+```
+
+## Forward Looking: Planned Changes in v5.0
+
+**Future Change (v5.0):**
+
+- The `fluent-operator-env` _ConfigMap_, which is used to provide backwards compatibility with fluent-operator =<3.5, will be completely removed

charts/fluent-operator/README.md

@@ -19,6 +19,30 @@ By default, all CRDs required for Fluent Operator will be installed.  To prevent
 
 ## Upgrading
 
+### Upgrading to v4.0
+
+⚠️ **v4.0 contains breaking changes.** Please review the [Migration Guide](MIGRATION-v4.md) before upgrading.
+
+**Key Changes:**
+
+- Default `containerRuntime` changed from `docker` to `containerd`
+- Removed initContainers for dynamic path detection for the `docker` runtime
+- Removed ability to configure custom log paths - uses fixed defaults based on `containerRuntime`
+
+**Quick Migration:**
+
+```yaml
+# If using Docker, explicitly set in your values:
+containerRuntime: docker
+
+# Note: Custom log paths are no longer supported
+# Each runtime uses its fixed default path
+```
+
+See [MIGRATION-v4.md](MIGRATION-v4.md) for complete migration instructions.
+
+### Upgrading
+
 Helm [does not manage the lifecycle of CRDs](https://helm.sh/docs/chart_best_practices/custom_resource_definitions/), so if the Fluent Operator CRDs already exist, subsequent
 chart upgrades will not add or remove CRDs even if they have changed.  During upgrades, users should manually update CRDs:
 

charts/fluent-operator/templates/_helpers.tpl

@@ -83,3 +83,14 @@ Util function for generating the image URL based on the provided options.
 {{- if .digest -}}{{ printf "@%s" .digest }}{{- else -}}{{ printf ":%s" (default $defaultTag .tag) }}{{- end -}}
 {{- end }}
 {{- end }}
+
+{{/*
+Determine the container log path based on containerRuntime
+*/}}
+{{- define "fluent-operator.containerLogPath" -}}
+{{- if eq .Values.containerRuntime "docker" -}}
+/var/lib/docker/containers
+{{- else -}}
+/var/log/containers
+{{- end -}}
+{{- end }}

charts/fluent-operator/templates/fluent-operator-configmap-env.yaml

@@ -0,0 +1,13 @@
+{{- if .Values.operator.enable }}
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: fluent-operator-env
+  namespace: {{ .Release.Namespace | quote }}
+  labels:
+    app.kubernetes.io/component: operator
+    app.kubernetes.io/name: fluent-operator
+data:
+  fluent-bit.env: |
+    CONTAINER_ROOT_DIR={{ include "fluent-operator.containerLogPath" . | trimSuffix "/containers" }}
+{{- end }}

charts/fluent-operator/templates/fluent-operator-deployment.yaml

@@ -27,64 +27,10 @@ spec:
         {{- toYaml . | nindent 8 }}
       {{- end }}
     spec:
-      {{- if eq .Values.containerRuntime "docker" }}
       volumes:
       - name: env
-        emptyDir: {}
-      - name: dockersock
-        hostPath:
-          path: /var/run/docker.sock
-      initContainers:
-      - name: setenv
-        image: {{ template "fluent-operator.image" (tuple .Values.operator.initcontainer.image "") }}
-        command:
-        - /bin/sh
-        - '-c'
-        - set -ex;
-          echo CONTAINER_ROOT_DIR=$(docker info -f '{{`{{.DockerRootDir}}`}}' 2> /dev/null) > /fluent-operator/fluent-bit.env
-        resources:
-          {{- toYaml .Values.operator.initcontainer.resources | nindent 10 }}
-        volumeMounts:
-        - name: env
-          mountPath: /fluent-operator
-        - name: dockersock
-          readOnly: true
-          mountPath: /var/run/docker.sock
-      {{- else if eq .Values.containerRuntime "containerd" }}
-      volumes:
-      - name: env
-        emptyDir: {}
-      initContainers:
-      - name: setenv
-        image: {{ template "fluent-operator.image" (tuple .Values.operator.initcontainer.image "") }}
-        command:
-        - /bin/sh
-        - '-c'
-        - set -ex;
-          echo CONTAINER_ROOT_DIR={{ .Values.operator.logPath.containerd }} > /fluent-operator/fluent-bit.env
-        resources:
-          {{- toYaml .Values.operator.initcontainer.resources | nindent 10 }}
-        volumeMounts:
-        - name: env
-          mountPath: /fluent-operator
-      {{- else if eq .Values.containerRuntime "crio" }}
-      volumes:
-        - name: env
-          emptyDir: {}
-      initContainers:
-      - name: setenv
-        image: {{ template "fluent-operator.image" (tuple .Values.operator.initcontainer.image "") }}
-        command:
-        - /bin/sh
-        - '-c'
-        - set -ex;
-          echo CONTAINER_ROOT_DIR={{ .Values.operator.logPath.crio }} > /fluent-operator/fluent-bit.env
-        resources:
-          {{- toYaml .Values.operator.initcontainer.resources | nindent 10 }}
-        volumeMounts:
-        - name: env
-          mountPath: /fluent-operator
-      {{- end }}
+        configMap:
+          name: fluent-operator-env
       containers:
       - name: fluent-operator
         image: {{ template "fluent-operator.image" (tuple .Values.operator.image $.Chart.AppVersion) }}
@@ -100,6 +46,8 @@ spec:
               fieldRef:
                 apiVersion: v1
                 fieldPath: metadata.namespace
+          - name: CONTAINER_LOG_PATH
+            value: {{ include "fluent-operator.containerLogPath" . | quote }}
         args:
           {{- with .Values.operator.extraArgs }}
           {{- toYaml . | nindent 10 }}

charts/fluent-operator/values.yaml

@@ -2,28 +2,14 @@
 # This is a YAML-formatted file.
 # Declare variables to be passed into your templates.
 
-# Set this to containerd or crio if you want to collect CRI format logs
-containerRuntime: docker
+# Container runtime used by your Kubernetes cluster
+# Supported values: containerd, crio, docker
+containerRuntime: containerd
 #  If you want to deploy a default Fluent Bit pipeline (including Fluent Bit Input, Filter, and output) to collect Kubernetes logs, you'll need to set the Kubernetes parameter to true
 # see https://github.com/fluent/fluent-operator/tree/master/manifests/logging-stack
 Kubernetes: true
 
 operator:
-  # The init container is to get the actual storage path of the docker log files so that it can be mounted to collect the logs.
-  # see https://github.com/fluent/fluent-operator/blob/master/manifests/setup/fluent-operator-deployment.yaml#L26
-  initcontainer:
-    image:
-      registry: docker.io
-      repository: docker
-      tag: "20.10"
-
-    resources:
-      limits:
-        cpu: 100m
-        memory: 64Mi
-      requests:
-        cpu: 50m
-        memory: 64Mi
   image:
     registry: ghcr.io
     repository: fluent/fluent-operator/fluent-operator
@@ -67,10 +53,6 @@ operator:
   # Reference one more key-value pairs of labels that should be attached to fluent-operator
   labels: {}
   #  myExampleLabel: someValue
-  logPath:
-    # The operator currently assumes a Docker container runtime path for the logs as the default, for other container runtimes you can set the location explicitly below.
-    # crio: /var/log
-    containerd: /var/log
   # By default, the operator provisions both Fluent Bit and FluentD controllers.
   # A specific controller can be disabled by setting the disableComponentControllers value.
   # The disableComponentControllers value can be either "fluent-bit" or "fluentd".
@@ -358,11 +340,11 @@ fluentbit:
     # See https://docs.fluentbit.io/manual/pipeline/outputs/loki
     loki:
       # Switch for generation of fluentbit loki ClusterOutput (and loki basic auth http user and pass secrets if required)
-      enable: false # Bool
+      enable: false  #  Bool
       retryLimit: "no_limits"
       logLevel: "info"
-      host: 127.0.0.1 # String
-      port: 3100 # Int
+      host: 127.0.0.1  #  String
+      port: 3100  #  Int
       # Either, give http{User,Password},tenantID string values specifying them directly
       httpUser: myuser
       httpPassword: mypass