OL8 docker image support, fluentd version uptake, logging analytics output plugin version upgrade to 2.0.2 for memory usage improvements

Author: Santhosh Kumar Vuda

CHANGELOG.md

@@ -0,0 +1,18 @@
+# Change Log
+
+## 2022-02-24
+### Added
+- Oracle Linux 8 based Docker Image support.
+### Changed
+- fluent-plugin-oci-logging-analytics version upgrade to 2.0.2 for memory usage improvements.
+- Fluentd version upgrade to 1.14.3.
+
+## 2022-02-7
+### Added
+- Helm chart v1.0.0 release.
+### Fixed
+- Fluentd config fix in CRI (configmap-cri.yaml) for Container Generic Logs.
+
+## 2022-01-20
+### Added
+- Initial release.

README.md

@@ -96,8 +96,10 @@ The following are the list of objects supported at present:
 
 We are in the process of building a docker image based off Oracle Linux 8 including Fluentd, OCI Logging Analytics Output Plugin and all the required dependencies. 
 All the dependencies will be build from source and installed into the image. This image soon would be available to use as a pre-built image as is (OR) to create a custom image using this image as a base image.
-At present, for testing purposes follow the below mentioned steps to build an image using official Fluentd Docker Image as base image (off Debian).
-- Download all the files from [this dir](logan/docker-images/v1.0/debian/) into a local machine having access to internet.
+At present, follow the below mentioned steps to build an image either using Dockerfile off Oracle Linux 8 as base image (OR) Dockerfile off Fluentd base image from Dockerhub (off Debian).
+- Download all the files from one of the below mentioned dirs into a local machine having access to internet.
+    - [OL8](logan/docker-images/v1.0/oraclelinux/8/)
+	- [Debian](logan/docker-images/v1.0/debian/) 
 - Run the following command to build the docker image.
     - *docker build -t fluentd_oci_la -f Dockerfile .*
 - The docker image built from the above step, can either be pushed to Docker Hub or OCI Container Registry (OCIR) or to a Local Docker Registry depending on the requirements.

logan/docker-images/v1.0/debian/Dockerfile

@@ -1,6 +1,6 @@
 ### Referred version from fluentd-kubernetes-daemonset gitgub repo ###
 
-FROM fluent/fluentd:v1.14.1-debian-1.0 
+FROM fluent/fluentd:v1.14.3-debian-1.0 
 
 USER root
 WORKDIR /home/fluent

logan/docker-images/v1.0/debian/Gemfile

@@ -1,7 +1,7 @@
 source "https://rubygems.org"
 
-gem "fluentd", "1.14.1"
-gem "fluent-plugin-oci-logging-analytics", "2.0.0"
+gem "fluentd", "1.14.3"
+gem "fluent-plugin-oci-logging-analytics", "2.0.2"
 gem "fluent-plugin-concat", "~> 2.5.0"
 gem "fluent-plugin-rewrite-tag-filter", "~> 2.4.0" 
 gem "fluent-plugin-parser-cri", "~> 0.1.1"

logan/docker-images/v1.0/oraclelinux/8/Dockerfile

@@ -0,0 +1,58 @@
+FROM container-registry.oracle.com/os/oraclelinux:8
+
+USER root
+WORKDIR /fluentd
+
+# Environment variables
+ENV PATH /fluentd/vendor/bundle/ruby/2.7.0/bin:$PATH
+ENV GEM_PATH /fluentd/vendor/bundle/ruby/2.7.0:$GEM_PATH
+ENV GEM_HOME /fluentd/vendor/bundle/ruby/2.7.0
+# skip runtime bundler installation
+ENV FLUENTD_DISABLE_BUNDLER_INJECTION 1
+ENV TINI_VERSION=0.19.0
+
+COPY Gemfile /fluentd/
+
+# Install ruby 2.7 along with rubygems and bundler.
+RUN dnf -y module enable ruby:2.7 \
+ && dnf -y install --nodocs ruby ruby-libs \ 
+ && dnf -y install --nodocs rubygems rubygem-openssl rubygem-psych \
+ && dnf -y install --nodocs rubygem-bundler rubygem-io-console \
+# Install development dependent packages for gems native installation
+ && dnf -y install --nodocs gcc make redhat-rpm-config openssl ruby-devel gcc-c++ libtool libffi-devel bzip2 \
+# Install Fluentd, it's dependencies along with other run time dependencies for OCI Logging Analytics Solution
+ && bundle config silence_root_warning true \
+ && bundle config --local path /fluentd/vendor/bundle \
+ && bundle install --gemfile=/fluentd/Gemfile \
+# Install tini, init for containers
+ && curl -L -o /usr/local/bin/tini "https://github.com/krallin/tini/releases/download/v${TINI_VERSION}/tini" \
+ && curl -L -o /usr/local/bin/tini.asc "https://github.com/krallin/tini/releases/download/v$TINI_VERSION/tini.asc" \
+ && export GNUPGHOME="$(mktemp -d)" \
+ && gpg --batch --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 595E85A6B1B4779EA4DAAEC70B588DFF0527A9B7 \
+ && gpg --batch --verify /usr/local/bin/tini.asc /usr/local/bin/tini \
+ && chmod +x /usr/local/bin/tini \
+# Install jemalloc
+ && curl -L -o /tmp/jemalloc-4.5.0.tar.bz2 https://github.com/jemalloc/jemalloc/releases/download/4.5.0/jemalloc-4.5.0.tar.bz2 \
+ && cd /tmp && tar -xjf jemalloc-4.5.0.tar.bz2 && cd jemalloc-4.5.0/ \
+ && ./configure && make \
+ && mv lib/libjemalloc.so.2 /usr/lib \      
+# Install hostname, required by fluent-plugin-rewrite-tag-filter
+ && dnf -y install hostname \
+# Remove all the development dependent packages
+ && dnf -y remove gcc make redhat-rpm-config openssl ruby-devel gcc-c++ libtool libffi-devel bzip2 \
+ && dnf clean all \
+ && rm -rf /var/cache/dnf /tmp/* /var/tmp/* /usr/lib/ruby/gems/*/cache/*.gem
+
+RUN mkdir -p /fluentd/etc /fluentd/plugins \
+ && touch /fluentd/etc/disable.conf
+
+# Environment variables
+ENV FLUENTD_CONF="/fluentd/etc/fluent.conf"
+ENV LD_PRELOAD="/usr/lib/libjemalloc.so.2"
+
+COPY entrypoint.sh /fluentd/entrypoint.sh
+# Give execution permission to entrypoint.sh
+RUN ["chmod", "+x", "/fluentd/entrypoint.sh"]
+
+# Overwrite ENTRYPOINT to run fluentd as root for /var/log / /var/lib
+ENTRYPOINT ["tini", "--", "/fluentd/entrypoint.sh"]

logan/docker-images/v1.0/oraclelinux/8/Gemfile

@@ -0,0 +1,12 @@
+source "https://rubygems.org"
+
+gem "oj", "3.10.18"
+gem "json", "2.4.1"
+gem "ext_monitor", "0.1.2"
+gem "fluentd", "1.14.3"
+gem "fluent-plugin-oci-logging-analytics", "2.0.2"
+gem "fluent-plugin-concat", "~> 2.5.0"
+gem "fluent-plugin-rewrite-tag-filter", "~> 2.4.0"
+gem "fluent-plugin-parser-cri", "~> 0.1.1"
+gem "fluent-plugin-kubernetes_metadata_filter", "2.9.1"
+gem "fluent-plugin-kubernetes-objects", "~> 1.1.7"

logan/docker-images/v1.0/oraclelinux/8/entrypoint.sh

@@ -0,0 +1,3 @@
+#!/usr/bin/env sh
+
+exec fluentd -c ${FLUENTD_CONF} -p /fluentd/plugins --gemfile /fluentd/Gemfile ${FLUENTD_OPT}

logan/helm-chart/Chart.yaml

@@ -15,11 +15,11 @@ type: application
 # This is the chart version. This version number should be incremented each time you make changes
 # to the chart and its templates, including the app version.
 # Versions are expected to follow Semantic Versioning (https://semver.org/)
-version: 1.0.0
+version: 1.0.1
 
 # This is the version number of the application being deployed. This version number should be
 # incremented each time you make changes to the application. Versions are not expected to
 # follow Semantic Versioning. They should reflect the version the application is using.
 # It is recommended to use it with quotes.
 # appVersion here reflects OCI Logging Analytics Fluentd Output Plugin version
-appVersion: "2.0.0"
+appVersion: "2.0.2"

logan/helm-chart/values.yaml

@@ -29,10 +29,10 @@ extraEnv: []
 ## parameters to set requests and limits for memory and cpu
 resources:
    limits:
-      memory: 750Mi
+      memory: 500Mi
    requests:
       cpu: 100m
-      memory: 500Mi
+      memory: 250Mi
 ## @param extraVolumes Extra volumes. Below is an example extra volume
 ##   - name: tmpDir
 ##     hostPath:

logan/kubernetes-resources/logs-collection/fluentd-daemonset.yaml

@@ -97,10 +97,10 @@ spec:
           ## parameters to limit the memory and requests for the pods
           resources:
             limits:
-              memory: 750Mi
+              memory: 500Mi
             requests:
               cpu: 100m
-              memory: 500Mi
+              memory: 250Mi
           volumeMounts:
             - name: varlog
               mountPath: /var/log