Fluentd Forwarding to different OpenShift Cluster

[chrisblatt]

Hi! First of all fluentd has been a fantastic tool for us! We recently moved all our apps running in a single OpenShift cluster to now many apps running in multiple OpenShift clusters. On our previous cluster and our new cluster our infrastrucuture team has left log collection up to the individual teams. Our team chose to implmented fluentd sending to an elasticsearch cluster. This was working fine for our apps when we were in a single OpenShift cluster, our fluentd service to elasticsearch was configured like this:

<source>
      @type http
      port 9881
      bind 0.0.0.0
      <parse>
        @type syslog
        message_format rfc5424
        with_priority true
      </parse>
    </source>
...
    <source>
      @type forward
      port 24224
      bind 0.0.0.0
    </source>
...
<match **>
      @type elasticsearch
      @id out_es
      @log_level debug
      include_tag_key false
      suppress_type_name true
      host "{{ .Values.es.endpoint }}"
      port "{{ .Values.es.port }}"
      scheme "{{ .Values.es.scheme }}"
      user "{{ .Values.elasticsearch.user }}"
      password "{{ .Values.elasticsearch.password }}"
      reconnect_on_error "true"
      reload_on_failure "true"
      log_es_400_reason "true"
      logstash_prefix "${tag}"
      logstash_dateformat "%Y.%m.%d"
      logstash_format "true"
      include_timestamp "true"
      request_timeout "60s"
      <buffer>
        slow_flush_log_threshold 30.0
        flush_thread_count "3"
        flush_interval "15s"
        chunk_limit_size "100M"
        retry_max_interval "30"
        retry_forever false
      </buffer>
    </match>

We created a service in OpenShift called fluentd-service and configured a sidecar like this to forward logs which is working:

<match **>
      @type forward
       send_timeout 60s
       recover_wait 10s
       hard_timeout 60s
       <server>
         name openshift-fluentd
         host fluentd-service.< namespace >.svc.cluster.local
         port 24224
         weight 60
       </server>
    </match>

This setup worked perfectly and we never had any issues with it with a single cluster. But now that we have moved to multiple clusters we still wanted to host the main fluentd service to elasticsearch on one cluster and just use sidecars in other clusters to forward logs to it. The sidecars on the same cluster use the configuration above using the local cluster dns name, but we are having problems with off cluster sidecars not being able to communicate.

Our fluentd service has port 24224 exposed for UDP and TCP but we haven't found a way to expose both protocols on the port using an OpenShift route, we created this route for sidecars outside our cluster to forward logs but it is not working:

kind: Route
apiVersion: route.openshift.io/v1
metadata:
  name: fluentd-preprod-fwd
  namespace: < namespace >
  labels:
    app.kubernetes.io/managed-by: Helm
  annotations:
    meta.helm.sh/release-name: fluentd-preprod
    meta.helm.sh/release-namespace: < namespace >
spec:
  host: preprod-log-fwd.example.com
  to:
    kind: Service
    name: fluentd-service
    weight: 100
  port:
    targetPort: fluentd-fwd-tcp # port 24224

We are assuming this route is not working when configured in the sidecars due to only the TCP and UDP route not exposed? Does anyone have any suggestions or is there a piece I am missing?

Here is the sidecare config we attempted for cross cluster forwarding:

<match **>
      @type forward
       send_timeout 60s
       recover_wait 10s
       hard_timeout 60s
       <server>
         name openshift-fluentd
         host preprod-log-fwd.example.com
         port 24224
         weight 60
       </server>
    </match>