Unable to add enhanceK8s global filter the config process is unable to connect to the kubernetes api

[vipinjn24]

Hi,

I am trying to add enhanceK8s global, but it seems the config job fails to connect to the cluster.

My config

apiVersion: logging.banzaicloud.io/v1beta1
kind: Logging
metadata:
  name: fluentd-resource
  namespace: logging
spec:
  controlNamespace: logging
  # globalFilters:
  #   - enhanceK8s:
  #       kubernetes_url: "https://kubernetes.default.svc.cluster.local"
  #       verify_ssl: false
  fluentd:
    disablePvc: true

I there is a specific requirements let me know.
I am using bare metal kubeadm cluster.

[vipinjn24]

Config check pod logs:

fluentd -c /fluentd/etc/fluent.conf --dry-run
2024-01-31 15:10:45 +0000 [info]: init supervisor logger path=nil rotate_age=nil rotate_size=nil
2024-01-31 15:10:45 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
2024-01-31 15:10:45 +0000 [info]: gem 'fluentd' version '1.16.3'
2024-01-31 15:10:45 +0000 [info]: gem 'fluent-mixin-config-placeholders' version '0.4.0'
2024-01-31 15:10:45 +0000 [info]: gem 'fluent-plugin-aliyun-oss' version '0.0.1'
2024-01-31 15:10:45 +0000 [info]: gem 'fluent-plugin-cloudwatch-logs' version '0.14.3'
2024-01-31 15:10:45 +0000 [info]: gem 'fluent-plugin-concat' version '2.5.0'
2024-01-31 15:10:45 +0000 [info]: gem 'fluent-plugin-datadog' version '0.14.2'
2024-01-31 15:10:45 +0000 [info]: gem 'fluent-plugin-datapoint' version '2.0.0'
2024-01-31 15:10:45 +0000 [info]: gem 'fluent-plugin-dedot_filter' version '1.0.0'
2024-01-31 15:10:45 +0000 [info]: gem 'fluent-plugin-detect-exceptions' version '0.0.15'
2024-01-31 15:10:45 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '5.4.2'
2024-01-31 15:10:45 +0000 [info]: gem 'fluent-plugin-enhance-k8s-metadata' version '2.0.0'
2024-01-31 15:10:45 +0000 [info]: gem 'fluent-plugin-events' version '2.0.0'
2024-01-31 15:10:45 +0000 [info]: gem 'fluent-plugin-gcs' version '0.4.3'
2024-01-31 15:10:45 +0000 [info]: gem 'fluent-plugin-gelf-hs' version '1.0.8'
2024-01-31 15:10:45 +0000 [info]: gem 'fluent-plugin-geoip' version '1.3.2'
2024-01-31 15:10:45 +0000 [info]: gem 'fluent-plugin-grafana-loki' version '1.2.20'
2024-01-31 15:10:45 +0000 [info]: gem 'fluent-plugin-grok-parser' version '2.6.2'
2024-01-31 15:10:45 +0000 [info]: gem 'fluent-plugin-kafka' version '0.19.2'
2024-01-31 15:10:45 +0000 [info]: gem 'fluent-plugin-kinesis' version '3.4.2'
2024-01-31 15:10:45 +0000 [info]: gem 'fluent-plugin-kube-events-timestamp' version '0.1.3'
2024-01-31 15:10:45 +0000 [info]: gem 'fluent-plugin-kubernetes-metadata-filter' version '2.5.3'
2024-01-31 15:10:45 +0000 [info]: gem 'fluent-plugin-kubernetes-sumologic' version '2.0.0'
2024-01-31 15:10:45 +0000 [info]: gem 'fluent-plugin-label-router' version '0.2.10'
2024-01-31 15:10:45 +0000 [info]: gem 'fluent-plugin-logzio' version '0.2.2'
2024-01-31 15:10:45 +0000 [info]: gem 'fluent-plugin-mattermost' version '0.2.2'
2024-01-31 15:10:45 +0000 [info]: gem 'fluent-plugin-multi-format-parser' version '1.1.0'
2024-01-31 15:10:45 +0000 [info]: gem 'fluent-plugin-mysqlslowquery' version '0.0.9'
2024-01-31 15:10:45 +0000 [info]: gem 'fluent-plugin-newrelic' version '1.2.2'
2024-01-31 15:10:45 +0000 [info]: gem 'fluent-plugin-opensearch' version '1.1.4'
2024-01-31 15:10:45 +0000 [info]: gem 'fluent-plugin-parser-logfmt' version '0.0.0'
2024-01-31 15:10:45 +0000 [info]: gem 'fluent-plugin-prometheus' version '2.1.0'
2024-01-31 15:10:45 +0000 [info]: gem 'fluent-plugin-prometheus-format' version '2.0.0'
2024-01-31 15:10:45 +0000 [info]: gem 'fluent-plugin-protobuf' version '2.0.0'
2024-01-31 15:10:45 +0000 [info]: gem 'fluent-plugin-record-modifier' version '2.2.0'
2024-01-31 15:10:45 +0000 [info]: gem 'fluent-plugin-record-modifier' version '2.1.1'
2024-01-31 15:10:45 +0000 [info]: gem 'fluent-plugin-redis' version '0.3.5'
2024-01-31 15:10:45 +0000 [info]: gem 'fluent-plugin-remote-syslog' version '1.1'
2024-01-31 15:10:45 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.4.0'
2024-01-31 15:10:45 +0000 [info]: gem 'fluent-plugin-s3' version '1.7.2'
2024-01-31 15:10:45 +0000 [info]: gem 'fluent-plugin-splunk-hec' version '1.3.3'
2024-01-31 15:10:45 +0000 [info]: gem 'fluent-plugin-sqs' version '3.0.0'
2024-01-31 15:10:45 +0000 [info]: gem 'fluent-plugin-sumologic_output' version '1.8.0'
2024-01-31 15:10:45 +0000 [info]: gem 'fluent-plugin-syslog_rfc5424' version '0.9.0.rc.8'
2024-01-31 15:10:45 +0000 [info]: gem 'fluent-plugin-systemd' version '1.0.5'
2024-01-31 15:10:45 +0000 [info]: gem 'fluent-plugin-tag-normaliser' version '0.1.2'
2024-01-31 15:10:45 +0000 [info]: gem 'fluent-plugin-throttle' version '0.0.5'
2024-01-31 15:10:45 +0000 [info]: gem 'fluent-plugin-ua-parser' version '1.2.0'
2024-01-31 15:10:45 +0000 [info]: gem 'fluent-plugin-vmware-log-intelligence' version '2.0.8'
2024-01-31 15:10:45 +0000 [info]: gem 'fluent-plugin-vmware-loginsight' version '1.4.2'
2024-01-31 15:10:45 +0000 [info]: gem 'fluent-plugin-webhdfs' version '1.5.0'
2024-01-31 15:10:45 +0000 [info]: gem 'fluentd' version '1.16.2'
2024-01-31 15:10:45 +0000 [info]: starting fluentd-1.16.3 as dry run mode ruby="3.2.2"
2024-01-31 15:10:45 +0000 [info]: [flow:backups:log-generator:1] DeDot will recurse nested hashes and arrays
2024-01-31 15:10:46 +0000 [info]: [flow:calico-apiserver:log-generator:1] DeDot will recurse nested hashes and arrays
2024-01-31 15:10:46 +0000 [info]: [flow:calico-system:log-generator:1] DeDot will recurse nested hashes and arrays
2024-01-31 15:10:46 +0000 [info]: [flow:cert-manager:log-generator:1] DeDot will recurse nested hashes and arrays
2024-01-31 15:10:46 +0000 [info]: [flow:cloud:log-generator:1] DeDot will recurse nested hashes and arrays
2024-01-31 15:10:46 +0000 [info]: [flow:cns:log-generator:1] DeDot will recurse nested hashes and arrays
2024-01-31 15:10:46 +0000 [info]: [flow:db:log-generator:1] DeDot will recurse nested hashes and arrays
2024-01-31 15:10:46 +0000 [info]: [flow:default:log-generator:1] DeDot will recurse nested hashes and arrays
2024-01-31 15:10:46 +0000 [info]: [flow:dns:log-generator:1] DeDot will recurse nested hashes and arrays
2024-01-31 15:10:46 +0000 [info]: [flow:git:log-generator:1] DeDot will recurse nested hashes and arrays
2024-01-31 15:10:46 +0000 [info]: [flow:home:log-generator:1] DeDot will recurse nested hashes and arrays
2024-01-31 15:10:46 +0000 [info]: [flow:kube-node-lease:log-generator:1] DeDot will recurse nested hashes and arrays
2024-01-31 15:10:46 +0000 [info]: [flow:kube-public:log-generator:1] DeDot will recurse nested hashes and arrays
2024-01-31 15:10:46 +0000 [info]: [flow:kube-system:log-generator:1] DeDot will recurse nested hashes and arrays
2024-01-31 15:10:46 +0000 [info]: [flow:kubernetes-dashboard:log-generator:1] DeDot will recurse nested hashes and arrays
2024-01-31 15:10:46 +0000 [info]: [flow:logging:kube-event-tailer-flow:1] DeDot will recurse nested hashes and arrays
2024-01-31 15:10:46 +0000 [info]: [flow:logging:kube-systemd-tailer-flow:1] DeDot will recurse nested hashes and arrays
2024-01-31 15:10:46 +0000 [info]: [flow:mail:log-generator:1] DeDot will recurse nested hashes and arrays
2024-01-31 15:10:46 +0000 [info]: [flow:metallb-system:log-generator:1] DeDot will recurse nested hashes and arrays
2024-01-31 15:10:46 +0000 [info]: [flow:minio-operator:log-generator:1] DeDot will recurse nested hashes and arrays
2024-01-31 15:10:46 +0000 [info]: [flow:minio-tenant-1:log-generator:1] DeDot will recurse nested hashes and arrays
2024-01-31 15:10:46 +0000 [info]: [flow:monitoring:log-generator:1] DeDot will recurse nested hashes and arrays
2024-01-31 15:10:46 +0000 [info]: [flow:opensearch:log-generator:1] DeDot will recurse nested hashes and arrays
2024-01-31 15:10:46 +0000 [info]: [flow:syc:log-generator:1] DeDot will recurse nested hashes and arrays
2024-01-31 15:10:46 +0000 [info]: [flow:tigera-operator:log-generator:1] DeDot will recurse nested hashes and arrays
2024-01-31 15:10:46 +0000 [info]: [flow:traefik:log-generator:1] DeDot will recurse nested hashes and arrays
2024-01-31 15:10:46 +0000 [info]: [flow:vjs:log-generator:1] DeDot will recurse nested hashes and arrays
2024-01-31 15:10:46 +0000 [info]: [globalFilter:0] cache_ttl: 7200, cache_size: 1000, cache_refresh: 3600, cache_refresh_variation: 900
2024-01-31 15:10:46 +0000 [info]: [globalFilter:0] Initializing Kubernetes API clients
2024-01-31 15:10:46 +0000 [info]: [globalFilter:0] create client with URL: https://kubernetes.default.svc.cluster.local/api and apiVersion: v1
2024-01-31 15:10:46 +0000 [error]: [globalFilter:0] Error creating client (:net_http_persistent is not registered on Faraday::Adapter), retrying in 2 second(s)...
2024-01-31 15:10:48 +0000 [info]: [globalFilter:0] create client with URL: https://kubernetes.default.svc.cluster.local/api and apiVersion: v1
2024-01-31 15:10:48 +0000 [error]: [globalFilter:0] Error creating client (:net_http_persistent is not registered on Faraday::Adapter), retrying in 4 second(s)...
2024-01-31 15:10:52 +0000 [info]: [globalFilter:0] create client with URL: https://kubernetes.default.svc.cluster.local/api and apiVersion: v1
2024-01-31 15:10:52 +0000 [error]: [globalFilter:0] Error creating client (:net_http_persistent is not registered on Faraday::Adapter), retrying in 8 second(s)...
2024-01-31 15:11:00 +0000 [info]: [globalFilter:0] create client with URL: https://kubernetes.default.svc.cluster.local/api and apiVersion: v1
2024-01-31 15:11:00 +0000 [error]: [globalFilter:0] Error creating client (:net_http_persistent is not registered on Faraday::Adapter), retrying in 16 second(s)...

[pepov]

I've created an issue based on this discussion item: #1659

[pepov]

@vipinjn24 what was the reason you wanted to use enhancek8s actually? With our latest release it is now possible to use attach namespace labels and annotations to the log messages, if that was the reason, see:
https://kube-logging.dev/docs/whats-new/#kubernetes-namespace-labels-and-annotations

[vipinjn24]

got it, i was under the impression that we can get the contains memory and cpu usages using this flag. if that is the case we can close the issue.

[pepov]

I'm not sure I understand, how do you get memory and cpu usages?

[vipinjn24]

i thought that this option gives us that, i dont know whether this is even possible or not.

[csatib02]

This was fixed in the follow-up release.