I have installed fluent bit in my kubernetes . but still i have not getting any response in my kibana dashboard. Also i have checked fluent bit pod logs not getting any errors. can anyone tell me how to resolve this issue

[kishorpacefin]

I have checked in fluent bit pods i am not getting any errors . but one thing i saw i am getting successfully connection to kubernetes but not getting any connection related to elastic search.

here is my details
fluent-bit.conf: |
[SERVICE]
Daemon Off
Flush 1
Log_Level info
Parsers_File /fluent-bit/etc/parsers.conf
Parsers_File /fluent-bit/etc/conf/custom_parsers.conf
HTTP_Server On
HTTP_Listen 0.0.0.0
HTTP_Port 2020
Health_Check On

[INPUT]
Name tail
Path /var/log/containers/.log
multiline.parser docker, cri
Tag kube.
Mem_Buf_Limit 512MB
Skip_Long_Lines On

[INPUT]
Name systemd
Tag host.*
Systemd_Filter _SYSTEMD_UNIT=kubelet.service
Read_From_Tail On

[FILTER]
Name kubernetes
Match kube.*
Merge_Log On
Keep_Log Off
K8S-Logging.Parser On
K8S-Logging.Exclude On

[OUTPUT]
Name es
Match kube.*
Host my_app.es.centralindia.azure.elastic-cloud.com
Port 443
Logstash_Format On
HTTP_User elastic
HTTP_Passwd password
Suppress_Type_Name On
Retry_Limit False
tls On
tls.verify Off
Replace_Dots On

[OUTPUT]
Name es
Match host.*
Host my_app.es.centralindia.azure.elastic-cloud.com
Port 443
Logstash_Format On
HTTP_User elastic
HTTP_Passwd password
Suppress_Type_Name On
Logstash_Prefix node
Retry_Limit False
tls On
tls.verify Off
Replace_Dots On
and here logs i am getting
Fluent Bit v2.2.0

Copyright (C) 2015-2023 The Fluent Bit Authors
Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
https://fluentbit.io/
[2023/11/21 08:56:59] [ info] [fluent bit] version=2.2.0, commit=db8487d123, pid=1
[2023/11/21 08:56:59] [ info] [storage] ver=1.5.1, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2023/11/21 08:56:59] [ info] [cmetrics] version=0.6.4
[2023/11/21 08:56:59] [ info] [ctraces ] version=0.3.1
[2023/11/21 08:56:59] [ info] [input:tail:tail.0] initializing
[2023/11/21 08:56:59] [ info] [input:tail:tail.0] storage_strategy='memory' (memory only)
[2023/11/21 08:56:59] [ info] [input:tail:tail.0] multiline core started
[2023/11/21 08:56:59] [ info] [input:systemd:systemd.1] initializing
[2023/11/21 08:56:59] [ info] [input:systemd:systemd.1] storage_strategy='memory' (memory only)
[2023/11/21 08:56:59] [ info] [filter:kubernetes:kubernetes.0] https=1 host=kubernetes.default.svc port=443
[2023/11/21 08:56:59] [ info] [filter:kubernetes:kubernetes.0] token updated
[2023/11/21 08:56:59] [ info] [filter:kubernetes:kubernetes.0] local POD info OK
[2023/11/21 08:56:59] [ info] [filter:kubernetes:kubernetes.0] testing connectivity with API server...
[2023/11/21 08:56:59] [ info] [filter:kubernetes:kubernetes.0] connectivity OK
[2023/11/21 08:56:59] [ info] [output🇪🇸es.0] worker #1 started
[2023/11/21 08:56:59] [ info] [output🇪🇸es.0] worker #0 started
[2023/11/21 08:56:59] [ info] [output🇪🇸es.1] worker #0 started
[2023/11/21 08:56:59] [ info] [output🇪🇸es.1] worker #1 started
[2023/11/21 08:56:59] [ info] [http_server] listen iface=0.0.0.0 tcp_port=2020
[2023/11/21 08:56:59] [ info] [sp] stream processor started
[2023/11/21 08:56:59] [ info] [input:tail:tail.0] inotify_fs_add(): inode=1829472 watch_fd=1 name=/var/log/containers/argocd-redis-7d8d46cc7f-sx7x8_argocd_redis-e8de5686da06e1d7e478a7bb2f621c2c175c8b9f40e6f6355ef56bcf94d862c7.log
[2023/11/21 08:56:59] [ info] [input:tail:tail.0] inotify_fs_add(): inode=4200490 watch_fd=2 name=/var/log/containers/argocd-server-5986f74c99-xs52l_argocd_argocd-server-9fda83cf12b5f7d96f8073759d2f188e88b01751de577f34dfa49eace8f25856.log
[2023/11/21 08:56:59] [ info] [input:tail:tail.0] inotify_fs_add(): inode=1827648 watch_fd=3 name=/var/log/containers/calico-kube-controllers-7c8c45649-kdn9q_calico-system_calico-kube-controllers-493e2b61885f1e044b1e067f38c1cd81dbbd2c922d51e75d09192a4553397ba1.log
[2023/11/21 08:56:59] [ info] [input:tail:tail.0] inotify_fs_add(): inode=4200818 watch_fd=4 name=/var/log/containers/calico-node-n7jn9_calico-system_calico-node-30536f2af8a8f745daad02a95f5dad31f80223a3b96b6d80d997580e4c0da013.log
[2023/11/21 08:56:59] [ info] [input:tail:tail.0] inotify_fs_add(): inode=4200725 watch_fd=5 name=/var/log/containers/calico-node-n7jn9_calico-system_flexvol-driver-0551b09373ac01ff6ae2dbf22a749d0d32ff766a0ba2eb764660ed898447312a.log
[2023/11/21 08:56:59] [ info] [input:tail:tail.0] inotify_fs_add(): inode=4200737 watch_fd=6 name=/var/log/containers/calico-node-n7jn9_calico-system_install-cni-c9c3d04c27d360fe0e18d8028a5d3ce278b735d679b038049b226eb05fc46fe9.log
[2023/11/21 08:56:59] [ info] [input:tail:tail.0] inotify_fs_add(): inode=1827504 watch_fd=7 name=/var/log/containers/calico-typha-566f69464-7lxvj_calico-system_calico-typha-88c3c7bdab1ae9af0c72497fcc6b9230bf8320ab21dbef5dda096befaa74193a.log
[2023/11/21 08:56:59] [ info] [input:tail:tail.0] inotify_fs_add(): inode=1806722 watch_fd=8 name=/var/log/containers/cloud-node-manager-729ct_kube-system_cloud-node-manager-d747502990e72749de6f4d47f97618988b846d60b7ffec58abe26c407bc51d07.log
[2023/11/21 08:56:59] [ info] [input:tail:tail.0] inotify_fs_add(): inode=4200588 watch_fd=9 name=/var/log/containers/cm-ftp-automation-84957cdf45-ngm6w_pocketful-dev_cm-ftp-automation-3211560af05b0ee7fc4450c46d332bef279641a2358220c4e2980cd5d79d8ea0.log
[2023/11/21 08:56:59] [ info] [input:tail:tail.0] inotify_fs_add(): inode=1827681 watch_fd=10 name=/var/log/containers/coredns-76b9877f49-v8fgw_kube-system_coredns-3b65111c78f1b0766220a5d7e9bcc54c12b986895f635cc41d1e4e45a4e52708.log
[2023/11/21 08:56:59] [ info] [input:tail:tail.0] inotify_fs_add(): inode=1829519 watch_fd=11 name=/var/log/containers/coredns-autoscaler-85f7d6b75d-896lh_kube-system_autoscaler-cd107e3cf9ae23313806d5b2ae9bd9d9d538a42bcd02914f99f08e0a33a9e508.log
[2023/11/21 08:56:59] [ info] [input:tail:tail.0] inotify_fs_add(): inode=1806856 watch_fd=12 name=/var/log/containers/csi-azuredisk-node-gfg9t_kube-system_azuredisk-6340e691d437f3f76f7c35d7ea88989ece1fba5dac1d0f91b33453c84e9eb348.log
[2023/11/21 08:56:59] [ info] [input:tail:tail.0] inotify_fs_add(): inode=1806724 watch_fd=13 name=/var/log/containers/csi-azuredisk-node-gfg9t_kube-system_liveness-probe-e397803789dd1aad24328dcfbfaccbc9e2ce0cac8579649f426abc345d4d026b.log
[2023/11/21 08:56:59] [ info] [input:tail:tail.0] inotify_fs_add(): inode=1806820 watch_fd=14 name=/var/log/containers/csi-azuredisk-node-gfg9t_kube-system_node-driver-registrar-b47b5c4e35aa21503651231c10fbd055cad607cb19d71ff5f8fc9585095fbb33.log
[2023/11/21 08:56:59] [ info] [input:tail:tail.0] inotify_fs_add(): inode=1806851 watch_fd=15 name=/var/log/containers/csi-azurefile-node-dv2c4_kube-system_azurefile-b3ce6864f6bef638b67e4c6cd060c6722f81e357fa8fc0aa7d4945afd2609efa.log
[2023/11/21 08:56:59] [ info] [input:tail:tail.0] inotify_fs_add(): inode=1806704 watch_fd=16 name=/var/log/containers/csi-azurefile-node-dv2c4_kube-system_liveness-probe-7c306ae32bf275e33de7bc40513abeb3ec2a7a2dc779317c8b14264ad94fa610.log
[2023/11/21 08:56:59] [ info] [input:tail:tail.0] inotify_fs_add(): inode=1806797 watch_fd=17 name=/var/log/containers/csi-azurefile-node-dv2c4_kube-system_node-driver-registrar-5aaedb51b220b3ddf5fc4cc7c9aeabe5c232dd3d02fe4b0fce7d7c05b3d40e2f.log
[2023/11/21 08:56:59] [ info] [input:tail:tail.0] inotify_fs_add(): inode=4176370 watch_fd=18 name=/var/log/containers/digiowebhook-app-65f76567fb-xrpl8_pocketful-dev_digiowebhook-app-da220796e260a489f7ee0b3fd48f28e1f7b7199f8a1b1c850682dfe463b2c2cb.log
[2023/11/21 08:56:59] [ info] [input:tail:tail.0] inotify_fs_add(): inode=1806569 watch_fd=19 name=/var/log/containers/ftp-websocket-7c56cd6b5-vbmn2_pocketful-dev_ftp-websocket-eda5486a0e710f75a28b044e85c3a23149773e8736ada51229fe2ce671d0e5c3.log
[2023/11/21 08:56:59] [ info] [input:tail:tail.0] inotify_fs_add(): inode=1827263 watch_fd=20 name=/var/log/containers/konnectivity-agent-5df89d895c-67wlg_kube-system_konnectivity-agent-a0abb1271cbe365225023f812672381227a12f6b1b75dc58bd3b4b18852dfed4.log
[2023/11/21 08:57:00] [ info] [input:tail:tail.0] inotify_fs_add(): inode=1806701 watch_fd=21 name=/var/log/containers/kube-proxy-vzmzp_kube-system_kube-proxy-bootstrap-1ca65abbccf52436fc20ec1fe173214f9f1fd44cec48a9971855b0da5fc26aaf.log
[2023/11/21 08:57:00] [ info] [input:tail:tail.0] inotify_fs_add(): inode=3899259 watch_fd=22 name=/var/log/containers/notifications-app-5d4c774bc5-grnms_pocketful-dev_notifications-app-ec0286586ee186aab9c279cc403bcd315871d94d69c2fbbe434e11b04d6d4c8d.log
[2023/11/21 08:57:00] [ info] [input:tail:tail.0] inotify_fs_add(): inode=4161973 watch_fd=23 name=/var/log/containers/pace-rbac-app-65cbc94798-fbw29_pocketful-dev_pace-rbac-app-2fa007b24c9d6e4ce8c6de7b5eaeef7e292c49ffd9299f2768c2610381f3f44e.log
[2023/11/21 08:57:00] [ info] [input:tail:tail.0] inotify_fs_add(): inode=4200513 watch_fd=24 name=/var/log/containers/pacejobs-app-887d4f8f6-vgggp_pocketful-dev_pacejobs-app-71f8bddfff4fa919eee629f9f170e04c63367cb7a617b98e68a4055f95b0f383.log
[2023/11/21 08:57:00] [ info] [input:tail:tail.0] inotify_fs_add(): inode=4200653 watch_fd=25 name=/var/log/containers/pacejobs-app-887d4f8f6-vgggp_pocketful-dev_pacejobs-app-aac87bdebcba4f55188c3f90196fef067047f6d841d40afada231947425b203a.log
[2023/11/21 08:57:00] [ info] [input:tail:tail.0] inotify_fs_add(): inode=3884256 watch_fd=26 name=/var/log/containers/paper-trading-app-7c94599c58-ch876_pocketful-dev_paper-trading-app-058474c0d7d6d2517790586bacfb66f2d1078ac086d2cf4b8001779438c4a678.log
[2023/11/21 08:57:00] [ info] [input:tail:tail.0] inotify_fs_add(): inode=4200558 watch_fd=27 name=/var/log/containers/payment-pocketful-app-866b7857db-bprcr_pocketful-dev_payment-pocketful-app-73e6439fe5310b8f20652d95a6b3ba5c5f93540ee3efabdb4e80e7c0232cc96b.log
[2023/11/21 08:57:00] [ info] [input:tail:tail.0] inotify_fs_add(): inode=4200845 watch_fd=28 name=/var/log/containers/pocketful-landing-page-app-547b477d99-vmwj2_pocketful-dev_pocketful-landing-page-app-ffe0bfcab81257f3683e196ff65a051aec1b9469945c87d91b7793c196a15241.log
[2023/11/21 08:57:00] [ info] [input:tail:tail.0] inotify_fs_add(): inode=4200532 watch_fd=29 name=/var/log/containers/pocketful-web-app-84d4b9b8f8-gvtl8_pocketful-dev_pocketful-web-app-9cc74a50f12b4e341298e909fa03d4aa2223f5c5b5df374938671b6f64e9eec9.log
[2023/11/21 08:57:00] [ info] [input:tail:tail.0] inotify_fs_add(): inode=4200743 watch_fd=30 name=/var/log/containers/pocketful-web-app-84d4b9b8f8-jdtwc_pocketful-dev_pocketful-web-app-429b70e18c68985d4621a6c0e4cbcd0aa0f09279ebc0052a1bc5edce00367062.log
[2023/11/21 08:57:00] [ info] [input:tail:tail.0] inotify_fs_add(): inode=4200772 watch_fd=31 name=/var/log/containers/pocketful-web-app-84d4b9b8f8-szjsz_pocketful-dev_pocketful-web-app-d7857d0c2a47177ac080ed12f14ca8aa843f390bab82d68ae96c4794b96066aa.log
[2023/11/21 08:57:00] [ info] [input:tail:tail.0] inotify_fs_add(): inode=4190841 watch_fd=32 name=/var/log/containers/qr-util-app-6c49979d87-s89w9_pocketful-dev_qr-util-app-7c80fc27453e4523ab05a4afef1b387cee4f50e0555bf27a55fc75e1f9b6a67c.log
[2023/11/21 08:57:00] [ info] [input:tail:tail.0] inotify_fs_add(): inode=4200624 watch_fd=33 name=/var/log/containers/redis-3a-6c8fdfd647-vcctq_pocketful-dev_redis-3a-6bee588addb767ef4a2f70f0a167f2a04eb0bbaa677c8ac824b340d68fae33b5.log
[2023/11/21 08:57:00] [ info] [input:tail:tail.0] inotify_fs_add(): inode=4200793 watch_fd=34 name=/var/log/containers/space-app-b8966d597-2k26r_pocketful-dev_space-app-5d28448abb3cb749368354a65483f745bdff8f81de420a1d7a57edd4ce660447.log
[2023/11/21 08:57:00] [ info] [input:tail:tail.0] inotify_fs_add(): inode=1807112 watch_fd=35 name=/var/log/containers/kube-proxy-vzmzp_kube-system_kube-proxy-5371bb2ed4e1a036f29465f091e98f0a67ef86124af43fd7f149f69c015aa9a7.log
[2023/11/21 08:57:00] [ info] [input:tail:tail.0] inotify_fs_add(): inode=4200512 watch_fd=36 name=/var/log/containers/fluent-bit-wq5dc_kube-logging_fluent-bit-b5649877da26dce225f086ce9660855a2c41cba56b7ad73672cfcea66005606d.log

here you can see no error in my logs but stiil not receiving any logs in my kibana

i have checked my elastc endpoint it is accessable from kubernetes
Test-NetConnection -ComputerName my_app.es.centralindia.azure.elastic-cloud.com -Port 443

ComputerName :my_app.es.centralindia.azure.elastic-cloud.com
RemoteAddress : //here i just hide my ip
RemotePort : 443
InterfaceAlias : Wi-Fi
SourceAddress : //hide my source ip
TcpTestSucceeded : True

[patrick-stephens]

Duplicate of #8207