filter_kubernetes: memory leak when Use_Pod_Association is enabled #11523
Description
Bug Report
Describe the bug
When Use_Pod_Association On is enabled in the kubernetes filter, fluent-bit memory grows linearly over time (~3-4 MB/hour), eventually causing OOMKill. Setting Use_Pod_Association Off immediately stabilizes memory.
To Reproduce
- Install the
amazon-cloudwatch-observabilityEKS addon (v4.10.0-eksbuild.1). Deploy withUse_Pod_Association Onin the kubernetes filter - Monitor memory over several hours
- Observe steady linear memory growth
Setting Use_Pod_Association Off stops the growth.
Expected behavior
Memory should remain stable over time when Use_Pod_Association On is enabled.
Screenshot
Massif heap profiler profiling
Ran Valgrind Massif on confirm the leak is from OPENSSL when calling fetch_pod_service_map periodically in plugins/filter_kubernetes/kubernetes_aws.c.
Below are two snapshots 4 and 79 to compare.
| Snapshot 4 (early) | Snapshot 79 (late) | Growth | |
|---|---|---|---|
| Total heap | 13.1 MB | 66.1 MB | +52.9 MB |
| CRYPTO_malloc | 8.2 MB | 54.9 MB | +46.7 MB |
| fetch_pod_service_map | ~2.8 MB | ~43.8 MB | +41.0 MB |
snapshot=4
#-----------
time=19025813146
mem_heap_B=13136045
mem_heap_extra_B=1383443
mem_stacks_B=0
heap_tree=detailed
n7: 13136045 (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
n10: 8227163 0x6245087: CRYPTO_malloc (in /usr/lib64/libcrypto.so.3.2.2)
n9: 3851520 0x6245297: CRYPTO_zalloc (in /usr/lib64/libcrypto.so.3.2.2)
n0: 1084936 in 190 places, all below massif's threshold (1.00%)
n3: 654952 0x61573BF: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n1: 355584 0x615761F: ASN1_item_new_ex (in /usr/lib64/libcrypto.so.3.2.2)
n2: 355584 0x62DBAF7: X509_new_ex (in /usr/lib64/libcrypto.so.3.2.2)
n1: 330624 0x626A3BB: PEM_X509_INFO_read_bio_ex (in /usr/lib64/libcrypto.so.3.2.2)
n1: 330624 0x62ADF8B: X509_load_cert_crl_file_ex (in /usr/lib64/libcrypto.so.3.2.2)
n1: 330624 0x62AE127: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n1: 330624 0x62CB33B: X509_STORE_load_file_ex (in /usr/lib64/libcrypto.so.3.2.2)
n2: 330624 0x5FBA20B: SSL_CTX_load_verify_locations (in /usr/lib64/libssl.so.3.2.2)
n1: 329472 0x68D4A3: load_system_certificates (openssl.c:684)
n1: 329472 0x68D7EF: tls_context_create (openssl.c:827)
n2: 329472 0x68EBE3: flb_tls_create (flb_tls.c:200)
n1: 164736 0xC8F717: cb_cloudwatch_init (cloudwatch_logs.c:246)
n1: 164736 0x57858B: flb_output_init_all (flb_output.c:1508)
n1: 164736 0x5ACDDB: flb_engine_start (flb_engine.c:972)
n1: 164736 0x52B563: flb_lib_worker (flb_lib.c:904)
n1: 164736 0x69F1B77: start_thread (in /usr/lib64/libc.so.6)
n0: 164736 0x6A5ECDB: thread_start (in /usr/lib64/libc.so.6)
n1: 164736 0xC8F7FF: cb_cloudwatch_init (cloudwatch_logs.c:261)
n1: 164736 0x57858B: flb_output_init_all (flb_output.c:1508)
n1: 164736 0x5ACDDB: flb_engine_start (flb_engine.c:972)
n1: 164736 0x52B563: flb_lib_worker (flb_lib.c:904)
n1: 164736 0x69F1B77: start_thread (in /usr/lib64/libc.so.6)
n0: 164736 0x6A5ECDB: thread_start (in /usr/lib64/libc.so.6)
n0: 1152 in 1 place, below massif's threshold (1.00%)
n0: 24960 in 3 places, all below massif's threshold (1.00%)
n2: 284552 0x614EB9B: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n2: 283016 0x615008F: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n2: 151096 0x614EB47: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n1: 151032 0x615008F: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n1: 151032 0x614EB47: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n1: 151032 0x614FE8B: ASN1_item_ex_d2i (in /usr/lib64/libcrypto.so.3.2.2)
n1: 151032 0x62DB3AB: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n2: 151032 0x614EB0F: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n1: 149568 0x61501F7: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n1: 149568 0x614EC33: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n1: 149568 0x61501F7: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n1: 149568 0x614EC33: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n2: 149568 0x614F28F: ASN1_item_d2i_ex (in /usr/lib64/libcrypto.so.3.2.2)
n1: 146160 0x626A41F: PEM_X509_INFO_read_bio_ex (in /usr/lib64/libcrypto.so.3.2.2)
n1: 146160 0x62ADF8B: X509_load_cert_crl_file_ex (in /usr/lib64/libcrypto.so.3.2.2)
n1: 146160 0x62AE127: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n1: 146160 0x62CB33B: X509_STORE_load_file_ex (in /usr/lib64/libcrypto.so.3.2.2)
n2: 146160 0x5FBA20B: SSL_CTX_load_verify_locations (in /usr/lib64/libssl.so.3.2.2)
n1: 146016 0x68D4A3: load_system_certificates (openssl.c:684)
n1: 146016 0x68D7EF: tls_context_create (openssl.c:827)
n1: 146016 0x68EBE3: flb_tls_create (flb_tls.c:200)
n0: 146016 in 2 places, all below massif's threshold (1.00%)
n0: 144 in 1 place, below massif's threshold (1.00%)
n0: 3408 in 2 places, all below massif's threshold (1.00%)
n0: 1464 in 1 place, below massif's threshold (1.00%)
n0: 64 in 1 place, below massif's threshold (1.00%)
n0: 131920 in 1 place, below massif's threshold (1.00%)
n0: 1536 in 2 places, all below massif's threshold (1.00%)
n0: 14816 in 1 place, below massif's threshold (1.00%)
n1: 549568 0x601E44F: ??? (in /usr/lib64/libssl.so.3.2.2)
n1: 549568 0x5FFE82F: ??? (in /usr/lib64/libssl.so.3.2.2)
n2: 549568 0x5FF744B: ??? (in /usr/lib64/libssl.so.3.2.2)
n3: 549568 0x5FDC417: ??? (in /usr/lib64/libssl.so.3.2.2)
n1: 274784 0x600FBF7: ??? (in /usr/lib64/libssl.so.3.2.2)
n1: 274784 0x600A7EF: ??? (in /usr/lib64/libssl.so.3.2.2)
n1: 274784 0x68E713: tls_net_handshake (openssl.c:1513)
n1: 274784 0x68F543: flb_tls_session_create (flb_tls.c:630)
n1: 274784 0x6FB687: flb_io_net_connect (flb_io.c:175)
n1: 274784 0x5D9847: create_conn (flb_upstream.c:629)
n2: 274784 0x5D9EAB: flb_upstream_conn_get (flb_upstream.c:833)
n1: 270352 0xD63DC3: fetch_pod_service_map (kubernetes_aws.c:228)
n1: 270352 0xD5D5CF: update_pod_service_map (kubernetes.c:66)
n1: 270352 0x69F1B77: start_thread (in /usr/lib64/libc.so.6)
n0: 270352 0x6A5ECDB: thread_start (in /usr/lib64/libc.so.6)
n0: 4432 in 2 places, all below massif's threshold (1.00%)
n1: 274784 0x60196CB: ??? (in /usr/lib64/libssl.so.3.2.2)
n1: 274784 0x600A9E7: ??? (in /usr/lib64/libssl.so.3.2.2)
n1: 274784 0x68E713: tls_net_handshake (openssl.c:1513)
n1: 274784 0x68F543: flb_tls_session_create (flb_tls.c:630)
n1: 274784 0x6FB687: flb_io_net_connect (flb_io.c:175)
n1: 274784 0x5D9847: create_conn (flb_upstream.c:629)
n2: 274784 0x5D9EAB: flb_upstream_conn_get (flb_upstream.c:833)
n1: 270352 0xD63DC3: fetch_pod_service_map (kubernetes_aws.c:228)
n1: 270352 0xD5D5CF: update_pod_service_map (kubernetes.c:66)
n1: 270352 0x69F1B77: start_thread (in /usr/lib64/libc.so.6)
n0: 270352 0x6A5ECDB: thread_start (in /usr/lib64/libc.so.6)
n0: 4432 in 2 places, all below massif's threshold (1.00%)
n0: 0 in 3 places, all below massif's threshold (1.00%)
n0: 0 in 2 places, all below massif's threshold (1.00%)
n2: 469232 0x6176CDF: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n2: 275176 0x617861F: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n1: 274920 0x6158AAB: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n1: 274920 0x614DC0F: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n1: 274920 0x614EE9F: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n1: 274920 0x61501F7: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n1: 274920 0x614EC33: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n2: 274920 0x614F28F: ASN1_item_d2i_ex (in /usr/lib64/libcrypto.so.3.2.2)
n1: 274656 0x6288E9B: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n1: 274656 0x62DC1F3: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n1: 274656 0x62DC3CF: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n1: 274656 0x614EA2F: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n1: 274656 0x614F28F: ASN1_item_d2i_ex (in /usr/lib64/libcrypto.so.3.2.2)
n1: 274656 0x63361BF: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n1: 274656 0x62D65B3: d2i_RSA_PUBKEY (in /usr/lib64/libcrypto.so.3.2.2)
n1: 274656 0x62F6677: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n1: 274656 0x61F0E0F: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n1: 274656 0x62F99A3: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n1: 274656 0x61F0E0F: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n1: 274656 0x61F10BF: OSSL_DECODER_from_bio (in /usr/lib64/libcrypto.so.3.2.2)
n1: 274656 0x61F13F7: OSSL_DECODER_from_data (in /usr/lib64/libcrypto.so.3.2.2)
n1: 274656 0x62DC45B: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n1: 274656 0x614EA2F: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n1: 274656 0x61501F7: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n1: 274656 0x614EC33: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n1: 274656 0x61501F7: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n1: 274656 0x614EC33: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n0: 274656 0x614F28F: ASN1_item_d2i_ex (in /usr/lib64/libcrypto.so.3.2.2)
n0: 264 in 1 place, below massif's threshold (1.00%)
n0: 256 in 16 places, all below massif's threshold (1.00%)
n0: 194056 in 18 places, all below massif's threshold (1.00%)
n1: 383408 0x5FBDD7B: ??? (in /usr/lib64/libssl.so.3.2.2)
n1: 383408 0x68DCD7: tls_session_create (openssl.c:1175)
n1: 383408 0x68F49B: flb_tls_session_create (flb_tls.c:611)
n1: 383408 0x6FB687: flb_io_net_connect (flb_io.c:175)
n1: 383408 0x5D9847: create_conn (flb_upstream.c:629)
n2: 383408 0x5D9EAB: flb_upstream_conn_get (flb_upstream.c:833)
n1: 377224 0xD63DC3: fetch_pod_service_map (kubernetes_aws.c:228)
n1: 377224 0xD5D5CF: update_pod_service_map (kubernetes.c:66)
n1: 377224 0x69F1B77: start_thread (in /usr/lib64/libc.so.6)
n0: 377224 0x6A5ECDB: thread_start (in /usr/lib64/libc.so.6)
n0: 6184 in 2 places, all below massif's threshold (1.00%)
n2: 244920 0x613BF27: ASN1_STRING_type_new (in /usr/lib64/libcrypto.so.3.2.2)
n1: 195480 0x6155683: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n1: 195480 0x6157287: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n1: 195480 0x615789F: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n2: 195480 0x615747F: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n1: 151032 0x614EB9B: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n1: 151032 0x615008F: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n1: 151032 0x614EB47: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n1: 151032 0x615008F: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n1: 151032 0x614EB47: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n1: 151032 0x614FE8B: ASN1_item_ex_d2i (in /usr/lib64/libcrypto.so.3.2.2)
n1: 151032 0x62DB3AB: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n2: 151032 0x614EB0F: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n1: 149568 0x61501F7: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n1: 149568 0x614EC33: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n1: 149568 0x61501F7: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n1: 149568 0x614EC33: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n2: 149568 0x614F28F: ASN1_item_d2i_ex (in /usr/lib64/libcrypto.so.3.2.2)
n1: 146160 0x626A41F: PEM_X509_INFO_read_bio_ex (in /usr/lib64/libcrypto.so.3.2.2)
n1: 146160 0x62ADF8B: X509_load_cert_crl_file_ex (in /usr/lib64/libcrypto.so.3.2.2)
n1: 146160 0x62AE127: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n1: 146160 0x62CB33B: X509_STORE_load_file_ex (in /usr/lib64/libcrypto.so.3.2.2)
n2: 146160 0x5FBA20B: SSL_CTX_load_verify_locations (in /usr/lib64/libssl.so.3.2.2)
n1: 146016 0x68D4A3: load_system_certificates (openssl.c:684)
n1: 146016 0x68D7EF: tls_context_create (openssl.c:827)
n1: 146016 0x68EBE3: flb_tls_create (flb_tls.c:200)
n0: 146016 in 2 places, all below massif's threshold (1.00%)
n0: 144 in 1 place, below massif's threshold (1.00%)
n0: 3408 in 2 places, all below massif's threshold (1.00%)
n0: 1464 in 1 place, below massif's threshold (1.00%)
n0: 44448 in 2 places, all below massif's threshold (1.00%)
n0: 49440 in 6 places, all below massif's threshold (1.00%)
n1: 159752 0x622EF0B: EVP_PKEY_new (in /usr/lib64/libcrypto.so.3.2.2)
n0: 159752 in 5 places, all below massif's threshold (1.00%)
n1: 154896 0x62577E7: CRYPTO_THREAD_lock_new (in /usr/lib64/libcrypto.so.3.2.2)
n0: 154896 in 35 places, all below massif's threshold (1.00%)
n1: 149856 0x628CEC7: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n1: 149856 0x62838CB: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n1: 149856 0x6157377: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n1: 149856 0x614EB9B: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n2: 149856 0x614F28F: ASN1_item_d2i_ex (in /usr/lib64/libcrypto.so.3.2.2)
n1: 149632 0x6288E9B: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n1: 149632 0x62DC1F3: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n1: 149632 0x62DC3CF: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n1: 149632 0x614EA2F: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n1: 149632 0x614F28F: ASN1_item_d2i_ex (in /usr/lib64/libcrypto.so.3.2.2)
n1: 149632 0x63361BF: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n1: 149632 0x62D65B3: d2i_RSA_PUBKEY (in /usr/lib64/libcrypto.so.3.2.2)
n1: 149632 0x62F6677: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n1: 149632 0x61F0E0F: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n1: 149632 0x62F99A3: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n1: 149632 0x61F0E0F: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n1: 149632 0x61F10BF: OSSL_DECODER_from_bio (in /usr/lib64/libcrypto.so.3.2.2)
n1: 149632 0x61F13F7: OSSL_DECODER_from_data (in /usr/lib64/libcrypto.so.3.2.2)
n1: 149632 0x62DC45B: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n1: 149632 0x614EA2F: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n1: 149632 0x61501F7: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n1: 149632 0x614EC33: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n1: 149632 0x61501F7: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n1: 149632 0x614EC33: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n1: 149632 0x614F28F: ASN1_item_d2i_ex (in /usr/lib64/libcrypto.so.3.2.2)
n0: 149632 in 3 places, all below massif's threshold (1.00%)
n0: 224 in 1 place, below massif's threshold (1.00%)
n1: 1036144 0x5FFD7D7: ??? (in /usr/lib64/libssl.so.3.2.2)
n1: 1036144 0x5FFDD13: ??? (in /usr/lib64/libssl.so.3.2.2)
n1: 1036144 0x5FF8B17: ??? (in /usr/lib64/libssl.so.3.2.2)
n2: 1036144 0x5FF66E7: ??? (in /usr/lib64/libssl.so.3.2.2)
n1: 1036144 0x5FAEE33: ??? (in /usr/lib64/libssl.so.3.2.2)
n1: 1036144 0x5FB7DE3: SSL_read (in /usr/lib64/libssl.so.3.2.2)
n2: 1036144 0x68E003: tls_net_read (openssl.c:1308)
n1: 1036144 0x68EFE3: flb_tls_net_read_async (flb_tls.c:384)
n1: 1036144 0x6FCDE7: flb_io_net_read (flb_io.c:747)
n1: 1036144 0x704287: flb_http_get_response_data (flb_http_client.c:1583)
n2: 1036144 0x704453: flb_http_do (flb_http_client.c:1642)
n1: 1019432 0xD63F7B: fetch_pod_service_map (kubernetes_aws.c:257)
n1: 1019432 0xD5D5CF: update_pod_service_map (kubernetes.c:66)
n1: 1019432 0x69F1B77: start_thread (in /usr/lib64/libc.so.6)
n0: 1019432 0x6A5ECDB: thread_start (in /usr/lib64/libc.so.6)
n0: 16712 in 1 place, below massif's threshold (1.00%)
n0: 0 in 1 place, below massif's threshold (1.00%)
n0: 0 in 1 place, below massif's threshold (1.00%)
n1: 1027278 0x601EA67: ??? (in /usr/lib64/libssl.so.3.2.2)
n1: 1027278 0x5FF8597: ??? (in /usr/lib64/libssl.so.3.2.2)
n1: 1027278 0x5FF9C63: ??? (in /usr/lib64/libssl.so.3.2.2)
n1: 1027278 0x5FFA46F: ??? (in /usr/lib64/libssl.so.3.2.2)
n2: 1027278 0x5FF519F: ??? (in /usr/lib64/libssl.so.3.2.2)
n1: 1027278 0x5FB8583: SSL_write (in /usr/lib64/libssl.so.3.2.2)
n2: 1027278 0x68E207: tls_net_write (openssl.c:1370)
n1: 1027278 0x68F22F: flb_tls_net_write_async (flb_tls.c:494)
n1: 1027278 0x6FC8A7: flb_io_net_write (flb_io.c:699)
n1: 1027278 0x703E5F: flb_http_do_request (flb_http_client.c:1472)
n2: 1027278 0x70442F: flb_http_do (flb_http_client.c:1630)
n1: 1010709 0xD63F7B: fetch_pod_service_map (kubernetes_aws.c:257)
n1: 1010709 0xD5D5CF: update_pod_service_map (kubernetes.c:66)
n1: 1010709 0x69F1B77: start_thread (in /usr/lib64/libc.so.6)
n0: 1010709 0x6A5ECDB: thread_start (in /usr/lib64/libc.so.6)
n0: 16569 in 1 place, below massif's threshold (1.00%)
n0: 0 in 1 place, below massif's threshold (1.00%)
n0: 0 in 1 place, below massif's threshold (1.00%)
n1: 656890 0x614F0D3: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n1: 656890 0x61501F7: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n1: 656890 0x614EC33: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n2: 656890 0x614F28F: ASN1_item_d2i_ex (in /usr/lib64/libcrypto.so.3.2.2)
n1: 617392 0x626A41F: PEM_X509_INFO_read_bio_ex (in /usr/lib64/libcrypto.so.3.2.2)
n1: 617392 0x62ADF8B: X509_load_cert_crl_file_ex (in /usr/lib64/libcrypto.so.3.2.2)
n1: 617392 0x62AE127: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n1: 617392 0x62CB33B: X509_STORE_load_file_ex (in /usr/lib64/libcrypto.so.3.2.2)
n2: 617392 0x5FBA20B: SSL_CTX_load_verify_locations (in /usr/lib64/libssl.so.3.2.2)
n1: 615888 0x68D4A3: load_system_certificates (openssl.c:684)
n1: 615888 0x68D7EF: tls_context_create (openssl.c:827)
n2: 615888 0x68EBE3: flb_tls_create (flb_tls.c:200)
n1: 307944 0xC8F717: cb_cloudwatch_init (cloudwatch_logs.c:246)
n1: 307944 0x57858B: flb_output_init_all (flb_output.c:1508)
n1: 307944 0x5ACDDB: flb_engine_start (flb_engine.c:972)
n1: 307944 0x52B563: flb_lib_worker (flb_lib.c:904)
n1: 307944 0x69F1B77: start_thread (in /usr/lib64/libc.so.6)
n0: 307944 0x6A5ECDB: thread_start (in /usr/lib64/libc.so.6)
n1: 307944 0xC8F7FF: cb_cloudwatch_init (cloudwatch_logs.c:261)
n1: 307944 0x57858B: flb_output_init_all (flb_output.c:1508)
n1: 307944 0x5ACDDB: flb_engine_start (flb_engine.c:972)
n1: 307944 0x52B563: flb_lib_worker (flb_lib.c:904)
n1: 307944 0x69F1B77: start_thread (in /usr/lib64/libc.so.6)
n0: 307944 0x6A5ECDB: thread_start (in /usr/lib64/libc.so.6)
n0: 1504 in 1 place, below massif's threshold (1.00%)
n0: 39498 in 2 places, all below massif's threshold (1.00%)
n1: 597998 0x614DEC7: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n2: 597998 0x614EE9F: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n1: 597998 0x61501F7: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n2: 597998 0x614EC33: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n1: 302910 0x614FE8B: ASN1_item_ex_d2i (in /usr/lib64/libcrypto.so.3.2.2)
n1: 302910 0x62DC393: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n2: 302910 0x614EA2F: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n1: 302910 0x61501F7: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n1: 302910 0x614EC33: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n1: 302910 0x61501F7: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n1: 302910 0x614EC33: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n2: 302910 0x614F28F: ASN1_item_d2i_ex (in /usr/lib64/libcrypto.so.3.2.2)
n1: 285360 0x626A41F: PEM_X509_INFO_read_bio_ex (in /usr/lib64/libcrypto.so.3.2.2)
n1: 285360 0x62ADF8B: X509_load_cert_crl_file_ex (in /usr/lib64/libcrypto.so.3.2.2)
n1: 285360 0x62AE127: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n1: 285360 0x62CB33B: X509_STORE_load_file_ex (in /usr/lib64/libcrypto.so.3.2.2)
n2: 285360 0x5FBA20B: SSL_CTX_load_verify_locations (in /usr/lib64/libssl.so.3.2.2)
n1: 284550 0x68D4A3: load_system_certificates (openssl.c:684)
n1: 284550 0x68D7EF: tls_context_create (openssl.c:827)
n1: 284550 0x68EBE3: flb_tls_create (flb_tls.c:200)
n0: 284550 in 2 places, all below massif's threshold (1.00%)
n0: 810 in 1 place, below massif's threshold (1.00%)
n0: 17550 in 2 places, all below massif's threshold (1.00%)
n0: 0 in 1 place, below massif's threshold (1.00%)
n2: 295088 0x614F28F: ASN1_item_d2i_ex (in /usr/lib64/libcrypto.so.3.2.2)
n1: 278448 0x626A41F: PEM_X509_INFO_read_bio_ex (in /usr/lib64/libcrypto.so.3.2.2)
n1: 278448 0x62ADF8B: X509_load_cert_crl_file_ex (in /usr/lib64/libcrypto.so.3.2.2)
n1: 278448 0x62AE127: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n1: 278448 0x62CB33B: X509_STORE_load_file_ex (in /usr/lib64/libcrypto.so.3.2.2)
n2: 278448 0x5FBA20B: SSL_CTX_load_verify_locations (in /usr/lib64/libssl.so.3.2.2)
n1: 277680 0x68D4A3: load_system_certificates (openssl.c:684)
n1: 277680 0x68D7EF: tls_context_create (openssl.c:827)
n1: 277680 0x68EBE3: flb_tls_create (flb_tls.c:200)
n0: 277680 in 2 places, all below massif's threshold (1.00%)
n0: 768 in 1 place, below massif's threshold (1.00%)
n0: 16640 in 3 places, all below massif's threshold (1.00%)
n0: 0 in 1 place, below massif's threshold (1.00%)
n0: 295223 in 89 places, all below massif's threshold (1.00%)
n2: 232600 0x6178247: BUF_MEM_grow (in /usr/lib64/libcrypto.so.3.2.2)
n1: 232600 0x62DB47B: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n2: 232600 0x614EB0F: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n1: 230648 0x61501F7: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n1: 230648 0x614EC33: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n1: 230648 0x61501F7: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n1: 230648 0x614EC33: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n2: 230648 0x614F28F: ASN1_item_d2i_ex (in /usr/lib64/libcrypto.so.3.2.2)
n1: 225840 0x626A41F: PEM_X509_INFO_read_bio_ex (in /usr/lib64/libcrypto.so.3.2.2)
n1: 225840 0x62ADF8B: X509_load_cert_crl_file_ex (in /usr/lib64/libcrypto.so.3.2.2)
n1: 225840 0x62AE127: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n1: 225840 0x62CB33B: X509_STORE_load_file_ex (in /usr/lib64/libcrypto.so.3.2.2)
n2: 225840 0x5FBA20B: SSL_CTX_load_verify_locations (in /usr/lib64/libssl.so.3.2.2)
n1: 225648 0x68D4A3: load_system_certificates (openssl.c:684)
n1: 225648 0x68D7EF: tls_context_create (openssl.c:827)
n1: 225648 0x68EBE3: flb_tls_create (flb_tls.c:200)
n0: 225648 in 2 places, all below massif's threshold (1.00%)
n0: 192 in 1 place, below massif's threshold (1.00%)
n0: 4808 in 2 places, all below massif's threshold (1.00%)
n0: 1952 in 1 place, below massif's threshold (1.00%)
n0: 0 in 5 places, all below massif's threshold (1.00%)
n2: 210670 0x613D84B: ASN1_STRING_set (in /usr/lib64/libcrypto.so.3.2.2)
n1: 195912 0x614E32B: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n0: 195912 in 2 places, all below massif's threshold (1.00%)
n0: 14758 in 3 places, all below massif's threshold (1.00%)
n1: 166706 0x62DB0A7: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n1: 166706 0x62DB58B: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n2: 166706 0x614EB0F: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n1: 165547 0x61501F7: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n1: 165547 0x614EC33: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n1: 165547 0x61501F7: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n1: 165547 0x614EC33: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n2: 165547 0x614F28F: ASN1_item_d2i_ex (in /usr/lib64/libcrypto.so.3.2.2)
n1: 162518 0x626A41F: PEM_X509_INFO_read_bio_ex (in /usr/lib64/libcrypto.so.3.2.2)
n1: 162518 0x62ADF8B: X509_load_cert_crl_file_ex (in /usr/lib64/libcrypto.so.3.2.2)
n1: 162518 0x62AE127: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n1: 162518 0x62CB33B: X509_STORE_load_file_ex (in /usr/lib64/libcrypto.so.3.2.2)
n2: 162518 0x5FBA20B: SSL_CTX_load_verify_locations (in /usr/lib64/libssl.so.3.2.2)
n1: 162396 0x68D4A3: load_system_certificates (openssl.c:684)
n1: 162396 0x68D7EF: tls_context_create (openssl.c:827)
n1: 162396 0x68EBE3: flb_tls_create (flb_tls.c:200)
n0: 162396 in 2 places, all below massif's threshold (1.00%)
n0: 122 in 1 place, below massif's threshold (1.00%)
n0: 3029 in 2 places, all below massif's threshold (1.00%)
n0: 1159 in 1 place, below massif's threshold (1.00%)
n1: 152134 0x6012AD7: ??? (in /usr/lib64/libssl.so.3.2.2)
n1: 152134 0x600A9E7: ??? (in /usr/lib64/libssl.so.3.2.2)
n1: 152134 0x5FF6B6F: ??? (in /usr/lib64/libssl.so.3.2.2)
n1: 152134 0x5FAEE33: ??? (in /usr/lib64/libssl.so.3.2.2)
n1: 152134 0x5FB7DE3: SSL_read (in /usr/lib64/libssl.so.3.2.2)
n2: 152134 0x68E003: tls_net_read (openssl.c:1308)
n1: 152134 0x68EFE3: flb_tls_net_read_async (flb_tls.c:384)
n1: 152134 0x6FCDE7: flb_io_net_read (flb_io.c:747)
n1: 152134 0x704287: flb_http_get_response_data (flb_http_client.c:1583)
n1: 152134 0x704453: flb_http_do (flb_http_client.c:1642)
n1: 152134 0xD63F7B: fetch_pod_service_map (kubernetes_aws.c:257)
n1: 152134 0xD5D5CF: update_pod_service_map (kubernetes.c:66)
n1: 152134 0x69F1B77: start_thread (in /usr/lib64/libc.so.6)
n0: 152134 0x6A5ECDB: thread_start (in /usr/lib64/libc.so.6)
n0: 0 in 1 place, below massif's threshold (1.00%)
n0: 1450566 in 325 places, all below massif's threshold (1.00%)
n1: 1187760 0x133F7DF: sqlite3MemMalloc (sqlite3.c:27572)
n1: 1187760 0x13403F7: mallocWithAlarm (sqlite3.c:31298)
n3: 1187760 0x13404AB: sqlite3Malloc (sqlite3.c:31344)
n1: 698944 0x13543AB: pcache1InitBulk (sqlite3.c:57284)
n1: 698944 0x135478F: pcache1AllocPage (sqlite3.c:57406)
n1: 698944 0x13554A7: pcache1FetchStage2 (sqlite3.c:57892)
n1: 698944 0x135564B: pcache1FetchNoMutex (sqlite3.c:57996)
n1: 698944 0x135567F: pcache1Fetch (sqlite3.c:58038)
n1: 698944 0x1353433: sqlite3PcacheFetch (sqlite3.c:56460)
n1: 698944 0x135B9D3: getPageNormal (sqlite3.c:64470)
n1: 698944 0x135BEC3: sqlite3PagerGet (sqlite3.c:64662)
n1: 698944 0x1366B13: btreeGetPage (sqlite3.c:74774)
n1: 698944 0x1368093: lockBtree (sqlite3.c:75723)
n1: 698944 0x13689CF: btreeBeginTrans (sqlite3.c:76118)
n2: 698944 0x1368D2F: sqlite3BtreeBeginTrans (sqlite3.c:76218)
n1: 698944 0x13D49CB: sqlite3InitOne (sqlite3.c:145907)
n1: 698944 0x13D4E83: sqlite3Init (sqlite3.c:146094)
n1: 698944 0x13D4F7B: sqlite3ReadSchema (sqlite3.c:146120)
n1: 698944 0x13B41A7: sqlite3StartTable (sqlite3.c:125609)
n1: 698944 0x140DBAF: yy_reduce (sqlite3.c:181207)
n1: 698944 0x1412567: sqlite3Parser (sqlite3.c:182932)
n1: 698944 0x1413C43: sqlite3RunParser (sqlite3.c:184274)
n1: 698944 0x13D5713: sqlite3Prepare (sqlite3.c:146431)
n1: 698944 0x13D592F: sqlite3LockAndPrepare (sqlite3.c:146506)
n1: 698944 0x13D5B0F: sqlite3_prepare_v2 (sqlite3.c:146595)
n1: 698944 0x13CD547: sqlite3_exec (sqlite3.c:140132)
n2: 698944 0xE6D7B7: flb_sqldb_query (flb_sqldb.c:158)
n1: 611576 0x8CBA97: flb_tail_db_open (tail_db.c:51)
n1: 611576 0x8ECECB: flb_tail_config_create (tail_config.c:362)
n0: 611576 0x8C1943: in_tail_init (tail.c:374)
n0: 87368 in 1 place, below massif's threshold (1.00%)
n0: 0 in 1 place, below massif's threshold (1.00%)
n1: 384064 0x1415573: setupLookaside (sqlite3.c:185660)
n1: 384064 0x141954B: openDatabase (sqlite3.c:188478)
n1: 384064 0x141961B: sqlite3_open (sqlite3.c:188517)
n2: 384064 0xE6D59F: flb_sqldb_open (flb_sqldb.c:88)
n1: 336056 0x8CBA67: flb_tail_db_open (tail_db.c:45)
n1: 336056 0x8ECECB: flb_tail_config_create (tail_config.c:362)
n1: 336056 0x8C1943: in_tail_init (tail.c:374)
n1: 336056 0x55606F: flb_input_instance_init (flb_input.c:1412)
n1: 336056 0x556237: flb_input_init_all (flb_input.c:1472)
n1: 336056 0x5ACD43: flb_engine_start (flb_engine.c:955)
n1: 336056 0x52B563: flb_lib_worker (flb_lib.c:904)
n1: 336056 0x69F1B77: start_thread (in /usr/lib64/libc.so.6)
n0: 336056 0x6A5ECDB: thread_start (in /usr/lib64/libc.so.6)
n0: 48008 in 1 place, below massif's threshold (1.00%)
n0: 104752 in 9 places, all below massif's threshold (1.00%)
n2: 979424 0x6D59DF: flb_calloc (flb_mem.h:95)
n4: 972096 0x6D9C8B: flb_hash_table_create (flb_hash_table.c:122)
n0: 468096 in 8 places, all below massif's threshold (1.00%)
n1: 168000 0x8ECA37: flb_tail_config_create (tail_config.c:290)
n1: 168000 0x8C1943: in_tail_init (tail.c:374)
n1: 168000 0x55606F: flb_input_instance_init (flb_input.c:1412)
n1: 168000 0x556237: flb_input_init_all (flb_input.c:1472)
n1: 168000 0x5ACD43: flb_engine_start (flb_engine.c:955)
n1: 168000 0x52B563: flb_lib_worker (flb_lib.c:904)
n1: 168000 0x69F1B77: start_thread (in /usr/lib64/libc.so.6)
n0: 168000 0x6A5ECDB: thread_start (in /usr/lib64/libc.so.6)
n1: 168000 0x8ECAEF: flb_tail_config_create (tail_config.c:297)
n1: 168000 0x8C1943: in_tail_init (tail.c:374)
n1: 168000 0x55606F: flb_input_instance_init (flb_input.c:1412)
n1: 168000 0x556237: flb_input_init_all (flb_input.c:1472)
n1: 168000 0x5ACD43: flb_engine_start (flb_engine.c:955)
n1: 168000 0x52B563: flb_lib_worker (flb_lib.c:904)
n1: 168000 0x69F1B77: start_thread (in /usr/lib64/libc.so.6)
n0: 168000 0x6A5ECDB: thread_start (in /usr/lib64/libc.so.6)
n1: 168000 0x8ECBA7: flb_tail_config_create (tail_config.c:305)
n1: 168000 0x8C1943: in_tail_init (tail.c:374)
n1: 168000 0x55606F: flb_input_instance_init (flb_input.c:1412)
n1: 168000 0x556237: flb_input_init_all (flb_input.c:1472)
n1: 168000 0x5ACD43: flb_engine_start (flb_engine.c:955)
n1: 168000 0x52B563: flb_lib_worker (flb_lib.c:904)
n1: 168000 0x69F1B77: start_thread (in /usr/lib64/libc.so.6)
n0: 168000 0x6A5ECDB: thread_start (in /usr/lib64/libc.so.6)
n0: 7328 in 2 places, all below massif's threshold (1.00%)
n2: 783462 0x8D6F97: flb_malloc (flb_mem.h:80)
n1: 688128 0x8DA7CF: flb_tail_file_append (tail_file.c:1323)
n1: 688128 0x8E7287: tail_scan_path (tail_scan_glob.c:277)
n1: 688128 0x8E7643: flb_tail_scan (tail_scan.c:65)
n1: 688128 0x8C19A7: in_tail_init (tail.c:388)
n1: 688128 0x55606F: flb_input_instance_init (flb_input.c:1412)
n1: 688128 0x556237: flb_input_init_all (flb_input.c:1472)
n1: 688128 0x5ACD43: flb_engine_start (flb_engine.c:955)
n1: 688128 0x52B563: flb_lib_worker (flb_lib.c:904)
n1: 688128 0x69F1B77: start_thread (in /usr/lib64/libc.so.6)
n0: 688128 0x6A5ECDB: thread_start (in /usr/lib64/libc.so.6)
n0: 95334 in 3 places, all below massif's threshold (1.00%)
n1: 302870 0x5409AF: flb_malloc (flb_mem.h:80)
n2: 302870 0x5449FF: sds_alloc (flb_sds.c:42)
n2: 262967 0x544B4B: flb_sds_create_size (flb_sds.c:94)
n2: 189198 0x717BB7: stream_group_create (flb_ml_stream.c:67)
n1: 172746 0x717F2F: stream_group_init (flb_ml_stream.c:172)
n1: 172746 0x718047: stream_create (flb_ml_stream.c:213)
n2: 172746 0x718163: flb_ml_stream_create (flb_ml_stream.c:264)
n1: 172746 0x8DA6F3: flb_tail_file_append (tail_file.c:1296)
n1: 172746 0x8E7287: tail_scan_path (tail_scan_glob.c:277)
n1: 172746 0x8E7643: flb_tail_scan (tail_scan.c:65)
n1: 172746 0x8C19A7: in_tail_init (tail.c:388)
n1: 172746 0x55606F: flb_input_instance_init (flb_input.c:1412)
n1: 172746 0x556237: flb_input_init_all (flb_input.c:1472)
n1: 172746 0x5ACD43: flb_engine_start (flb_engine.c:955)
n1: 172746 0x52B563: flb_lib_worker (flb_lib.c:904)
n1: 172746 0x69F1B77: start_thread (in /usr/lib64/libc.so.6)
n0: 172746 0x6A5ECDB: thread_start (in /usr/lib64/libc.so.6)
n0: 0 in 1 place, below massif's threshold (1.00%)
n0: 16452 in 1 place, below massif's threshold (1.00%)
n0: 73769 in 19 places, all below massif's threshold (1.00%)
n0: 39903 in 1 place, below massif's threshold (1.00%)
n1: 204800 0x622767: msgpack_sbuffer_write (sbuffer.h:81)
n2: 204800 0x6225C3: msgpack_pack_str_body (pack_template.h:784)
n1: 163840 0x622D93: flb_log_event_encoder_emit_record (flb_log_event_encoder.c:212)
n1: 163840 0x622EB7: flb_log_event_encoder_commit_record (flb_log_event_encoder.c:267)
n0: 163840 in 5 places, all below massif's threshold (1.00%)
n0: 40960 in 1 place, below massif's threshold (1.00%)
snapshot=79
#-----------
time=246928884076
mem_heap_B=66061891
mem_heap_extra_B=2787013
mem_stacks_B=0
heap_tree=detailed
n6: 66061891 (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
n8: 54934447 0x6245087: CRYPTO_malloc (in /usr/lib64/libcrypto.so.3.2.2)
n6: 21214632 0x6245297: CRYPTO_zalloc (in /usr/lib64/libcrypto.so.3.2.2)
n1: 7233024 0x601E44F: ??? (in /usr/lib64/libssl.so.3.2.2)
n1: 7233024 0x5FFE82F: ??? (in /usr/lib64/libssl.so.3.2.2)
n2: 7233024 0x5FF744B: ??? (in /usr/lib64/libssl.so.3.2.2)
n3: 7233024 0x5FDC417: ??? (in /usr/lib64/libssl.so.3.2.2)
n1: 3616512 0x600FBF7: ??? (in /usr/lib64/libssl.so.3.2.2)
n1: 3616512 0x600A7EF: ??? (in /usr/lib64/libssl.so.3.2.2)
n1: 3616512 0x68E713: tls_net_handshake (openssl.c:1513)
n1: 3616512 0x68F543: flb_tls_session_create (flb_tls.c:630)
n1: 3616512 0x6FB687: flb_io_net_connect (flb_io.c:175)
n1: 3616512 0x5D9847: create_conn (flb_upstream.c:629)
n2: 3616512 0x5D9EAB: flb_upstream_conn_get (flb_upstream.c:833)
n1: 3612080 0xD63DC3: fetch_pod_service_map (kubernetes_aws.c:228)
n1: 3612080 0xD5D5CF: update_pod_service_map (kubernetes.c:66)
n1: 3612080 0x69F1B77: start_thread (in /usr/lib64/libc.so.6)
n0: 3612080 0x6A5ECDB: thread_start (in /usr/lib64/libc.so.6)
n0: 4432 in 2 places, all below massif's threshold (1.00%)
n1: 3616512 0x60196CB: ??? (in /usr/lib64/libssl.so.3.2.2)
n1: 3616512 0x600A9E7: ??? (in /usr/lib64/libssl.so.3.2.2)
n1: 3616512 0x68E713: tls_net_handshake (openssl.c:1513)
n1: 3616512 0x68F543: flb_tls_session_create (flb_tls.c:630)
n1: 3616512 0x6FB687: flb_io_net_connect (flb_io.c:175)
n1: 3616512 0x5D9847: create_conn (flb_upstream.c:629)
n2: 3616512 0x5D9EAB: flb_upstream_conn_get (flb_upstream.c:833)
n1: 3612080 0xD63DC3: fetch_pod_service_map (kubernetes_aws.c:228)
n1: 3612080 0xD5D5CF: update_pod_service_map (kubernetes.c:66)
n1: 3612080 0x69F1B77: start_thread (in /usr/lib64/libc.so.6)
n0: 3612080 0x6A5ECDB: thread_start (in /usr/lib64/libc.so.6)
n0: 4432 in 2 places, all below massif's threshold (1.00%)
n0: 0 in 3 places, all below massif's threshold (1.00%)
n0: 0 in 2 places, all below massif's threshold (1.00%)
n1: 5046144 0x5FBDD7B: ??? (in /usr/lib64/libssl.so.3.2.2)
n1: 5046144 0x68DCD7: tls_session_create (openssl.c:1175)
n1: 5046144 0x68F49B: flb_tls_session_create (flb_tls.c:611)
n1: 5046144 0x6FB687: flb_io_net_connect (flb_io.c:175)
n1: 5046144 0x5D9847: create_conn (flb_upstream.c:629)
n2: 5046144 0x5D9EAB: flb_upstream_conn_get (flb_upstream.c:833)
n1: 5039960 0xD63DC3: fetch_pod_service_map (kubernetes_aws.c:228)
n1: 5039960 0xD5D5CF: update_pod_service_map (kubernetes.c:66)
n1: 5039960 0x69F1B77: start_thread (in /usr/lib64/libc.so.6)
n0: 5039960 0x6A5ECDB: thread_start (in /usr/lib64/libc.so.6)
n0: 6184 in 2 places, all below massif's threshold (1.00%)
n0: 4935616 in 193 places, all below massif's threshold (1.00%)
n1: 1566720 0x62DE8F7: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n1: 1566720 0x621F57B: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n1: 1566720 0x5FF1767: ??? (in /usr/lib64/libssl.so.3.2.2)
n1: 1566720 0x5FFE90B: ??? (in /usr/lib64/libssl.so.3.2.2)
n1: 1566720 0x5FF744B: ??? (in /usr/lib64/libssl.so.3.2.2)
n3: 1566720 0x5FDC417: ??? (in /usr/lib64/libssl.so.3.2.2)
n1: 783360 0x600FBF7: ??? (in /usr/lib64/libssl.so.3.2.2)
n1: 783360 0x600A7EF: ??? (in /usr/lib64/libssl.so.3.2.2)
n1: 783360 0x68E713: tls_net_handshake (openssl.c:1513)
n1: 783360 0x68F543: flb_tls_session_create (flb_tls.c:630)
n1: 783360 0x6FB687: flb_io_net_connect (flb_io.c:175)
n1: 783360 0x5D9847: create_conn (flb_upstream.c:629)
n2: 783360 0x5D9EAB: flb_upstream_conn_get (flb_upstream.c:833)
n1: 782400 0xD63DC3: fetch_pod_service_map (kubernetes_aws.c:228)
n1: 782400 0xD5D5CF: update_pod_service_map (kubernetes.c:66)
n1: 782400 0x69F1B77: start_thread (in /usr/lib64/libc.so.6)
n0: 782400 0x6A5ECDB: thread_start (in /usr/lib64/libc.so.6)
n0: 960 in 2 places, all below massif's threshold (1.00%)
n1: 783360 0x60196CB: ??? (in /usr/lib64/libssl.so.3.2.2)
n1: 783360 0x600A9E7: ??? (in /usr/lib64/libssl.so.3.2.2)
n1: 783360 0x68E713: tls_net_handshake (openssl.c:1513)
n1: 783360 0x68F543: flb_tls_session_create (flb_tls.c:630)
n1: 783360 0x6FB687: flb_io_net_connect (flb_io.c:175)
n1: 783360 0x5D9847: create_conn (flb_upstream.c:629)
n2: 783360 0x5D9EAB: flb_upstream_conn_get (flb_upstream.c:833)
n1: 782400 0xD63DC3: fetch_pod_service_map (kubernetes_aws.c:228)
n1: 782400 0xD5D5CF: update_pod_service_map (kubernetes.c:66)
n1: 782400 0x69F1B77: start_thread (in /usr/lib64/libc.so.6)
n0: 782400 0x6A5ECDB: thread_start (in /usr/lib64/libc.so.6)
n0: 960 in 2 places, all below massif's threshold (1.00%)
n0: 0 in 3 places, all below massif's threshold (1.00%)
n1: 1253392 0x6176CDF: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n0: 1253392 in 19 places, all below massif's threshold (1.00%)
n1: 1179736 0x61573BF: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n0: 1179736 in 3 places, all below massif's threshold (1.00%)
n1: 13636992 0x5FFD7D7: ??? (in /usr/lib64/libssl.so.3.2.2)
n1: 13636992 0x5FFDD13: ??? (in /usr/lib64/libssl.so.3.2.2)
n1: 13636992 0x5FF8B17: ??? (in /usr/lib64/libssl.so.3.2.2)
n2: 13636992 0x5FF66E7: ??? (in /usr/lib64/libssl.so.3.2.2)
n1: 13636992 0x5FAEE33: ??? (in /usr/lib64/libssl.so.3.2.2)
n1: 13636992 0x5FB7DE3: SSL_read (in /usr/lib64/libssl.so.3.2.2)
n2: 13636992 0x68E003: tls_net_read (openssl.c:1308)
n1: 13636992 0x68EFE3: flb_tls_net_read_async (flb_tls.c:384)
n1: 13636992 0x6FCDE7: flb_io_net_read (flb_io.c:747)
n1: 13636992 0x704287: flb_http_get_response_data (flb_http_client.c:1583)
n2: 13636992 0x704453: flb_http_do (flb_http_client.c:1642)
n1: 13620280 0xD63F7B: fetch_pod_service_map (kubernetes_aws.c:257)
n1: 13620280 0xD5D5CF: update_pod_service_map (kubernetes.c:66)
n1: 13620280 0x69F1B77: start_thread (in /usr/lib64/libc.so.6)
n0: 13620280 0x6A5ECDB: thread_start (in /usr/lib64/libc.so.6)
n0: 16712 in 1 place, below massif's threshold (1.00%)
n0: 0 in 1 place, below massif's threshold (1.00%)
n0: 0 in 1 place, below massif's threshold (1.00%)
n1: 13520304 0x601EA67: ??? (in /usr/lib64/libssl.so.3.2.2)
n1: 13520304 0x5FF8597: ??? (in /usr/lib64/libssl.so.3.2.2)
n1: 13520304 0x5FF9C63: ??? (in /usr/lib64/libssl.so.3.2.2)
n1: 13520304 0x5FFA46F: ??? (in /usr/lib64/libssl.so.3.2.2)
n2: 13520304 0x5FF519F: ??? (in /usr/lib64/libssl.so.3.2.2)
n1: 13520304 0x5FB8583: SSL_write (in /usr/lib64/libssl.so.3.2.2)
n2: 13520304 0x68E207: tls_net_write (openssl.c:1370)
n1: 13520304 0x68F22F: flb_tls_net_write_async (flb_tls.c:494)
n1: 13520304 0x6FC8A7: flb_io_net_write (flb_io.c:699)
n1: 13520304 0x703E5F: flb_http_do_request (flb_http_client.c:1472)
n2: 13520304 0x70442F: flb_http_do (flb_http_client.c:1630)
n1: 13503735 0xD63F7B: fetch_pod_service_map (kubernetes_aws.c:257)
n1: 13503735 0xD5D5CF: update_pod_service_map (kubernetes.c:66)
n1: 13503735 0x69F1B77: start_thread (in /usr/lib64/libc.so.6)
n0: 13503735 0x6A5ECDB: thread_start (in /usr/lib64/libc.so.6)
n0: 16569 in 1 place, below massif's threshold (1.00%)
n0: 0 in 1 place, below massif's threshold (1.00%)
n0: 0 in 1 place, below massif's threshold (1.00%)
n1: 2032610 0x6012AD7: ??? (in /usr/lib64/libssl.so.3.2.2)
n1: 2032610 0x600A9E7: ??? (in /usr/lib64/libssl.so.3.2.2)
n1: 2032610 0x5FF6B6F: ??? (in /usr/lib64/libssl.so.3.2.2)
n1: 2032610 0x5FAEE33: ??? (in /usr/lib64/libssl.so.3.2.2)
⠋ n1: 2032610 0x5FB7DE3: SSL_read (in /usr/lib64/libssl.so.3.2.2)
n2: 2032610 0x68E003: tls_net_read (openssl.c:1308)
n1: 2032610 0x68EFE3: flb_tls_net_read_async (flb_tls.c:384)
n1: 2032610 0x6FCDE7: flb_io_net_read (flb_io.c:747)
n1: 2032610 0x704287: flb_http_get_response_data (flb_http_client.c:1583)
Thinking... n1: 2032610 0x704453: flb_http_do (flb_http_client.c:1642)
n1: 2032610 0xD63F7B: fetch_pod_service_map (kubernetes_aws.c:257)
n1: 2032610 0xD5D5CF: update_pod_service_map (kubernetes.c:66)
n1: 2032610 0x69F1B77: start_thread (in /usr/lib64/libc.so.6)
n0: 2032610 0x6A5ECDB: thread_start (in /usr/lib64/libc.so.6)
n0: 0 in 1 place, below massif's threshold (1.00%)
n0: 1681761 in 91 places, all below massif's threshold (1.00%)
n1: 1097226 0x614F0D3: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n1: 1097226 0x61501F7: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n1: 1097226 0x614EC33: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n1: 1097226 0x614F28F: ASN1_item_d2i_ex (in /usr/lib64/libcrypto.so.3.2.2)
n0: 1097226 in 3 places, all below massif's threshold (1.00%)
n1: 994602 0x614DEC7: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n2: 994602 0x614EE9F: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n1: 994602 0x61501F7: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n1: 994602 0x614EC33: ??? (in /usr/lib64/libcrypto.so.3.2.2)
n0: 994602 in 2 places, all below massif's threshold (1.00%)
n0: 0 in 1 place, below massif's threshold (1.00%)
n1: 756320 0x5FC5C37: ??? (in /usr/lib64/libssl.so.3.2.2)
n1: 756320 0x6012A33: ??? (in /usr/lib64/libssl.so.3.2.2)
n1: 756320 0x600A9E7: ??? (in /usr/lib64/libssl.so.3.2.2)
n1: 756320 0x5FF6B6F: ??? (in /usr/lib64/libssl.so.3.2.2)
n1: 756320 0x5FAEE33: ??? (in /usr/lib64/libssl.so.3.2.2)
n1: 756320 0x5FB7DE3: SSL_read (in /usr/lib64/libssl.so.3.2.2)
n2: 756320 0x68E003: tls_net_read (openssl.c:1308)
n1: 756320 0x68EFE3: flb_tls_net_read_async (flb_tls.c:384)
n1: 756320 0x6FCDE7: flb_io_net_read (flb_io.c:747)
n1: 756320 0x704287: flb_http_get_response_data (flb_http_client.c:1583)
n1: 756320 0x704453: flb_http_do (flb_http_client.c:1642)
n1: 756320 0xD63F7B: fetch_pod_service_map (kubernetes_aws.c:257)
n1: 756320 0xD5D5CF: update_pod_service_map (kubernetes.c:66)
n1: 756320 0x69F1B77: start_thread (in /usr/lib64/libc.so.6)
n0: 756320 0x6A5ECDB: thread_start (in /usr/lib64/libc.so.6)
n0: 0 in 1 place, below massif's threshold (1.00%)
n3: 4674304 0xC8E35B: flb_malloc (flb_mem.h:80)
n1: 2097152 0xC900FB: new_buffer (cloudwatch_logs.c:412)
n1: 2097152 0xC90243: cb_cloudwatch_flush (cloudwatch_logs.c:452)
n1: 2097152 0x57D84B: output_pre_cb_flush (flb_output.h:706)
n3: 2097152 0x189FE17: co_switch (aarch64.c:133)
n0: 1048576 0xCA: ???
n0: 1048576 0x65747962AB656D60: ???
n0: 0 in 1354 places, all below massif's threshold (1.00%)
n1: 2097152 0xC90157: new_buffer (cloudwatch_logs.c:420)
n1: 2097152 0xC90243: cb_cloudwatch_flush (cloudwatch_logs.c:452)
n1: 2097152 0x57D84B: output_pre_cb_flush (flb_output.h:706)
n3: 2097152 0x189FE17: co_switch (aarch64.c:133)
n0: 1048576 0xCA: ???
n0: 1048576 0x65747962AB656D60: ???
n0: 0 in 1354 places, all below massif's threshold (1.00%)
n0: 480000 in 1 place, below massif's threshold (1.00%)
n0: 3388863 in 327 places, all below massif's threshold (1.00%)
n1: 1187760 0x133F7DF: sqlite3MemMalloc (sqlite3.c:27572)
n1: 1187760 0x13403F7: mallocWithAlarm (sqlite3.c:31298)
n2: 1187760 0x13404AB: sqlite3Malloc (sqlite3.c:31344)
n1: 698944 0x13543AB: pcache1InitBulk (sqlite3.c:57284)
n1: 698944 0x135478F: pcache1AllocPage (sqlite3.c:57406)
n1: 698944 0x13554A7: pcache1FetchStage2 (sqlite3.c:57892)
n1: 698944 0x135564B: pcache1FetchNoMutex (sqlite3.c:57996)
n1: 698944 0x135567F: pcache1Fetch (sqlite3.c:58038)
n1: 698944 0x1353433: sqlite3PcacheFetch (sqlite3.c:56460)
n1: 698944 0x135B9D3: getPageNormal (sqlite3.c:64470)
n1: 698944 0x135BEC3: sqlite3PagerGet (sqlite3.c:64662)
n1: 698944 0x1366B13: btreeGetPage (sqlite3.c:74774)
n1: 698944 0x1368093: lockBtree (sqlite3.c:75723)
n1: 698944 0x13689CF: btreeBeginTrans (sqlite3.c:76118)
n2: 698944 0x1368D2F: sqlite3BtreeBeginTrans (sqlite3.c:76218)
n1: 698944 0x13D49CB: sqlite3InitOne (sqlite3.c:145907)
n1: 698944 0x13D4E83: sqlite3Init (sqlite3.c:146094)
n1: 698944 0x13D4F7B: sqlite3ReadSchema (sqlite3.c:146120)
n1: 698944 0x13B41A7: sqlite3StartTable (sqlite3.c:125609)
n1: 698944 0x140DBAF: yy_reduce (sqlite3.c:181207)
n1: 698944 0x1412567: sqlite3Parser (sqlite3.c:182932)
n1: 698944 0x1413C43: sqlite3RunParser (sqlite3.c:184274)
n1: 698944 0x13D5713: sqlite3Prepare (sqlite3.c:146431)
n1: 698944 0x13D592F: sqlite3LockAndPrepare (sqlite3.c:146506)
n1: 698944 0x13D5B0F: sqlite3_prepare_v2 (sqlite3.c:146595)
n1: 698944 0x13CD547: sqlite3_exec (sqlite3.c:140132)
n1: 698944 0xE6D7B7: flb_sqldb_query (flb_sqldb.c:158)
n0: 698944 in 2 places, all below massif's threshold (1.00%)
n0: 0 in 1 place, below massif's threshold (1.00%)
n0: 488816 in 10 places, all below massif's threshold (1.00%)
n2: 981056 0x6D59DF: flb_calloc (flb_mem.h:95)
n1: 972096 0x6D9C8B: flb_hash_table_create (flb_hash_table.c:122)
n0: 972096 in 11 places, all below massif's threshold (1.00%)
n0: 8960 in 2 places, all below massif's threshold (1.00%)
n2: 895461 0x8D6F97: flb_malloc (flb_mem.h:80)
n1: 786432 0x8DA7CF: flb_tail_file_append (tail_file.c:1323)
n1: 786432 0x8E7287: tail_scan_path (tail_scan_glob.c:277)
n1: 786432 0x8E7643: flb_tail_scan (tail_scan.c:65)
n0: 786432 in 2 places, all below massif's threshold (1.00%)
n0: 109029 in 3 places, all below massif's threshold (1.00%)
#-----------
Logs
Debug logs from a fluent-bit pod show the background thread fetching the pod-to-service map every 60 seconds:
[debug] [filter:kubernetes:kubernetes.2] fetch pod to service map
[debug] [filter:kubernetes:kubernetes.2] Request (uri = /kubernetes/pod-to-service-env-map) http_do=0, HTTP Status: 200
[debug] [filter:kubernetes:kubernetes.2] HTTP response payload : {}
[debug] [filter:kubernetes:kubernetes.2] started parsing pod to service map
[debug] [filter:kubernetes:kubernetes.2] ended parsing pod to service map
[debug] [filter:kubernetes:kubernetes.2] Updating pod to service map after 60 seconds
This cycle repeats every 60 seconds. Memory grows steadily with each cycle and never stabilizes.
Note: Not all debug-level logs from the update_pod_service_map background thread are visible. flb_log_check() in flb_log.h suppresses debug-level output for threads without a worker context - this includes connection and TLS-related logs.
Your Environment
- Version used:
602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/observability/aws-for-fluent-bit:3.2.0 - Configuration: kubernetes filter with
Use_Pod_Association On, defaultaws_pod_service_map_refresh_interval 60 - Environment name and version: EKS Auto Mode, addon
amazon-cloudwatch-observabilityv4.10.0-eksbuild.1 - Server type and version: ARM64 (Graviton)
- Operating System and version: Bottlerocket
- Filters and plugins: kubernetes filter (pod association feature)
[FILTER]
Name kubernetes
Match application.*
Kube_URL https://kubernetes.default.svc:443
Kube_Tag_Prefix application.var.log.containers.
Merge_Log On
Merge_Log_Key log_processed
K8S-Logging.Parser On
K8S-Logging.Exclude Off
Labels Off
Annotations Off
Use_Kubelet On
Kubelet_Port 10250
Buffer_Size 0
Use_Pod_Association On