in_tail: fix memory leak when using generic unicode conversion (backport #10781)

[pwhelan]

Summary

Backport of #10781.

---

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

• [N/A] Example configuration file for the change
• [N/A] Debug log output from testing the change

• [N/A] Attached Valgrind output that shows no leaks or memory corruption was found

Backporting

• Backport to latest stable release.

---

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

Summary by CodeRabbit

• New Features

  • Added Prometheus textfile input.
  • Expanded Node Exporter metrics (sockstat, hwmon, rootfs path) and NGINX scrape_interval.
  • Introduced Windows Exporter cache and TCP metrics.
  • Added Elasticsearch API key auth.
  • Expanded Azure Kusto auth (service principal, managed identity, workload identity).
  • Added CloudWatch Logs entity support.
  • Windows TLS CertStore support for outputs.

• Improvements

  • Output latency histogram; skip emitting empty metrics.
  • Firehose/Kinesis configurable ports; Windows builds updated (ltsc2025).
  • Version bump to 4.0.9.

• Bug Fixes

  • OTEL logs null/type handling; JSON numeric/overflow; histogram bucket and allocation; Kafka commit logic; syslog/tail memory; Windows timezone.

• Tests

  • New runtime and internal tests for Prometheus textfile, Chronicle, OTEL, and JSON packing.

[coderabbitai]

Caution

Review failed

Failed to post review comments.

Walkthrough

Adds new input/output features, Windows TLS support, proxy env ignore, metrics latency histogram, Prometheus textfile input, Node/Windows exporter collectors, Azure Kusto auth modes, CloudWatch entity handling, assorted bug fixes, tests, packaging/version bumps, and CI workflow tweaks.

Changes

Cohort / File(s)	Summary
CI workflows .github/workflows/staging-build.yaml, .github/workflows/staging-release.yaml	Add disk space cleanup step; ensure image is pulled before retag/push.
Version bumps CMakeLists.txt, lib/cmetrics/CMakeLists.txt, dockerfiles/Dockerfile, fluent-bit-4.0.9.bb, snap/snapcraft.yaml	Bump core/CMetrics/package/snap versions to 4.0.9 and 1.0.5.
CMake/options cmake/msgpack.cmake, cmake/plugins_options.cmake, plugins/CMakeLists.txt	Define MSGPACK_EMBED_STACK_SIZE=64; add FLB_IN_PROMETHEUS_TEXTFILE option and register plugin.
Prometheus textfile input plugins/in_prometheus_textfile/*	New input plugin reading Prometheus text files with globbing; config map for path and scrape interval; runtime tests/data added.
Node Exporter metrics plugins/in_node_exporter_metrics/*	Add sockstat and hwmon collectors, path.rootfs support, config entries, new gauges, regex filters, Linux/non-Linux shims.
Windows Exporter metrics plugins/in_windows_exporter_metrics/*	Add cache and TCP collectors, new counters/specs, logical disk API-based size/free gauges, perflib enhancements, timezone update, build adds iphlpapi.
Windows TLS support include/fluent-bit/tls/flb_tls.h, src/tls/flb_tls.c, src/tls/openssl.c, include/fluent-bit/flb_output.h, src/flb_output.c	Add Windows CertStore name/enterprise store config, backend hooks, setters, and output instance fields; load system certs accordingly.
Proxy env ignore include/fluent-bit/flb_network.h, src/flb_network.c, src/flb_upstream.c, src/flb_output.c	Add proxy_env_ignore to net setup; config map; upstream/proxy handling respects ignore flag.
Input chunk timing & output latency include/fluent-bit/flb_input_chunk.h, src/flb_input_chunk.c, src/flb_engine.c, include/fluent-bit/flb_output.h, src/flb_output.c	Track input chunk create time; observe per-output latency histogram; add cmt_latency handle.
Metrics encode/emptiness include/fluent-bit/flb_metrics.h, src/flb_metrics.c, src/flb_input_metric.c, include/fluent-bit/flb_macros.h	Add flb_metrics_is_empty; unify encode path and skip empty; switch FLB_TRUE/FLB_FALSE to stdbool.
OpenTelemetry include/fluent-bit/flb_opentelemetry.h, src/opentelemetry/flb_opentelemetry_logs.c, src/opentelemetry/flb_opentelemetry_utils.c, plugins/in_opentelemetry/opentelemetry_logs.c, tests/internal/*	Replace error-code macros with enum and add new code; gate trace/span IDs by exact sizes; stricter kvlist conversions; nil handling; add tests and updated test data.
Kafka/Syslog/Tail plugins/in_kafka/in_kafka.c, plugins/in_syslog/syslog_prot.c, plugins/in_tail/tail_file.c	Commit Kafka offsets after successful append; defer syslog emission and reset encoder per-batch; always free decoded buffer.
Calyptia Fleet timestamp logic plugins/in_calyptia_fleet/in_calyptia_fleet.c	Use numeric timestamps; glob existing files; create only if newer; update helpers.
CloudWatch Logs entity plugins/out_cloudwatch_logs/*	Add entity construction/filtering, macros, new config add_entity, track per-stream entity, Kubernetes metadata flag.
Elasticsearch API key plugins/out_es/es.c, plugins/out_es/es.h	Add http_api_key config/field; send Authorization: ApiKey header.
Azure Kusto auth & MSI plugins/out_azure_kusto/*	Add auth type enum and config; support service principal, managed identity, workload identity; workload token exchange; resource loading with mutex; random refresh jitter.
Kinesis ports plugins/out_kinesis_firehose/*, plugins/out_kinesis_streams/*	Add per-instance port (uint16_t), default 443 macro, range handling; use configured port in client/upstream.
ChunkIO Windows lib/chunkio/src/*	Replace Shlwapi usage; implement recursive path creation; add path_is_directory; drop Shlwapi.lib link.
CMetrics fixes lib/cmetrics/src/*	Fix histogram bucket loop and allocation off-by-one.
Packaging/docker dockerfiles/Dockerfile.windows, packaging/distros/debian/Dockerfile, dockerfiles/Dockerfile, docker_compose/node-exporter-dashboard/docker-compose.yml, conf/fluent-bit-metrics.conf	Update Windows base/toolchain and cleanup; patch Debian sources to archive; version arg bump; switch to path.rootfs mount and document rootfs in conf.
Input collector lifecycle src/flb_input.c	Remove extra timer socket close; adjust pause/destroy for timers.
AWS EC2 creds upstream src/aws/flb_aws_credentials_ec2.c	Restore upstream net settings after flb_output_upstream_set; reset timeouts/keepalive.
JSON packer src/flb_pack.c, tests/internal/pack.c	Accurate token count; robust numeric packing (int/uint/float); add stress and correctness tests.
Nginx exporter plugins/in_nginx_exporter_metrics/nginx.*	Add configurable scrape_interval; use in collector scheduling.

Sequence Diagram(s)

sequenceDiagram
  autonumber
  participant User
  participant FB as Fluent Bit
  participant PT as Prometheus Textfile In
  participant FS as Filesystem
  participant CMT as CMetrics

  User->>FB: Configure in_prometheus_textfile (paths, scrape_interval)
  FB->>PT: Init plugin (bind config, register collector)
  loop Every scrape_interval
    PT->>FS: Glob/Read matching files
    FS-->>PT: File contents
    PT->>CMT: cmt_decode_prometheus_create(...)
    alt Decode OK
      PT->>FB: flb_input_metrics_append(cmt)
    else Decode Error/Empty
      PT-->>FB: Skip/log
    end
  end

Loading

sequenceDiagram
  autonumber
  participant Cfg as Config
  participant K as Azure Kusto Out
  participant Auth as Auth Selector
  participant MSI as Managed Identity
  participant WI as Workload Identity
  participant SP as Service Principal

  Cfg->>K: auth_type, tenant_id, client_id, secret/token_file
  K->>Auth: get_azure_kusto_token()
  alt WORKLOAD_IDENTITY
    Auth->>WI: flb_azure_workload_identity_token_get(...)
    WI-->>Auth: access_token or error
  else MANAGED_IDENTITY
    Auth->>MSI: flb_azure_msiauth_token_get()
    MSI-->>Auth: access_token or error
  else SERVICE_PRINCIPAL
    Auth->>SP: OAuth2 client_credentials
    SP-->>Auth: access_token or error
  end
  Auth-->>K: Token or failure

Loading

sequenceDiagram
  autonumber
  participant IN as Input
  participant ENG as Engine
  participant OUT as Output
  participant MET as CMetrics

  IN->>ENG: Create chunk (record create_time)
  ENG->>OUT: Dispatch task
  OUT-->>ENG: Result (FLB_OK/ERR)
  alt FLB_OK and OUT has cmt_latency
    ENG->>MET: cmt_histogram_observe(latency_seconds, labels=input,chunk)
  end

Loading

sequenceDiagram
  autonumber
  participant Rec as Record
  participant CW as CloudWatch Out
  participant Ent as Entity Builder

  Rec-->>CW: msgpack map
  alt add_entity enabled
    CW->>Ent: parse_entity/update_or_create_entity(...)
    Ent-->>CW: entity state
    CW->>CW: remove aws_entity* keys from maps
    CW-->>AWS: PutLogEvents payload with entity block
  else
    CW-->>AWS: PutLogEvents payload (no entity)
  end

Loading

Estimated code review effort

🎯 5 (Critical) | ⏱️ ~120 minutes

Possibly related PRs

• in_node_exporter_metrics: add Linux hwmon collector #10719 — Adds hwmon collector to in_node_exporter_metrics, matching the new hwmon sources/headers and config wiring here.
• out_cloudwatch: add entity support and remove unnecessary log content #10585 — Introduces CloudWatch Logs entity support with similar helpers/macros and config additions.
• out_kinesis_firehose: introduce port parameter #10660 — Implements per-instance port handling in out_kinesis_firehose consistent with this port/config update.

Suggested labels

backport to v4.0.x

Suggested reviewers

• edsiper
• koleini
• fujimotos

Poem

A rabbit taps the build with gentle paws,
New metrics bloom, and Windows tweaks applause.
Textfiles sing; sockstat hums along—
Kusto chooses keys so strong.
Entities dance in cloudlit logs,
While histograms time the hops of frogs. 🐇⏱️📈

Tip

🔌 Remote MCP (Model Context Protocol) integration is now available!

Pro plan users can now connect to remote MCP servers from the Integrations page. Connect with popular remote MCPs such as Notion and Linear to add more context to your reviews and chats.

✨ Finishing Touches

• 📝 Generate Docstrings

🧪 Generate unit tests

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch pwhelan-backport-10781

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

• Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  • I pushed a fix in commit <commit_id>, please review it.
  • Open a follow-up GitHub issue for this discussion.
• Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query.
• PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
  • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

CodeRabbit Commands (Invoked using PR/Issue comments)

Type @coderabbitai help to get the list of available commands.

Other keywords and placeholders

• Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
• Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
• Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

• You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
• Please see the configuration documentation for more information.
• If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Status, Documentation and Community

• Visit our Status Page to check the current availability of CodeRabbit.
• Visit our Documentation for detailed information on how to use CodeRabbit.
• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.