Fluent-bit Splitting Large JSON Log into Partial Messages #932
Description
Describe the question/issue
All JSON-formatted logs are being parsed correctly, except for one particular log from an application that generates an unusually large log entry. The size details of this log are:
Character Count: ~415,000
Byte Size (UTF-8): ~415 KB
Line Count (approx.): 41
File Size on Disk: ~415 KB
Fluent Bit appears to split this single log into multiple partial_message chunks and stores them under the log field in
"partial_ordinal": "1",
"partial_ordinal": "2",
"partial_ordinal": "3",
and so on...
Question:
How does Fluent Bit handle such large log entries?
What are the best practices or possible solutions to reliably process and forward long logs without splitting into partial_message
Configuration
Fluentbit running as a sidecar container in each Fargate task:
"logConfiguration": {
"logDriver": "awsfirelens",
"options": {
"AWS_Region": "us-west-1",
"Logstash_Format": "On",
"Suppress_Type_Name": "On",
"Port": "443",
"Logstash_Prefix": "index-name-dummy",
"tls": "On",
"Name": "es"
},
Fluent Bit Log Output
{
"_index": "test-name-cloud-2025.03.22",
"_id": "dfd343eefdfer4343",
"_version": 1,
"_score": null,
"_source": {
"@timestamp": "2025-05-27T12:45:26.052Z",
"partial_last": "false",
"container_id": "dfdfdfdfdfdfdfdfdfddd-3825463810",
"container_name": "dummy-name",
"source": "stdout",
"log": "{\"host\":\"ip-172-31-1-dfdfddddl\",\"short_message\":\"Dummy message outgoing message:\\n{"full_message\": "Very log message like this: aid\\\":\\\"oAAAAVcAMA==\\\",\\\aid\\\":\\\"oAAAAVcAMA==\\\",\\\aid\\\":\\\"oAAAAVcAMA==\\\",\\\aid\\\":\\\"oAAAAVcAMA==\\\",\\\",
"partial_message": "true",
"partial_id": "4343klj4l3jl43l43lk43lk43lkl3k4l3kl3k4l4l",
"partial_ordinal": "1",
"ecs_cluster": "test-name-cloud",
"ecs_task_arn": "arn:aws:ecs:eu-west-1:343434334343:task/test-name-cloud/efererefd3434jkdjkfdjkf",
"ecs_task_definition": "test-name-cloud:42"
},
Fluent Bit Version Info
Fluentbit: 1.9
public.ecr.aws/aws-observability/aws-for-fluent-bit:2.32.5
Which AWS for Fluent Bit Versions have you tried?*
AWS managed lastest version use version 1.9...
Cluster Details
AWS Opensearch Service
Micros running in AWS ECS Fargate
Fluentbit running as a sidecar container with awsfirelense
Input plugin: farward
Ingesting logs to Opensearch.
Using AWS Managed Fluent-bit image: public.ecr.aws/aws-observability/aws-for-fluent-bit:2.32.5