Can we specify a custom fluentd config file in the IAC, rather than creating an image or hosting the config file in S3? #1022
Description
Describe the question/issue
Is there a way to specify a custom fluentd config file in the IAC, rather than creating an image or hosting the config file in S3?
We'd like to reduce our infra as much as possible.
Configuration
I'm using terraform. I'm trying to define the whole config file as a terraform variable above and inject it into the container definition: Here's part of the container resource block:
container_definitions = jsonencode([
{
"name" : "backend-container",
"image" : "${data.aws_ecr_repository.backend_repository.repository_url}:${var.backend_image_tag}",
"cpu" : "${var.backend_cpu}",
"memory" : "${var.backend_memory}",
"essential" : true,
"portMappings" : [
{
"containerPort" : 8000,
"hostPort" : 8000,
"name" : "backend-port"
}
],
# Send all stdout/stderr through FireLens so it can route normal vs audit logs to different CloudWatch log groups
"logConfiguration" : {
"logDriver" : "awsfirelens"
},
},
# FireLens sidecar for routing logs to CloudWatch audit log group
# https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_FirelensConfiguration.html
{
"name" : "log-router",
"image" : "amazon/aws-for-fluent-bit:stable",
"essential" : true,
"firelensConfiguration" : {
"type" : "fluentbit",
"options" : {
"config-file-type" : "file",
"config-file-value" : "/fluent-bit/etc/extra.conf"
}
},
"entryPoint" : ["/bin/sh", "-c"],
"command" : [
"cat > /fluent-bit/etc/extra.conf << 'CONF_EOF'\n${local.fluentbit_config_content}\nCONF_EOF\necho '=== Generated Fluent Bit Config ==='\ncat /fluent-bit/etc/extra.conf\necho '=== Starting Fluent Bit ==='\nexec /fluent-bit/bin/fluent-bit -c /fluent-bit/etc/extra.conf"
],
# Send the log router's own logs to the standard backend log group for troubleshooting
"logConfiguration" : {
"logDriver" : "awslogs",
"options" : {
"awslogs-group" : "${aws_cloudwatch_log_group.backend_logs.name}",
"awslogs-region" : "${var.region}",
"awslogs-stream-prefix" : "firelens"
}
}
}
])
Here's the variable with the config file content:
locals {
fluentbit_config_content = <<-EOT
[SERVICE]
Flush 1
Log_Level info
[INPUT]
Name forward
Listen 0.0.0.0
Port 24224
# Tag audit lines (look for substring 'audit: True/true') by retagging
[FILTER]
Name rewrite_tag
Match *-firelens-*
Rule .*audit: [Tt]rue.* audit.$TAG true
Emitter_Name audit_rewrite
# Output all logs to backend standard log group
[OUTPUT]
Name cloudwatch_logs
Match *
region ${var.region}
log_group_name ${aws_cloudwatch_log_group.backend_logs.name}
log_stream_prefix backend
auto_create_group false
# Output only audit-tagged logs to audit log group (shared services account)
[OUTPUT]
Name cloudwatch_logs
Match audit.*
region us-east-1
log_group_name ${aws_cloudwatch_log_group.app_audit_logs.name}
log_stream_prefix backend-audit
auto_create_group false
EOT
}
Fluent Bit Log Output
Here's the output, showing a sigterm:
October 15, 2025, 09:47
[2025/10/15 16:47:27] [ info] [output:cloudwatch_logs:cloudwatch_logs.1] thread worker #0 stopping...
888e326acb004294b3f6a3550fa71145
log-router
October 15, 2025, 09:47
[2025/10/15 16:47:27] [ info] [output:cloudwatch_logs:cloudwatch_logs.1] thread worker #0 stopped
888e326acb004294b3f6a3550fa71145
log-router
October 15, 2025, 09:47
[2025/10/15 16:47:27] [ info] [engine] service has stopped (0 pending tasks)
888e326acb004294b3f6a3550fa71145
log-router
October 15, 2025, 09:47
[2025/10/15 16:47:27] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] thread worker #0 stopping...
888e326acb004294b3f6a3550fa71145
log-router
October 15, 2025, 09:47
[2025/10/15 16:47:27] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] thread worker #0 stopped
888e326acb004294b3f6a3550fa71145
log-router
October 15, 2025, 09:47
[2025/10/15 16:47:26] [ warn] [engine] service will shutdown in max 5 seconds
888e326acb004294b3f6a3550fa71145
log-router
October 15, 2025, 09:47
[2025/10/15 16:47:26] [engine] caught signal (SIGTERM)
888e326acb004294b3f6a3550fa71145
log-router
October 15, 2025, 09:47
[2025/10/15 16:47:13] [ info] [output:cloudwatch_logs:cloudwatch_logs.1] worker #0 started
888e326acb004294b3f6a3550fa71145
log-router
October 15, 2025, 09:47
[2025/10/15 16:47:13] [ info] [sp] stream processor started
888e326acb004294b3f6a3550fa71145
log-router
October 15, 2025, 09:47
[2025/10/15 16:47:13] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] worker #0 started
888e326acb004294b3f6a3550fa71145
log-router
October 15, 2025, 09:47
[2025/10/15 16:47:13] [ warn] [filter:rewrite_tag:rewrite_tag.0] 'Match' may cause infinite loop.
888e326acb004294b3f6a3550fa71145
log-router
October 15, 2025, 09:47
[2025/10/15 16:47:13] [ info] [storage] version=1.4.0, type=memory-only, sync=normal, checksum=disabled, max_chunks_up=128
888e326acb004294b3f6a3550fa71145
log-router
October 15, 2025, 09:47
[2025/10/15 16:47:13] [ info] [cmetrics] version=0.3.7
888e326acb004294b3f6a3550fa71145
log-router
October 15, 2025, 09:47
[2025/10/15 16:47:13] [ info] [fluent bit] version=1.9.10, commit=e4a311c35f, pid=1
888e326acb004294b3f6a3550fa71145
log-router
October 15, 2025, 09:47
Fluent Bit v1.9.10
888e326acb004294b3f6a3550fa71145
log-router
October 15, 2025, 09:47
* Copyright (C) 2015-2022 The Fluent Bit Authors
888e326acb004294b3f6a3550fa71145
log-router
October 15, 2025, 09:47
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
888e326acb004294b3f6a3550fa71145
log-router
October 15, 2025, 09:47
* https://fluentbit.io
888e326acb004294b3f6a3550fa71145
log-router
October 15, 2025, 09:47
[SERVICE]
888e326acb004294b3f6a3550fa71145
log-router
October 15, 2025, 09:47
Flush 1
888e326acb004294b3f6a3550fa71145
log-router
October 15, 2025, 09:47
Log_Level info
888e326acb004294b3f6a3550fa71145
log-router
October 15, 2025, 09:47
# Tag audit lines (look for substring 'audit: True/true') by retagging
888e326acb004294b3f6a3550fa71145
log-router
October 15, 2025, 09:47
[FILTER]
888e326acb004294b3f6a3550fa71145
log-router
October 15, 2025, 09:47
Name rewrite_tag
888e326acb004294b3f6a3550fa71145
log-router
October 15, 2025, 09:47
Match *
888e326acb004294b3f6a3550fa71145
log-router
October 15, 2025, 09:47
Rule .*audit: [Tt]rue.* audit.$TAG true
888e326acb004294b3f6a3550fa71145
log-router
October 15, 2025, 09:47
Emitter_Name audit_rewrite
888e326acb004294b3f6a3550fa71145
log-router
October 15, 2025, 09:47
# Output all logs to backend standard log group
888e326acb004294b3f6a3550fa71145
log-router
October 15, 2025, 09:47
[OUTPUT]
888e326acb004294b3f6a3550fa71145
log-router
October 15, 2025, 09:47
Name cloudwatch_logs
888e326acb004294b3f6a3550fa71145
log-router
October 15, 2025, 09:47
Match *
888e326acb004294b3f6a3550fa71145
log-router
October 15, 2025, 09:47
region us-east-1
888e326acb004294b3f6a3550fa71145
log-router
October 15, 2025, 09:47
log_group_name backend-logs
888e326acb004294b3f6a3550fa71145
log-router
October 15, 2025, 09:47
log_stream_prefix backend
888e326acb004294b3f6a3550fa71145
log-router
October 15, 2025, 09:47
auto_create_group false
888e326acb004294b3f6a3550fa71145
log-router
October 15, 2025, 09:47
# Output only audit-tagged logs to audit log group (shared services account)
888e326acb004294b3f6a3550fa71145
log-router
October 15, 2025, 09:47
[OUTPUT]
888e326acb004294b3f6a3550fa71145
log-router
October 15, 2025, 09:47
Name cloudwatch_logs
888e326acb004294b3f6a3550fa71145
log-router
October 15, 2025, 09:47
Match audit.*
888e326acb004294b3f6a3550fa71145
log-router
October 15, 2025, 09:47
region us-east-1
888e326acb004294b3f6a3550fa71145
log-router
October 15, 2025, 09:47
log_group_name app-audit-logs-sandbox
888e326acb004294b3f6a3550fa71145
log-router
October 15, 2025, 09:47
log_stream_prefix backend-audit
888e326acb004294b3f6a3550fa71145
log-router
October 15, 2025, 09:47
auto_create_group false
888e326acb004294b3f6a3550fa71145
log-router
October 15, 2025, 09:45
[2025/10/15 16:45:53] [ info] [output:cloudwatch_logs:cloudwatch_logs.1] thread worker #0 stopping...
Fluent Bit Version Info
"image" : "amazon/aws-for-fluent-bit:stable" October 15th 2025
Cluster Details
- ECS with Fargate
- Sidecar deployment for Fluent Bit
-->
Steps to reproduce issue
Applying the terraform config causes the container to be terminated upon start.