Migrated base image from Amazon Linux 2 to Amazon Linux 2023 #913

jenademoodley · 2025-03-27T15:46:27Z

Summary

At present, aws-for-fluent-bit images are unable to access logs from systemd on the host when systemd version is greater than 246. Further discussed on fluent-bit systemd input plugin cannot read zstd compressed/hash collision hardened journal files in systemd >= 246 fluent/fluent-bit#2998
This especially effects container instances running AL2023 which does not have rsyslog installed, so all logs (containerd, docker for ECS) must be obtained from systemd journal:
https://docs.aws.amazon.com/linux/al2023/ug/journald.html
Fix is to update the systemd libs (systemd-devel) to an updated version, however base Amazon Linux 2 does not have the version required
This PR moves to AL2023 as the base, which has the updated libs

Issue #, if available:
#831

Testing

make debug succeeded: yes
Integ tests succeeded: yes
New tests cover the changes:

Description for the changelog

Migrated base image from Amazon Linux 2 to Amazon Linux 2023

Licensing

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

sparrc · 2025-03-27T16:29:34Z

scripts/dockerfiles/Dockerfile.build

@@ -11,7 +11,23 @@ RUN mkdir -p /fluent-bit/bin /fluent-bit/etc /fluent-bit/log /tmp/fluent-bit-mas
 RUN curl -sL -o /bin/gimme https://raw.githubusercontent.com/travis-ci/gimme/master/gimme
 RUN chmod +x /bin/gimme
 RUN yum upgrade -y
-RUN amazon-linux-extras install -y epel && yum install -y libASL --skip-broken


We may want to consider just removing libASL from our build container if we move to AL2023, rather than adding in a dependency on a fedora-maintained package repo

Fluent-bit also only requires libsasl:
fluent/fluent-bit#1490 (comment)

This is included already in cyrus-sasl-devel so entire libASl package installation is unneccessary

sparrc · 2025-03-27T16:30:20Z

@jenademoodley Do you have steps to reproduce this issue in the context of ECS?

jenademoodley · 2025-03-27T16:52:43Z

On ECS, you can use an init container to bootstrap a config to a shared volume (or bind mount it from the host or EFS volume)

Alternatively, use thr new image as a base and bake the config in.

Example config:

[SERVICE]
    Flush                     5
    Grace                     30
    Log_Level                 error
    Daemon                    off
    storage.path              /var/fluent-bit/state/flb-storage/
    storage.sync              normal
    storage.checksum          off
    storage.backlog.mem_limit 5M

[INPUT]
    Name                systemd
    Tag                 systemd-*
    Systemd_Filter      _SYSTEMD_UNIT=docker.service
    Systemd_Filter      _SYSTEMD_UNIT=containerd.service
    DB                  /var/fluent-bit/state/systemd.db
    Path                ${SYSTEMD_PATH}

[OUTPUT]
    Name                cloudwatch_logs
    Match               *
    region              ${AWS_REGION}
    log_group_name      test-systemd
    log_stream_prefix   test-systemd-
    auto_create_group   true

SYSTEMD_PATH is /var/log/journal on most instances (including AL2023) but some OS such as CentOS use /run/log/journal

I previously tested using the bootstrap method:

{
    "family": "fluent-bit-daemon-service",
    "taskRoleArn": "{{task-role-arn}}",
    "executionRoleArn": "{{execution-role-arn}}",
    "containerDefinitions": [
        {
            "name": "fluent-bit-config-init",
            "image": "public.ecr.aws/aws-cli/aws-cli:2.18.14",
            "cpu": 0,
            "essential": false,
            "entryPoint": [
                "/bin/sh",
                "-c"
            ],
            "command": [
                "set -ex && if [ -z $( ls -A '/run/log/journal' ) ]; then echo @SET SYSTEMD_PATH=/var/log/journal >> /fluent-bit/etc/fluent-bit.conf; else echo @SET SYSTEMD_PATH=/run/log/journal >> /fluent-bit/etc/fluent-bit.conf; fi && echo @SET INSTANCE_ID=$(cat /var/lib/cloud/data/instance-id) >> /fluent-bit/etc/fluent-bit.conf && if [ -z \"${FLUENT_BIT_CONFIG_FILE}\" ]; then echo 'W1NFUlZJQ0VdCiAgICBGbHVzaCAgICAgICAgICAgICAgICAgICAgIDUKICAgIEdyYWNlICAgICAgICAgICAgICAgICAgICAgMzAKICAgIExvZ19MZXZlbCAgICAgICAgICAgICAgICAgZXJyb3IKICAgIERhZW1vbiAgICAgICAgICAgICAgICAgICAgb2ZmCiAgICBQYXJzZXJzX0ZpbGUgICAgICAgICAgICAgIHBhcnNlcnMuY29uZgogICAgc3RvcmFnZS5wYXRoICAgICAgICAgICAgICAvdmFyL2ZsdWVudC1iaXQvc3RhdGUvZmxiLXN0b3JhZ2UvCiAgICBzdG9yYWdlLnN5bmMgICAgICAgICAgICAgIG5vcm1hbAogICAgc3RvcmFnZS5jaGVja3N1bSAgICAgICAgICBvZmYKICAgIHN0b3JhZ2UuYmFja2xvZy5tZW1fbGltaXQgNU0KCltJTlBVVF0KICAgIE5hbWUgICAgICAgICAgICAgICAgc3lzdGVtZAogICAgVGFnICAgICAgICAgICAgICAgICBzeXN0ZW1kLSoKICAgIFN5c3RlbWRfRmlsdGVyICAgICAgX1NZU1RFTURfVU5JVD1kb2NrZXIuc2VydmljZQogICAgU3lzdGVtZF9GaWx0ZXIgICAgICBfU1lTVEVNRF9VTklUPWNvbnRhaW5lcmQuc2VydmljZQogICAgREIgICAgICAgICAgICAgICAgICAvdmFyL2ZsdWVudC1iaXQvc3RhdGUvc3lzdGVtZC5kYgogICAgUGF0aCAgICAgICAgICAgICAgICAke1NZU1RFTURfUEFUSH0KCltJTlBVVF0KICAgIE5hbWUgICAgICAgICAgICAgICAgc3lzdGVtZAogICAgVGFnICAgICAgICAgICAgICAgICBzeXN0ZW1kLXN5c2xvZwogICAgREIgICAgICAgICAgICAgICAgICAvdmFyL2ZsdWVudC1iaXQvc3RhdGUvc3lzdGVtZC1zeXNsb2cuZGIKICAgIFBhdGggICAgICAgICAgICAgICAgJHtTWVNURU1EX1BBVEh9CiAgICAKW0lOUFVUXQogICAgTmFtZSAgICAgICAgICAgICAgICBzeXN0ZW1kCiAgICBUYWcgICAgICAgICAgICAgICAgIHN5c3RlbWQtZG1lc2cuKgogICAgU3lzdGVtZF9GaWx0ZXIgICAgICBfVFJBTlNQT1JUPWtlcm5lbAogICAgREIgICAgICAgICAgICAgICAgICAvdmFyL2ZsdWVudC1iaXQvc3RhdGUvZG1lc2cuZGIKICAgIFBhdGggICAgICAgICAgICAgICAgJHtTWVNURU1EX1BBVEh9CgpbSU5QVVRdCiAgICBOYW1lICAgICAgICAgICAgICAgIHRhaWwKICAgIFRhZyAgICAgICAgICAgICAgICAgc2VjdXJlCiAgICBQYXRoICAgICAgICAgICAgICAgIC92YXIvbG9nL3NlY3VyZQogICAgUGFyc2VyICAgICAgICAgICAgICBzeXNsb2cKICAgIERCICAgICAgICAgICAgICAgICAgL3Zhci9mbHVlbnQtYml0L3N0YXRlL2ZsYl9zZWN1cmUuZGIKICAgIE1lbV9CdWZfTGltaXQgICAgICAgNU1CCiAgICBTa2lwX0xvbmdfTGluZXMgICAgIE9uCiAgICBSZWZyZXNoX0ludGVydmFsICAgIDEwCiAgICBSZWFkX2Zyb21fSGVhZCAgICAgIE9uCgpbSU5QVVRdCiAgICBOYW1lICAgICAgICAgICAgICAgIHRhaWwKICAgIFRhZyAgICAgICAgICAgICAgICAgZWNzLWFnZW50LmxvZwogICAgUGF0aCAgICAgICAgICAgICAgICAvdmFyL2xvZy9lY3MvZWNzLWFnZW50LmxvZwogICAgREIgICAgICAgICAgICAgICAgICAvdmFyL2ZsdWVudC1iaXQvc3RhdGUvZmxiX2Vjc19hZ2VudC5kYgogICAgTWVtX0J1Zl9MaW1pdCAgICAgICA1TUIKICAgIFNraXBfTG9uZ19MaW5lcyAgICAgT24KICAgIFJlZnJlc2hfSW50ZXJ2YWwgICAgMTAKICAgIFJlYWRfZnJvbV9IZWFkICAgICAgT24KCltJTlBVVF0KICAgIE5hbWUgICAgICAgICAgICAgICAgdGFpbAogICAgVGFnICAgICAgICAgICAgICAgICBlY3MtaW5pdC5sb2cKICAgIFBhdGggICAgICAgICAgICAgICAgL3Zhci9sb2cvZWNzL2Vjcy1pbml0LmxvZwogICAgREIgICAgICAgICAgICAgICAgICAvdmFyL2ZsdWVudC1iaXQvc3RhdGUvZmxiX2Vjc19pbml0LmRiCiAgICBNZW1fQnVmX0xpbWl0ICAgICAgIDVNQgogICAgU2tpcF9Mb25nX0xpbmVzICAgICBPbgogICAgUmVmcmVzaF9JbnRlcnZhbCAgICAxMAogICAgUmVhZF9mcm9tX0hlYWQgICAgICBPbgoKW0lOUFVUXQogICAgTmFtZSAgICAgICAgICAgICAgICB0YWlsCiAgICBUYWcgICAgICAgICAgICAgICAgIGVjcy1hdWRpdC5sb2cKICAgIFBhdGggICAgICAgICAgICAgICAgL3Zhci9sb2cvZWNzL2F1ZGl0LmxvZwogICAgREIgICAgICAgICAgICAgICAgICAvdmFyL2ZsdWVudC1iaXQvc3RhdGUvZmxiX2Vjc19hdWRpdC5kYgogICAgTWVtX0J1Zl9MaW1pdCAgICAgICA1TUIKICAgIFNraXBfTG9uZ19MaW5lcyAgICAgT24KICAgIFJlZnJlc2hfSW50ZXJ2YWwgICAgMTAKICAgIFJlYWRfZnJvbV9IZWFkICAgICAgT24KCltJTlBVVF0KICAgIE5hbWUgICAgICAgICAgICAgICAgdGFpbAogICAgVGFnICAgICAgICAgICAgICAgICBlY3Mtdm9sdW1lLXBsdWdpbi5sb2cKICAgIFBhdGggICAgICAgICAgICAgICAgL3Zhci9sb2cvZWNzL2Vjcy12b2x1bWUtcGx1Z2luLmxvZwogICAgREIgICAgICAgICAgICAgICAgICAvdmFyL2ZsdWVudC1iaXQvc3RhdGUvZmxiX2Vjc192b2x1bWVfcGx1Z2luLmRiCiAgICBNZW1fQnVmX0xpbWl0ICAgICAgIDVNQgogICAgU2tpcF9Mb25nX0xpbmVzICAgICBPbgogICAgUmVmcmVzaF9JbnRlcnZhbCAgICAxMAogICAgUmVhZF9mcm9tX0hlYWQgICAgICBPbiAKCltGSUxURVJdCiAgICBOYW1lICAgICAgICAgICAgICAgIG1vZGlmeQogICAgTWF0Y2ggICAgICAgICAgICAgICBzeXN0ZW1kLSoKICAgIFJlbmFtZSAgICAgICAgICAgICAgX0hPU1ROQU1FICAgICAgICAgICAgICAgICAgIGhvc3RuYW1lCiAgICBSZW5hbWUgICAgICAgICAgICAgIF9TWVNURU1EX1VOSVQgICAgICAgICAgICAgICBzeXN0ZW1kX3VuaXQKICAgIFJlbmFtZSAgICAgICAgICAgICAgTUVTU0FHRSAgICAgICAgICAgICAgICAgICAgIG1lc3NhZ2UKICAgIFJlbW92ZV9yZWdleCAgICAgICAgXigoPyFob3N0bmFtZXxzeXN0ZW1kX3VuaXR8bWVzc2FnZSkuKSokCgpbRklMVEVSXQogICAgTmFtZSAgICAgICAgICAgICAgICBhd3MKICAgIE1hdGNoICAgICAgICAgICAgICAgKgogICAgaW1kc192ZXJzaW9uICAgICAgICB2MgoKW09VVFBVVF0KICAgIE5hbWUgICAgICAgICAgICAgICAgY2xvdWR3YXRjaF9sb2dzCiAgICBNYXRjaCAgICAgICAgICAgICAgICoKICAgIHJlZ2lvbiAgICAgICAgICAgICAgJHtBV1NfUkVHSU9OfQogICAgbG9nX2dyb3VwX25hbWUgICAgICAvYXdzL2Vjcy9jb250YWluZXJpbnNpZ2h0cy8ke0NMVVNURVJfTkFNRX0vaW5zdGFuY2UtbG9ncwogICAgbG9nX3N0cmVhbV9wcmVmaXggICAke0lOU1RBTkNFX0lEfS0KICAgIGF1dG9fY3JlYXRlX2dyb3VwICAgdHJ1ZQ==' | base64 -d >> /fluent-bit/etc/fluent-bit.conf; else aws s3 cp $FLUENT_BIT_CONFIG_FILE /tmp/fluent-bit-temp.conf && cat /tmp/fluent-bit-temp.conf >> /fluent-bit/etc/fluent-bit.conf; fi && if [ -z \"${FLUENT_BIT_PARSERS_FILE}\" ]; then echo 'W1BBUlNFUl0KICAgIE5hbWUgICAgICAgICAgICAgICAgc3lzbG9nCiAgICBGb3JtYXQgICAgICAgICAgICAgIHJlZ2V4CiAgICBSZWdleCAgICAgICAgICAgICAgIF4oPzx0aW1lPlteIF0qIHsxLDJ9W14gXSogW14gXSopICg/PGhvc3Q+W14gXSopICg/PGlkZW50PlthLXpBLVowLTlfXC9cLlwtXSopKD86XFsoPzxwaWQ+WzAtOV0rKVxdKT8oPzpbXlw6XSpcOik/ICooPzxtZXNzYWdlPi4qKSQKICAgIFRpbWVfS2V5ICAgICAgICAgICAgdGltZQogICAgVGltZV9Gb3JtYXQgICAgICAgICAlYiAlZCAlSDolTTolUw==' | base64 -d >>  /fluent-bit/etc/parsers.conf; else aws s3 cp $FLUENT_BIT_PARSERS_FILE  /tmp/parsers-temp.conf && cat /tmp/parsers-temp.conf >> /fluent-bit/etc/parsers.conf; fi\n"
            ],
            "mountPoints": [
                {
                    "sourceVolume": "fluent-bit-config",
                    "containerPath": "/fluent-bit/etc/"
                },
                {
                    "sourceVolume": "instance-id",
                    "containerPath": "/var/lib/cloud/data/instance-id",
                    "readOnly": true
                },
                {
                    "sourceVolume": "systemd-journal",
                    "containerPath": "/run/log/journal",
                    "readOnly": true
                }
            ],
            "logConfiguration": {
                "logDriver": "awslogs",
                "options": {
                    "awslogs-group": "/ecs/ecs-fluent-bit-daemon-service",
                    "awslogs-create-group": "True",
                    "awslogs-region": "{{aws-region}}",
                    "awslogs-stream-prefix": "ecs"
                }
            }
        },
        {
            "name": "fluent-bit",
            "image": "{{fluent-bit-image}} ",
            "cpu": 0,
            "essential": true,
            "environment": [
                {
                    "name": "AWS_REGION",
                    "value": "{{aws-region}}"
                },
                {
                    "name": "CLUSTER_NAME",
                    "value": "{{ecs-cluster-name}}"
                }
            ],
            "mountPoints": [
                {
                    "sourceVolume": "fluent-bit-config",
                    "containerPath": "/fluent-bit/etc/"
                },
                {
                    "sourceVolume": "fluent-bit-state",
                    "containerPath": "/var/fluent-bit/state"
                },
                {
                    "sourceVolume": "instance-logs",
                    "containerPath": "/var/log",
                    "readOnly": true
                },
                {
                    "sourceVolume": "systemd-journal",
                    "containerPath": "/run/log/journal",
                    "readOnly": true
                },
                {
                    "sourceVolume": "machine-id",
                    "containerPath": "/etc/machine-id",
                    "readOnly": true
                }
            ],
            "dependsOn": [
                {
                    "containerName": "fluent-bit-config-init",
                    "condition": "COMPLETE"
                }
            ],
            "logConfiguration": {
                "logDriver": "awslogs",
                "options": {
                    "awslogs-group": "/ecs/ecs-fluent-bit-daemon-service",
                    "awslogs-create-group": "True",
                    "awslogs-region": "{{aws-region}}",
                    "awslogs-stream-prefix": "ecs"
                }
            }
        }
    ],
    "networkMode": "bridge",
    "volumes": [
        {
            "name": "fluent-bit-state",
            "host": {
                "sourcePath": "/var/fluent-bit/state"
            }
        },
        {
            "name": "instance-logs",
            "host": {
                "sourcePath": "/var/log"
            }
        },
        {
            "name": "instance-id",
            "host": {
                "sourcePath": "/var/lib/cloud/data/instance-id"
            }
        },
        {
            "name": "systemd-journal",
            "host": {
                "sourcePath": "/run/log/journal"
            }
        },
        {
            "name": "machine-id",
            "host": {
                "sourcePath": "/etc/machine-id"
            }
        },
        {
            "name": "fluent-bit-config",
            "dockerVolumeConfiguration": {
                "scope": "task",
                "driver": "local"
            }
        }
    ],
    "requiresCompatibilities": [
        "EC2"
    ],
    "cpu": "512",
    "memory": "256"
}

For EKS, I just used fluent-bit included with container insights, and just replaced the image in the daemonset with the updated image. Config is included in the configmap

sparrc · 2025-03-27T18:44:06Z

scripts/dockerfiles/Dockerfile.main-release

@@ -17,9 +17,9 @@ RUN make -j $(getconf _NPROCESSORS_ONLN)
 RUN install bin/fluent-bit /fluent-bit/bin/

 # Build lightweight release image
-FROM public.ecr.aws/amazonlinux/amazonlinux:2
+FROM public.ecr.aws/amazonlinux/amazonlinux:2023


the release container could probably be based off of 2023-minimal: public.ecr.aws/amazonlinux/amazonlinux:2023-minimal

Good callout, will adjust. 2023-minimal does not have yum installed and uses dnf (specifically microdnf). Will modify yum commands in release to cater for this

… for release

jenademoodley · 2025-06-17T05:19:22Z

Any update on this merge request?

ShelbyZ · 2025-06-24T17:25:56Z

Any update on this merge request?

We have an internal effort for AL2 --> AL2023 migration as part of aws/containers-roadmap#2297. I had a large portion of this duplicated before noticing this PR. We do not expect the 1.9.10 version to get migrated from AL2 --> AL2023.

We can leave this open for now in case someone wants to use this for building custom images, but expect to see it close when new version builds are introduced.

ShelbyZ · 2025-08-26T17:39:08Z

With the PR for adding AL2023 dockerfiles, #974, it is possible provide overrides for:

AL_TAG
FLB_VERSION
FLB_REPOSITORY

To create a local AL2023 build against newer fluent-bit versions by modifying the Makefile to add a new target like:

.PHONY: release-al2023
release-al2023: AL_TAG=2023
release-al2023: FLB_VERSION=v4.0.8
release-al2023: FLB_REPOSITORY=https://github.com/fluent/fluent-bit.git
release-al2023: release

This will build images leading up to the final runtime image (with exception for plugins) under the AL_TAG override value and pull fluent-bit source from the FLB_REPOSITORY override URL and select the version from supplied FLB_VERSION override. FLB_VERSION can be a branch or tag. In the example above it is a v4.0.8 tag.

jenademoodley added 2 commits March 27, 2025 15:16

Migrated from Amazon Linux 2 to Amazon Linux 2023

34db30c

Migrated from Amazon Linux 2 to Amazon Linux 2023

1d49158

jenademoodley requested a review from a team as a code owner March 27, 2025 15:46

jenademoodley mentioned this pull request Mar 27, 2025

aws-for-fluent-bit image migration to AL2023 #831

Open

sparrc reviewed Mar 27, 2025

View reviewed changes

Removed libASL package and switched to 2023-minimal base AL2023 image…

904b48f

… for release

jenademoodley force-pushed the mainline branch from 6d1d8a3 to 904b48f Compare April 24, 2025 09:57

ShelbyZ added the close-post-4.x Revisit after 4.x and close or reevaluate label Jun 24, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Migrated base image from Amazon Linux 2 to Amazon Linux 2023 #913

Migrated base image from Amazon Linux 2 to Amazon Linux 2023 #913

Uh oh!

jenademoodley commented Mar 27, 2025

Uh oh!

sparrc Mar 27, 2025

Uh oh!

jenademoodley Apr 24, 2025

Uh oh!

sparrc commented Mar 27, 2025

Uh oh!

jenademoodley commented Mar 27, 2025

Uh oh!

sparrc Mar 27, 2025

Uh oh!

jenademoodley Apr 24, 2025

Uh oh!

jenademoodley commented Jun 17, 2025

Uh oh!

ShelbyZ commented Jun 24, 2025

Uh oh!

ShelbyZ commented Aug 26, 2025

Uh oh!

Uh oh!

Migrated base image from Amazon Linux 2 to Amazon Linux 2023 #913

Are you sure you want to change the base?

Migrated base image from Amazon Linux 2 to Amazon Linux 2023 #913

Uh oh!

Conversation

jenademoodley commented Mar 27, 2025

Summary

Testing

Description for the changelog

Licensing

Uh oh!

sparrc Mar 27, 2025

Choose a reason for hiding this comment

Uh oh!

jenademoodley Apr 24, 2025

Choose a reason for hiding this comment

Uh oh!

sparrc commented Mar 27, 2025

Uh oh!

jenademoodley commented Mar 27, 2025

Uh oh!

sparrc Mar 27, 2025

Choose a reason for hiding this comment

Uh oh!

jenademoodley Apr 24, 2025

Choose a reason for hiding this comment

Uh oh!

jenademoodley commented Jun 17, 2025

Uh oh!

ShelbyZ commented Jun 24, 2025

Uh oh!

ShelbyZ commented Aug 26, 2025

Uh oh!

Uh oh!