Update FLB_VERSION to allow overriding default branch or tag (#949)

* Update FLB_VERSION to allow overriding default branch or tag

- Change FLB_VERSION from ENV to ARG
- Set default value in Makefile 1.9.10
- Remove FLB_TARBALL as it may not make sense (nor was used to pull
  source archives)
- Reworked /tmp/fluent-bit as source directory and updated a few
  versioned directories
- Update README.md, debugging.md to reflect changes to FLB_VERSION
- Add local build steps for using overrides (repository/version)

Author: ShelbyZ

Makefile

@@ -15,6 +15,8 @@ all: release
 
 # Execute set-cache to turn docker cache back on for faster development.
 DOCKER_BUILD_FLAGS := "--no-cache"
+# Fluent Bit version (branch or tag) to checkout, will use value if not set 
+FLB_VERSION ?= "1.9.10"
 # Fluent Bit repository to checkout, will use value if not set
 FLB_REPOSITORY ?= "https://github.com/amazon-contributing/upstream-to-fluent-bit.git"
 
@@ -36,7 +38,7 @@ debug: main-debug init-debug
 .PHONY: build
 build:
 	docker system prune -f
-	docker build $(DOCKER_BUILD_FLAGS) --build-arg FLB_REPOSITORY=${FLB_REPOSITORY} -t amazon/aws-for-fluent-bit:build -f ./scripts/dockerfiles/Dockerfile.build .
+	docker build $(DOCKER_BUILD_FLAGS) --build-arg FLB_VERSION=${FLB_VERSION} --build-arg FLB_REPOSITORY=${FLB_REPOSITORY} -t amazon/aws-for-fluent-bit:build -f ./scripts/dockerfiles/Dockerfile.build .
 
 .PHONY: build-init
 build-init:

README.md

@@ -28,6 +28,7 @@ Welcome to AWS for Fluent Bit! Before using this Docker Image, please read this
 - [Using the AWS Plugins outside of a container](#using-the-aws-plugins-outside-of-a-container)
 - [Running aws-for-fluent-bit Windows containers](#running-aws-for-fluent-bit-windows-containers)
 - [Development](#development)
+    - [Local build](#local-build)
     - [Local integ testing](#local-integ-testing)
     - [Developing Features in the AWS Plugins](#developing-features-in-the-aws-plugins)
 - [Fluent Bit Examples](#fluent-bit-examples)
@@ -354,6 +355,55 @@ For more details about running Fluent Bit Windows containers in Amazon ECS, plea
 
 ### Development
 
+#### Local build
+
+AWS for Fluent Bit can be built locally using the following commands:
+
+- `make release`: Builds the image with the `--no-cache` option, ensuring a clean build
+- `make dev`: Builds the image with Docker caching enabled for faster development iterations
+
+##### Customizing Fluent Bit Version and Repository
+
+You can customize which version of Fluent Bit is built and which repository it's sourced from. The default values are:
+
+- `FLB_VERSION`: "1.9.10" (can be either a branch name or tag name within the repository)
+- `FLB_REPOSITORY`: "https://github.com/amazon-contributing/upstream-to-fluent-bit.git"
+
+There are two ways to customize these values:
+
+**Method 1: Using environment variables**
+
+Set the environment variables when running the make command:
+
+```bash
+# Build with a specific Fluent Bit version tag
+FLB_VERSION="2.0.8" make release
+
+# Build with a specific branch name
+FLB_VERSION="feature-branch" make release
+
+# Build from a different repository
+FLB_VERSION="your-branch" FLB_REPOSITORY="https://github.com/your-username/fluent-bit.git" make release
+
+# Combine with dev mode for faster builds with caching
+FLB_VERSION="your-branch" FLB_REPOSITORY="https://github.com/your-username/fluent-bit.git" make dev
+```
+
+**Method 2: Updating the Makefile directly**
+
+You can also modify the default values in the Makefile:
+
+1. Open the Makefile in your editor
+2. Locate these lines (near the top):
+   ```
+   # Fluent Bit version (branch or tag) to checkout, will use value if not set 
+   FLB_VERSION ?= "1.9.10"
+   # Fluent Bit repository to checkout, will use value if not set
+   FLB_REPOSITORY ?= "https://github.com/amazon-contributing/upstream-to-fluent-bit.git"
+   ```
+3. Update the values as needed
+4. Save the file and run `make release` or `make dev`
+
 #### Local integ testing
 
 Use `make dev` to build the image.

scripts/dockerfiles/Dockerfile.build

@@ -1,14 +1,13 @@
 FROM public.ecr.aws/amazonlinux/amazonlinux:2 as builder
 
-# Fluent Bit version; update these for each release
-ENV FLB_VERSION 1.9.10
+# Fluent Bit version (branch or tag) to checkout
+ARG FLB_VERSION=1.9.10
 # Fluent Bit repository to checkout
-ARG FLB_REPOSITORY ${FLB_REPOSITORY}
+ARG FLB_REPOSITORY=https://github.com/amazon-contributing/upstream-to-fluent-bit.git
 # branch to pull parsers from in github.com/fluent/fluent-bit-docker-image
 ENV FLB_DOCKER_BRANCH 1.8
 
-ENV FLB_TARBALL http://github.com/fluent/fluent-bit/archive/v$FLB_VERSION.zip
-RUN mkdir -p /fluent-bit/bin /fluent-bit/etc /fluent-bit/log /tmp/fluent-bit-master/
+RUN mkdir -p /fluent-bit/bin /fluent-bit/etc /fluent-bit/log
 
 RUN curl -sL -o /bin/gimme https://raw.githubusercontent.com/travis-ci/gimme/master/gimme
 RUN chmod +x /bin/gimme
@@ -77,9 +76,8 @@ ADD configs/plugin-metrics-parser.conf /fluent-bit/configs/
 FROM builder as compile
 
 # Get Fluent Bit source code
-WORKDIR /tmp/fluent-bit-$FLB_VERSION/
-RUN git clone --single-branch --branch $FLB_VERSION ${FLB_REPOSITORY} /tmp/fluent-bit-$FLB_VERSION/
-WORKDIR /tmp/fluent-bit-$FLB_VERSION/build/
+RUN git clone --single-branch --branch ${FLB_VERSION} ${FLB_REPOSITORY} /tmp/fluent-bit/
+WORKDIR /tmp/fluent-bit/build/
 
 # Apply Fluent Bit patches to base version
 COPY AWS_FLB_CHERRY_PICKS \

troubleshooting/debugging.md

@@ -778,7 +778,7 @@ CMD valgrind --leak-check=full --error-limit=no /fluent-bit/bin/fluent-bit -c /f
 The best option, which is most likely to catch any leak or segfault is to create a fresh build of the image using the `make debug-valgrind` target. This will create a fresh build with debug mode and valgrind support enabled, which gives the highest chance that Valgrind will be able to produce useful diagnostic information about the issue.
 
 1. Check out the git tag for the version that saw the problem
-2. Make sure the `FLB_VERSION` at the top of the `scripts/dockerfiles/Dockerfile.build` is set to the same version as the main Dockerfile for that tag.
+2. Make sure the `FLB_VERSION` at the top of the `Makefile` is set to the same version as the main Dockerfile for that tag.
 3. Build this dockerfile with the `make debug-valgrind` target. The image will be tagged with the `amazon/aws-for-fluent-bit:debug-valgrind` tag.
 
 ##### Other Options: Other Debug Builds
@@ -1233,7 +1233,7 @@ When you recieve a SIGSEGV/crash report from a customer, perform the following s
 
 #### 1. Build and distribute a core dump S3 uploader image
 
-For debug images, we update the `debug-latest` tag and add a tag as `debug-<Version>`. You can find them in [Docker Hub](https://hub.docker.com/r/amazon/aws-for-fluent-bit), [Amazon ECR Public Gallery](https://gallery.ecr.aws/aws-observability/aws-for-fluent-bit) and Amazon ECR. If you need a customized image build for the specific version/case you are testing. Make sure the `ENV FLB_VERSION` is set to the right version in the `Dockerfile.build` and make sure the `AWS_FLB_CHERRY_PICKS` file has the right contents for the release you are testing.
+For debug images, we update the `debug-latest` tag and add a tag as `debug-<Version>`. You can find them in [Docker Hub](https://hub.docker.com/r/amazon/aws-for-fluent-bit), [Amazon ECR Public Gallery](https://gallery.ecr.aws/aws-observability/aws-for-fluent-bit) and Amazon ECR. If you need a customized image build for the specific version/case you are testing. Make sure `FLB_VERSION` is set to the right version in `Makefile` and make sure the `AWS_FLB_CHERRY_PICKS` file has the right contents for the release you are testing.
 
 Then simply run:
 ```

troubleshooting/tutorials/remote-core-dump/README.md

@@ -27,12 +27,12 @@ Clone the AWS for Fluent Bit source code, and run `make debug` for a plain debug
 When Fluent Bit crashes, a zipped core file, stacktrace, and the Fluent Bit executable will be output to the `/cores` directory and the files will also be uploaded to S3.
 
 There are couple of things to note about the debug target for the core file debugging use case:
-- The Fluent Bit upstream base version is specified with `ENV FLB_VERSION`
+- The Fluent Bit upstream base version is specified with `FLB_VERSION` in `Makefile`
 - Fluent Bit is compiled with CMake flag `-DFLB_DEBUG=On`
 - `gdb` is installed in the final stage of the Docker build.
 - `aws` cli is installed to copy files to the S3 bucket.
 
-When you clone AWS for Fluent Bit, you will automatically get the latest Dockerfile for our latest release on the mainline branch. To create a debug build of a different version, either check out the tag for that version, or modify the `ENV FLB_VERSION` at the top of the `/scripts/dockerfiles/Dockerfile.build` to install the desired Fluent Bit base version.
+When you clone AWS for Fluent Bit, you will automatically get the latest Dockerfile for our latest release on the mainline branch. To create a debug build of a different version, either check out the tag for that version, or modify the `FLB_VERSION` at the top of the `Makefile` to install the desired Fluent Bit base version.
 
 Once you are ready, build the debug image: