Incorporating code review comments

Author: hariohmprasath

README.md

@@ -55,7 +55,7 @@ In this repository you will find a number of demos and sample projects from AWS
 
 * [Demo: Logs to Amazon S3 extension: container image ](s3-logs-extension-demo-container-image/): Demo logs extension to receive logs directly from Lambda and send them to S3. This example packages the extension and function as separate container images. The demo is deployed using AWS SAM.
 
-* [Demo: Logs to Kinesis firehose Logs API extension](kinesisfirehose-logs-extension-demo/): How to get a basic logs API extension for AWS Kinesis Firehose written in Go. The extension explains the overall approach to streamline and and centralize log collection using Kinesis firehose. The extension runs a local HTTP listener and subscribes to a stream of function and platform logs using the Logs API. It buffers them and sends them to Kinesis firehose periodically. The demo gets deployed using AWS SAM.
+* [Demo: Logs to Kinesis firehose Logs API extension](kinesisfirehose-logs-extension-demo/): How to get a basic logs API extension for Amazon Kinesis Data Firehose written in Go. The extension explains the overall approach to streamline and centralize log collection using Amazon Kinesis Data firehose. The extension runs a local HTTP listener and subscribes to a stream of function and platform logs using the Logs API. It buffers them and sends them to Amazon Kinesis Data firehose periodically, which streams logs to Amazon S3. The demo gets deployed using AWS SAM.
 
 ### Other extension examples
 

kinesisfirehose-logs-extension-demo/README.md

@@ -1,18 +1,18 @@
-# Centralize log collection with Kinesis Firehose using Lambda Extensions
+# Centralize log collection with Amazon Kinesis Data Firehose using Lambda Extensions
 
 ## Introduction
 
-This pattern walks through an approach to centralize log collection for lambda function with Kinesis firehose using external extensions. The provided code sample shows how to get send logs directly to kinesis firehose without sending them to AWS CloudWatch service.
+This pattern walks through an approach to centralize log collection for Lambda function with Kinesis firehose using external extensions. The provided code sample shows how to get send logs directly to Kinesis firehose without sending them to AWS CloudWatch service.
 
 > Note: This is a simple example extension to help you investigate an approach to centralize the log aggregation. This example code is not production ready. Use it with your own discretion after testing thoroughly.
 
 This sample extension:
 
 * Subscribes to receive `platform` and `function` logs.
 * Runs with a main, and a helper goroutine: The main goroutine registers to `ExtensionAPI` and process its `invoke` and `shutdown` events. The helper goroutine:
-  * starts a local HTTP server at the provided port (default 1234) that receives requests from Logs API with `NextEvent` method call
+  * starts a local HTTP server at the provided port (default 1234, the port can be overridden with Lambda environment variable `HTTP_LOGS_LISTENER_PORT` ) that receives requests from Logs API with `NextEvent` method call
   * puts the logs in a synchronized queue (Producer) to be processed by the main goroutine (Consumer)
-* The main goroutine writes the received logs to AWS Kinesis firehose, which gets stored in AWS S3
+* The main goroutine writes the received logs to Amazon Kinesis firehose, which gets stored in Amazon S3
 
 ## Amazon Kinesis Data firehose
 
@@ -31,24 +31,24 @@ Lambda Extensions, a new way to easily integrate Lambda with your favorite monit
 
 read more about it [here](https://aws.amazon.com/blogs/compute/introducing-aws-lambda-extensions-in-preview/)
 
-> Note: The code sample provided part of this pattern uses **external** extension to listen to log events from the lambda function
+> Note: The code sample provided part of this pattern uses **external** extension to listen to log events from the Lambda function
 
 ## Need to centralize log collection
 
-Having a centralized log collection mechanism using kinesis firehose provides the following benefits:
+Having a centralized log collecting mechanism using Kinesis firehose provides the following benefits:
 
-* Helps to collect logs from different sources in one place. Even though the sample provided sends logs from Lambda, log routers like `Fluentbit` and `Firelens` can send logs directly to kinesis firehose from container orchestrators like `EKS` and `ECS`.
+* Helps to collect logs from different sources in one place. Even though the sample provided sends logs from Lambda, log routers like `Fluentbit` and `Firelens` can send logs directly to Kinesis Data firehose from container orchestrators like `EKS` and `ECS`.
 * Define and standardize the transformations before the log gets delivered to downstream systems like S3, elastic search, redshift, etc
 * Provides a secure storage area for log data, before it gets written out to the disk. In the event of machine/application failure, we still have access to the logs emitted from the source machine/application
 
 ## Architecture
 
 ### AWS Services
 
-* AWS Lambda
-* AWS Lambda extension
-* AWS KinesisFirehose
-* AWS S3
+* Amazon Lambda
+* Amazon Lambda extension
+* Amazon Kinesis Data Firehose
+* Amazon S3
 
 ### High level architecture
 
@@ -64,7 +64,7 @@ Once deployed the overall flow looks like below:
 
 > Note: Firehose stream name gets specified as an environment variable (`AWS_KINESIS_STREAM_NAME`)
 
-* The lambda function won't be able to send any logs events to AWS CloudWatch service due to the following explict `DENY` policy:
+* The Lambda function won't be able to send any logs events to Amazon CloudWatch service due to the following explicit `DENY` policy:
 
 ```yaml
 Sid: CloudWatchLogsDeny
@@ -91,8 +91,9 @@ AWS SAM template available part of the root directory can be used for deploying
 Check out the code by running the following command:
 
 ```bash
-mkdir kinesisfirehose-logs-extension-demo && cd kinesisfirehose-logs-extension-demo
-git clone https://github.com/hariohmprasath/load-testing-serverless-apps.git .
+mkdir aws-lambda-extensions && cd aws-lambda-extensions
+git clone https://github.com/aws-samples/aws-lambda-extensions.git .
+cd kinesisfirehose-logs-extension-demo
 ```
 
 Run the following command from the root directory
@@ -128,7 +129,7 @@ Commands you can use next
 
 ### Deployment
 
-Run the following command to deploy the sample lambda function with the extension
+Run the following command to deploy the sample Lambda function with the extension
 
 ```bash
 sam deploy --guided
@@ -183,7 +184,7 @@ aws lambda invoke \
     --log-type Tail
 ```
 
->Note: Make sure to replace `function-name` with the actual lambda function name
+>Note: Make sure to replace `function-name` with the actual Lambda function name
 
 The function should return ```"StatusCode": 200```, with the below output
 
@@ -195,15 +196,15 @@ The function should return ```"StatusCode": 200```, with the below output
 }
 ```
 
-In a few minutes after the successfully invocation of the lambda function, we should start seeing the log messages from the example extension written to an S3 bucket.
+In a few minutes after the successful invocation of the Lambda function, we should start seeing the log messages from the example extension sent to Amazon Data Firehose which sends the messages to a Amazon S3 bucket.
 
 * Login to AWS console:
   * Navigate to the S3 bucket mentioned under the parameter `BucketName` in the SAM output.
-  * We can see the logs successly written to the S3 bucket, partitioned based on date in `GZIP` format.
+  * We can see the logs successfully written to the S3 bucket, partitioned based on date in `GZIP` format.
   ![s3](images/S3.png)
 
   * Navigate to `"/aws/lambda/${functionname}"` log group inside AWS CloudWatch service.
-  * We shouldn't see any logs created under this log group as we have denied access to write any logs from the lambda function.
+  * We shouldn't see any logs created under this log group as we have denied access to write any logs from the Lambda function.
   ![cloudwatch](images/CloudWatch.png)
 
 ## Cleanup

kinesisfirehose-logs-extension-demo/agent/http.go

@@ -17,8 +17,8 @@ import (
 	"github.com/golang-collections/go-datastructures/queue"
 )
 
-// DefaultHttpListenerPort is used to set the URL where the logs will be sent by Logs API
-const DefaultHttpListenerPort = "1234"
+// HttpListenerPort - Env variable to override the default logs http listener port
+const HttpListenerPort = "HTTP_LOGS_LISTENER_PORT"
 
 // LogsApiHttpListener is used to listen to the Logs API using HTTP
 type LogsApiHttpListener struct {
@@ -37,12 +37,13 @@ func NewLogsApiHttpListener(lq *queue.Queue) (*LogsApiHttpListener, error) {
 }
 
 func ListenOnAddress() string {
+	httpListenerPort:= listenerPort()
 	env_aws_local, ok := os.LookupEnv("AWS_SAM_LOCAL")
 	if ok && "true" == env_aws_local {
-		return ":" + DefaultHttpListenerPort
+		return ":" + httpListenerPort
 	}
 
-	return "sandbox:" + DefaultHttpListenerPort
+	return "sandbox.localdomain:" + httpListenerPort
 }
 
 // Start initiates the server in a goroutine where the logs will be sent
@@ -143,12 +144,10 @@ func (a HttpAgent) Init(agentID string) error {
 		MaxBytes:  262144,
 		TimeoutMS: 1000,
 	}
-	if err != nil {
-		return err
-	}
+	
 	destination := logsapi.Destination{
 		Protocol:   logsapi.HttpProto,
-		URI:        logsapi.URI(fmt.Sprintf("http://sandbox:%s", DefaultHttpListenerPort)),
+		URI:        logsapi.URI(fmt.Sprintf("http://sandbox.localdomain:%s", listenerPort())),
 		HttpMethod: logsapi.HttpPost,
 		Encoding:   logsapi.JSON,
 	}
@@ -166,3 +165,14 @@ func (a *HttpAgent) Shutdown() {
 
 	a.listener.Shutdown()
 }
+
+func listenerPort() string{
+	// Default listener port can be overridden by Lambda environment variable
+	httpListenerPort := os.Getenv(HttpListenerPort)
+	if httpListenerPort == "" {
+		httpListenerPort = "1234"
+	}
+
+	return httpListenerPort
+}
+