Support Fluentbit to CloudWatch logs (#140)

* Import and customize fluenbit add-on

* Enable fluent bit logs

* Bump helm addon version

* Dropping account id as it seems to create scraping errors

* Create separate log groups per namespace

* Apply pre-commit

* Remove conflicting global label

* Add config object for logs

* Enable logs in examples

* Add logs docs

* Fix broken link

* Add screenshots

* Update docs

* Typos

Author: bonclay7

README.md

@@ -4,13 +4,14 @@
 
 Welcome to the AWS Observability Accelerator for Terraform!
 
-The AWS Observability Accelerator for Terraform is a set of opinionated modules to
-help you set up observability for your AWS environments with
+The AWS Observability Accelerator for Terraform is a set of opinionated modules
+to help you set up observability for your AWS environments with
 AWS-managed observability services such as Amazon Managed Service for Prometheus,
-Amazon Managed Grafana and AWS Distro for OpenTelemetry (ADOT).
+Amazon Managed Grafana, AWS Distro for OpenTelemetry (ADOT) and Amazon CloudWatch.
 
-We provide curated metrics, traces collection, alerting rules and Grafana dashboards
-for your EKS infrastructure, Java/JMX, NGINX based workloads and custom applications.
+We provide curated metrics, logs, traces collection, alerting rules and Grafana
+dashboards for your EKS infrastructure, Java/JMX, NGINX based workloads and
+your custom applications.
 
 You also can monitor your Amazon Managed Service for Prometheus workspaces ingestion,
 costs, active series with [this module](./modules/managed-prometheus-monitoring).
@@ -42,18 +43,20 @@ v2+ releases introduces couple of breaking changes compared to previous versions
 
 ### Base Module
 
-The base module allows you to configure the AWS Observability services for your cluster and
-the AWS Distro for OpenTelemetry (ADOT) Operator as the signals collection mechanism.
+The base module allows you to configure the AWS Observability services for your
+cluster and the AWS Distro for OpenTelemetry (ADOT) Operator as the signals
+collection mechanism.
 
-This is the minimum configuration to have a new Amazon Managed Service for Prometheus Workspace
-and ADOT Operator deployed for you and ready to receive your data.
-The base module serve as an anchor to the workload modules and cannot run on its own.
+This is the minimum configuration to have a new Amazon Managed Service for
+Prometheus Workspace and ADOT Operator deployed for you and ready to receive
+your data. The base module serve as an anchor to the workload modules and
+cannot run on its own.
 
 ```hcl
 module "aws_observability_accelerator" {
   # use release tags and check for the latest versions
   # https://github.com/aws-observability/terraform-aws-observability-accelerator/releases
-  source = "github.com/aws-observability/terraform-aws-observability-accelerator?ref=v1.6.1"
+  source = "github.com/aws-observability/terraform-aws-observability-accelerator?ref=v2.1.0"
 
   aws_region     = "eu-west-1"
   eks_cluster_id = "my-eks-cluster"
@@ -70,7 +73,7 @@ You can optionally reuse an existing Amazon Managed Servce for Prometheus Worksp
 module "aws_observability_accelerator" {
   # use release tags and check for the latest versions
   # https://github.com/aws-observability/terraform-aws-observability-accelerator/releases
-  source = "github.com/aws-observability/terraform-aws-observability-accelerator?ref=v1.6.1"
+  source = "github.com/aws-observability/terraform-aws-observability-accelerator?ref=v2.1.0"
 
   aws_region     = "eu-west-1"
   eks_cluster_id = "my-eks-cluster"
@@ -91,13 +94,13 @@ View all the configuration options in the module documentation below.
 ### Workload modules
 
 [Workloads modules](./modules) are provided, which essentially provide curated
-metrics collection, alerting rules and Grafana dashboards.
+metrics, logs, traces collection, alerting rules and Grafana dashboards.
 
-#### Infrastructure monitoring
+#### Amazon EKS monitoring
 
 ```hcl
-module "workloads_infra" {
-  source = "aws-observability/terraform-aws-observability-accelerator/workloads/infra"
+module "eks_monitoring" {
+  source = "github.com/aws-observability/terraform-aws-observability-accelerator//modules/eks-monitoring?ref=v2.1.0"
 
   eks_cluster_id = module.eks_observability_accelerator.eks_cluster_id
 
@@ -106,6 +109,9 @@ module "workloads_infra" {
 
   managed_prometheus_workspace_endpoint = module.eks_observability_accelerator.managed_prometheus_workspace_endpoint
   managed_prometheus_workspace_region   = module.eks_observability_accelerator.managed_prometheus_workspace_region
+
+  enable_logs = true
+  enable_tracing = true
 }
 ```
 
@@ -118,17 +124,30 @@ Check the the [complete example](./examples/existing-cluster-with-base-and-infra
 
 ## Motivation
 
-Kubernetes is a powerful and extensible container orchestration technology that allows you to deploy and manage containerized applications at scale. The extensible nature of Kubernetes also allows you to use a wide range of popular open-source tools, commonly referred to as add-ons, in Kubernetes clusters. With such a large number of tools and design choices available, building a tailored EKS cluster that meets your application’s specific needs can take a significant amount of time. It involves integrating a wide range of open-source tools and AWS services and requires deep expertise in AWS and Kubernetes.
+To gain deep visibility into your workloads and environments, AWS proposes a
+set of secure, scalable, highly available, production-grade managed open
+source services such as Amazon Managed Service for Prometheus, Amazon Managed
+Grafana and Amazon OpenSearch.
+
+AWS customers have asked for best-practices and guidance to collect metrics, logs
+and traces from their containerized applications and microservices with ease of
+deployment. Customers can use the AWS Observability Accelerator to configure their
+metrics and traces collection, leveraging [AWS Distro for OpenTelemetry](https://aws-otel.github.io/),
+to have opinionated dashboards and alerts available in only minutes.
 
-AWS customers have asked for examples that demonstrate how to integrate the landscape of Kubernetes tools and make it easy for them to provision complete, opinionated EKS clusters that meet specific application requirements. Customers can use AWS Observability Accelerator to configure and deploy purpose built EKS clusters, and start onboarding workloads in days, rather than months.
 
 ## Support & Feedback
 
-AWS Observability Accelerator for Terraform is maintained by AWS Solution Architects. It is not part of an AWS service and support is provided best-effort by the AWS Observability Accelerator community.
+AWS Observability Accelerator for Terraform is maintained by AWS Solution
+Architects. It is not part of an AWS service and support is provided best-effort
+by the AWS Observability Accelerator community.
 
-To post feedback, submit feature ideas, or report bugs, please use the [Issues](https://github.com/aws-observability/terraform-aws-observability-accelerator/issues) section of this GitHub repo.
+To post feedback, submit feature ideas, or report bugs, please use the
+[Issues](https://github.com/aws-observability/terraform-aws-observability-accelerator/issues)
+section of this GitHub repo.
 
-If you are interested in contributing, see the [Contribution guide](https://github.com/aws-observability/terraform-aws-observability-accelerator/blob/main/CONTRIBUTING.md).
+If you are interested in contributing, see the
+[Contribution guide](https://github.com/aws-observability/terraform-aws-observability-accelerator/blob/main/CONTRIBUTING.md).
 
 ---
 

docs/concepts.md

@@ -123,4 +123,6 @@ classDiagram
 
 ## Getting started with AWS Observability services
 
-If you are new to AWS Observability services, or want to dive deeper into them, check our [One Observability Workshop](https://catalog.workshops.aws/observability/) for a hands-on experience in a self-paced environement or at an AWS venue.
+If you are new to AWS Observability services, or want to dive deeper into them,
+check our [One Observability Workshop](https://catalog.workshops.aws/observability/)
+for a hands-on experience in a self-paced environement or at an AWS venue.

docs/eks/logs.md

@@ -0,0 +1,67 @@
+# Viewing Logs
+
+By default, we deploy a FluentBit daemon set in the cluster to collect worker
+logs for all namespaces. Logs collection can be disabled with
+`enable_logs = false`. Logs are collected and exported to Amazon CloudWatch Logs,
+which enables you to centralize the logs from all of your systems, applications,
+and AWS services that you use, in a single, highly scalable service.
+
+Further configuration options are available in the [module documentation](https://github.com/aws-observability/terraform-aws-observability-accelerator/tree/main/modules/eks-monitoring#inputs).
+This guide shows how you can leverage CloudWatch Logs in Amazon Managed Grafana
+for your cluster and application logs.
+
+## Using CloudWatch Logs as data source in Grafana
+
+Follow [the documentation](https://docs.aws.amazon.com/grafana/latest/userguide/using-amazon-cloudwatch-in-AMG.html)
+to enable Amazon CloudWatch as a data source. Make sure to provide permissions.
+
+!!! tip
+    If you created your workspace with our [provided example](https://aws-observability.github.io/terraform-aws-observability-accelerator/helpers/managed-grafana/),
+    Amazon CloudWatch data source has already been setup for you.
+
+All logs are delivered in the following CloudWatch Log groups naming pattern:
+`/aws/eks/observability-accelerator/{cluster-name}/{namespace}`. Log streams
+follow `{container-name}.{pod-name}`. In Grafana, querying and analyzing logs
+is done with [CloudWatch Logs Insights](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/AnalyzingLogData.html)
+
+### Example - ADOT collector logs
+
+Select one or many log groups and run the following query. The example below,
+queries AWS Distro for OpenTelemetry (ADOT) logs
+
+```console
+fields @timestamp, log
+| order @timestamp desc
+| limit 100
+```
+
+<img width="1987" alt="Screenshot 2023-03-27 at 19 08 35" src="https://user-images.githubusercontent.com/10175027/228037030-95005f47-ff46-4f7a-af74-d31809c52fcd.png">
+
+
+### Example - Using time series visualizations
+
+[CloudWatch Logs syntax](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_QuerySyntax.html)
+provide powerful functions to extract data from your logs. The `stats()`
+function allows you to calculate aggregate statistics with log field values.
+This is useful to have visualization on non-metric data from your applications.
+
+In the example below, we use the following query to graph the number of metrics
+collected by the ADOT collector
+
+```console
+fields @timestamp, log
+| parse log /"#metrics": (?<metrics_count>\d+)}/
+| stats avg(metrics_count) by bin(5m)
+| limit 100
+```
+
+!!! tip
+    You can add logs in your dashboards with logs panel types or time series
+    depending on your query results type.
+
+<img width="2056" alt="image" src="https://user-images.githubusercontent.com/10175027/228037186-12691590-0bfe-465b-a83b-5c4f583ebf96.png">
+
+!!! warning
+    Querying CloudWatch logs will incur costs per GB scanned. Use small time
+    windows and limits in your queries. Checkout the CloudWatch
+    [pricing page](https://aws.amazon.com/cloudwatch/pricing/) for more infos.

docs/index.md

@@ -5,32 +5,36 @@ Welcome to the AWS Observability Accelerator for Terraform!
 The AWS Observability Accelerator for Terraform is a set of opinionated modules to
 help you set up observability for your AWS environments with
 AWS-managed observability services such as Amazon Managed Service for Prometheus,
-Amazon Managed Grafana and AWS Distro for OpenTelemetry (ADOT).
+Amazon Managed Grafana, AWS Distro for OpenTelemetry (ADOT) and Amazon CloudWatch.
 
-We provide curated metrics, traces collection, alerting rules and Grafana dashboards
-for your EKS infrastructure, Java/JMX, NGINX based workloads and custom applications.
+We provide curated metrics, logs, traces collection, alerting rules and Grafana
+dashboards for your EKS infrastructure, Java/JMX, NGINX based workloads and
+your custom applications.
+
+You also can monitor your Amazon Managed Service for Prometheus workspaces ingestion,
+costs, active series with [this module](https://aws-observability.github.io/terraform-aws-observability-accelerator/workloads/managed-prometheus/).
 
 <img width="1501" alt="image" src="images/dark-o11y-accelerator-amp-xray.png">
 
 ## Getting started
 
-This project provides a set of Terraform modules to enable metrics and traces collection,
-dashboards and alerts for monitoring:
+This project provides a set of Terraform modules to enable metrics, logs and
+traces collection, dashboards and alerts for monitoring:
 
-- Amazon EKS clusters infrastructure
+- Amazon EKS clusters infrastructure and applications
 - NGINX workloads (running on Amazon EKS)
 - Java/JMX workloads (running on Amazon EKS)
 - Amazon Managed Service for Prometheus workspaces with Amazon CloudWatch
 
-These modules can be directly configured in your existing Terraform configurations or ready
-to be deployed in our packaged
+These modules can be directly configured in your existing Terraform
+configurations or ready to be deployed in our packaged
 [examples](https://github.com/aws-observability/terraform-aws-observability-accelerator/tree/main/examples)
 
 !!! tip
     We have supporting examples for quick setup such as:
 
     - Creating a new Amazon EKS cluster and a VPC
-    - Creating and configure an Amazon Managed Grafana workspace with SSO (coming soon)
+    - Creating and configure an Amazon Managed Grafana workspace with SSO
 
 ## Motivation
 

examples/existing-cluster-java/main.tf

@@ -84,6 +84,8 @@ module "eks_monitoring" {
     scrape_sample_limit    = 2000
   }
 
+  enable_logs = true
+
   tags = local.tags
 
   depends_on = [

examples/existing-cluster-nginx/main.tf

@@ -73,6 +73,8 @@ module "eks_monitoring" {
   managed_prometheus_workspace_endpoint = module.aws_observability_accelerator.managed_prometheus_workspace_endpoint
   managed_prometheus_workspace_region   = module.aws_observability_accelerator.managed_prometheus_workspace_region
 
+  enable_logs = true
+
   tags = local.tags
 
   depends_on = [

examples/existing-cluster-with-base-and-infra/main.tf

@@ -89,6 +89,8 @@ module "eks_monitoring" {
     global_scrape_timeout  = "15s"
   }
 
+  enable_logs = true
+
   tags = local.tags
 
   depends_on = [

mkdocs.yml

@@ -29,6 +29,7 @@ nav:
       - Infrastructure monitoring: eks/index.md
       - Java/JMX: eks/java.md
       - Nginx: eks/nginx.md
+      - Viewing logs: eks/logs.md
       - Teardown: eks/destroy.md
   - Monitoring Managed Service for Prometheus Workspaces: workloads/managed-prometheus.md
   - Supporting Examples:

modules/eks-monitoring/README.md

@@ -2,11 +2,12 @@
 
 This module provides EKS cluster monitoring with the following resources:
 
-- AWS Distro For OpenTelemetry Operator and Collector
+- AWS Distro For OpenTelemetry Operator and Collector for Metrics and Traces
+- Logs with [AWS for FluentBit](https://github.com/aws/aws-for-fluent-bit)
 - AWS Managed Grafana Dashboard and data source
 - Alerts and recording rules with AWS Managed Service for Prometheus
 
-This module is inspired from the open source [kube-prometheus-stack](https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack)
+This module makes use of the open source [kube-prometheus-stack](https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack)
 
 <!-- BEGINNING OF PRE-COMMIT-TERRAFORM DOCS HOOK -->
 ## Requirements
@@ -32,7 +33,8 @@ This module is inspired from the open source [kube-prometheus-stack](https://git
 
 | Name | Source | Version |
 |------|--------|---------|
-| <a name="module_helm_addon"></a> [helm\_addon](#module\_helm\_addon) | github.com/aws-ia/terraform-aws-eks-blueprints//modules/kubernetes-addons/helm-addon | v4.13.1 |
+| <a name="module_fluentbit_logs"></a> [fluentbit\_logs](#module\_fluentbit\_logs) | ./add-ons/aws-for-fluentbit | n/a |
+| <a name="module_helm_addon"></a> [helm\_addon](#module\_helm\_addon) | github.com/aws-ia/terraform-aws-eks-blueprints//modules/kubernetes-addons/helm-addon | v4.26.0 |
 | <a name="module_java_monitoring"></a> [java\_monitoring](#module\_java\_monitoring) | ./patterns/java | n/a |
 | <a name="module_nginx_monitoring"></a> [nginx\_monitoring](#module\_nginx\_monitoring) | ./patterns/nginx | n/a |
 | <a name="module_operator"></a> [operator](#module\_operator) | ./add-ons/adot-operator | n/a |
@@ -70,6 +72,7 @@ This module is inspired from the open source [kube-prometheus-stack](https://git
 | <a name="input_enable_dashboards"></a> [enable\_dashboards](#input\_enable\_dashboards) | Enables or disables curated dashboards | `bool` | `true` | no |
 | <a name="input_enable_java"></a> [enable\_java](#input\_enable\_java) | Enable Java workloads monitoring, alerting and default dashboards | `bool` | `false` | no |
 | <a name="input_enable_kube_state_metrics"></a> [enable\_kube\_state\_metrics](#input\_enable\_kube\_state\_metrics) | Enables or disables Kube State metrics exporter. Disabling this might affect some data in the dashboards | `bool` | `true` | no |
+| <a name="input_enable_logs"></a> [enable\_logs](#input\_enable\_logs) | Using AWS For FluentBit to collect cluster and application logs to Amazon CloudWatch | `bool` | `true` | no |
 | <a name="input_enable_nginx"></a> [enable\_nginx](#input\_enable\_nginx) | Enable NGINX workloads monitoring, alerting and default dashboards | `bool` | `false` | no |
 | <a name="input_enable_node_exporter"></a> [enable\_node\_exporter](#input\_enable\_node\_exporter) | Enables or disables Node exporter. Disabling this might affect some data in the dashboards | `bool` | `true` | no |
 | <a name="input_enable_tracing"></a> [enable\_tracing](#input\_enable\_tracing) | (Experimental) Enables tracing with AWS X-Ray. This changes the deploy mode of the collector to daemon set. Requirement: adot add-on <= 0.58-build.0 | `bool` | `false` | no |
@@ -78,6 +81,7 @@ This module is inspired from the open source [kube-prometheus-stack](https://git
 | <a name="input_irsa_iam_role_path"></a> [irsa\_iam\_role\_path](#input\_irsa\_iam\_role\_path) | IAM role path for IRSA roles | `string` | `"/"` | no |
 | <a name="input_java_config"></a> [java\_config](#input\_java\_config) | Configuration object for Java/JMX monitoring | <pre>object({<br>    enable_alerting_rules = bool<br>    scrape_sample_limit   = number<br>  })</pre> | <pre>{<br>  "enable_alerting_rules": true,<br>  "scrape_sample_limit": 1000<br>}</pre> | no |
 | <a name="input_ksm_config"></a> [ksm\_config](#input\_ksm\_config) | Kube State metrics configuration | <pre>object({<br>    create_namespace   = bool<br>    k8s_namespace      = string<br>    helm_chart_name    = string<br>    helm_chart_version = string<br>    helm_release_name  = string<br>    helm_repo_url      = string<br>    helm_settings      = map(string)<br>    helm_values        = map(any)<br><br>    scrape_interval = string<br>    scrape_timeout  = string<br>  })</pre> | <pre>{<br>  "create_namespace": true,<br>  "helm_chart_name": "kube-state-metrics",<br>  "helm_chart_version": "4.24.0",<br>  "helm_release_name": "kube-state-metrics",<br>  "helm_repo_url": "https://prometheus-community.github.io/helm-charts",<br>  "helm_settings": {},<br>  "helm_values": {},<br>  "k8s_namespace": "kube-system",<br>  "scrape_interval": "60s",<br>  "scrape_timeout": "15s"<br>}</pre> | no |
+| <a name="input_logs_config"></a> [logs\_config](#input\_logs\_config) | Configuration object for logs collection | <pre>object({<br>    cw_log_retention_days = number<br>  })</pre> | <pre>{<br>  "cw_log_retention_days": 90<br>}</pre> | no |
 | <a name="input_managed_prometheus_workspace_endpoint"></a> [managed\_prometheus\_workspace\_endpoint](#input\_managed\_prometheus\_workspace\_endpoint) | Amazon Managed Prometheus Workspace Endpoint | `string` | `""` | no |
 | <a name="input_managed_prometheus_workspace_id"></a> [managed\_prometheus\_workspace\_id](#input\_managed\_prometheus\_workspace\_id) | Amazon Managed Prometheus Workspace ID | `string` | `null` | no |
 | <a name="input_managed_prometheus_workspace_region"></a> [managed\_prometheus\_workspace\_region](#input\_managed\_prometheus\_workspace\_region) | Amazon Managed Prometheus Workspace's Region | `string` | `null` | no |