feat: add Logs Agent

README.md

@@ -15,12 +15,21 @@ This module deploys the following observability agents to a Red Hat OpenShift Co
 <!-- BEGIN OVERVIEW HOOK -->
 ## Overview
 * [terraform-ibm-observability-agents](#terraform-ibm-observability-agents)
+* [Submodules](./modules)
+    * [logs-agent-module](./modules/logs-agent-module)
 * [Examples](./examples)
-    * [Deploy basic observability agents](./examples/basic)
+    * [Log Analysis agent](./examples/basic)
+    * [Monitoring agent + Cloud Logs agent on Kubernetes using CSE ingress endpoint with an apikey](./examples/logs-agent-iks)
+    * [Monitoring agent + Cloud Logs agent on OCP using VPE ingress endpoint with a Trusted Profile](./examples/logs-agent-roks)
 * [Contributing](#contributing)
 <!-- END OVERVIEW HOOK -->
 
 ## terraform-ibm-observability-agents
+
+### Deprecated: Log Analysis
+
+**Important:** IBM Log Analysis will be discontinued on 30 March 2025 and replaced by IBM Cloud Logs.
+
 ### Usage
 
 ```hcl
@@ -64,6 +73,12 @@ module "observability_agents" {
   log_analysis_instance_region     = "us-south"
   cloud_monitoring_access_key      = "XXXXXXXX"
   cloud_monitoring_instance_region = "us-south"
+  # Logs Agent variables
+  logs_agent_trusted_profile  = "XXXXXXXX"
+  logs_agent_namespace        = "ibm-observe"
+  logs_agent_name             = "logs-agent"
+  cloud_logs_ingress_endpoint = "<cloud-logs-instance-guid>.ingress.us-south.logs.cloud.ibm.com"
+  cloud_logs_ingress_port     = 443
 }
 ```
 
@@ -111,13 +126,15 @@ You need the following permissions to run this module.
 
 | Name | Version |
 |------|---------|
-| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 1.1.0 |
-| <a name="requirement_helm"></a> [helm](#requirement\_helm) | >= 2.8.0, <3.0.0 |
+| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 1.9.0 |
+| <a name="requirement_helm"></a> [helm](#requirement\_helm) | >= 2.11.0, <3.0.0 |
 | <a name="requirement_ibm"></a> [ibm](#requirement\_ibm) | >= 1.59.0, <2.0.0 |
 
 ### Modules
 
-No modules.
+| Name | Source | Version |
+|------|--------|---------|
+| <a name="module_logs_agents"></a> [logs\_agents](#module\_logs\_agents) | ./modules/logs-agent-module | n/a |
 
 ### Resources
 
@@ -133,6 +150,8 @@ No modules.
 
 | Name | Description | Type | Default | Required |
 |------|-------------|------|---------|:--------:|
+| <a name="input_cloud_logs_ingress_endpoint"></a> [cloud\_logs\_ingress\_endpoint](#input\_cloud\_logs\_ingress\_endpoint) | The host for IBM Cloud Logs ingestion. Ensure you use the ingress endpoint. See https://cloud.ibm.com/docs/cloud-logs?topic=cloud-logs-endpoints_ingress. | `string` | `null` | no |
+| <a name="input_cloud_logs_ingress_port"></a> [cloud\_logs\_ingress\_port](#input\_cloud\_logs\_ingress\_port) | The target port for the IBM Cloud Logs ingestion endpoint. The port must be 443 if you connect by using a VPE gateway, or port 3443 when you connect by using CSEs. | `number` | `3443` | no |
 | <a name="input_cloud_monitoring_access_key"></a> [cloud\_monitoring\_access\_key](#input\_cloud\_monitoring\_access\_key) | Access key used by the IBM Cloud Monitoring agent to communicate with the instance | `string` | `null` | no |
 | <a name="input_cloud_monitoring_add_cluster_name"></a> [cloud\_monitoring\_add\_cluster\_name](#input\_cloud\_monitoring\_add\_cluster\_name) | If true, configure the cloud monitoring agent to attach a tag containing the cluster name to all metric data. | `bool` | `true` | no |
 | <a name="input_cloud_monitoring_agent_name"></a> [cloud\_monitoring\_agent\_name](#input\_cloud\_monitoring\_agent\_name) | Cloud Monitoring agent name. Used for naming all kubernetes and helm resources on the cluster. | `string` | `"sysdig-agent"` | no |
@@ -155,11 +174,25 @@ No modules.
 | <a name="input_log_analysis_agent_namespace"></a> [log\_analysis\_agent\_namespace](#input\_log\_analysis\_agent\_namespace) | Namespace where to deploy the Log Analysis agent. Default value is 'ibm-observe' | `string` | `"ibm-observe"` | no |
 | <a name="input_log_analysis_agent_tags"></a> [log\_analysis\_agent\_tags](#input\_log\_analysis\_agent\_tags) | List of tags to associate to all log records that the agent collects so that you can identify the agent's data quicker in the logging UI. NOTE: Use the 'log\_analysis\_add\_cluster\_name' variable to add the cluster name as a tag. | `list(string)` | `[]` | no |
 | <a name="input_log_analysis_agent_tolerations"></a> [log\_analysis\_agent\_tolerations](#input\_log\_analysis\_agent\_tolerations) | List of tolerations to apply to Log Analysis agent. | <pre>list(object({<br/>    key               = optional(string)<br/>    operator          = optional(string)<br/>    value             = optional(string)<br/>    effect            = optional(string)<br/>    tolerationSeconds = optional(number)<br/>  }))</pre> | <pre>[<br/>  {<br/>    "operator": "Exists"<br/>  }<br/>]</pre> | no |
-| <a name="input_log_analysis_enabled"></a> [log\_analysis\_enabled](#input\_log\_analysis\_enabled) | Deploy IBM Cloud Logging agent | `bool` | `true` | no |
+| <a name="input_log_analysis_enabled"></a> [log\_analysis\_enabled](#input\_log\_analysis\_enabled) | Deploy IBM Cloud Logging agent | `bool` | `false` | no |
 | <a name="input_log_analysis_endpoint_type"></a> [log\_analysis\_endpoint\_type](#input\_log\_analysis\_endpoint\_type) | Specify the IBM Log Analysis instance endpoint type (public or private) to use. Used to construct the ingestion endpoint. | `string` | `"private"` | no |
 | <a name="input_log_analysis_ingestion_key"></a> [log\_analysis\_ingestion\_key](#input\_log\_analysis\_ingestion\_key) | Ingestion key for the IBM Cloud Logging agent to communicate with the instance | `string` | `null` | no |
 | <a name="input_log_analysis_instance_region"></a> [log\_analysis\_instance\_region](#input\_log\_analysis\_instance\_region) | The IBM Log Analysis instance region. Used to construct the ingestion endpoint. | `string` | `null` | no |
 | <a name="input_log_analysis_secret_name"></a> [log\_analysis\_secret\_name](#input\_log\_analysis\_secret\_name) | The name of the secret which will store the ingestion key. | `string` | `"logdna-agent"` | no |
+| <a name="input_logs_agent_additional_log_source_paths"></a> [logs\_agent\_additional\_log\_source\_paths](#input\_logs\_agent\_additional\_log\_source\_paths) | The list of additional log sources. By default, the Logs agent collects logs from a single source at `/var/log/containers/*.log`. | `list(string)` | `[]` | no |
+| <a name="input_logs_agent_additional_metadata"></a> [logs\_agent\_additional\_metadata](#input\_logs\_agent\_additional\_metadata) | The list of additional metadata fields to add to the routed logs. | <pre>list(object({<br/>    key   = optional(string)<br/>    value = optional(string)<br/>  }))</pre> | `[]` | no |
+| <a name="input_logs_agent_agent_tolerations"></a> [logs\_agent\_agent\_tolerations](#input\_logs\_agent\_agent\_tolerations) | List of tolerations to apply to Logs agent. The default value means a pod will run on every node. | <pre>list(object({<br/>    key               = optional(string)<br/>    operator          = optional(string)<br/>    value             = optional(string)<br/>    effect            = optional(string)<br/>    tolerationSeconds = optional(number)<br/>  }))</pre> | <pre>[<br/>  {<br/>    "operator": "Exists"<br/>  }<br/>]</pre> | no |
+| <a name="input_logs_agent_enable_scc"></a> [logs\_agent\_enable\_scc](#input\_logs\_agent\_enable\_scc) | Whether to enable creation of Security Context Constraints in Openshift. When installing on an OpenShift cluster, this setting is mandatory to configure permissions for pods within your cluster. | `bool` | `true` | no |
+| <a name="input_logs_agent_enabled"></a> [logs\_agent\_enabled](#input\_logs\_agent\_enabled) | Whether to deploy the Logs agent. | `bool` | `true` | no |
+| <a name="input_logs_agent_exclude_log_source_paths"></a> [logs\_agent\_exclude\_log\_source\_paths](#input\_logs\_agent\_exclude\_log\_source\_paths) | The list of log sources to exclude. Specify the paths that the Logs agent ignores. | `list(string)` | `[]` | no |
+| <a name="input_logs_agent_iam_api_key"></a> [logs\_agent\_iam\_api\_key](#input\_logs\_agent\_iam\_api\_key) | The IBM Cloud API key for the Logs agent to authenticate and communicate with the IBM Cloud Logs. It is required if `logs_agent_iam_mode` is set to `IAMAPIKey`. | `string` | `null` | no |
+| <a name="input_logs_agent_iam_environment"></a> [logs\_agent\_iam\_environment](#input\_logs\_agent\_iam\_environment) | IAM authentication Environment: `Production` or `PrivateProduction` or `Staging` or `PrivateStaging`. `Production` specifies the public endpoint & `PrivateProduction` specifies the private endpoint. | `string` | `"PrivateProduction"` | no |
+| <a name="input_logs_agent_iam_mode"></a> [logs\_agent\_iam\_mode](#input\_logs\_agent\_iam\_mode) | IAM authentication mode: `TrustedProfile` or `IAMAPIKey`. | `string` | `"TrustedProfile"` | no |
+| <a name="input_logs_agent_log_source_namespaces"></a> [logs\_agent\_log\_source\_namespaces](#input\_logs\_agent\_log\_source\_namespaces) | The list of namespaces from which logs should be forwarded by agent. If namespaces are not listed, logs from all namespaces will be sent. | `list(string)` | `[]` | no |
+| <a name="input_logs_agent_name"></a> [logs\_agent\_name](#input\_logs\_agent\_name) | The name of the Logs agent. The name is used in all Kubernetes and Helm resources in the cluster. | `string` | `"logs-agent"` | no |
+| <a name="input_logs_agent_namespace"></a> [logs\_agent\_namespace](#input\_logs\_agent\_namespace) | The namespace where the Logs agent is deployed. The default value is `ibm-observe`. | `string` | `"ibm-observe"` | no |
+| <a name="input_logs_agent_selected_log_source_paths"></a> [logs\_agent\_selected\_log\_source\_paths](#input\_logs\_agent\_selected\_log\_source\_paths) | The list of specific log sources paths. Logs will only be collected from the specified log source paths. If no paths are specified, it will send logs from `/var/log/containers`. | `list(string)` | `[]` | no |
+| <a name="input_logs_agent_trusted_profile"></a> [logs\_agent\_trusted\_profile](#input\_logs\_agent\_trusted\_profile) | The IBM Cloud trusted profile ID. Used only when `logs_agent_iam_mode` is set to `TrustedProfile`. The trusted profile must have an IBM Cloud Logs `Sender` role. | `string` | `null` | no |
 
 ### Outputs
 

examples/basic/README.md

@@ -1,7 +1,13 @@
-# Deploy basic observability agents
+# Log Analysis agent
+
+## Deprecated: Log Analysis
+
+**Important:** IBM Log Analysis will be discontinued on 30 March 2025 and replaced by IBM Cloud Logs.
 
 An end-to-end example that uses the module's default variable values.
 
 The example sets up the logging agent for [Kubernetes metadata filtering](https://github.com/logdna/logdna-agent-v2/blob/3.8/docs/KUBERNETES.md#configuration-for-kubernetes-metadata-filtering).
 
 The example configures the agent to include all log lines coming from the `default` Kubernetes namespace and excludes anything with a label `app.kubernetes.io/name` and value `sample-app` or an annotation `annotation.user` with the value `sample-user`.
+
+:exclamation: The service IBM Cloud Log Analysis is now deprecated and new instances cannot be provisioned after November 30, 2024, and all existing instances will be destroyed on March 30, 2025. For more information, see https://cloud.ibm.com/docs/log-analysis?topic=log-analysis-getting-started

examples/basic/main.tf

@@ -159,6 +159,8 @@ module "observability_agents" {
   cluster_id                    = var.is_vpc_cluster ? ibm_container_vpc_cluster.cluster[0].id : ibm_container_cluster.cluster[0].id
   cluster_resource_group_id     = module.resource_group.resource_group_id
   log_analysis_instance_region  = module.observability_instances.region
+  logs_agent_enabled            = false
+  log_analysis_enabled          = true
   log_analysis_ingestion_key    = module.observability_instances.log_analysis_ingestion_key
   cloud_monitoring_access_key   = module.observability_instances.cloud_monitoring_access_key
   log_analysis_agent_tags       = var.resource_tags

examples/basic/version.tf

@@ -1,6 +1,6 @@
 terraform {
   # module uses nullable feature which is only available in versions >= 1.1.0
-  required_version = ">= 1.1.0"
+  required_version = ">= 1.9.0"
 
   required_providers {
     ibm = {

examples/logs-agent-iks/README.md

@@ -0,0 +1,11 @@
+# Monitoring agent + Cloud Logs agent on Kubernetes using CSE ingress endpoint with an apikey
+
+An example that shows how to deploy Logs Routing agents and Monitoring agent in a Kubernetes cluster to send Logs directly to IBM Cloud Logs and Cloud Monitoring instance respectively.
+
+The example provisions the following resources:
+- A new resource group, if an existing one is not passed in.
+- A basic VPC.
+- A Kubernetes cluster.
+- A Service ID with `Sender` role to `logs` service.
+- An IBM Cloud Logs and Cloud Monitoring instance
+- Logs agents and Monitoring agent

examples/logs-agent-iks/main.tf

@@ -0,0 +1,135 @@
+##############################################################################
+# Resource Group
+##############################################################################
+
+module "resource_group" {
+  source = "git::https://github.com/terraform-ibm-modules/terraform-ibm-resource-group.git?ref=v1.1.6"
+  # if an existing resource group is not set (null) create a new one using prefix
+  resource_group_name          = var.resource_group == null ? "${var.prefix}-resource-group" : null
+  existing_resource_group_name = var.resource_group
+}
+
+##############################################################################
+# Trusted Profile
+##############################################################################
+
+locals {
+  logs_agent_namespace = "ibm-observe"
+  logs_agent_name      = "logs-agent"
+}
+
+# As a `Sender`, you can send logs to your IBM Cloud Logs service instance - but not query or tail logs. This role is meant to be used by agents and routers sending logs.
+module "iam_service_id" {
+  source                          = "terraform-ibm-modules/iam-service-id/ibm"
+  version                         = "1.2.0"
+  iam_service_id_name             = "${var.prefix}-service-id"
+  iam_service_id_description      = "Logs Agent service id"
+  iam_service_id_apikey_provision = true
+  iam_service_policies = {
+    logs = {
+      roles = ["Sender"]
+      resources = [{
+        service = "logs"
+      }]
+    }
+  }
+}
+
+##############################################################################
+# Create VPC and IKS Cluster
+##############################################################################
+
+resource "ibm_is_vpc" "example_vpc" {
+  name           = "${var.prefix}-vpc"
+  resource_group = module.resource_group.resource_group_id
+  tags           = var.resource_tags
+}
+
+resource "ibm_is_subnet" "testacc_subnet" {
+  name                     = "${var.prefix}-subnet"
+  vpc                      = ibm_is_vpc.example_vpc.id
+  zone                     = "${var.region}-1"
+  total_ipv4_address_count = 256
+  resource_group           = module.resource_group.resource_group_id
+}
+
+# Lookup the current default kube version
+data "ibm_container_cluster_versions" "cluster_versions" {}
+locals {
+  default_version = data.ibm_container_cluster_versions.cluster_versions.default_kube_version
+}
+
+resource "ibm_container_vpc_cluster" "cluster" {
+  name                 = var.prefix
+  vpc_id               = ibm_is_vpc.example_vpc.id
+  kube_version         = local.default_version
+  flavor               = "bx2.4x16"
+  worker_count         = "2"
+  force_delete_storage = true
+  wait_till            = "IngressReady"
+  zones {
+    subnet_id = ibm_is_subnet.testacc_subnet.id
+    name      = "${var.region}-1"
+  }
+  resource_group_id = module.resource_group.resource_group_id
+  tags              = var.resource_tags
+}
+
+data "ibm_container_cluster_config" "cluster_config" {
+  cluster_name_id   = ibm_container_vpc_cluster.cluster.id
+  resource_group_id = module.resource_group.resource_group_id
+}
+
+# Sleep to allow RBAC sync on cluster
+resource "time_sleep" "wait_operators" {
+  depends_on      = [data.ibm_container_cluster_config.cluster_config]
+  create_duration = "45s"
+}
+
+##############################################################################
+# Observability Instance
+##############################################################################
+
+
+module "observability_instances" {
+  source  = "terraform-ibm-modules/observability-instances/ibm"
+  version = "2.18.1"
+  providers = {
+    logdna.at = logdna.at
+    logdna.ld = logdna.ld
+  }
+  resource_group_id              = module.resource_group.resource_group_id
+  region                         = var.region
+  cloud_logs_plan                = "standard"
+  cloud_monitoring_plan          = "graduated-tier"
+  activity_tracker_provision     = false
+  enable_platform_logs           = false
+  enable_platform_metrics        = false
+  log_analysis_provision         = false
+  cloud_logs_instance_name       = "${var.prefix}-cloud-logs"
+  cloud_monitoring_instance_name = "${var.prefix}-cloud-monitoring"
+}
+
+##############################################################################
+# Observability Agents
+##############################################################################
+
+module "observability_agents" {
+  source                    = "../../modules/logs-agent-module"
+  depends_on                = [time_sleep.wait_operators]
+  cluster_id                = ibm_container_vpc_cluster.cluster.id
+  cluster_resource_group_id = module.resource_group.resource_group_id
+  # Logs Agent
+  # logs_agent_enabled          = true
+  logs_agent_iam_mode         = "IAMAPIKey"
+  logs_agent_iam_api_key      = module.iam_service_id.service_id_apikey
+  logs_agent_namespace        = local.logs_agent_namespace
+  logs_agent_name             = local.logs_agent_name
+  cloud_logs_ingress_endpoint = module.observability_instances.cloud_logs_ingress_private_endpoint
+  cloud_logs_ingress_port     = 3443
+  logs_agent_enable_scc       = false
+  # # Monitoring agent
+  # cloud_monitoring_enabled         = true
+  # cloud_monitoring_access_key      = module.observability_instances.cloud_monitoring_access_key
+  # cloud_monitoring_instance_region = module.observability_instances.region
+}

examples/logs-agent-iks/outputs.tf

@@ -0,0 +1,11 @@
+##############################################################################
+# Outputs
+##############################################################################
+
+#output "myoutput" {
+#  description = "Description of my output"
+#  value       = "value"
+#  depends_on  = [<some resource>]
+#}
+
+##############################################################################

examples/logs-agent-iks/provider.tf

@@ -0,0 +1,34 @@
+provider "ibm" {
+  ibmcloud_api_key = var.ibmcloud_api_key
+  region           = var.region
+}
+
+provider "helm" {
+  kubernetes {
+    host                   = data.ibm_container_cluster_config.cluster_config.host
+    token                  = data.ibm_container_cluster_config.cluster_config.token
+    cluster_ca_certificate = data.ibm_container_cluster_config.cluster_config.ca_certificate
+  }
+}
+
+provider "kubernetes" {
+  host                   = data.ibm_container_cluster_config.cluster_config.host
+  token                  = data.ibm_container_cluster_config.cluster_config.token
+  cluster_ca_certificate = data.ibm_container_cluster_config.cluster_config.ca_certificate
+}
+
+locals {
+  at_endpoint = "https://api.${var.region}.logging.cloud.ibm.com"
+}
+
+provider "logdna" {
+  alias      = "at"
+  servicekey = module.observability_instances.activity_tracker_resource_key != null ? module.observability_instances.activity_tracker_resource_key : ""
+  url        = local.at_endpoint
+}
+
+provider "logdna" {
+  alias      = "ld"
+  servicekey = module.observability_instances.log_analysis_resource_key != null ? module.observability_instances.log_analysis_resource_key : ""
+  url        = local.at_endpoint
+}