ST with secrets - mysql, elasticsearch, postgresql. Customer facing doc for setting secrets as env variable (#4945)

* Doc for postgresql ST with some correction in doc for postgresql app

* Rearranging folder structure and adding a note for metric collection support in different version of otel agent and postgresql

* Correct file names, links, sidebars

* Uncomment postgresql source template docs

* ST with secrets - postgresql updated. Mysql and elasticsearch doc added

* Changes from master merged

* Changelog for postgresql

* Changes to sidebars st to include postgresql, mysql and elasticsearch and redirection link for them. Added an additional note for setting env in windows for stwith secrets

* adding slug id to mysql and elasticsearch

* minor fixes

* cid redirect correction for st with secrets

* mysql and elasticsearch changes for prerequisite for metrics - setting env variable

* correction for processing rule referece which was calling PR check failure

* Apply suggestions from code review

Co-authored-by: Kim (Sumo Logic) <[email protected]>

* Apply suggestions from code review

Co-authored-by: Kim (Sumo Logic) <[email protected]>

* monitor changes - adding note to let the user know naming convention for env varibales

* Update in note for setting env varibales rules

* Changes to put env setting doc in sidebar - rearraging its hierarchy and updating path in associated docs

* cid-redirect update after changing location for st-with-secret doc

* removing code snippet  intended for beta and/or hidden docs

* SUMO-255467 | note for not prefixing http in endpoint string for postgresql

---------

Co-authored-by: Kim Pohas <[email protected]>
Co-authored-by: Kim (Sumo Logic) <[email protected]>
Co-authored-by: Jagadisha V <[email protected]>

Author: 3 people

cid-redirects.json

@@ -1764,7 +1764,10 @@
   "/cid/10321": "/docs/send-data/opentelemetry-collector/remote-management/source-templates/windows",
   "/cid/10322": "/docs/send-data/opentelemetry-collector/remote-management/source-templates/docker",
   "/cid/10323": "/docs/send-data/opentelemetry-collector/remote-management/source-templates/nginx",
-  "/cid/10324": "/docs/send-data/opentelemetry-collector/remote-management/source-templates/kafka",
+  "/cid/10340": "/docs/send-data/opentelemetry-collector/remote-management/source-templates/kafka",
+  "/cid/10341": "/docs/send-data/opentelemetry-collector/remote-management/source-templates/postgresql",
+  "/cid/10342": "/docs/send-data/opentelemetry-collector/remote-management/source-templates/mysql",
+  "/cid/10343": "/docs/send-data/opentelemetry-collector/remote-management/source-templates/elasticsearch",
   "/cid/10325": "/docs/send-data/opentelemetry-collector/remote-management/source-templates/apache/changelog",
   "/cid/10326": "/docs/send-data/opentelemetry-collector/remote-management/source-templates/linux/changelog",
   "/cid/10327": "/docs/send-data/opentelemetry-collector/remote-management/source-templates/localfile/changelog",
@@ -1776,6 +1779,10 @@
   "/cid/10337": "/docs/send-data/opentelemetry-collector/remote-management/source-templates/docker/changelog",
   "/cid/10338": "/docs/send-data/opentelemetry-collector/remote-management/source-templates/nginx/changelog",
   "/cid/10339": "/docs/send-data/opentelemetry-collector/remote-management/source-templates/kafka/changelog",
+  "/cid/10344": "/docs/send-data/opentelemetry-collector/remote-management/source-templates/postgresql/changelog",
+  "/cid/10345": "/docs/send-data/opentelemetry-collector/remote-management/source-templates/mysql/changelog",
+  "/cid/10346": "/docs/send-data/opentelemetry-collector/remote-management/source-templates/elasticsearch/changelog",
+  "/cid/10347": "/docs/send-data/opentelemetry-collector/remote-management/source-templates/st-with-secrets",
   "/cid/10822": "/docs/manage/manage-subscription/create-manage-orgs-flex",
   "/cid/10817": "/docs/integrations/sumo-apps/cse",
   "/cid/10818": "/docs/integrations/sumo-apps/cse",

docs/integrations/databases/opentelemetry/postgresql-opentelemetry.md

@@ -19,7 +19,7 @@ This app supports PostgreSQL version 9.6+.
 
 We use the OpenTelemetry collector for PostgreSQL metric collection and for collecting PostgreSQL logs.
 
-The diagram below illustrates the components of the PostgreSQL collection for each database server. OpenTelemetry collector runs on the same host as PostgreSQL, and uses the [PostgreSQL receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/postgresqlreceiver) to obtain PostgreSQL metrics, and the [Sumo Logic OpenTelemetry Exporter](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/sumologicexporter) to send the metrics to Sumo Logic. MySQL logs are sent to Sumo Logic through a [filelog receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/filelogreceiver).
+The diagram below illustrates the components of the PostgreSQL collection for each database server. OpenTelemetry collector runs on the same host as PostgreSQL, and uses the [PostgreSQL receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/postgresqlreceiver) to obtain PostgreSQL metrics, and the [Sumo Logic OpenTelemetry Exporter](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/sumologicexporter) to send the metrics to Sumo Logic. PostgreSQL logs are sent to Sumo Logic through a [filelog receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/filelogreceiver).
 
 <img src='https://sumologic-app-data-v2.s3.amazonaws.com/dashboards/Postgresql-OpenTelemetry/PostgreSQL-Schematics.png' alt="Schematics" />
 
@@ -107,7 +107,7 @@ import SetupColl from '../../../reuse/apps/opentelemetry/set-up-collector.md';
 
 ### Step 2: Configure integration
 
-In this step, you will configure the yaml file required for Mysql collection.
+In this step, you will configure the yaml file required for PostgreSQL collection.
 
 Below is the required input:
 

docs/send-data/opentelemetry-collector/remote-management/source-templates/elasticsearch/changelog.md

@@ -0,0 +1,11 @@
+---
+id: changelog
+title: Changelog
+sidebar_label: Changelog
+description: Changelog for Elasticsearch source template for OpenTelemetry.
+---
+
+## [1.0.0] - 2025-01-30
+
+### Added
+- Initial version of Elasticsearch source template.

docs/send-data/opentelemetry-collector/remote-management/source-templates/elasticsearch/index.md

@@ -0,0 +1,95 @@
+---
+slug: /send-data/opentelemetry-collector/remote-management/source-templates/elasticsearch
+title: Elasticsearch Source Template
+sidebar_label: Elasticsearch
+description: Learn about the Sumo Logic Elasticsearch source template for OpenTelemetry.
+---
+
+import useBaseUrl from '@docusaurus/useBaseUrl';
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+
+<img src={useBaseUrl('img/integrations/databases/elasticsearch.png')} alt="Thumbnail icon" width="100"/> <img src={useBaseUrl('img/send-data/otel-color.svg')} alt="Thumbnail icon" width="30"/>
+
+The Elasticsearch source template creates an OpenTelemetry configuration that can be pushed to a remotely managed OpenTelemetry collector (abbreviated as otelcol). By creating this source template and pushing the config to the appropriate OpenTelemetry agent, you can ensure collection of Elasticsearch logs and metrics to Sumo Logic.
+
+## Fields creation in Sumo Logic for Local File
+
+If not already present, the following [Fields](/docs/manage/fields/) are created as part of source template creation.
+
+- **`sumo.datasource`**. Fixed value of **elasticsearch**.
+- **`deployment.environment`**. This is a user-configured field set at the time of collector installation. It identifies the environment where the host resides, such as `dev`, `prod`, or `qa`.
+- **`db.cluster.name`**. User configured. Enter a uniquely identifiable name for your elasticsearch cluster to show in the Sumo Logic dashboards.
+- **`db.node.name`**. Includes the value of the hostname of the machine which is being monitored.
+
+## Prerequisites
+
+### For metrics collection
+
+- The Elasticsearch metrics [receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/elasticsearchreceiver) queries the Elasticsearch [node stats](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-nodes-stats.html), [cluster health](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-health.html) and [index stats](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-stats.html) endpoints in order to scrape metrics from a running Elasticsearch cluster. This receiver supports Elasticsearch versions 7.9+
+
+- Make sure to set the Elasticsearch user's password as an environment variable for OpenTelemetry agent. Refer to the [Setting Environment Variables with Secret Values for Source Templates.](../st-with-secrets.md).
+
+### For logs collection
+
+Elasticsearch supports logging via local text log files. Elasticsearch logs have four levels of verbosity. To select a level, set `loglevel` to one of:
+
+* **debug**. A lot of information, useful for development/testing.
+* **verbose**. Includes information not often needed, but logs less than debug.
+* **notice** (default value). Moderately verbose, ideal for production environments.
+* **warning**. Only important/critical messages are logged.
+
+All logging settings are located in [Elasticsearch.conf](https://www.elastic.co/guide/en/elasticsearch/reference/current/logging.html). By default, Elasticsearch logs are stored in `/var/log/elasticsearch/ELK-<Clustername>.log`. The default directory for log files is listed in the Elasticsearch.conf file.
+
+Ensure that the otelcol has adequate permissions to access all log file paths. Execute the following command:
+
+```bash
+sudo setfacl -R -m d:u:otelcol-sumo:r-x,u:otelcol-sumo:r-x,g:otelcol-sumo:r-x <PATH_TO_LOG_FILE>
+```
+
+import LogsCollectionPrereqisites from '../../../../../reuse/apps/logs-collection-prereqisites.md';
+
+<LogsCollectionPrereqisites/>
+
+import OtelWindowsLogPrereq from '../../../../../reuse/apps/opentelemetry/log-collection-prerequisite-windows.md';
+
+<OtelWindowsLogPrereq/>
+
+## Configuring the Elasticsearch source template
+
+You can follow the below steps to set a remotely managed OpenTelemetry collector and push the source template to it.
+
+### Step 1: Set up remotely managed OpenTelemetry collector
+
+import CollectorInstallation from '../../../../../reuse/apps/opentelemetry/collector-installation.md';
+
+<CollectorInstallation/>
+
+### Step 2: Configure the source template
+
+In this step, you will configure the yaml required for Elasticsearch collection. Below are the inputs required for configuration:
+
+- **Name**. Name of the source template.
+- **Description**. Description for the source template.	
+- **Log Filepath**. Location where the Elasticsearch logs are logged. Please refer to your elasticsearch.conf file.
+- **Endpoint**. Enter the url of the server you need to monitor. (default: `localhost:9200`).
+- **Username**. Enter the Elasticsearch username.
+- **Password Environment Variable Name**. Enter the Elasticsearch password environment variable name.
+- **Fields/Metadata**. You can provide any customer fields to be tagged with the data collected. By default, Sumo Logic tags `_sourceCategory` with the value otel/elasticsearch user needs to provide the value for `db.cluster.name`.
+
+import OtelLogAdvanceOption from '../../../../../reuse/apps/opentelemetry/logs-advance-option-otel.md';
+
+<OtelLogAdvanceOption/>
+
+**Processing Rules**. You can add **processing rules** for logs/metrics collected. To learn more, refer to [Processing Rules](../../processing-rules/index.md).
+
+### Step 3: Push the source template to the desired remotely managed collectors
+
+import DataConfiguration from '../../../../../reuse/apps/opentelemetry/data-configuration.md';
+
+<DataConfiguration/>
+
+:::info
+Refer to the [changelog](changelog.md) for information on periodic updates to this source template.
+:::

docs/send-data/opentelemetry-collector/remote-management/source-templates/mysql/changelog.md

@@ -0,0 +1,11 @@
+---
+id: changelog
+title: Changelog
+sidebar_label: Changelog
+description: Changelog for MySQL source template for OpenTelemetry.
+---
+
+## [1.0.0] - 2025-01-30
+
+### Added
+- Initial version of MySQL source template.

docs/send-data/opentelemetry-collector/remote-management/source-templates/mysql/index.md

@@ -0,0 +1,105 @@
+---
+slug: /send-data/opentelemetry-collector/remote-management/source-templates/mysql
+title: MySQL Source Template
+sidebar_label: MySQL
+description: Learn about the Sumo Logic MySQL source template for OpenTelemetry.
+---
+
+import useBaseUrl from '@docusaurus/useBaseUrl';
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+
+
+<img src={useBaseUrl('img/integrations/databases/mysql.png')} alt="Thumbnail icon" width="100"/> <img src={useBaseUrl('img/send-data/otel-color.svg')} alt="Thumbnail icon" width="30"/>
+
+The MySQL source template creates an OpenTelemetry configuration that can be pushed to a remotely managed OpenTelemetry collector (abbreviated as otelcol). By creating this source template and pushing the config to the appropriate OpenTelemetry agent, you can ensure collection of MySQL logs and metrics to Sumo Logic.
+
+## Fields creation in Sumo Logic for Local File
+
+If not already present, the following [Fields](/docs/manage/fields/) are created as part of source template creation.
+
+- **`sumo.datasource`**. Fixed value of **mysql**.
+- **`deployment.environment`**. This is a user-configured field set at the time of collector installation. It identifies the environment where the host resides, such as `dev`, `prod`, or `qa`.
+- **`db.cluster.name`**. User configured. Enter a uniquely identifiable name for your mysql cluster to show in the Sumo Logic dashboards.
+- **`db.node.name`**. Includes the value of the hostname of the machine which is being monitored.
+
+## Prerequisites
+
+### For metrics collection
+
+- The MySQL metrics [receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/mysqlreceiver) collects metrics by querying MySQL's global status and InnoDB tables. This app has been tested with following MySQL versions: 8.0
+
+- Make sure to set the MySQL user's password as an environment variable for the OpenTelemetry agent. Refer to [Setting Environment Variables with Secret Values for Source Templates](../st-with-secrets.md).
+
+### For logs collection
+
+MySQL logs are stored in log files. Slow query logs must be explicitly enabled to be able to be written to a log file. To configure the MySQL log file(s), locate your local `my.cnf` configuration file in the database directory.
+   1. Open `my.cnf` in a text editor.
+   2. Set the following parameters in the `[mysqld]` section:
+   ```sql
+   [mysqld]
+       log_error = /var/log/mysql/error.log
+       slow_query_log=1
+       slow_query_log_file = /var/log/mysql/mysql-slow.log
+       long_query_time=2
+   ```
+     * [Error Logs](https://dev.mysql.com/doc/refman/5.7/en/slow-query-log.html). By default, error logs are enabled and are logged at file specified by the `log_error` key.
+     * [Slow Query Logs](https://dev.mysql.com/doc/refman/5.7/en/slow-query-log.html). `slow_query_log=1` enables logging of slow queries to the file specified by `slow_query_log_file`. Setting `long_query_time=2` will cause queries that take more than two seconds to execute to be logged. The default value of `long_query_time` is 10 seconds.
+     * [General Query Logs](https://dev.mysql.com/doc/refman/5.7/en/query-log.html). We do not recommend enabling `general_log` for performance reasons. These logs are not used by the Sumo Logic MySQL app.
+   3. Save the `my.cnf` file.
+   4. Restart the MySQL server:
+    ```bash
+        sudo mysql.server restart
+    ```
+Ensure that the otelcol has adequate permissions to access all log file paths. Execute the following command:
+
+```bash
+sudo setfacl -R -m d:u:otelcol-sumo:r-x,u:otelcol-sumo:r-x,g:otelcol-sumo:r-x <PATH_TO_LOG_FILE>
+```
+
+import LogsCollectionPrereqisites from '../../../../../reuse/apps/logs-collection-prereqisites.md';
+
+<LogsCollectionPrereqisites/>
+
+import OtelWindowsLogPrereq from '../../../../../reuse/apps/opentelemetry/log-collection-prerequisite-windows.md';
+
+<OtelWindowsLogPrereq/>
+
+## Configuring the MySQL source template
+
+You can follow the below steps to set a remotely managed OpenTelemetry collector and push the source template to it.
+
+### Step 1: Set up remotely managed OpenTelemetry collector
+
+import CollectorInstallation from '../../../../../reuse/apps/opentelemetry/collector-installation.md';
+
+<CollectorInstallation/>
+
+### Step 2: Configure the source template
+
+In this step, you will configure the yaml required for MySQL collection. Below are the inputs required for configuration:
+
+- **Name**. Name of the source template.
+- **Description**. Description for the source template.	
+- **Error log path**. Location where the SQL Errors are logged. Please refer to your my.cnf file.
+- **Slow Transaction log file path (optional)**. Location where the Slow SQL transactions are logged. Please refer to your my.cnf file.
+- **Endpoint**. The URL of the MySQL endpoint (default: `localhost:3306`).
+- **Username**. Enter the MySQL username.
+- **Password Environment Variable Name**. Enter the MySQL password environment variable name.
+- **Fields/Metadata**. You can provide any customer fields to be tagged with the data collected. By default, Sumo Logic tags `_sourceCategory` with the value otel/mysql user needs to provide the value for `db.cluster.name`.
+
+import OtelLogAdvanceOption from '../../../../../reuse/apps/opentelemetry/logs-advance-option-otel.md';
+
+<OtelLogAdvanceOption/>
+
+**Processing Rules**. You can add **processing rules** for logs/metrics collected. To learn more, refer to [Processing Rules](../../processing-rules/index.md).
+
+### Step 3: Push the source template to the desired remotely managed collectors
+
+import DataConfiguration from '../../../../../reuse/apps/opentelemetry/data-configuration.md';
+
+<DataConfiguration/>
+
+:::info
+Refer to the [changelog](changelog.md) for information on periodic updates to this source template.
+:::

docs/send-data/opentelemetry-collector/remote-management/source-templates/postgresql/changelog.md

@@ -0,0 +1,9 @@
+---
+id: changelog
+title: Changelog
+sidebar_label: Changelog
+description: Changelog for PostgreSQL source template for OpenTelemetry.
+---
+## [1.0.0] - 2025-01-30
+### Added
+- Initial version of PostgreSQL source template.