diff --git a/content/en/observability_pipelines/sources/amazon_data_firehose.md b/content/en/observability_pipelines/sources/amazon_data_firehose.md index 49a5dce808c..7cfe2028f13 100644 --- a/content/en/observability_pipelines/sources/amazon_data_firehose.md +++ b/content/en/observability_pipelines/sources/amazon_data_firehose.md @@ -18,9 +18,9 @@ Use Observability Pipelines' Amazon Data Firehose source to receive logs from Am Select and set up this source when you [set up a pipeline][1]. The information below is for the source settings in the pipeline UI. -- Enter the identifier for your Amazon Data Firehose address. - - **Note**: Only enter the identifier for the address. Do **not** enter the actual address. - - If left blank, the default is used: `SOURCE_AWS_DATA_FIREHOSE_ADDRESS`. +
Only enter the identifiers for the Amazon Data Firehose address and, if applicable, the TLS key pass. Do not enter the actual values.
+ +- Enter the identifier for your Amazon Data Firehose address. If you leave it blank, the [default](#set-secrets) is used. ### Optional settings @@ -33,17 +33,33 @@ Select an **AWS authentication** option. If you select **Assume role**: #### Enable TLS Toggle the switch to **Enable TLS**. If you enable TLS, the following certificate and key files are required.
**Note**: All file paths are made relative to the configuration data directory, which is `/var/lib/observability-pipelines-worker/config/` by default. See [Advanced Worker Configurations][2] for more information. The file must be owned by the `observability-pipelines-worker group` and `observability-pipelines-worker` user, or at least readable by the group or user. -- Enter the identifier for your Amazon Data Firehose key pass. - - **Note**: Only enter the identifier for the key pass. Do **not** enter the actual key pass. - - If left blank, the default is used: `SOURCE_AWS_DATA_FIREHOSE_KEY_PASS`. +- Enter the identifier for your Amazon Data Firehose key pass. If you leave it blank, the [default](#set-secrets) is used. - `Server Certificate Path`: The path to the certificate file that has been signed by your Certificate Authority (CA) root file in DER or PEM (X.509). -- `CA Certificate Path`: The path to the certificate file that is your Certificate Authority (CA) Root File in DER or PEM (X.509). +- `CA Certificate Path`: The path to the certificate file that is your Certificate Authority (CA) root file in DER or PEM (X.509). - `Private Key Path`: The path to the `.key` private key file that belongs to your Server Certificate Path in DER or PEM (PKCS#8) format. -## Set the environment variables +## Set secrets + +{{% observability_pipelines/set_secrets_intro %}} + +{{< tabs >}} +{{% tab "Secrets Management" %}} + +- Amazon Data Firehose address identifier: + - References the socket address on which the Observability Pipelines Worker listens to receive logs. + - The default identifier is `SOURCE_AWS_DATA_FIREHOSE_ADDRESS`. +- Amazon Data Firehose TLS passphrase identifier (when TLS is enabled): + - The default identifier is `SOURCE_AWS_DATA_FIREHOSE_KEY_PASS`. + +{{% /tab %}} + +{{% tab "Environment variables" %}} {{% observability_pipelines/configure_existing_pipelines/source_env_vars/amazon_data_firehose %}} +{{% /tab %}} +{{< /tabs >}} + ## Send logs to the Observability Pipelines Worker over Amazon Data Firehose {{% observability_pipelines/log_source_configuration/amazon_data_firehose %}} diff --git a/content/en/observability_pipelines/sources/amazon_s3.md b/content/en/observability_pipelines/sources/amazon_s3.md index 8f65a864643..47249cd8cb7 100644 --- a/content/en/observability_pipelines/sources/amazon_s3.md +++ b/content/en/observability_pipelines/sources/amazon_s3.md @@ -13,9 +13,9 @@ Use Observability Pipelines' Amazon S3 source to receive logs from Amazon S3. Se Select and set up this source when you [set up a pipeline][1]. The information below is for the source settings in the pipeline UI. -1. Enter the identifier for your Amazon S3 URL. - - **Note**: Only enter the identifier for the URL. Do **not** enter the actual URL. - - If left blank, the default is used: `SOURCE_AWS_S3_SQS_URL`. +
Only enter the identifiers for the Amazon S3 URL and, if applicable, the TLS key pass. Do not enter the actual values.
+ +1. Enter the identifier for your Amazon S3 URL. If you leave it blank, the [default](#set-secrets) is used. 1. Enter the AWS region. ### Optional settings @@ -29,17 +29,33 @@ Select an **AWS authentication** option. If you select **Assume role**: #### Enable TLS Toggle the switch to **Enable TLS**. If you enable TLS, the following certificate and key files are required.
**Note**: All file paths are made relative to the configuration data directory, which is `/var/lib/observability-pipelines-worker/config/` by default. See [Advanced Worker Configurations][2] for more information. The file must be owned by the `observability-pipelines-worker group` and `observability-pipelines-worker` user, or at least readable by the group or user. -- Enter the identifier for your Amazon S3 key pass. - - **Note**: Only enter the identifier for the key pass. Do **not** enter the actual key pass. - - If left blank, the default is used: `SOURCE_AWS_S3_KEY_PASS`. +- Enter the identifier for your Amazon S3 key pass. If you leave it blank, the [default](#set-secrets) is used. - `Server Certificate Path`: The path to the certificate file that has been signed by your Certificate Authority (CA) root file in DER or PEM (X.509). -- `CA Certificate Path`: The path to the certificate file that is your Certificate Authority (CA) Root File in DER or PEM (X.509). +- `CA Certificate Path`: The path to the certificate file that is your Certificate Authority (CA) root file in DER or PEM (X.509). - `Private Key Path`: The path to the `.key` private key file that belongs to your Server Certificate Path in DER or PEM (PKCS#8) format. -## Set the environment variables +## Set secrets + +{{% observability_pipelines/set_secrets_intro %}} + +{{< tabs >}} +{{% tab "Secrets Management" %}} + +- Amazon S3 URL identifier: + - References the URL of the SQS queue to which the S3 bucket sends the notification events. + - The default identifier is `SOURCE_AWS_S3_SQS_URL`. +- Amazon S3 TLS passphrase identifier (when TLS is enabled): + - The default identifier is `SOURCE_AWS_S3_KEY_PASS`. + +{{% /tab %}} + +{{% tab "Environment Variables" %}} {{% observability_pipelines/configure_existing_pipelines/source_env_vars/amazon_s3 %}} +{{% /tab %}} +{{< /tabs >}} + ## AWS Authentication {{% observability_pipelines/aws_authentication/instructions %}} diff --git a/content/en/observability_pipelines/sources/datadog_agent.md b/content/en/observability_pipelines/sources/datadog_agent.md index 249e7286fcd..ab68ed6f8f6 100644 --- a/content/en/observability_pipelines/sources/datadog_agent.md +++ b/content/en/observability_pipelines/sources/datadog_agent.md @@ -25,8 +25,8 @@ Use Observability Pipelines' Datadog Agent source to receive logs or metrics ({{ ## Set up the source in the pipeline UI Optionally, toggle the switch to enable TLS. If you enable TLS, the following certificate and key files are required. - - `Server Certificate Path`: The path to the certificate file that has been signed by your Certificate Authority (CA) Root File in DER or PEM (X.509) format. - - `CA Certificate Path`: The path to the certificate file that is your Certificate Authority (CA) Root File in DER or PEM (X.509) format. + - `Server Certificate Path`: The path to the certificate file that has been signed by your Certificate Authority (CA) root file in DER or PEM (X.509) format. + - `CA Certificate Path`: The path to the certificate file that is your Certificate Authority (CA) root file in DER or PEM (X.509) format. - `Private Key Path`: The path to the `.key` private key file that belongs to your Server Certificate Path in DER or PEM (PKCS#8) format. **Note**: All file paths are made relative to the configuration data directory, which is `/var/lib/observability-pipelines-worker/config/` by default. See [Advanced Worker Configurations][5] for more information. The file must be owned by the `observability-pipelines-worker group` and `observability-pipelines-worker` user, or at least readable by the group or user. diff --git a/content/en/observability_pipelines/sources/fluent.md b/content/en/observability_pipelines/sources/fluent.md index 2537fdde5e9..d397bfb4db6 100644 --- a/content/en/observability_pipelines/sources/fluent.md +++ b/content/en/observability_pipelines/sources/fluent.md @@ -18,24 +18,40 @@ Use Observability Pipelines' Fluentd or Fluent Bit source to receive logs from t Select and set up this source when you [set up a pipeline][1]. The information below are for the source settings in the pipeline UI. -- 1. Enter the identifier for your Fluent address. - - **Note**: Only enter the identifier for the address. Do **not** enter the actual address. - - If left blank, the default is used: `SOURCE_FLUENT_ADDRESS`. +
Only enter the identifiers for the Fluent address and, if applicable, the TLS key pass. Do not enter the actual values.
+ +1. Enter the identifier for your Fluent address. If you leave it blank, the [default](#set-secrets) is used. ### Optional settings Toggle the switch to **Enable TLS**. If you enable TLS, the following certificate and key files are required.
**Note**: All file paths are made relative to the configuration data directory, which is `/var/lib/observability-pipelines-worker/config/` by default. See [Advanced Worker Configurations][2] for more information. The file must be owned by the `observability-pipelines-worker group` and `observability-pipelines-worker` user, or at least readable by the group or user. -- Enter the identifier for your Fluent key pass. - - **Note**: Only enter the identifier for the key pass. Do **not** enter the actual key pass. - - If left blank, the default is used: `SOURCE_FLUENT_KEY_PASS`. -- `Server Certificate Path`: The path to the certificate file that has been signed by your Certificate Authority (CA) Root File in DER or PEM (X.509) format. -- `CA Certificate Path`: The path to the certificate file that is your Certificate Authority (CA) Root File in DER or PEM (X.509) format. +- Enter the identifier for your Fluent key pass. If you leave it blank, the [default](#set-secrets) is used. +- `Server Certificate Path`: The path to the certificate file that has been signed by your Certificate Authority (CA) root file in DER or PEM (X.509) format. +- `CA Certificate Path`: The path to the certificate file that is your Certificate Authority (CA) root file in DER or PEM (X.509) format. - `Private Key Path`: The path to the `.key` private key file that belongs to your Server Certificate Path in DER or PEM (PKCS#8) format. -## Set the environment variables +## Set secrets + +{{% observability_pipelines/set_secrets_intro %}} + +{{< tabs >}} +{{% tab "Secrets Management" %}} + +- Fluent address identifier: + - References the address on which the Observability Pipelines Worker listens for incoming log messages. + - The default identifier is `SOURCE_FLUENT_ADDRESS`. +- Fluent TLS passphrase identifier (when TLS is enabled): + - The default identifier is `SOURCE_FLUENT_KEY_PASS`. + +{{% /tab %}} + +{{% tab "Environment Variables" %}} {{% observability_pipelines/configure_existing_pipelines/source_env_vars/fluent %}} +{{% /tab %}} +{{< /tabs >}} + ## Send logs to the Observability Pipelines Worker over Fluent {{% observability_pipelines/log_source_configuration/fluent %}} diff --git a/content/en/observability_pipelines/sources/google_pubsub.md b/content/en/observability_pipelines/sources/google_pubsub.md index d06429fd0da..72076e3ba18 100644 --- a/content/en/observability_pipelines/sources/google_pubsub.md +++ b/content/en/observability_pipelines/sources/google_pubsub.md @@ -26,13 +26,27 @@ Select and set up this source when you [set up a pipeline][1]. The information b 1. Select the decoder you want to use (Bytes, GELF, JSON, syslog). 1. Optionally, toggle the switch to enable TLS. If you enable TLS, the following certificate and key files are required.
**Note**: All file paths are made relative to the configuration data directory, which is `/var/lib/observability-pipelines-worker/config/` by default. See [Advanced Worker Configurations][3] for more information. The file must be owned by the `observability-pipelines-worker group` and `observability-pipelines-worker` user, or at least readable by the group or user. - `Server Certificate Path`: The path to the certificate file that has been signed by your Certificate Authority (CA) root file in DER or PEM (X.509). - - `CA Certificate Path`: The path to the certificate file that is your Certificate Authority (CA) Root File in DER or PEM (X.509). + - `CA Certificate Path`: The path to the certificate file that is your Certificate Authority (CA) root file in DER or PEM (X.509). - `Private Key Path`: The path to the `.key` private key file that belongs to your Server Certificate Path in DER or PEM (PKCS #8) format. -## Set the environment variables +## Set secrets + +{{% observability_pipelines/set_secrets_intro %}} + +{{< tabs >}} +{{% tab "Secrets Management" %}} + +There are no default secret identifiers for this source. + +{{% /tab %}} + +{{% tab "Environment Variables" %}} {{% observability_pipelines/configure_existing_pipelines/source_env_vars/google_pubsub %}} +{{% /tab %}} +{{< /tabs >}} + [1]: /observability_pipelines/configuration/set_up_pipelines/ [2]: https://cloud.google.com/kubernetes-engine/docs/concepts/workload-identity [3]: /observability_pipelines/configuration/install_the_worker/advanced_worker_configurations/ diff --git a/content/en/observability_pipelines/sources/http_client.md b/content/en/observability_pipelines/sources/http_client.md index 86995c8b314..a1d54f79a4b 100644 --- a/content/en/observability_pipelines/sources/http_client.md +++ b/content/en/observability_pipelines/sources/http_client.md @@ -20,28 +20,26 @@ Select and set up this source when you [set up a pipeline][1]. The information b To configure your HTTP/S Client source: -1. Enter the identifier for your HTTP Client endpoint URL. - - **Note**: Only enter the identifier for the endpoint URL. Do **not** enter the actual endpoint URL. - - If left blank, the default is used: `SOURCE_HTTP_CLIENT_ENDPOINT_URL`. + +
Only enter the identifiers for the HTTP Client endpoint URL and, if applicable, your authorization strategy secrets. Do not enter the actual values.
+ +1. Enter the identifier for your HTTP Client endpoint URL. If you leave it blank, the [default](#set-secrets) is used. 1. Select your authorization strategy. If you selected: - **Basic**: - - Enter the identifier for your HTTP Client username. - - If left blank, the default is used: `SOURCE_HTTP_CLIENT_USERNAME`. - Enter the identifier for your HTTP Client password. - - If left blank, the default is used: `SOURCE_HTTP_CLIENT_PASSWORD`. - - **Bearer**: Enter the identifier for your bearer token. - - If left blank, the default is used: `SOURCE_HTTP_CLIENT_BEARER_TOKEN`. + - Enter the identifier for your HTTP Client username. If you leave it blank, the [default](#set-secrets) is used. + - Enter the identifier for your HTTP Client password. If you leave it blank, the [default](#set-secrets) is used. + - **Bearer**: Enter the identifier for your bearer token. If you leave it blank, the [default](#set-secrets) is used. 1. Select the decoder you want to use on the HTTP messages. Logs pulled from the HTTP source must be in this format. ### Optional settings #### Enable TLS + Toggle the switch to **Enable TLS**. If you enable TLS, the following certificate and key files are required.
**Note**: All file paths are made relative to the configuration data directory, which is `/var/lib/observability-pipelines-worker/config/` by default. See [Advanced Worker Configurations][2] for more information. The file must be owned by the `observability-pipelines-worker group` and `observability-pipelines-worker` user, or at least readable by the group or user. - - Enter the identifier for your HTTP Client key pass. + - Enter the identifier for your HTTP Client key pass. If you leave it blank, the [default](#set-secrets) is used. - **Note**: Only enter the identifier for the key pass. Do **not** enter the actual key pass. - - If left blank, the default is used: `SOURCE_HTTP_CLIENT_KEY_PASS` - - `Server Certificate Path`: The path to the certificate file that has been signed by your Certificate Authority (CA) Root File in DER or PEM (X.509) format. - - `CA Certificate Path`: The path to the certificate file that is your Certificate Authority (CA) Root File in DER or PEM (X.509) format. + - `Server Certificate Path`: The path to the certificate file that has been signed by your Certificate Authority (CA) root file in DER or PEM (X.509) format. + - `CA Certificate Path`: The path to the certificate file that is your Certificate Authority (CA) root file in DER or PEM (X.509) format. - `Private Key Path`: The path to the `.key` private key file that belongs to your Server Certificate Path in DER or PEM (PKCS#8) format. #### Scrape settings @@ -51,9 +49,35 @@ Toggle the switch to **Enable TLS**. If you enable TLS, the following certificat - Since requests run concurrently, if a scrape takes longer than the interval given, a new scrape is started, which can consume extra resources. Set the timeout to a value lower than the scrape interval to prevent this from happening. - Enter the timeout for each scrape request. -## Set the environment variables +## Set secrets + +{{% observability_pipelines/set_secrets_intro %}} + +{{< tabs >}} +{{% tab "Secrets Management" %}} + +- HTTP Client endpoint URL identifier: + - References the endpoint from which the Observability Pipelines Worker collects log events. + - The default identifier is `SOURCE_HTTP_CLIENT_ENDPOINT_URL`. +- HTTP Client TLS passphrase identifier (when TLS is enabled): + - The default identifier is `SOURCE_HTTP_CLIENT_KEY_PASS`. +- If you are using basic authentication: + - HTTP Client username identifier: + - The default identifier is `SOURCE_HTTP_CLIENT_USERNAME`. + - HTTP Client password identifier: + - The default identifier is `SOURCE_HTTP_CLIENT_PASSWORD`. +- If you are using bearer authentication: + - HTTP Client bearer token identifier: + - The default identifier is `SOURCE_HTTP_CLIENT_BEARER_TOKEN`. + +{{% /tab %}} + +{{% tab "Environment Variables" %}} {{% observability_pipelines/configure_existing_pipelines/source_env_vars/http_client %}} +{{% /tab %}} +{{< /tabs >}} + [1]: /observability_pipelines/configuration/set_up_pipelines/ [2]: /observability_pipelines/configuration/install_the_worker/advanced_worker_configurations/ \ No newline at end of file diff --git a/content/en/observability_pipelines/sources/http_server.md b/content/en/observability_pipelines/sources/http_server.md index 49d552fa6d4..b874fce4ca1 100644 --- a/content/en/observability_pipelines/sources/http_server.md +++ b/content/en/observability_pipelines/sources/http_server.md @@ -22,30 +22,50 @@ Select and set up this source when you [set up a pipeline][1]. The information b To configure your HTTP/S Server source, enter the following: -1. Enter the identifier for your HTTP Server address. +
Only enter the identifiers for the HTTP Server address and, if applicable, the username and password for basic authorization and the TLS key pass. Do not enter the actual values.
+ +1. Enter the identifier for your HTTP Server address. If you leave it blank, the [default](#set-secrets) is used. - **Note**: Only enter the identifier for the address. Do **not** enter the actual address. - - If left blank, the default is used: `SOURCE_HTTP_SERVER_ADDRESS`. 1. Select your authorization strategy. If you selected **Basic**: - - Enter the identifier for your HTTP Server username. - - If left blank, the default is used: `SOURCE_HTTP_SERVER_USERNAME`. - Enter the identifier for your HTTP Server password. - - If left blank, the default is used: `SOURCE_HTTP_SERVER_PASSWORD`. + - Enter the identifier for your HTTP Server username. If you leave it blank, the [default](#set-secrets) is used. + - Enter the identifier for your HTTP Server password. If you leave it blank, the [default](#set-secrets) is used. 1. Select the decoder you want to use on the HTTP messages. Your HTTP client logs must be in this format. **Note**: If you select `bytes` decoding, the raw log is stored in the `message` field. ### Optional settings Toggle the switch to **Enable TLS**. If you enable TLS, the following certificate and key files are required.
**Note**: All file paths are made relative to the configuration data directory, which is `/var/lib/observability-pipelines-worker/config/` by default. See [Advanced Worker Configurations][2] for more information. The file must be owned by the `observability-pipelines-worker group` and `observability-pipelines-worker` user, or at least readable by the group or user. -- Enter the identifier for your HTTP Server key pass. - - **Note**: Only enter the identifier for the key pass. Do **not** enter the actual key pass. - - If left blank, the default is used: `SOURCE_HTTP_SERVER_KEY_PASS`. +- Enter the identifier for your HTTP Server key pass. If you leave it blank, the [default](#set-secrets) is used. - `Server Certificate Path`: The path to the certificate file that has been signed by your Certificate Authority (CA) root file in DER or PEM (X.509). -- `CA Certificate Path`: The path to the certificate file that is your Certificate Authority (CA) Root File in DER or PEM (X.509). +- `CA Certificate Path`: The path to the certificate file that is your Certificate Authority (CA) root file in DER or PEM (X.509). - `Private Key Path`: The path to the `.key` private key file that belongs to your Server Certificate Path in DER or PEM (PKCS #8) format. -## Set the environment variables +## Set secrets + +{{% observability_pipelines/set_secrets_intro %}} + +{{< tabs >}} +{{% tab "Secrets Management" %}} + +- HTTP Server address identifier: + - References the socket address, such as `0.0.0.0:9997`, on which the Observability Pipelines Worker listens for HTTP client logs. + - The default identifier is `SOURCE_HTTP_SERVER_ADDRESS`. +- HTTP Server TLS passphrase identifier (when TLS is enabled): + - The default identifier is `SOURCE_HTTP_SERVER_KEY_PASS`. +- If you are using basic authentication: + - HTTP Server username identifier: + - The default identifier is `SOURCE_HTTP_SERVER_USERNAME`. + - HTTP Server password identifier: + - The default identifier is `SOURCE_HTTP_SERVER_PASSWORD`. + +{{% /tab %}} + +{{% tab "Environment Variables" %}} {{% observability_pipelines/configure_existing_pipelines/source_env_vars/http_server %}} +{{% /tab %}} +{{< /tabs >}} + ## Send AWS vended logs with the Datadog Lambda Forwarder to Observability Pipelines To send AWS vended logs to Observability Pipelines with the HTTP/S Server source: diff --git a/content/en/observability_pipelines/sources/kafka.md b/content/en/observability_pipelines/sources/kafka.md index b4a7096e189..c26c3226655 100644 --- a/content/en/observability_pipelines/sources/kafka.md +++ b/content/en/observability_pipelines/sources/kafka.md @@ -20,14 +20,11 @@ You can also [send Azure Event Hub logs to Observability Pipelines using the Kaf Select and set up this source when you [set up a pipeline][1]. The information below is for the source settings in the pipeline UI. -
Only enter the identifiers for the Kafka servers, username, and password. Do not enter the actual values.
- -1. Enter the identifier for your Kafka servers. - - If left blank, the default is used: `SOURCE_KAFKA_BOOTSTRAP_SERVERS`. -1. Enter the identifier for your Kafka username. - - If left blank, the default is used: `SOURCE_KAFKA_SASL_USERNAME`. -1. Enter the identifier for your Kafka password. - - If left blank, the default is used: `SOURCE_KAFKA_SASL_PASSWORD`. +
Only enter the identifiers for the Kafka servers, username, password, and if applicable, the TLS key pass. Do not enter the actual values.
+ +1. Enter the identifier for your Kafka servers. If you leave it blank, the [default](#set-secrets) is used. +1. Enter the identifier for your Kafka username. If you leave it blank, the [default](#set-secrets) is used. +1. Enter the identifier for your Kafka password. If you leave it blank, the [default](#set-secrets) is used. 1. Enter the group ID. 1. Enter the topic name. If there is more than one, click **Add Field** to add additional topics. @@ -41,11 +38,9 @@ Select and set up this source when you [set up a pipeline][1]. The information b #### Enable TLS Toggle the switch to **Enable TLS**. If you enable TLS, the following certificate and key files are required.
**Note**: All file paths are made relative to the configuration data directory, which is `/var/lib/observability-pipelines-worker/config/` by default. See [Advanced Worker Configurations][5] for more information. The file must be owned by the `observability-pipelines-worker group` and `observability-pipelines-worker` user, or at least readable by the group or user. -- Enter the identifier for your Kafka key pass. - - **Note**: Only enter the identifier for the key pass. Do **not** enter the actual key pass. - - If left blank, the default is used: `SOURCE_KAFKA_KEY_PASS`. +- Enter the identifier for your Kafka key pass. If you leave it blank, the [default](#set-secrets) is used. - `Server Certificate Path`: The path to the certificate file that has been signed by your Certificate Authority (CA) root file in DER or PEM (X.509). -- `CA Certificate Path`: The path to the certificate file that is your Certificate Authority (CA) Root File in DER or PEM (X.509). +- `CA Certificate Path`: The path to the certificate file that is your Certificate Authority (CA) root file in DER or PEM (X.509). - `Private Key Path`: The path to the `.key` private key file that belongs to your Server Certificate Path in DER or PEM (PKCS#8) format. #### Add additional librdkafka options @@ -56,10 +51,33 @@ Toggle the switch to **Enable TLS**. If you enable TLS, the following certificat 1. Check your values against the [librdkafka documentation][4] to make sure they have the correct type and are within the set range. 1. Click **Add Option** to add another librdkafka option. -## Set the environment variables +## Set secrets + +{{% observability_pipelines/set_secrets_intro %}} + +{{< tabs >}} +{{% tab "Secrets Management" %}} + +- Kafka bootstrap servers identifier: + - References the bootstrap server that the client uses to connect to the Kafka cluster and discover all the other hosts in the cluster. + - In your secrets manager, the host and port must be entered in the format of `host:port`, such as `10.14.22.123:9092`. If there is more than one server, use commas to separate them. + - The default identifier is `SOURCE_KAFKA_BOOTSTRAP_SERVERS`. +- Kafka SASL username identifier: + - The default identifier is `SOURCE_KAFKA_SASL_USERNAME`. +- Kafka SASL password identifier: + - The default identifier is `SOURCE_KAFKA_SASL_PASSWORD`. +- Kafka TLS passphrase identifier (when TLS is enabled): + - The default identifier is `SOURCE_KAFKA_KEY_PASS`. + +{{% /tab %}} + +{{% tab "Environment Variables" %}} {{% observability_pipelines/configure_existing_pipelines/source_env_vars/kafka %}} +{{% /tab %}} +{{< /tabs >}} + ## librdkafka options These are the available librdkafka options: diff --git a/content/en/observability_pipelines/sources/logstash.md b/content/en/observability_pipelines/sources/logstash.md index ccd5d208c47..ad166e97e41 100644 --- a/content/en/observability_pipelines/sources/logstash.md +++ b/content/en/observability_pipelines/sources/logstash.md @@ -20,24 +20,40 @@ You can also use the Logstash source to [send logs to Observability Pipelines us Select and set up this source when you [set up a pipeline][1]. The information below is for the source settings in the pipeline UI. -- Enter the identifier for your Logstash address. - - **Note**: Only enter the identifier for the address. Do **not** enter the actual address. - - If left blank, the default is used: `SOURCE_LOGSTASH_ADDRESS`. +
Only enter the identifiers for the Logstash address and, if applicable, the TLS key pass. Do not enter the actual values.
+ +- Enter the identifier for your Logstash address. If you leave it blank, the [default](#set-secrets) is used. ### Optional settings Toggle the switch to **Enable TLS**. If you enable TLS, the following certificate and key files are required.
**Note**: All file paths are made relative to the configuration data directory, which is `/var/lib/observability-pipelines-worker/config/` by default. See [Advanced Worker Configurations][3] for more information. The file must be owned by the `observability-pipelines-worker group` and `observability-pipelines-worker` user, or at least readable by the group or user. -- Enter the identifier for your Logstash key pass. - - **Note**: Only enter the identifier for the key pass. Do **not** enter the actual key pass. - - If left blank, the default is used: `SOURCE_LOGSTASH_KEY_PASS`. +- Enter the identifier for your Logstash key pass. If you leave it blank, the [default](#set-secrets) is used. - `Server Certificate Path`: The path to the certificate file that has been signed by your Certificate Authority (CA) root file in DER or PEM (X.509). -- `CA Certificate Path`: The path to the certificate file that is your Certificate Authority (CA) Root File in DER or PEM (X.509). +- `CA Certificate Path`: The path to the certificate file that is your Certificate Authority (CA) root file in DER or PEM (X.509). - `Private Key Path`: The path to the `.key` private key file that belongs to your Server Certificate Path in DER or PEM (PKCS #8) format. -## Set the environment variables +## Set secrets + +{{% observability_pipelines/set_secrets_intro %}} + +{{< tabs >}} +{{% tab "Secrets Management" %}} + +- Logstash address identifier: + - References the address on which the Observability Pipelines Worker listens for incoming log messages. + - The default identifier is `SOURCE_LOGSTASH_ADDRESS`. +- Logstash TLS passphrase identifier (when TLS is enabled): + - The default identifier is `SOURCE_LOGSTASH_KEY_PASS`. + +{{% /tab %}} + +{{% tab "Environment Variables" %}} {{% observability_pipelines/configure_existing_pipelines/source_env_vars/logstash %}} +{{% /tab %}} +{{< /tabs >}} + ## Send logs to the Observability Pipelines Worker over Logstash {{% observability_pipelines/log_source_configuration/logstash %}} diff --git a/content/en/observability_pipelines/sources/opentelemetry.md b/content/en/observability_pipelines/sources/opentelemetry.md index 83feae0ee3b..318f71dc5b6 100644 --- a/content/en/observability_pipelines/sources/opentelemetry.md +++ b/content/en/observability_pipelines/sources/opentelemetry.md @@ -36,7 +36,7 @@ If your forwarders are globally configured to enable SSL, you need the appropria Optionally, toggle the switch to enable TLS. If you enable TLS, the following certificate and key files are required.
**Note**: All file paths are made relative to the configuration data directory, which is `/var/lib/observability-pipelines-worker/config/` by default. See [Advanced Worker Configurations][3] for more information. The file must be owned by the `observability-pipelines-worker group` and `observability-pipelines-worker` user, or at least readable by the group or user. - `Server Certificate Path`: The path to the certificate file that has been signed by your Certificate Authority (CA) root file in DER or PEM (X.509). -- `CA Certificate Path`: The path to the certificate file that is your Certificate Authority (CA) Root File in DER or PEM (X.509). +- `CA Certificate Path`: The path to the certificate file that is your Certificate Authority (CA) root file in DER or PEM (X.509). - `Private Key Path`: The path to the `.key` private key file that belongs to your Server Certificate Path in DER or PEM (PKCS #8) format. {{< img src="observability_pipelines/sources/otel_settings.png" alt="The OpenTelemetry source settings" style="width:35%;" >}} diff --git a/content/en/observability_pipelines/sources/socket.md b/content/en/observability_pipelines/sources/socket.md index afd59c9036d..65bff5ca3bc 100644 --- a/content/en/observability_pipelines/sources/socket.md +++ b/content/en/observability_pipelines/sources/socket.md @@ -18,9 +18,9 @@ Use Observability Pipelines' Socket source to send logs to the Worker over a soc Select and set up this source when you [set up a pipeline][1]. The information below is for the source settings in the pipeline UI. -1. Enter the identifier for your socket address. - - **Note**: Only enter the identifier for the address. Do **not** enter the actual address. - - If left blank, the default is used: `SOURCE_SOCKET_ADDRESS`. +
Only enter the identifiers for the socket address and, if applicable, the TLS key pass. Do not enter the actual values.
+ +1. Enter the identifier for your socket address. If you leave it blank, the [default](#set-secrets) is used. 1. In the **Mode** dropdown menu, select the socket type to use. 1. In the **Framing** dropdown menu, select how to delimit the stream of events. @@ -59,16 +59,32 @@ Select and set up this source when you [set up a pipeline][1]. The information b ### Optional settings If you selected **TCP** mode, toggle the switch to **Enable TLS**. The following certificate and key files are required for TLS.
**Note**: All file paths are made relative to the configuration data directory, which is `/var/lib/observability-pipelines-worker/config/` by default. See [Advanced Worker Configurations][2] for more information. The file must be owned by the `observability-pipelines-worker group` and `observability-pipelines-worker` user, or at least readable by the group or user. -- Enter the identifier for your socket key pass. - - **Note**: Only enter the identifier for the key pass. Do **not** enter the actual key pass. - - If left blank, the default is used: `SOURCE_SOCKET_KEY_PASS`. +- Enter the identifier for your socket key pass. If you leave it blank, the [default](#set-secrets) is used. - `Server Certificate Path`: The path to the certificate file that has been signed by your Certificate Authority (CA) root file in DER or PEM (X.509). -- `CA Certificate Path`: The path to the certificate file that is your Certificate Authority (CA) Root File in DER or PEM (X.509). +- `CA Certificate Path`: The path to the certificate file that is your Certificate Authority (CA) root file in DER or PEM (X.509). - `Private Key Path`: The path to the `.key` private key file that belongs to your Server Certificate Path in DER or PEM (PKCS #8) format. -## Set the environment variables +## Set secrets + +{{% observability_pipelines/set_secrets_intro %}} + +{{< tabs >}} +{{% tab "Secrets Management" %}} + +- Socket address identifier: + - References the address and port where the Observability Pipelines Worker listens for incoming logs. + - The default identifier is `SOURCE_SOCKET_ADDRESS`. +- Socket TLS passphrase identifier (when TLS is enabled): + - The default identifier is `SOURCE_SOCKET_KEY_PASS`. + +{{% /tab %}} + +{{% tab "Environment Variables" %}} {{% observability_pipelines/configure_existing_pipelines/source_env_vars/socket %}} +{{% /tab %}} +{{< /tabs >}} + [1]: /observability_pipelines/configuration/set_up_pipelines/ [2]: /observability_pipelines/configuration/install_the_worker/advanced_worker_configurations/ \ No newline at end of file diff --git a/content/en/observability_pipelines/sources/splunk_hec.md b/content/en/observability_pipelines/sources/splunk_hec.md index 40e66d21d93..60769c7b067 100644 --- a/content/en/observability_pipelines/sources/splunk_hec.md +++ b/content/en/observability_pipelines/sources/splunk_hec.md @@ -20,24 +20,40 @@ Use Observability Pipelines' Splunk HTTP Event Collector (HEC) source to receive Select and set up this source when you [set up a pipeline][1]. The information below is for the source settings in the pipeline UI. -- Enter the identifier for your Splunk HEC address. - - **Note**: Only enter the identifier for the address. Do **not** enter the actual address. - - If left blank, the default is used: `SOURCE_SPLUNK_HEC_ADDRESS`. +
Only enter the identifiers for the Splunk HEC address and, if applicable, the TLS key pass. Do not enter the actual values.
+ +- Enter the identifier for your Splunk HEC address. If you leave it blank, the [default](#set-secrets) is used. ### Optional settings Toggle the switch to **Enable TLS**. If you enable TLS, the following certificate and key files are required.
**Note**: All file paths are made relative to the configuration data directory, which is `/var/lib/observability-pipelines-worker/config/` by default. See [Advanced Worker Configurations][5] for more information. The file must be owned by the `observability-pipelines-worker group` and `observability-pipelines-worker` user, or at least readable by the group or user. -- Enter the identifier for your Splunk HEC key pass. - - **Note**: Only enter the identifier for the key pass. Do **not** enter the actual key pass. - - If left blank, the default is used: `SOURCE_SPLUNK_HEC_KEY_PASS`. +- Enter the identifier for your Splunk HEC key pass. If you leave it blank, the [default](#set-secrets) is used. - `Server Certificate Path`: The path to the certificate file that has been signed by your Certificate Authority (CA) root file in DER or PEM (X.509). -- `CA Certificate Path`: The path to the certificate file that is your Certificate Authority (CA) Root File in DER or PEM (X.509). +- `CA Certificate Path`: The path to the certificate file that is your Certificate Authority (CA) root file in DER or PEM (X.509). - `Private Key Path`: The path to the `.key` private key file that belongs to your Server Certificate Path in DER or PEM (PKCS#8) format. -## Set the environment variables +## Set secrets + +{{% observability_pipelines/set_secrets_intro %}} + +{{< tabs >}} +{{% tab "Secrets Management" %}} + +- Splunk HEC address identifier: + - References the bind address, such as `0.0.0.0:8088`, on which your Observability Pipelines Worker listens to receive logs originally intended for the Splunk indexer. + - The default identifier is `SOURCE_SPLUNK_HEC_ADDRESS`. +- Splunk HEC TLS passphrase identifier (when TLS is enabled): + - The default identifier is `SOURCE_SPLUNK_HEC_KEY_PASS`. + +{{% /tab %}} + +{{% tab "Environment Variables" %}} {{% observability_pipelines/configure_existing_pipelines/source_env_vars/splunk_hec %}} +{{% /tab %}} +{{< /tabs >}} + {{% observability_pipelines/log_source_configuration/splunk_hec %}} ## Send logs from the Splunk Distribution of the OpenTelemetry Collector to Observability Pipelines diff --git a/content/en/observability_pipelines/sources/splunk_tcp.md b/content/en/observability_pipelines/sources/splunk_tcp.md index 859c37174b6..36b3c216250 100644 --- a/content/en/observability_pipelines/sources/splunk_tcp.md +++ b/content/en/observability_pipelines/sources/splunk_tcp.md @@ -18,24 +18,40 @@ Use Observability Pipelines' Splunk Heavy and Universal Forwards (TCP) source to Select and set up this source when you [set up a pipeline][1]. The information below is for the source settings in the pipeline UI. -- Enter the identifier for your Splunk TCP address. - - **Note**: Only enter the identifier for the address. Do **not** enter the actual address. - - If left blank, the default is used: `SOURCE_SPLUNK_TCP_ADDRESS`. +
Only enter the identifiers for the Splunk TCP address and, if applicable, the TLS key pass. Do not enter the actual values.
+ +- Enter the identifier for your Splunk TCP address. If you leave it blank, the [default](#set-secrets) is used. ### Optional settings Click the toggle to **Enable TLS**. If you enable TLS, the following certificate and key files are required: -- Enter the identifier for your Splunk TCP key pass. - - **Note**: Only enter the identifier for the key pass. Do **not** enter the actual key pass. - - If left blank, the default is used: `SOURCE_SPLUNK_TCP_KEY_PASS`. +- Enter the identifier for your Splunk TCP key pass. If you leave it blank, the [default](#set-secrets) is used. - `Server Certificate Path`: The path to the certificate file that has been signed by your Certificate Authority (CA) root file in DER or PEM (X.509). -- `CA Certificate Path`: The path to the certificate file that is your Certificate Authority (CA) Root File in either DER or PEM (X.509). +- `CA Certificate Path`: The path to the certificate file that is your Certificate Authority (CA) root file in either DER or PEM (X.509). - `Private Key Path`: The path to the `.key` private key file that belongs to your Server Certificate Path in DER or PEM (PKCS#8) format. -## Set the environment variables +## Set secrets + +{{% observability_pipelines/set_secrets_intro %}} + +{{< tabs >}} +{{% tab "Secrets Management" %}} + +- Splunk TCP address identifier: + - References the socket address, such as `0.0.0.0:9997` on which the Observability Pipelines Worker listens to receive logs from the Splunk Forwarder. + - The default identifier is `SOURCE_SPLUNK_TCP_ADDRESS`. +- Splunk TCP TLS passphrase identifier (when TLS is enabled): + - The default identifier is `SOURCE_SPLUNK_TCP_KEY_PASS`. + +{{% /tab %}} + +{{% tab "Environment Variables" %}} {{% observability_pipelines/configure_existing_pipelines/source_env_vars/splunk_tcp %}} +{{% /tab %}} +{{< /tabs >}} + {{% observability_pipelines/log_source_configuration/splunk_tcp %}} [1]: /observability_pipelines/configuration/set_up_pipelines/ diff --git a/content/en/observability_pipelines/sources/sumo_logic.md b/content/en/observability_pipelines/sources/sumo_logic.md index 2a2ddf2ebc0..4330c221b80 100644 --- a/content/en/observability_pipelines/sources/sumo_logic.md +++ b/content/en/observability_pipelines/sources/sumo_logic.md @@ -18,18 +18,33 @@ Use Observability Pipelines' Sumo Logic Hosted Collector source to receive logs Select and set up this source when you [set up a pipeline][1]. The information below is for the source settings in the pipeline UI. -- Enter the identifier for your Sumo Logic address. +- Enter the identifier for your Sumo Logic address. If you leave it blank, the [default](#set-secrets) is used. - **Note**: Only enter the identifier for the address. Do **not** enter the actual address. - - If left blank, the default is used: `SOURCE_SUMO_LOGIC_ADDRESS`. ### Optional settings In the **Decoding** dropdown menu, select whether your input format is raw **Bytes**, **JSON**, Graylog Extended Log Format (**Gelf**), or **Syslog**. If no decoding is selected, the decoding defaults to JSON. -## Set the environment variables +## Set secrets + +{{% observability_pipelines/set_secrets_intro %}} + +{{< tabs >}} +{{% tab "Secrets Management" %}} + +- Sumo Logic address identifier: + - References the bind address, such as `0.0.0.0:80.`, that your Observability Pipelines Worker listens on to receive logs originally intended for the Sumo Logic HTTP Source. + - The default identifier is `SOURCE_SUMO_LOGIC_ADDRESS`. + +{{% /tab %}} + +{{% tab "Environment Variables" %}} {{% observability_pipelines/configure_existing_pipelines/source_env_vars/sumo_logic %}} +{{% /tab %}} +{{< /tabs >}} + {{% observability_pipelines/log_source_configuration/sumo_logic %}} [1]: /observability_pipelines/configuration/set_up_pipelines/ \ No newline at end of file diff --git a/content/en/observability_pipelines/sources/syslog.md b/content/en/observability_pipelines/sources/syslog.md index 6354f3e817b..0c011e65e19 100644 --- a/content/en/observability_pipelines/sources/syslog.md +++ b/content/en/observability_pipelines/sources/syslog.md @@ -22,25 +22,41 @@ Select and set up this source when you [set up a pipeline][1]. The information b To configure your Syslog source: -1. Enter the identifier for your syslog address. - - **Note**: Only enter the identifier for the address. Do **not** enter the actual address. - - If left blank, the default is used: `SOURCE_SYSLOG_ADDRESS`. +
Only enter the identifiers for the syslog address and, if applicable, the TLS key pass. Do not enter the actual values.
+ +1. Enter the identifier for your syslog address. If you leave it blank, the [default](#set-secrets) is used. 1. In the **Socket Type** dropdown menu, select the communication protocol you want to use: **TCP** or **UDP**. ### Optional settings Toggle the switch to **Enable TLS**. If you enable TLS, the following certificate and key files are required.
**Note**: All file paths are made relative to the configuration data directory, which is `/var/lib/observability-pipelines-worker/config/` by default. See [Advanced Worker Configurations][6] for more information. The file must be owned by the `observability-pipelines-worker group` and `observability-pipelines-worker` user, or at least readable by the group or user. -- Enter the identifier for your syslog key pass. - - **Note**: Only enter the identifier for the key pass. Do **not** enter the actual key pass. - - If left blank, the default is used: `SOURCE_SYSLOG_KEY_PASS`. -- `Server Certificate Path`: The path to the certificate file that has been signed by your Certificate Authority (CA) Root File in DER or PEM (X.509) format. -- `CA Certificate Path`: The path to the certificate file that is your Certificate Authority (CA) Root File in DER or PEM (X.509) format. +- Enter the identifier for your syslog key pass. If you leave it blank, the [default](#set-secrets) is used. +- `Server Certificate Path`: The path to the certificate file that has been signed by your Certificate Authority (CA) root file in DER or PEM (X.509) format. +- `CA Certificate Path`: The path to the certificate file that is your Certificate Authority (CA) root file in DER or PEM (X.509) format. - `Private Key Path`: The path to the `.key` private key file that belongs to your Server Certificate Path in DER or PEM (PKCS#8) format. -## Set the environment variables +## Set secrets + +{{% observability_pipelines/set_secrets_intro %}} + +{{< tabs >}} +{{% tab "Secrets Management" %}} + +- rsyslog or syslog-ng address identifier: + - References the bind address, such `0.0.0.0:9997`, on which the Observability Pipelines Worker listens to receive logs from the Syslog forwarder. + - The default identifier is `SOURCE_SYSLOG_ADDRESS`. +- rsyslog or syslog-ng TLS passphrase identifier (when TLS is enabled): + - The default identifier is `SOURCE_SYSLOG_KEY_PASS`. + +{{% /tab %}} + +{{% tab "Environment Variables" %}} {{% observability_pipelines/configure_existing_pipelines/source_env_vars/syslog %}} +{{% /tab %}} +{{< /tabs >}} + ## Send logs to the Observability Pipelines Worker over syslog {{% observability_pipelines/log_source_configuration/syslog %}}