Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -18,9 +18,9 @@ Use Observability Pipelines' Amazon Data Firehose source to receive logs from Am

Select and set up this source when you [set up a pipeline][1]. The information below is for the source settings in the pipeline UI.

- Enter the identifier for your Amazon Data Firehose address.
- **Note**: Only enter the identifier for the address. Do **not** enter the actual address.
- If left blank, the default is used: `SOURCE_AWS_DATA_FIREHOSE_ADDRESS`.
<div class="alert alert-danger">Only enter the identifiers for the Amazon Data Firehose address and, if applicable, the TLS key pass. Do <b>not</b> enter the actual values.</div>

- Enter the identifier for your Amazon Data Firehose address. If you leave it blank, the [default](#set-secrets) is used.

### Optional settings

Expand All @@ -33,17 +33,33 @@ Select an **AWS authentication** option. If you select **Assume role**:
#### Enable TLS

Toggle the switch to **Enable TLS**. If you enable TLS, the following certificate and key files are required.<br>**Note**: All file paths are made relative to the configuration data directory, which is `/var/lib/observability-pipelines-worker/config/` by default. See [Advanced Worker Configurations][2] for more information. The file must be owned by the `observability-pipelines-worker group` and `observability-pipelines-worker` user, or at least readable by the group or user.
- Enter the identifier for your Amazon Data Firehose key pass.
- **Note**: Only enter the identifier for the key pass. Do **not** enter the actual key pass.
- If left blank, the default is used: `SOURCE_AWS_DATA_FIREHOSE_KEY_PASS`.
- Enter the identifier for your Amazon Data Firehose key pass. If you leave it blank, the [default](#set-secrets) is used.
- `Server Certificate Path`: The path to the certificate file that has been signed by your Certificate Authority (CA) root file in DER or PEM (X.509).
- `CA Certificate Path`: The path to the certificate file that is your Certificate Authority (CA) Root File in DER or PEM (X.509).
- `CA Certificate Path`: The path to the certificate file that is your Certificate Authority (CA) root file in DER or PEM (X.509).
- `Private Key Path`: The path to the `.key` private key file that belongs to your Server Certificate Path in DER or PEM (PKCS#8) format.

## Set the environment variables
## Set secrets

{{% observability_pipelines/set_secrets_intro %}}

{{< tabs >}}
{{% tab "Secrets Management" %}}

- Amazon Data Firehose address identifier:
- References the socket address on which the Observability Pipelines Worker listens to receive logs.
- The default identifier is `SOURCE_AWS_DATA_FIREHOSE_ADDRESS`.
- Amazon Data Firehose TLS passphrase identifier (when TLS is enabled):
- The default identifier is `SOURCE_AWS_DATA_FIREHOSE_KEY_PASS`.

{{% /tab %}}

{{% tab "Environment variables" %}}

{{% observability_pipelines/configure_existing_pipelines/source_env_vars/amazon_data_firehose %}}

{{% /tab %}}
{{< /tabs >}}

## Send logs to the Observability Pipelines Worker over Amazon Data Firehose

{{% observability_pipelines/log_source_configuration/amazon_data_firehose %}}
Expand Down
32 changes: 24 additions & 8 deletions content/en/observability_pipelines/sources/amazon_s3.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,9 +13,9 @@ Use Observability Pipelines' Amazon S3 source to receive logs from Amazon S3. Se

Select and set up this source when you [set up a pipeline][1]. The information below is for the source settings in the pipeline UI.

1. Enter the identifier for your Amazon S3 URL.
- **Note**: Only enter the identifier for the URL. Do **not** enter the actual URL.
- If left blank, the default is used: `SOURCE_AWS_S3_SQS_URL`.
<div class="alert alert-danger">Only enter the identifiers for the Amazon S3 URL and, if applicable, the TLS key pass. Do <b>not</b> enter the actual values.</div>

1. Enter the identifier for your Amazon S3 URL. If you leave it blank, the [default](#set-secrets) is used.
1. Enter the AWS region.

### Optional settings
Expand All @@ -29,17 +29,33 @@ Select an **AWS authentication** option. If you select **Assume role**:
#### Enable TLS

Toggle the switch to **Enable TLS**. If you enable TLS, the following certificate and key files are required.<br>**Note**: All file paths are made relative to the configuration data directory, which is `/var/lib/observability-pipelines-worker/config/` by default. See [Advanced Worker Configurations][2] for more information. The file must be owned by the `observability-pipelines-worker group` and `observability-pipelines-worker` user, or at least readable by the group or user.
- Enter the identifier for your Amazon S3 key pass.
- **Note**: Only enter the identifier for the key pass. Do **not** enter the actual key pass.
- If left blank, the default is used: `SOURCE_AWS_S3_KEY_PASS`.
- Enter the identifier for your Amazon S3 key pass. If you leave it blank, the [default](#set-secrets) is used.
- `Server Certificate Path`: The path to the certificate file that has been signed by your Certificate Authority (CA) root file in DER or PEM (X.509).
- `CA Certificate Path`: The path to the certificate file that is your Certificate Authority (CA) Root File in DER or PEM (X.509).
- `CA Certificate Path`: The path to the certificate file that is your Certificate Authority (CA) root file in DER or PEM (X.509).
- `Private Key Path`: The path to the `.key` private key file that belongs to your Server Certificate Path in DER or PEM (PKCS#8) format.

## Set the environment variables
## Set secrets

{{% observability_pipelines/set_secrets_intro %}}

{{< tabs >}}
{{% tab "Secrets Management" %}}

- Amazon S3 URL identifier:
- References the URL of the SQS queue to which the S3 bucket sends the notification events.
- The default identifier is `SOURCE_AWS_S3_SQS_URL`.
- Amazon S3 TLS passphrase identifier (when TLS is enabled):
- The default identifier is `SOURCE_AWS_S3_KEY_PASS`.

{{% /tab %}}

{{% tab "Environment Variables" %}}

{{% observability_pipelines/configure_existing_pipelines/source_env_vars/amazon_s3 %}}

{{% /tab %}}
{{< /tabs >}}

## AWS Authentication

{{% observability_pipelines/aws_authentication/instructions %}}
Expand Down
4 changes: 2 additions & 2 deletions content/en/observability_pipelines/sources/datadog_agent.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,8 +25,8 @@ Use Observability Pipelines' Datadog Agent source to receive logs or metrics ({{
## Set up the source in the pipeline UI

Optionally, toggle the switch to enable TLS. If you enable TLS, the following certificate and key files are required.
- `Server Certificate Path`: The path to the certificate file that has been signed by your Certificate Authority (CA) Root File in DER or PEM (X.509) format.
- `CA Certificate Path`: The path to the certificate file that is your Certificate Authority (CA) Root File in DER or PEM (X.509) format.
- `Server Certificate Path`: The path to the certificate file that has been signed by your Certificate Authority (CA) root file in DER or PEM (X.509) format.
- `CA Certificate Path`: The path to the certificate file that is your Certificate Authority (CA) root file in DER or PEM (X.509) format.
- `Private Key Path`: The path to the `.key` private key file that belongs to your Server Certificate Path in DER or PEM (PKCS#8) format.

**Note**: All file paths are made relative to the configuration data directory, which is `/var/lib/observability-pipelines-worker/config/` by default. See [Advanced Worker Configurations][5] for more information. The file must be owned by the `observability-pipelines-worker group` and `observability-pipelines-worker` user, or at least readable by the group or user.
Expand Down
34 changes: 25 additions & 9 deletions content/en/observability_pipelines/sources/fluent.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,24 +18,40 @@ Use Observability Pipelines' Fluentd or Fluent Bit source to receive logs from t

Select and set up this source when you [set up a pipeline][1]. The information below are for the source settings in the pipeline UI.

- 1. Enter the identifier for your Fluent address.
- **Note**: Only enter the identifier for the address. Do **not** enter the actual address.
- If left blank, the default is used: `SOURCE_FLUENT_ADDRESS`.
<div class="alert alert-danger">Only enter the identifiers for the Fluent address and, if applicable, the TLS key pass. Do <b>not</b> enter the actual values.</div>

1. Enter the identifier for your Fluent address. If you leave it blank, the [default](#set-secrets) is used.

### Optional settings

Toggle the switch to **Enable TLS**. If you enable TLS, the following certificate and key files are required.<br>**Note**: All file paths are made relative to the configuration data directory, which is `/var/lib/observability-pipelines-worker/config/` by default. See [Advanced Worker Configurations][2] for more information. The file must be owned by the `observability-pipelines-worker group` and `observability-pipelines-worker` user, or at least readable by the group or user.
- Enter the identifier for your Fluent key pass.
- **Note**: Only enter the identifier for the key pass. Do **not** enter the actual key pass.
- If left blank, the default is used: `SOURCE_FLUENT_KEY_PASS`.
- `Server Certificate Path`: The path to the certificate file that has been signed by your Certificate Authority (CA) Root File in DER or PEM (X.509) format.
- `CA Certificate Path`: The path to the certificate file that is your Certificate Authority (CA) Root File in DER or PEM (X.509) format.
- Enter the identifier for your Fluent key pass. If you leave it blank, the [default](#set-secrets) is used.
- `Server Certificate Path`: The path to the certificate file that has been signed by your Certificate Authority (CA) root file in DER or PEM (X.509) format.
- `CA Certificate Path`: The path to the certificate file that is your Certificate Authority (CA) root file in DER or PEM (X.509) format.
- `Private Key Path`: The path to the `.key` private key file that belongs to your Server Certificate Path in DER or PEM (PKCS#8) format.

## Set the environment variables
## Set secrets

{{% observability_pipelines/set_secrets_intro %}}

{{< tabs >}}
{{% tab "Secrets Management" %}}

- Fluent address identifier:
- References the address on which the Observability Pipelines Worker listens for incoming log messages.
- The default identifier is `SOURCE_FLUENT_ADDRESS`.
- Fluent TLS passphrase identifier (when TLS is enabled):
- The default identifier is `SOURCE_FLUENT_KEY_PASS`.

{{% /tab %}}

{{% tab "Environment Variables" %}}

{{% observability_pipelines/configure_existing_pipelines/source_env_vars/fluent %}}

{{% /tab %}}
{{< /tabs >}}

## Send logs to the Observability Pipelines Worker over Fluent

{{% observability_pipelines/log_source_configuration/fluent %}}
Expand Down
18 changes: 16 additions & 2 deletions content/en/observability_pipelines/sources/google_pubsub.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,13 +26,27 @@ Select and set up this source when you [set up a pipeline][1]. The information b
1. Select the decoder you want to use (Bytes, GELF, JSON, syslog).
1. Optionally, toggle the switch to enable TLS. If you enable TLS, the following certificate and key files are required.<br>**Note**: All file paths are made relative to the configuration data directory, which is `/var/lib/observability-pipelines-worker/config/` by default. See [Advanced Worker Configurations][3] for more information. The file must be owned by the `observability-pipelines-worker group` and `observability-pipelines-worker` user, or at least readable by the group or user.
- `Server Certificate Path`: The path to the certificate file that has been signed by your Certificate Authority (CA) root file in DER or PEM (X.509).
- `CA Certificate Path`: The path to the certificate file that is your Certificate Authority (CA) Root File in DER or PEM (X.509).
- `CA Certificate Path`: The path to the certificate file that is your Certificate Authority (CA) root file in DER or PEM (X.509).
- `Private Key Path`: The path to the `.key` private key file that belongs to your Server Certificate Path in DER or PEM (PKCS #8) format.

## Set the environment variables
## Set secrets

{{% observability_pipelines/set_secrets_intro %}}

{{< tabs >}}
{{% tab "Secrets Management" %}}

There are no default secret identifiers for this source.

{{% /tab %}}

{{% tab "Environment Variables" %}}

{{% observability_pipelines/configure_existing_pipelines/source_env_vars/google_pubsub %}}

{{% /tab %}}
{{< /tabs >}}

[1]: /observability_pipelines/configuration/set_up_pipelines/
[2]: https://cloud.google.com/kubernetes-engine/docs/concepts/workload-identity
[3]: /observability_pipelines/configuration/install_the_worker/advanced_worker_configurations/
Expand Down
52 changes: 38 additions & 14 deletions content/en/observability_pipelines/sources/http_client.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,28 +20,26 @@ Select and set up this source when you [set up a pipeline][1]. The information b

To configure your HTTP/S Client source:

1. Enter the identifier for your HTTP Client endpoint URL.
- **Note**: Only enter the identifier for the endpoint URL. Do **not** enter the actual endpoint URL.
- If left blank, the default is used: `SOURCE_HTTP_CLIENT_ENDPOINT_URL`.

<div class="alert alert-danger">Only enter the identifiers for the HTTP Client endpoint URL and, if applicable, your authorization strategy secrets. Do <b>not</b> enter the actual values.</div>

1. Enter the identifier for your HTTP Client endpoint URL. If you leave it blank, the [default](#set-secrets) is used.
1. Select your authorization strategy. If you selected:
- **Basic**:
- Enter the identifier for your HTTP Client username.
- If left blank, the default is used: `SOURCE_HTTP_CLIENT_USERNAME`.
Enter the identifier for your HTTP Client password.
- If left blank, the default is used: `SOURCE_HTTP_CLIENT_PASSWORD`.
- **Bearer**: Enter the identifier for your bearer token.
- If left blank, the default is used: `SOURCE_HTTP_CLIENT_BEARER_TOKEN`.
- Enter the identifier for your HTTP Client username. If you leave it blank, the [default](#set-secrets) is used.
- Enter the identifier for your HTTP Client password. If you leave it blank, the [default](#set-secrets) is used.
- **Bearer**: Enter the identifier for your bearer token. If you leave it blank, the [default](#set-secrets) is used.
1. Select the decoder you want to use on the HTTP messages. Logs pulled from the HTTP source must be in this format.

### Optional settings

#### Enable TLS

Toggle the switch to **Enable TLS**. If you enable TLS, the following certificate and key files are required.<br>**Note**: All file paths are made relative to the configuration data directory, which is `/var/lib/observability-pipelines-worker/config/` by default. See [Advanced Worker Configurations][2] for more information. The file must be owned by the `observability-pipelines-worker group` and `observability-pipelines-worker` user, or at least readable by the group or user.
- Enter the identifier for your HTTP Client key pass.
- Enter the identifier for your HTTP Client key pass. If you leave it blank, the [default](#set-secrets) is used.
- **Note**: Only enter the identifier for the key pass. Do **not** enter the actual key pass.
- If left blank, the default is used: `SOURCE_HTTP_CLIENT_KEY_PASS`
- `Server Certificate Path`: The path to the certificate file that has been signed by your Certificate Authority (CA) Root File in DER or PEM (X.509) format.
- `CA Certificate Path`: The path to the certificate file that is your Certificate Authority (CA) Root File in DER or PEM (X.509) format.
- `Server Certificate Path`: The path to the certificate file that has been signed by your Certificate Authority (CA) root file in DER or PEM (X.509) format.
- `CA Certificate Path`: The path to the certificate file that is your Certificate Authority (CA) root file in DER or PEM (X.509) format.
- `Private Key Path`: The path to the `.key` private key file that belongs to your Server Certificate Path in DER or PEM (PKCS#8) format.

#### Scrape settings
Expand All @@ -51,9 +49,35 @@ Toggle the switch to **Enable TLS**. If you enable TLS, the following certificat
- Since requests run concurrently, if a scrape takes longer than the interval given, a new scrape is started, which can consume extra resources. Set the timeout to a value lower than the scrape interval to prevent this from happening.
- Enter the timeout for each scrape request.

## Set the environment variables
## Set secrets

{{% observability_pipelines/set_secrets_intro %}}

{{< tabs >}}
{{% tab "Secrets Management" %}}

- HTTP Client endpoint URL identifier:
- References the endpoint from which the Observability Pipelines Worker collects log events.
- The default identifier is `SOURCE_HTTP_CLIENT_ENDPOINT_URL`.
- HTTP Client TLS passphrase identifier (when TLS is enabled):
- The default identifier is `SOURCE_HTTP_CLIENT_KEY_PASS`.
- If you are using basic authentication:
- HTTP Client username identifier:
- The default identifier is `SOURCE_HTTP_CLIENT_USERNAME`.
- HTTP Client password identifier:
- The default identifier is `SOURCE_HTTP_CLIENT_PASSWORD`.
- If you are using bearer authentication:
- HTTP Client bearer token identifier:
- The default identifier is `SOURCE_HTTP_CLIENT_BEARER_TOKEN`.

{{% /tab %}}

{{% tab "Environment Variables" %}}

{{% observability_pipelines/configure_existing_pipelines/source_env_vars/http_client %}}

{{% /tab %}}
{{< /tabs >}}

[1]: /observability_pipelines/configuration/set_up_pipelines/
[2]: /observability_pipelines/configuration/install_the_worker/advanced_worker_configurations/
Loading
Loading