|
| 1 | +--- |
| 2 | +canonical: https://grafana.com/docs/alloy/latest/reference/components/otelcol/otelcol.receiver.awss3/ |
| 3 | +description: Learn about otelcol.receiver.awss3 |
| 4 | +labels: |
| 5 | + stage: experimental |
| 6 | + products: |
| 7 | + - oss |
| 8 | +title: otelcol.receiver.awss3 |
| 9 | +--- |
| 10 | + |
| 11 | +# `otelcol.receiver.awss3` |
| 12 | + |
| 13 | +{{< docs/shared lookup="stability/experimental.md" source="alloy" version="<ALLOY_VERSION>" >}} |
| 14 | + |
| 15 | +`otelcol.receiver.awss3` receives telemetry stored in S3 by the [AWS S3 Exporter](./otelcol.exporter.awss3.md). |
| 16 | + |
| 17 | +{{< admonition type="warning" >}} |
| 18 | +`otelcol.receiver.awss3` is a wrapper over the upstream OpenTelemetry Collector [`awss3`][] receiver. |
| 19 | +Bug reports or feature requests will be redirected to the upstream repository, if necessary. |
| 20 | + |
| 21 | +[`awss3`]: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/{{< param "OTEL_VERSION" >}}/receiver/awss3receiver |
| 22 | +{{< /admonition >}} |
| 23 | + |
| 24 | +The receiver has two modes of operation: |
| 25 | + |
| 26 | +- **Time Range Mode** - Specify start and end to fetch data from a specific time range. |
| 27 | +- **SQS Message Mode** - Subscribe to SQS messages to process objects as they arrive. |
| 28 | + |
| 29 | +The receiver supports the following encodings: |
| 30 | + |
| 31 | +- `otlp_json` (OpenTelemetry Protocol format represented as JSON) with a suffix of `.json` |
| 32 | +- `otlp_proto` (OpenTelemetry Protocol format represented as Protocol Buffers) with a suffix of `.binpb` |
| 33 | + |
| 34 | +{{< admonition type="note" >}} |
| 35 | +Currently, `otelcol.receiver.awss3` receiver doesn't support encoding extensions. |
| 36 | +{{< /admonition >}} |
| 37 | + |
| 38 | +You can specify multiple `otelcol.receiver.awss3` components by giving them different labels. |
| 39 | + |
| 40 | +## Usage |
| 41 | + |
| 42 | +```alloy |
| 43 | +otelcol.receiver.awss3 "<LABEL>" { |
| 44 | + start_time = "..." |
| 45 | + end_time = "..." |
| 46 | +
|
| 47 | + s3downloader { |
| 48 | + s3_bucket = "..." |
| 49 | + s3_prefix = "..." |
| 50 | + } |
| 51 | +
|
| 52 | + output { |
| 53 | + logs = [...] |
| 54 | + metrics = [...] |
| 55 | + trace = [...] |
| 56 | + } |
| 57 | +} |
| 58 | +``` |
| 59 | + |
| 60 | +## Arguments |
| 61 | + |
| 62 | +You can use the following arguments with `otelcol.receiver.awss3`: |
| 63 | + |
| 64 | +| Name | Type | Description | Default | Required | |
| 65 | +| ------------ | -------- | ------------------------------------------- | ------- | ----------------------------- | |
| 66 | +| `start_time` | `string` | The time at which to start retrieving data. | | Required if fetching by time. | |
| 67 | +| `end_time` | `string` | The time at which to stop retrieving data. | | Required if fetching by time. | |
| 68 | + |
| 69 | +The `start_time` and `end_time` fields use one of the following time formats: RFC3339, `YYYY-MM-DD HH:MM`, or `YYYY-MM-DD`. When using `YYYY-MM-DD`, the time defaults to `00:00`. |
| 70 | + |
| 71 | +{{< admonition type="note" >}} |
| 72 | +Time-based configuration (`start_time` and `end_time` arguments) can't be combined together with [`sqs`][] block. |
| 73 | + |
| 74 | +[`sqs`]: #sqs |
| 75 | + |
| 76 | +{{< /admonition >}} |
| 77 | + |
| 78 | +Refer to the upstream receiver [documentation](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/{{< param "OTEL_VERSION" >}}/receiver/awss3receiver#configuration) for more details. |
| 79 | + |
| 80 | +## Blocks |
| 81 | + |
| 82 | +You can use the following blocks with `otelcol.receiver.awss3`: |
| 83 | + |
| 84 | +| Block | Description | Required | |
| 85 | +| ------------------------------ | ---------------------------------------------------------------------------- | ----------------------------------------- | |
| 86 | +| [`s3downloader`][s3downloader] | Configures S3 downloader. | yes | |
| 87 | +| [`sqs`][sqs] | Configures SQS queue configuration for receiving object change notification. | Required if fetching by SQS notification. | |
| 88 | +| [`output`][output] | Configures where to send received telemetry data. | yes | |
| 89 | + |
| 90 | +[s3downloader]: #s3downloader |
| 91 | +[sqs]: #sqs |
| 92 | +[output]: #output |
| 93 | + |
| 94 | +### `s3downloader` |
| 95 | + |
| 96 | +{{< badge text="Required" >}} |
| 97 | + |
| 98 | +The `s3downloader` block contains AWS S3 downloader related configuration to control things like bucket, prefix, batching, connections, retries, etc. |
| 99 | + |
| 100 | +The following arguments are supported: |
| 101 | + |
| 102 | +| Name | Type | Description | Default | Required | |
| 103 | +| ----------------------- | -------- | ----------------------------------------------------------------------------------------------------- | ------------- | -------- | |
| 104 | +| `s3_bucket` | `string` | S3 bucket. | | yes | |
| 105 | +| `s3_prefix` | `string` | Prefix for the S3 key (root directory inside bucket). | | yes | |
| 106 | +| `endpoint_partition_id` | `string` | Partition id to use if `endpoint` is specified. | `"aws"` | no | |
| 107 | +| `endpoint` | `string` | Overrides the endpoint used by the exporter instead of constructing it from `region` and `s3_bucket`. | | no | |
| 108 | +| `file_prefix` | `string` | Prefix used to filter files for download. | | no | |
| 109 | +| `region` | `string` | AWS region. | `"us-east-1"` | no | |
| 110 | +| `s3_force_path_style` | `bool` | When enabled, forces the request to use [path-style addressing][s3-force-path-style-ref]. | `false` | no | |
| 111 | +| `s3_partition` | `string` | Time granularity of S3 key: hour or minute. | `"minute"` | no | |
| 112 | + |
| 113 | +[s3-force-path-style-ref]: http://docs.aws.amazon.com/AmazonS3/latest/dev/VirtualHosting.html |
| 114 | + |
| 115 | +### `sqs` |
| 116 | + |
| 117 | +The `sqs` block holds SQS queue configuration for receiving object change notifications. |
| 118 | + |
| 119 | +The following arguments are supported: |
| 120 | + |
| 121 | +| Name | Type | Description | Default | Required | |
| 122 | +| ------------------------ | -------- | --------------------------------------------------------------- | ------- | -------- | |
| 123 | +| `queue_url` | `string` | The URL of the SQS queue that receives S3 bucket notifications. | | yes | |
| 124 | +| `region` | `string` | AWS region of the SQS queue. | | yes | |
| 125 | +| `endpoint` | `string` | Custom endpoint for the SQS service. | | no | |
| 126 | +| `max_number_of_messages` | `int` | Maximum number of messages to retrieve in a single SQS request. | `10` | no | |
| 127 | +| `wait_time_seconds` | `int` | Wait time in seconds for long polling SQS requests. | `20` | no | |
| 128 | + |
| 129 | +{{< admonition type="note" >}} |
| 130 | +You must configure your S3 bucket to send event notifications to the SQS queue. |
| 131 | +Time-based configuration (`start_time`/`end_time`) and SQS configuration can't be used together. |
| 132 | +{{< /admonition >}} |
| 133 | + |
| 134 | +### `output` |
| 135 | + |
| 136 | +{{< badge text="Required" >}} |
| 137 | + |
| 138 | +{{< docs/shared lookup="reference/components/output-block.md" source="alloy" version="<ALLOY_VERSION>" >}} |
| 139 | + |
| 140 | +## Exported fields |
| 141 | + |
| 142 | +`otelcol.receiver.awss3` doesn't export any fields. |
| 143 | + |
| 144 | +## Component health |
| 145 | + |
| 146 | +`otelcol.receiver.awss3` is only reported as unhealthy if given an invalid configuration. |
| 147 | + |
| 148 | +## Debug information |
| 149 | + |
| 150 | +`otelcol.receiver.awss3` doesn't expose any component-specific debug information. |
| 151 | + |
| 152 | +## Example |
| 153 | + |
| 154 | +This example forwards received traces through a batch processor before finally sending it to an OTLP-capable endpoint: |
| 155 | + |
| 156 | +```alloy |
| 157 | +// Time range mode: |
| 158 | +otelcol.receiver.awss3 "default" { |
| 159 | + start_time = "2024-01-01 01:00" |
| 160 | + end_time = "2024-01-02" |
| 161 | +
|
| 162 | + s3downloader { |
| 163 | + region = "us-west-1" |
| 164 | + s3_bucket = "mybucket" |
| 165 | + s3_prefix = "trace" |
| 166 | + s3_partition = "minute" |
| 167 | + } |
| 168 | +
|
| 169 | + output { |
| 170 | + traces = [otelcol.processor.batch.default.input] |
| 171 | + } |
| 172 | +} |
| 173 | +
|
| 174 | +// SQS message mode: |
| 175 | +otelcol.receiver.awss3 "sqs_traces" { |
| 176 | + s3downloader { |
| 177 | + region = "us-east-1" |
| 178 | + s3_bucket = "mybucket" |
| 179 | + s3_prefix = "mytrace" |
| 180 | + } |
| 181 | +
|
| 182 | + sqs { |
| 183 | + queue_url = "https://sqs.us-east-1.amazonaws.com/123456789012/test-queue" |
| 184 | + region = "us-east-1" |
| 185 | + } |
| 186 | +
|
| 187 | + output { |
| 188 | + traces = [otelcol.processor.batch.default.input] |
| 189 | + } |
| 190 | +} |
| 191 | +
|
| 192 | +otelcol.processor.batch "default" { |
| 193 | + output { |
| 194 | + metrics = [otelcol.exporter.otlp.default.input] |
| 195 | + logs = [otelcol.exporter.otlp.default.input] |
| 196 | + traces = [otelcol.exporter.otlp.default.input] |
| 197 | + } |
| 198 | +} |
| 199 | +
|
| 200 | +otelcol.exporter.otlp "default" { |
| 201 | + client { |
| 202 | + endpoint = sys.env("<OTLP_ENDPOINT>") |
| 203 | + } |
| 204 | +} |
| 205 | +``` |
| 206 | + |
| 207 | +<!-- START GENERATED COMPATIBLE COMPONENTS --> |
| 208 | + |
| 209 | +## Compatible components |
| 210 | + |
| 211 | +`otelcol.receiver.awss3` can accept arguments from the following components: |
| 212 | + |
| 213 | +- Components that export [OpenTelemetry `otelcol.Consumer`](../../../compatibility/#opentelemetry-otelcolconsumer-exporters) |
| 214 | + |
| 215 | + |
| 216 | +{{< admonition type="note" >}} |
| 217 | +Connecting some components may not be sensible or components may require further configuration to make the connection work correctly. |
| 218 | +Refer to the linked documentation for more details. |
| 219 | +{{< /admonition >}} |
| 220 | + |
| 221 | +<!-- END GENERATED COMPATIBLE COMPONENTS --> |
0 commit comments