diff --git a/content/en/docs/collector/custom-collector.md b/content/en/docs/collector/custom-collector.md index 3bfcaf17b198..2594438a928d 100644 --- a/content/en/docs/collector/custom-collector.md +++ b/content/en/docs/collector/custom-collector.md @@ -5,12 +5,12 @@ weight: 29 cSpell:ignore: chipset darwin debugexporter gomod otlpexporter otlpreceiver wyrtw --- -If you are planning to build and debug custom collector receivers, processors, -extensions, or exporters, you are going to need your own Collector instance. -That will allow you to launch and debug your OpenTelemetry Collector components +If you are building and debugging custom Collector receivers, processors, +extensions, or exporters, you need your own Collector instance. +This allows you to launch and debug your OpenTelemetry Collector components directly within your favorite Golang IDE. -The other interesting aspect of approaching the component development this way +Another benefit of this approach is that you can use all the debugging features from your IDE (stack traces are great teachers!) to understand how the Collector itself interacts with your component code. @@ -20,13 +20,12 @@ builder][ocb] (or `ocb` for short) to assist people in assembling their own distribution, making it easy to build a distribution that includes their custom components along with components that are publicly available. -As part of the process the `ocb` will generate the Collector's source code, -which you can use to help build and debug your own custom components, so let's -get started. +As part of the process, the `ocb` generates the Collector's source code, +which you can use to build and debug your own custom components. ## Step 1 - Install the builder -{{% alert color="primary" title="Note" %}} +{{% alert title="Note" %}} The `ocb` tool requires Go to build the Collector distribution. [Install Go](https://go.dev/doc/install) on your machine, if you haven't done so @@ -35,17 +34,22 @@ already. {{% /alert %}} The `ocb` binary is available as a downloadable asset from OpenTelemetry -Collector [releases with `cmd/builder` tags][tags]. You will find a list of +Collector [releases with `cmd/builder` tags](https://github.com/open-telemetry/opentelemetry-collector-releases/tags). You can find a list of assets named based on OS and chipset, so download the one that fits your configuration: -{{< tabpane text=true >}} +- Linux (AMD 64) +- Linux (ARM 64) +- Linux (ppc64le) +- macOS (AMD 64) +- macOS (ARM 64) +- Windows (AMD 64) -{{% tab "Linux (AMD 64)" %}} +{{< tabpane text=true >}} {{% tab "Linux (AMD 64)" %}} ```sh curl --proto '=https' --tlsv1.2 -fL -o ocb \ -https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/cmd%2Fbuilder%2F{{% version-from-registry collector-builder %}}/ocb_{{% version-from-registry collector-builder noPrefix %}}_linux_amd64 +https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/cmd%2Fbuilder%2Fv0.139.0/ocb_0.139.0_linux_amd64 chmod +x ocb ``` @@ -53,15 +57,15 @@ chmod +x ocb ```sh curl --proto '=https' --tlsv1.2 -fL -o ocb \ -https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/cmd%2Fbuilder%2F{{% version-from-registry collector-builder %}}/ocb_{{% version-from-registry collector-builder noPrefix %}}_linux_arm64 +https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/cmd%2Fbuilder%2Fv0.139.0/ocb_0.139.0_linux_arm64 chmod +x ocb ``` -{{% /tab %}} {{% tab "Linux (ppc64le) "%}} +{{% /tab %}} {{% tab "Linux (ppc64le)" %}} ```sh curl --proto '=https' --tlsv1.2 -fL -o ocb \ -https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/cmd%2Fbuilder%2F{{% version-from-registry collector-builder %}}/ocb_{{% version-from-registry collector-builder noPrefix %}}_linux_ppc64le +https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/cmd%2Fbuilder%2Fv0.139.0/ocb_0.139.0_linux_ppc64le chmod +x ocb ``` @@ -69,7 +73,7 @@ chmod +x ocb ```sh curl --proto '=https' --tlsv1.2 -fL -o ocb \ -https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/cmd%2Fbuilder%2F{{% version-from-registry collector-builder %}}/ocb_{{% version-from-registry collector-builder noPrefix %}}_darwin_amd64 +https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/cmd%2Fbuilder%2Fv0.139.0/ocb_0.139.0_darwin_amd64 chmod +x ocb ``` @@ -77,51 +81,51 @@ chmod +x ocb ```sh curl --proto '=https' --tlsv1.2 -fL -o ocb \ -https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/cmd%2Fbuilder%2F{{% version-from-registry collector-builder %}}/ocb_{{% version-from-registry collector-builder noPrefix %}}_darwin_arm64 +https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/cmd%2Fbuilder%2Fv0.139.0/ocb_0.139.0_darwin_arm64 chmod +x ocb ``` -{{% /tab %}} {{% tab "Windows (AMD 64)" %}} +{{% /tab %}} {{% tab "Windows" %}} -```sh -Invoke-WebRequest -Uri "https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/cmd%2Fbuilder%2F{{% version-from-registry collector-builder %}}/ocb_{{% version-from-registry collector-builder noPrefix %}}_windows_amd64.exe" -OutFile "ocb.exe" +```powershell +Invoke-WebRequest -Uri "https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/cmd%2Fbuilder%2Fv0.139.0/ocb_0.139.0_windows_amd64.exe" -OutFile "ocb.exe" Unblock-File -Path "ocb.exe" ``` {{% /tab %}} {{< /tabpane >}} -To make sure the `ocb` is ready to be used, go to your terminal and type -`./ocb help`, and once you hit enter you should have the output of the `help` -command showing up in your console. +To make sure the `ocb` is ready to use, go to your terminal and type +`./ocb help`. When you press enter, you should see the output of the `help` +command in your console. ## Step 2 - Create a builder manifest file -The builder's `manifest` file is a `yaml` where you pass information about the -code generation and compile process combined with the components that you would +The builder's `manifest` file is a YAML file where you provide information about the +code generation and compile process combined with the components you would like to add to your Collector's distribution. The `manifest` starts with a map named `dist` which contains tags to help you configure the code generation and compile process. In fact, all the tags for -`dist` are the equivalent of the `ocb` command line `flags`. +`dist` are equivalent to the `ocb` command line `flags`. Here are the tags for the `dist` map: -| Tag | Description | Optional | Default Value | -| ------------ | -------------------------------------------------------------------------------------------------- | -------- | --------------------------------------------------------------------------------- | -| module: | The module name for the new distribution, following Go mod conventions. Optional, but recommended. | Yes | `go.opentelemetry.io/collector/cmd/builder` | -| name: | The binary name for your distribution | Yes | `otelcol-custom` | -| description: | A long name for the application. | Yes | `Custom OpenTelemetry Collector distribution` | -| output_path: | The path to write the output (sources and binary). | Yes | `/var/folders/86/s7l1czb16g124tng0d7wyrtw0000gn/T/otelcol-distribution3618633831` | -| version: | The version for your custom OpenTelemetry Collector. | Yes | `1.0.0` | -| go: | Which Go binary to use to compile the generated sources. | Yes | go from the PATH | - -As you can see on the table above, all the `dist` tags are optional, so you will -be adding custom values for them depending if your intentions to make your +| Tag | Description | Optional | Default Value | +| ---------------- | -------------------------------------------------------------------------- | -------- | ---------------------------------------------------------------------------------------- | +| module: | The module name for the new distribution, following Go mod conventions. | Yes | `go.opentelemetry.io/collector/cmd/builder` | +| name: | The binary name for your distribution | Yes | `otelcol-custom` | +| description: | A long name for the application. | Yes | `Custom OpenTelemetry Collector distribution` | +| output_path: | The path to write the output (sources and binary). | Yes | `/var/folders/86/s7l1czb16g124tng0d7wyrtw0000gn/T/otelcol-distribution3618633831` | +| version: | The version for your custom OpenTelemetry Collector. | Yes | `1.0.0` | +| go: | Which Go binary to use to compile the generated sources. | Yes | go from the PATH | + +As you can see in the table above, all the `dist` tags are optional. You add +custom values for them depending on whether you intend to make your custom Collector distribution available for consumption by other users or if you -are simply leveraging the `ocb` to bootstrap your component development and +are simply using the `ocb` to bootstrap your component development and testing environment. -For this tutorial, you will be creating a Collector's distribution to support +For this tutorial, you create a Collector's distribution to support the development and testing of components. Go ahead and create a manifest file named `builder-config.yaml` with the @@ -134,22 +138,21 @@ dist: output_path: ./otelcol-dev ``` -Now you need to add the modules representing the components you want to be +Now add the modules representing the components you want incorporated in this custom Collector distribution. Take a look at the [ocb configuration documentation](https://github.com/open-telemetry/opentelemetry-collector/tree/main/cmd/builder#configuration) to understand the different modules and how to add the components. -We will be adding the following components to our development and testing -collector distribution: +Add the following components to the development and testing +Collector distribution: - Exporters: OTLP and Debug - Receivers: OTLP - Processors: Batch -The `builder-config.yaml` manifest file will look like this after adding the +The `builder-config.yaml` manifest file looks like this after adding the components: - ```yaml dist: name: otelcol-dev @@ -178,9 +181,9 @@ providers: - gomod: go.opentelemetry.io/collector/confmap/provider/yamlprovider v1.18.0 ``` -{{% alert color="primary" title="Tip" %}} +{{% alert title="Tip" %}} -For a list of components that you can add to your custom collector, see the +For a list of components that you can add to your custom Collector, see the [OpenTelemetry Registry](/ecosystem/registry/?language=collector). Note that registry entries provide the full name and version you need to add to your `builder-config.yaml`. @@ -189,27 +192,27 @@ registry entries provide the full name and version you need to add to your ## Step 3a - Generate the code and build your Collector's distribution -{{% alert color="primary" title="Note" %}} +{{% alert title="Note" %}} -This step is used to build your custom collector distribution using the `ocb` +This step builds your custom Collector distribution using the `ocb` binary. If you would like to build and deploy your custom Collector distribution to a container orchestrator (for example, Kubernetes), skip this step and go to [Step 3b](#step-3b---containerize-your-collectors-distribution). {{% /alert %}} -All you need now is to let the `ocb` do its job, so go to your terminal and type +Now let the `ocb` do its job. Go to your terminal and type the following command: -```cmd +```sh ./ocb --config builder-config.yaml ``` -If everything went well, here is what the output of the command should look +If everything went well, here is what the output of the command looks like: -```nocode -2022-06-13T14:25:03.037-0500 INFO internal/command.go:85 OpenTelemetry Collector distribution builder {"version": "{{% version-from-registry collector-builder noPrefix %}}", "date": "2023-01-03T15:05:37Z"} +```text +2022-06-13T14:25:03.037-0500 INFO internal/command.go:85 OpenTelemetry Collector distribution builder {"version": "0.139.0", "date": "2023-01-03T15:05:37Z"} 2022-06-13T14:25:03.039-0500 INFO internal/command.go:108 Using config file {"path": "builder-config.yaml"} 2022-06-13T14:25:03.040-0500 INFO builder/config.go:99 Using go {"go-executable": "/usr/local/go/bin/go"} 2022-06-13T14:25:03.041-0500 INFO builder/main.go:76 Sources created {"path": "./otelcol-dev"} @@ -222,9 +225,9 @@ As defined in the `dist` section of your config file, you now have a folder named `otelcol-dev` containing all the source code and the binary for your Collector's distribution. -The folder structure should look like this: +The folder structure looks like this: -```console +```text . ├── builder-config.yaml ├── ocb @@ -240,22 +243,22 @@ The folder structure should look like this: ``` You can now use the generated code to bootstrap your component development -projects and easily build and distribute your own collector distribution with +projects and easily build and distribute your own Collector distribution with your components. ## Step 3b - Containerize your Collector's distribution -{{% alert color="primary" title="Note" %}} +{{% alert title="Note" %}} -This step will build your collector distribution inside a `Dockerfile`. Follow +This step builds your Collector distribution inside a `Dockerfile`. Follow this step if you need to deploy your Collector distribution to a container -orchestrator (for example, Kubernetes). If you would like to _only_ build your -collector distribution without containerization, go to +orchestrator (for example, Kubernetes). If you would like to *only* build your +Collector distribution without containerization, go to [Step 3a](#step-3a---generate-the-code-and-build-your-collectors-distribution). {{% /alert %}} -You need to add two new files to your project: +Add two new files to your project: - `Dockerfile` - Container image definition of your Collector distribution - `collector-config.yaml` - Minimalist Collector configuration YAML for testing @@ -263,7 +266,7 @@ You need to add two new files to your project: After adding these files, your file structure looks like this: -```console +```text . ├── builder-config.yaml ├── collector-config.yaml @@ -274,8 +277,7 @@ The following `Dockerfile` builds your Collector distribution in-place, ensuring that the resulting Collector distribution binary matches the target container architecture (for example, `linux/arm64`, `linux/amd64`): - -```yaml +```dockerfile FROM alpine:3.19 AS certs RUN apk --update add ca-certificates @@ -284,7 +286,7 @@ WORKDIR /build COPY ./builder-config.yaml builder-config.yaml -RUN --mount=type=cache,target=/root/.cache/go-build GO111MODULE=on go install go.opentelemetry.io/collector/cmd/builder@{{% version-from-registry collector-builder %}} +RUN --mount=type=cache,target=/root/.cache/go-build GO111MODULE=on go install go.opentelemetry.io/collector/cmd/builder@v0.139.0 RUN --mount=type=cache,target=/root/.cache/go-build builder --config builder-config.yaml FROM gcr.io/distroless/base:latest @@ -301,7 +303,6 @@ CMD ["--config", "/otelcol/collector-config.yaml"] EXPOSE 4317 4318 12001 ``` - The following is the minimalist `collector-config.yaml` definition: @@ -353,11 +354,7 @@ docker run -it --rm -p 4317:4317 -p 4318:4318 \ --name otelcol : ``` -## Further reading: - -- [Building a Trace Receiver](/docs/collector/building/receiver) -- [Building a Connector](/docs/collector/building/connector) +## Further reading -[ocb]: - https://github.com/open-telemetry/opentelemetry-collector/tree/main/cmd/builder -[tags]: https://github.com/open-telemetry/opentelemetry-collector-releases/tags +- [Building a trace receiver](/docs/collector/building/receiver) +- [Building a connector](/docs/collector/building/connector)