Skip to content

Commit e2e66ed

Browse files
authored
Add custom build support for lambdacomponents (open-telemetry#1427)
* Add custom build support for lambdacomponents * fix typo * move custmized collector build readme section to the collectors README * use BUILDTAGS env variable for customized build with go build tags
1 parent 397af0e commit e2e66ed

31 files changed

+724
-3
lines changed

README.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,8 @@ Some layers include the corresponding OTel language SDK for the Lambda. This all
2525

2626
* **What exporters/receivers/processors are included from the OpenTelemetry Collector?**
2727
> You can check out [the stripped-down collector's imports](https://github.com/open-telemetry/opentelemetry-lambda/blob/main/collector/lambdacomponents/default.go#L18) in this repository for a full list of currently included components.
28+
29+
> Self-built binaries of the collector have **experimental** support for a custom set of connectors/exporters/receivers/processors. For more information, see [(Experimental) Customized collector build](./collector/README.md#experimental-customized-collector-build)
2830
* **Is the Lambda layer provided or do I need to build it and distribute it myself?**
2931
> This repository provides pre-built Lambda layers, their ARNs are available in the [Releases](https://github.com/open-telemetry/opentelemetry-lambda/releases). You can also build the layers manually and publish them in your AWS account. This repo has files to facilitate doing that. More information is provided in [the Collector folder's README](collector/README.md).
3032

collector/Makefile

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ LAYER_NAME:=otel-collector
77
OTELCOL_VERSION="UNSET" # run recipe set-otelcol-version to get this
88
GIT_SHA=$(shell git rev-parse HEAD)
99
GOARCH ?= amd64
10-
GOBUILD=GO111MODULE=on CGO_ENABLED=0 installsuffix=cgo go build -trimpath
10+
GOBUILD=GO111MODULE=on CGO_ENABLED=0 installsuffix=cgo go build -trimpath$(if ${BUILDTAGS}, -tags "${BUILDTAGS}",)
1111
BUILD_INFO_IMPORT_PATH=main
1212

1313
LDFLAGS=-ldflags "-s -w -X $(BUILD_INFO_IMPORT_PATH).GitHash=$(GIT_SHA) -X $(BUILD_INFO_IMPORT_PATH).Version=$(OTELCOL_VERSION) \

collector/README.md

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,45 @@ Be sure to:
1313
* Install [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html)
1414
* Config [AWS credential](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html)
1515

16+
## (Experimental) Customized collector build
17+
The collector can be built with a customized set of connectors/exporters/receivers/processors. This feature is **experimental** and is only supported for self-built binaries of the collector.
18+
19+
### Build Tags
20+
The build-tag `lambdacomponents.custom` must always be provided to opt-in for a custom build.
21+
Once this build-tag is present, you need provide additional build-tags to include your desired components in the resulting binary:
22+
23+
- `lambdacomponents.all` includes all available components
24+
- `lambdacomponents.connector.all` includes all available connectors
25+
- `lambdacomponents.exporter.all` includes all available exporters
26+
- `lambdacomponents.extension.all` includes all available extensions
27+
- `lambdacomponents.processor.all` includes all available processors
28+
- `lambdacomponents.receiver.all` includes all available receivers
29+
30+
Each available component can also be included explicitly by using its specific build-tag. For a full-list of available components, have a look into the [lambdacomponents](./collector/lambdacomponents) package.
31+
32+
As an example, the full command to publish OpenTelemetry Collector Lambda layer in your AWS account and get its ARN including the following components:
33+
- All receivers
34+
- All processors
35+
- No extensions
36+
- Only the otlphttp exporter
37+
- Only the spanmetrics connector
38+
39+
would be the following:
40+
```shell
41+
cd collector && BUILDTAGS="lambdacomponents.custom,lambdacomponents.receiver.all,lambdacomponents.processor.all,lambdacomponents.exporter.otlphttp,lambdacomponents.connector.spanmetrics" make publish-layer
42+
```
43+
44+
### Adding additional options
45+
To add more options for a customized build, you can add your desired component to the [lambdacomponents](./collector/lambdacomponents) package.
46+
Make sure to always restrict your addition using the appropriate build-tags.
47+
48+
For example, if you want to add the extension `foo`, the file providing this extension should be located in the [extension](./collector/lambdacomponents/extension) directory have the following build restriction:
49+
```
50+
//go:build lambdacomponents.custom && (lambdacomponents.all || lambdacomponents.extension.all || lambdacomponents.extension.foo)
51+
```
52+
53+
You can provide your addition as a pull-request to this repository. Before doing so, please also read through the details of [Contributing](#contributing) to this project.
54+
1655
## Installing
1756
To install the OpenTelemetry Collector Lambda layer to an existing Lambda function using the `aws` CLI:
1857

collector/go.mod

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -79,14 +79,17 @@ require (
7979
github.com/google/uuid v1.6.0 // indirect
8080
github.com/grpc-ecosystem/grpc-gateway/v2 v2.20.0 // indirect
8181
github.com/hashicorp/go-version v1.7.0 // indirect
82+
github.com/hashicorp/golang-lru v1.0.2 // indirect
8283
github.com/hashicorp/golang-lru/v2 v2.0.7 // indirect
8384
github.com/iancoleman/strcase v0.3.0 // indirect
8485
github.com/inconshreveable/mousetrap v1.1.0 // indirect
86+
github.com/jonboulle/clockwork v0.4.0 // indirect
8587
github.com/json-iterator/go v1.1.12 // indirect
8688
github.com/klauspost/compress v1.17.9 // indirect
8789
github.com/knadh/koanf/maps v0.1.1 // indirect
8890
github.com/knadh/koanf/providers/confmap v0.1.0 // indirect
8991
github.com/knadh/koanf/v2 v2.1.1 // indirect
92+
github.com/lightstep/go-expohisto v1.0.0 // indirect
9093
github.com/lufia/plan9stats v0.0.0-20211012122336-39d0f177ccd0 // indirect
9194
github.com/magefile/mage v1.15.0 // indirect
9295
github.com/mitchellh/copystructure v1.2.0 // indirect
@@ -95,6 +98,7 @@ require (
9598
github.com/modern-go/reflect2 v1.0.2 // indirect
9699
github.com/mostynb/go-grpc-compression v1.2.3 // indirect
97100
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
101+
github.com/open-telemetry/opentelemetry-collector-contrib/connector/spanmetricsconnector v0.107.0 // indirect
98102
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusremotewriteexporter v0.107.0 // indirect
99103
github.com/open-telemetry/opentelemetry-collector-contrib/extension/sigv4authextension v0.107.0 // indirect
100104
github.com/open-telemetry/opentelemetry-collector-contrib/internal/coreinternal v0.107.0 // indirect

collector/go.sum

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -90,6 +90,8 @@ github.com/grpc-ecosystem/grpc-gateway/v2 v2.20.0 h1:bkypFPDjIYGfCYD5mRBvpqxfYX1
9090
github.com/grpc-ecosystem/grpc-gateway/v2 v2.20.0/go.mod h1:P+Lt/0by1T8bfcF3z737NnSbmxQAppXMRziHUxPOC8k=
9191
github.com/hashicorp/go-version v1.7.0 h1:5tqGy27NaOTB8yJKUZELlFAS/LTKJkrmONwQKeRZfjY=
9292
github.com/hashicorp/go-version v1.7.0/go.mod h1:fltr4n8CU8Ke44wwGCBoEymUuxUHl09ZGVZPK5anwXA=
93+
github.com/hashicorp/golang-lru v1.0.2 h1:dV3g9Z/unq5DpblPpw+Oqcv4dU/1omnb4Ok8iPY6p1c=
94+
github.com/hashicorp/golang-lru v1.0.2/go.mod h1:iADmTwqILo4mZ8BN3D2Q6+9jd8WM5uGBxy+E8yxSoD4=
9395
github.com/hashicorp/golang-lru/v2 v2.0.7 h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs4luLUK2k=
9496
github.com/hashicorp/golang-lru/v2 v2.0.7/go.mod h1:QeFd9opnmA6QUJc5vARoKUSoFhyfM2/ZepoAG6RGpeM=
9597
github.com/hexops/gotextdiff v1.0.3 h1:gitA9+qJrrTCsiCl7+kh75nPqQt1cx4ZkudSTLoUqJM=
@@ -98,6 +100,8 @@ github.com/iancoleman/strcase v0.3.0 h1:nTXanmYxhfFAMjZL34Ov6gkzEsSJZ5DbhxWjvSAS
98100
github.com/iancoleman/strcase v0.3.0/go.mod h1:iwCmte+B7n89clKwxIoIXy/HfoL7AsD47ZCWhYzw7ho=
99101
github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8=
100102
github.com/inconshreveable/mousetrap v1.1.0/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw=
103+
github.com/jonboulle/clockwork v0.4.0 h1:p4Cf1aMWXnXAUh8lVfewRBx1zaTSYKrKMF2g3ST4RZ4=
104+
github.com/jonboulle/clockwork v0.4.0/go.mod h1:xgRqUGwRcjKCO1vbZUEtSLrqKoPSsUpK7fnezOII0kc=
101105
github.com/json-iterator/go v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnrnM=
102106
github.com/json-iterator/go v1.1.12/go.mod h1:e30LSqwooZae/UwlEbR2852Gd8hjQvJoHmT4TnhNGBo=
103107
github.com/kisielk/errcheck v1.5.0/go.mod h1:pFxgyoBC7bSaBwPgfKdkLd5X25qrDl4LWUI2bnpBCr8=
@@ -114,6 +118,8 @@ github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE=
114118
github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk=
115119
github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
116120
github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
121+
github.com/lightstep/go-expohisto v1.0.0 h1:UPtTS1rGdtehbbAF7o/dhkWLTDI73UifG8LbfQI7cA4=
122+
github.com/lightstep/go-expohisto v1.0.0/go.mod h1:xDXD0++Mu2FOaItXtdDfksfgxfV0z1TMPa+e/EUd0cs=
117123
github.com/lufia/plan9stats v0.0.0-20211012122336-39d0f177ccd0 h1:6E+4a0GO5zZEnZ81pIr0yLvtUWk2if982qA3F3QD6H4=
118124
github.com/lufia/plan9stats v0.0.0-20211012122336-39d0f177ccd0/go.mod h1:zJYVVT2jmtg6P3p1VtQj7WsuWi/y4VnjVBn7F8KPB3I=
119125
github.com/magefile/mage v1.15.0 h1:BvGheCMAsG3bWUDbZ8AyXXpCNwU9u5CB6sM+HNb9HYg=
@@ -135,6 +141,8 @@ github.com/open-telemetry/opentelemetry-collector-contrib/confmap/provider/s3pro
135141
github.com/open-telemetry/opentelemetry-collector-contrib/confmap/provider/s3provider v0.107.0/go.mod h1:BIZvmh0Jx3qxmmUpQs4Wmp+yw3eZTinKBd1uRAdDRPg=
136142
github.com/open-telemetry/opentelemetry-collector-contrib/confmap/provider/secretsmanagerprovider v0.107.0 h1:XZGFxuM2NpjCDPKi/zrVXAaf+JN6Rlp/HlkI9lg+jc4=
137143
github.com/open-telemetry/opentelemetry-collector-contrib/confmap/provider/secretsmanagerprovider v0.107.0/go.mod h1:Fo7KPO1ElNawOHDL4peoRgU3V+DmoRF65R2i7Ox1I/c=
144+
github.com/open-telemetry/opentelemetry-collector-contrib/connector/spanmetricsconnector v0.107.0 h1:1S+oa3bMVrSi7PclgMvxtyPAi8mTdHsuDqAq8XA06qU=
145+
github.com/open-telemetry/opentelemetry-collector-contrib/connector/spanmetricsconnector v0.107.0/go.mod h1:jrqMHeaEBFtmRKKocfoVrfGjRYWp2+nj9C2aGdEN7xo=
138146
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusremotewriteexporter v0.107.0 h1:okRANT1e80Lva/ck1gWZXF4Z1caH4SIz5A4crYPfylw=
139147
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusremotewriteexporter v0.107.0/go.mod h1:GJoWZ9tTTev7+HjiXW0uV61tYDGTorNmy1Wdg7UmaQU=
140148
github.com/open-telemetry/opentelemetry-collector-contrib/extension/sigv4authextension v0.107.0 h1:Xt9yRUYN3vyzjgc3rRwQ10OZ+WTnrIsergAdrEF7usQ=
Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
// Copyright The OpenTelemetry Authors
2+
//
3+
// Licensed under the Apache License, Version 2.0 (the "License");
4+
// you may not use this file except in compliance with the License.
5+
// You may obtain a copy of the License at
6+
//
7+
// http://www.apache.org/licenses/LICENSE-2.0
8+
//
9+
// Unless required by applicable law or agreed to in writing, software
10+
// distributed under the License is distributed on an "AS IS" BASIS,
11+
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
// See the License for the specific language governing permissions and
13+
// limitations under the License.
14+
15+
package connector
16+
17+
import "go.opentelemetry.io/collector/connector"
18+
19+
var Factories []func(extensionId string) connector.Factory
Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
//go:build lambdacomponents.custom && (lambdacomponents.all || lambdacomponents.connector.all || lambdacomponents.connector.spanmetrics)
2+
3+
// Copyright The OpenTelemetry Authors
4+
//
5+
// Licensed under the Apache License, Version 2.0 (the "License");
6+
// you may not use this file except in compliance with the License.
7+
// You may obtain a copy of the License at
8+
//
9+
// http://www.apache.org/licenses/LICENSE-2.0
10+
//
11+
// Unless required by applicable law or agreed to in writing, software
12+
// distributed under the License is distributed on an "AS IS" BASIS,
13+
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
// See the License for the specific language governing permissions and
15+
// limitations under the License.
16+
17+
package connector
18+
19+
import (
20+
"github.com/open-telemetry/opentelemetry-collector-contrib/connector/spanmetricsconnector"
21+
"go.opentelemetry.io/collector/connector"
22+
)
23+
24+
func init() {
25+
Factories = append(Factories, func(extensionId string) connector.Factory {
26+
return spanmetricsconnector.NewFactory()
27+
})
28+
}
Lines changed: 83 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,83 @@
1+
//go:build lambdacomponents.custom
2+
3+
// Copyright The OpenTelemetry Authors
4+
//
5+
// Licensed under the Apache License, Version 2.0 (the "License");
6+
// you may not use this file except in compliance with the License.
7+
// You may obtain a copy of the License at
8+
//
9+
// http://www.apache.org/licenses/LICENSE-2.0
10+
//
11+
// Unless required by applicable law or agreed to in writing, software
12+
// distributed under the License is distributed on an "AS IS" BASIS,
13+
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
// See the License for the specific language governing permissions and
15+
// limitations under the License.
16+
17+
package lambdacomponents
18+
19+
import (
20+
custom_connector "github.com/open-telemetry/opentelemetry-lambda/collector/lambdacomponents/connector"
21+
custom_exporter "github.com/open-telemetry/opentelemetry-lambda/collector/lambdacomponents/exporter"
22+
custom_extension "github.com/open-telemetry/opentelemetry-lambda/collector/lambdacomponents/extension"
23+
custom_processor "github.com/open-telemetry/opentelemetry-lambda/collector/lambdacomponents/processor"
24+
custom_receiver "github.com/open-telemetry/opentelemetry-lambda/collector/lambdacomponents/receiver"
25+
26+
"go.opentelemetry.io/collector/component"
27+
"go.opentelemetry.io/collector/connector"
28+
"go.opentelemetry.io/collector/exporter"
29+
"go.opentelemetry.io/collector/extension"
30+
"go.opentelemetry.io/collector/otelcol"
31+
"go.opentelemetry.io/collector/processor"
32+
"go.opentelemetry.io/collector/receiver"
33+
34+
"go.uber.org/multierr"
35+
)
36+
37+
func Components(extensionID string) (otelcol.Factories, error) {
38+
var errs []error
39+
40+
receivers, err := makeFactoryMap(custom_receiver.Factories, receiver.MakeFactoryMap, extensionID)
41+
if err != nil {
42+
errs = append(errs, err)
43+
}
44+
45+
processors, err := makeFactoryMap(custom_processor.Factories, processor.MakeFactoryMap, extensionID)
46+
if err != nil {
47+
errs = append(errs, err)
48+
}
49+
50+
exporters, err := makeFactoryMap(custom_exporter.Factories, exporter.MakeFactoryMap, extensionID)
51+
if err != nil {
52+
errs = append(errs, err)
53+
}
54+
55+
extensions, err := makeFactoryMap(custom_extension.Factories, extension.MakeFactoryMap, extensionID)
56+
if err != nil {
57+
errs = append(errs, err)
58+
}
59+
60+
connectors, err := makeFactoryMap(custom_connector.Factories, connector.MakeFactoryMap, extensionID)
61+
if err != nil {
62+
errs = append(errs, err)
63+
}
64+
65+
factories := otelcol.Factories{
66+
Receivers: receivers,
67+
Processors: processors,
68+
Exporters: exporters,
69+
Extensions: extensions,
70+
Connectors: connectors,
71+
}
72+
73+
return factories, multierr.Combine(errs...)
74+
}
75+
76+
func makeFactoryMap[F any](factories []func(extensionId string) F, fn func(...F) (map[component.Type]F, error), extensionId string) (map[component.Type]F, error) {
77+
preprocessedFactories := make([]F, len(factories))
78+
for i, f := range factories {
79+
preprocessedFactories[i] = f(extensionId)
80+
}
81+
82+
return fn(preprocessedFactories...)
83+
}

collector/lambdacomponents/default.go

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,5 @@
1+
//go:build !lambdacomponents.custom
2+
13
// Copyright The OpenTelemetry Authors
24
//
35
// Licensed under the Apache License, Version 2.0 (the "License");
Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
//go:build lambdacomponents.custom && (lambdacomponents.all || lambdacomponents.exporter.all || lambdacomponents.exporter.logging)
2+
3+
// Copyright The OpenTelemetry Authors
4+
//
5+
// Licensed under the Apache License, Version 2.0 (the "License");
6+
// you may not use this file except in compliance with the License.
7+
// You may obtain a copy of the License at
8+
//
9+
// http://www.apache.org/licenses/LICENSE-2.0
10+
//
11+
// Unless required by applicable law or agreed to in writing, software
12+
// distributed under the License is distributed on an "AS IS" BASIS,
13+
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
// See the License for the specific language governing permissions and
15+
// limitations under the License.
16+
17+
package exporter
18+
19+
import (
20+
"go.opentelemetry.io/collector/exporter"
21+
"go.opentelemetry.io/collector/exporter/loggingexporter"
22+
)
23+
24+
func init() {
25+
Factories = append(Factories, func(extensionId string) exporter.Factory {
26+
return loggingexporter.NewFactory()
27+
})
28+
}

0 commit comments

Comments
 (0)