Skip to content

Commit 9e1aeb7

Browse files
jack-bergjack-berg
authored
Define instrumentation configuration API (#4128)
Resolves #3535. This introduces an API component to file configuration, which has been limited to SDK (i.e. end user facing) up until this point. The configuration model recently added the first surface area related to instrumentation configuration properties in open-telemetry/opentelemetry-configuration#91. The API proposed in this PR is collectively called the "Instrumentation config API", and provides a mechanism for instrumentation libraries to participate in file configuration and read relevant properties during initialization. The intent is for both OpenTelemetry-authored and native instrumentation alike to be able to be configured by users in a standard way. New API surface area is necessary to accomplish this to avoid instrumentation libraries from needing to take a dependency on SDK artifacts. The following summarizes the additions: - Introduce ConfigProvider, the instrumentation config API analog of TracerProvider, MeterProvider, LoggerProvider. This is the entry point to the API. - Define "Get instrumentation config" operation for ConfigProvider. This returns something called ConfigProperties, which is a programmatic representation of a YAML mapping node. The ConfigProperties returned by "Get instrumentation config" represents the [`.instrumentation`](https://github.com/open-telemetry/opentelemetry-configuration/blob/670901762dd5cce1eecee423b8660e69f71ef4be/examples/kitchen-sink.yaml#L438-L439) node defined in a config file. - Rebrand "file configuration" to "declarative configuration". This expresses the intent without coupling to the file representation, which although will be the most popular way to consume these features is just one possible way to represent the configuration model and use these tools. - Break out dedicated `api.md`, `data-model.md`, and `sdk.md` files for respective API, data model, and SDK portions of declarative configuration. This aligns with other portions of the spec. The separation should improve clarity regarding what should and should not be exposed in the API. I've prototyped this new API in `opentelemetry-java` here: open-telemetry/opentelemetry-java#6549 cc @open-telemetry/configuration-maintainers, @open-telemetry/specs-semconv-maintainers
1 parent 254b78f commit 9e1aeb7

File tree

10 files changed

+649
-526
lines changed

10 files changed

+649
-526
lines changed

‎spec-compliance-matrix.md‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -294,10 +294,11 @@ Note: Support for environment variables is optional.
294294
| OTEL_EXPORTER_OTLP_METRICS_DEFAULT_HISTOGRAM_AGGREGATION | | + | | | | | | | | | |
295295
| OTEL_EXPERIMENTAL_CONFIG_FILE | | | | | | | | | | | |
296296

297-
## File Configuration
297+
## Declarative configuration
298298

299-
See [File Configuration](./specification/configuration/file-configuration.md)
299+
See [declarative configuration](./specification/configuration/README.md#declarative-configuration)
300300
for details.
301+
Disclaimer: Declarative configuration is currently in Development status - work in progress.
301302

302303
| Feature | Go | Java | JS | Python | Ruby | Erlang | PHP | Rust | C++ | .NET | Swift |
303304
|-------------------------------------------------------------------------------------------------------------------------|----|------|----|--------|------|--------|-----|------|-----|------|-------|

‎specification/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,7 @@ path_base_for_github_subdir:
3636
- [Metrics](metrics/sdk.md)
3737
- [Logs](logs/sdk.md)
3838
- [Resource](resource/sdk.md)
39-
- [Configuration](configuration/sdk-configuration.md)
39+
- [Configuration](configuration/README.md)
4040
- Data Specification
4141
- [Semantic Conventions](overview.md#semantic-conventions)
4242
- [Protocol](protocol/README.md)

‎specification/configuration/README.md‎

Lines changed: 56 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,4 +4,59 @@ path_base_for_github_subdir:
44
to: configuration/README.md
55
--->
66

7-
# Configuration
7+
# Overview
8+
9+
OpenTelemetry SDK components are highly configurable. This specification
10+
outlines the mechanisms by which OpenTelemetry components can be configured. It
11+
does not attempt to specify the details of what can be configured.
12+
13+
## Configuration Interfaces
14+
15+
### Programmatic
16+
17+
The SDK MUST provide a programmatic interface for all configuration.
18+
This interface SHOULD be written in the language of the SDK itself.
19+
All other configuration mechanisms SHOULD be built on top of this interface.
20+
21+
An example of this programmatic interface is accepting a well-defined
22+
struct on an SDK builder class. From that, one could build a CLI that accepts a
23+
file (YAML, JSON, TOML, ...) and then transforms into that well-defined struct
24+
consumable by the programmatic interface (
25+
see [declarative configuration](#declarative-configuration)).
26+
27+
### Environment variables
28+
29+
Environment variable configuration defines a set of language agnostic
30+
environment variables for common configuration goals.
31+
32+
See [OpenTelemetry Environment Variable Specification](./sdk-environment-variables.md).
33+
34+
### Declarative configuration
35+
36+
Declarative configuration provides a mechanism for configuring OpenTelemetry
37+
which is more expressive and full-featured than
38+
the [environment variable](#environment-variables) based scheme, and language
39+
agnostic in a way not possible with [programmatic configuration](#programmatic).
40+
Notably, declarative configuration defines tooling allowing users to load
41+
OpenTelemetry components according to a file-based representation of a
42+
standardized configuration data model.
43+
44+
Declarative configuration consists of the following main components:
45+
46+
* [Data model](./data-model.md) defines data structures which allow users to
47+
specify an intended configuration of OpenTelemetry SDK components and
48+
instrumentation. The data model includes a file-based representation.
49+
* [Instrumentation configuration API](./api.md) allows
50+
instrumentation libraries to consume configuration by reading relevant
51+
configuration options during initialization.
52+
* [Configuration SDK](./sdk.md) defines SDK capabilities around file
53+
configuration, including an In-Memory configuration model, support for
54+
referencing custom extension plugin interfaces in configuration files, and
55+
operations to parse configuration files and interpret the configuration data
56+
model.
57+
58+
### Other Mechanisms
59+
60+
Additional configuration mechanisms SHOULD be provided in whatever
61+
language/format/style is idiomatic for the language of the SDK. The
62+
SDK can include as many configuration mechanisms as appropriate.

‎specification/configuration/api.md‎

Lines changed: 80 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,80 @@
1+
# Instrumentation Configuration API
2+
3+
**Status**: [Development](../document-status.md)
4+
5+
<!-- toc -->
6+
7+
- [Overview](#overview)
8+
* [ConfigProvider](#configprovider)
9+
+ [ConfigProvider operations](#configprovider-operations)
10+
- [Get instrumentation config](#get-instrumentation-config)
11+
* [ConfigProperties](#configproperties)
12+
13+
<!-- tocstop -->
14+
15+
## Overview
16+
17+
The instrumentation configuration API is part of
18+
the [declarative configuration interface](./README.md#declarative-configuration).
19+
20+
The API allows [instrumentation libraries](../glossary.md#instrumentation-library)
21+
to consume configuration by reading relevant configuration during
22+
initialization. For example, an instrumentation library for an HTTP client can
23+
read the set of HTTP request and response headers to capture.
24+
25+
It consists of the following main components:
26+
27+
* [ConfigProvider](#configprovider) is the entry point of the API.
28+
* [ConfigProperties](#configproperties) is a programmatic representation of a
29+
configuration mapping node.
30+
31+
### ConfigProvider
32+
33+
`ConfigProvider` provides access to configuration properties relevant to
34+
instrumentation.
35+
36+
Instrumentation libraries access `ConfigProvider` during
37+
initialization. `ConfigProvider` may be passed as an argument to the
38+
instrumentation library, or the instrumentation library may access it from a
39+
central place. Thus, the API SHOULD provide a way to access a global
40+
default `ConfigProvider`, and set/register it.
41+
42+
#### ConfigProvider operations
43+
44+
The `ConfigProvider` MUST provide the following functions:
45+
46+
* [Get instrumentation config](#get-instrumentation-config)
47+
48+
TODO: decide if additional operations are needed to improve API ergonomics
49+
50+
##### Get instrumentation config
51+
52+
Obtain configuration relevant to instrumentation libraries.
53+
54+
**Returns:** [`ConfigProperties`](#configproperties) representing
55+
the [`.instrumentation`](https://github.com/open-telemetry/opentelemetry-configuration/blob/670901762dd5cce1eecee423b8660e69f71ef4be/examples/kitchen-sink.yaml#L438-L439)
56+
configuration mapping node.
57+
58+
If the `.instrumentation` node is not set, get instrumentation config MUST
59+
return nil, null, undefined or another language-specific idiomatic pattern
60+
denoting empty.
61+
62+
### ConfigProperties
63+
64+
`ConfigProperties` is a programmatic representation of a configuration mapping
65+
node (i.e. a YAML mapping node).
66+
67+
`ConfigProperties` MUST provide accessors for reading all properties from the
68+
mapping node it represents, including:
69+
70+
* scalars (string, boolean, double precision floating point, 64-bit integer)
71+
* mappings, which SHOULD be represented as `ConfigProperties`
72+
* sequences of scalars
73+
* sequences of mappings, which SHOULD be represented as `ConfigProperties`
74+
* the set of property keys present
75+
76+
`ConfigProperties` SHOULD provide access to properties in a type safe manner,
77+
based on what is idiomatic in the language.
78+
79+
`ConfigProperties` SHOULD allow a caller to determine if a property is present
80+
with a null value, versus not set.

‎specification/configuration/data-model.md‎

Lines changed: 148 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,148 @@
1+
# Configuration Data Model
2+
3+
**Status**: [Development](../document-status.md)
4+
5+
<!-- toc -->
6+
7+
- [Overview](#overview)
8+
* [Stability definition](#stability-definition)
9+
* [File-based configuration model](#file-based-configuration-model)
10+
+ [YAML file format](#yaml-file-format)
11+
+ [Environment variable substitution](#environment-variable-substitution)
12+
13+
<!-- tocstop -->
14+
15+
## Overview
16+
17+
The OpenTelemetry configuration data model is part of
18+
the [declarative configuration interface](./README.md#declarative-configuration).
19+
20+
The data model defines data structures which allow users to specify an intended
21+
configuration of OpenTelemetry SDK components and instrumentation.
22+
23+
The data model is defined
24+
in [opentelemetry-configuration](https://github.com/open-telemetry/opentelemetry-configuration)
25+
using [JSON Schema](https://json-schema.org/).
26+
27+
The data model itself is an abstraction with multiple built-in representations:
28+
29+
* [File-based configuration model](#file-based-configuration-model)
30+
* [SDK in-memory configuration model](./sdk.md#in-memory-configuration-model)
31+
32+
### Stability definition
33+
34+
TODO: define stability guarantees and backwards compatibility
35+
36+
### File-based configuration model
37+
38+
A configuration file is a serialized file-based representation of
39+
the configuration data model.
40+
41+
Configuration files SHOULD use one the following serialization formats:
42+
43+
* [YAML file format](#yaml-file-format)
44+
45+
#### YAML file format
46+
47+
[YAML](https://yaml.org/spec/1.2.2/) configuration files SHOULD follow YAML spec
48+
revision >= 1.2.
49+
50+
YAML configuration files SHOULD be parsed using [v1.2 YAML core schema](https://yaml.org/spec/1.2.2/#103-core-schema).
51+
52+
YAML configuration files MUST use file extensions `.yaml` or `.yml`.
53+
54+
#### Environment variable substitution
55+
56+
Configuration files support environment variables substitution for references
57+
which match the following PCRE2 regular expression:
58+
59+
```regexp
60+
\$\{(?:env:)?(?<ENV_NAME>[a-zA-Z_][a-zA-Z0-9_]*)(:-(?<DEFAULT_VALUE>[^\n]*))?\}
61+
```
62+
63+
The `ENV_NAME` MUST start with an alphabetic or `_` character, and is followed
64+
by 0 or more alphanumeric or `_` characters.
65+
66+
For example, `${API_KEY}` and `${env:API_KEY}` are valid, while `${1API_KEY}`
67+
and `${API_$KEY}` are invalid.
68+
69+
Environment variable substitution MUST only apply to scalar values. Mapping keys
70+
are not candidates for substitution.
71+
72+
The `DEFAULT_VALUE` is an optional fallback value which is substituted
73+
if `ENV_NAME` is null, empty, or undefined. `DEFAULT_VALUE` consists of 0 or
74+
more non line break characters (i.e. any character except `\n`). If a referenced
75+
environment variable is not defined and does not have a `DEFAULT_VALUE`, it MUST
76+
be replaced with an empty value.
77+
78+
When parsing a configuration file that contains a reference not matching
79+
the references regular expression but does match the following PCRE2
80+
regular expression, the parser MUST return an empty result (no partial
81+
results are allowed) and an error describing the parse failure to the user.
82+
83+
```regexp
84+
\$\{(?<INVALID_IDENTIFIER>[^}]+)\}
85+
```
86+
87+
Node types MUST be interpreted after environment variable substitution takes
88+
place. This ensures the environment string representation of boolean, integer,
89+
or floating point fields can be properly converted to expected types.
90+
91+
It MUST NOT be possible to inject YAML structures by environment variables. For
92+
example, see references to `INVALID_MAP_VALUE` environment variable below.
93+
94+
It MUST NOT be possible to inject environment variable by environment variables.
95+
For example, see references to `DO_NOT_REPLACE_ME` environment variable below.
96+
97+
For example, consider the following environment variables,
98+
and [YAML](#yaml-file-format) configuration file:
99+
100+
```shell
101+
export STRING_VALUE="value"
102+
export BOOL_VALUE="true"
103+
export INT_VALUE="1"
104+
export FLOAT_VALUE="1.1"
105+
export HEX_VALUE="0xdeadbeef" # A valid integer value written in hexadecimal
106+
export INVALID_MAP_VALUE="value\nkey:value" # An invalid attempt to inject a map key into the YAML
107+
export DO_NOT_REPLACE_ME="Never use this value" # An unused environment variable
108+
export REPLACE_ME='${DO_NOT_REPLACE_ME}' # A valid replacement text, used verbatim, not replaced with "Never use this value"
109+
```
110+
111+
```yaml
112+
string_key: ${STRING_VALUE} # Valid reference to STRING_VALUE
113+
env_string_key: ${env:STRING_VALUE} # Valid reference to STRING_VALUE
114+
other_string_key: "${STRING_VALUE}" # Valid reference to STRING_VALUE inside double quotes
115+
another_string_key: "${BOOL_VALUE}" # Valid reference to BOOL_VALUE inside double quotes
116+
string_key_with_quoted_hex_value: "${HEX_VALUE}" # Valid reference to HEX_VALUE inside double quotes
117+
yet_another_string_key: ${INVALID_MAP_VALUE} # Valid reference to INVALID_MAP_VALUE, but YAML structure from INVALID_MAP_VALUE MUST NOT be injected
118+
bool_key: ${BOOL_VALUE} # Valid reference to BOOL_VALUE
119+
int_key: ${INT_VALUE} # Valid reference to INT_VALUE
120+
int_key_with_unquoted_hex_value: ${HEX_VALUE} # Valid reference to HEX_VALUE without quotes
121+
float_key: ${FLOAT_VALUE} # Valid reference to FLOAT_VALUE
122+
combo_string_key: foo ${STRING_VALUE} ${FLOAT_VALUE} # Valid reference to STRING_VALUE and FLOAT_VALUE
123+
string_key_with_default: ${UNDEFINED_KEY:-fallback} # UNDEFINED_KEY is not defined but a default value is included
124+
undefined_key: ${UNDEFINED_KEY} # Invalid reference, UNDEFINED_KEY is not defined and is replaced with ""
125+
${STRING_VALUE}: value # Invalid reference, substitution is not valid in mapping keys and reference is ignored
126+
recursive_key: ${REPLACE_ME} # Valid reference to REPLACE_ME
127+
# invalid_identifier_key: ${STRING_VALUE:?error} # If uncommented, this is an invalid identifier, it would fail to parse
128+
```
129+
130+
Environment variable substitution results in the following YAML:
131+
132+
```yaml
133+
string_key: value # Interpreted as type string, tag URI tag:yaml.org,2002:str
134+
env_string_key: value # Interpreted as type string, tag URI tag:yaml.org,2002:str
135+
other_string_key: "value" # Interpreted as type string, tag URI tag:yaml.org,2002:str
136+
another_string_key: "true" # Interpreted as type string, tag URI tag:yaml.org,2002:str
137+
string_key_with_quoted_hex_value: "0xdeadbeef" # Interpreted as type string, tag URI tag:yaml.org,2002:str
138+
yet_another_string_key: "value\nkey:value" # Interpreted as type string, tag URI tag:yaml.org,2002:str
139+
bool_key: true # Interpreted as type bool, tag URI tag:yaml.org,2002:bool
140+
int_key: 1 # Interpreted as type int, tag URI tag:yaml.org,2002:int
141+
int_key_with_unquoted_hex_value: 3735928559 # Interpreted as type int, tag URI tag:yaml.org,2002:int
142+
float_key: 1.1 # Interpreted as type float, tag URI tag:yaml.org,2002:float
143+
combo_string_key: foo value 1.1 # Interpreted as type string, tag URI tag:yaml.org,2002:str
144+
string_key_with_default: fallback # Interpreted as type string, tag URI tag:yaml.org,2002:str
145+
undefined_key: # Interpreted as type null, tag URI tag:yaml.org,2002:null
146+
${STRING_VALUE}: value # Interpreted as type string, tag URI tag:yaml.org,2002:str
147+
recursive_key: ${DO_NOT_REPLACE_ME} # Interpreted as type string, tag URI tag:yaml.org,2002:str
148+
```

0 commit comments

Comments
 (0)