diff --git a/content/en/blog/2025/declarative-config.md b/content/en/blog/2025/declarative-config.md new file mode 100644 index 000000000000..866afdb3cf69 --- /dev/null +++ b/content/en/blog/2025/declarative-config.md @@ -0,0 +1,326 @@ +--- +title: + 'The Declarative configuration journey: Why it took 5 years to ignore health + check endpoints in tracing' +linkTitle: Declarative configuration journey +date: 2025-10-17 +author: >- + [Gregor Zeitlinger](https://github.com/zeitlinger)(Grafana Labs), [Jay + DeLuca](https://github.com/jaydeluca) (Grafana Labs), [Marylia + Gutierrez](https://github.com/maryliag) (Grafana Labs) +cSpell:ignore: Dotel marylia otelconf zeitlinger +--- + +One of the most persistent and popular feature requests for Java OpenTelemetry +over the past couple of years has been the ability to efficiently [drop spans +for health check endpoints][drop-spans-issue] – or any other low-value, +cost-driving endpoints. This issue was first raised in August 2020, yet a +comprehensive solution remained elusive for a surprisingly long time. Why did it +take us five years to address this seemingly straightforward problem? The answer +lies in the fundamental principles of OpenTelemetry's configuration system and +the journey towards a more robust, flexible approach: declarative configuration. + +From the outset, OpenTelemetry relied on environment variables for +configuration, a choice driven by their universal availability across languages +and ease of parsing. However, as the need for more complex configuration use +cases grew, the limitations of simple string-based environment variables became +increasingly apparent, making advanced configurations cumbersome and difficult +to manage. + +Enter declarative configuration, a powerful evolution that leverages YAML files +to define OpenTelemetry settings. This shift allows for reading data from any +tree-shaped source, fundamentally transforming how we approach complex +configurations. Throughout this post, we'll explore how declarative +configuration provides an elegant solution to the challenges of the past, and +demonstrate its immediate impact with practical use cases like health check +exclusion in Java. + +## Getting started + +The configuration file is language agnostic, so once you create one file, you +can use it for all your SDKs. The only exceptions are the parameters with the +specific language name that are only relevant to that language (for example, +`instrumentation/development.java.spring_batch` parameter). Keep in mind that +declarative configuration is **experimental**, so things might still change. + +The following example is a basic configuration file you can use to get started: + +```yaml +file_format: '1.0-rc.1' + +resource: + attributes_list: ${OTEL_RESOURCE_ATTRIBUTES} + detection/development: + detectors: + - service: # will add "service.instance.id" and "service.name" from OTEL_SERVICE_NAME + +tracer_provider: + processors: + - batch: + exporter: + otlp_http: + endpoint: ${OTEL_EXPORTER_OTLP_TRACES_ENDPOINT:-http://localhost:4318}/v1/traces + +meter_provider: + readers: + - periodic: + exporter: + otlp_http: + endpoint: ${OTEL_EXPORTER_OTLP_METRICS_ENDPOINT:-http://localhost:4318}/v1/metrics + +logger_provider: + processors: + - batch: + exporter: + otlp_http: + endpoint: ${OTEL_EXPORTER_OTLP_LOGS_ENDPOINT:-http://localhost:4318}/v1/logs +``` + +All you have to do is pass +`OTEL_EXPERIMENTAL_CONFIG_FILE=/path/to/otel-config.yaml` to the application to +activate the experimental declarative configuration option. This variable only +works in the Java agent and JavaScript at the time of writing. + +## Declarative configuration in Java + +Let's now look at the broader implementation of declarative configuration within +the Java ecosystem. As the pioneering language in this area, Java agent 2.21+ +now fully supports declarative configuration, with most instrumentations and +features already functional. We are working to incorporate the remaining +features throughout 2026, and you can track our progress on the [project +board][java-project] and see the [list of features not yet +supported][list-not-supported]. + +Depending on whether you are starting fresh or migrating from using environment +variables, there’s a few resources you can leverage: + +- The basic (language agnostic) configuration file example from the previous + section is the quickest way to get started when you don’t need any further + customizations. +- The [migration configuration file][migration-file] maps the old environment + variables into the YAML schema, allowing for a drop in replacement for users + using workloads already configured with environment variables. +- The [full configuration file][full-file] (“kitchen sink”) shows the entire + schema, annotated with documentation as comments. This is useful for users who + want to see all available options and their defaults. + +All of the above files work for any language that supports declarative +configuration. + +In addition, there are many settings specific to Java agent that go into the +instrumentation section of your configuration file. For example, if you have the +system property `otel.instrumentation.spring-batch.experimental.chunk.new-trace` +in your application, you can create the declarative configuration file by +removing the `otel.instrumentation` prefix, splitting at . and converting - to +\_. + +```yaml +file_format: '1.0-rc.1' + +# ... + +instrumentation/development: + java: + spring_batch: + experimental: + chunk: + new_trace: true +``` + +With this configuration in place, developers can continue to use their Java +instrumentation as they normally would, sending telemetry data to their chosen +observability backend. Furthermore, the declarative configuration file provides +the flexibility to expand and add more parameters as needed, allowing for highly +customized and nuanced control over the observability setup. + +## Health check exclusion + +As mentioned in the introduction, one of the most popular feature requests in +the Java community was to be able to exclude health checks (or other unimportant +or noisy resources) from generating traces. + +To achieve this, you need to add a new `sampler` block within your +`tracer_provider` configuration, as shown below: + +```yaml +file_format: '1.0-rc.1' + +# ... the rest of the configuration .... + +tracer_provider: + # Configure sampling to exclude health check endpoints. + sampler: + rule_based_routing: + fallback_sampler: + always_on: + span_kind: SERVER + rules: + # Action to take when the rule matches. Must be DROP or RECORD_AND_SAMPLE. + - action: DROP + # The span attribute to match against. + attribute: url.path + # The pattern to compare the span attribute to. + pattern: /actuator.* +# ... the rest of the tracer_provider configuration ... +``` + +See the [Java sampler documentation][java-sampler] for more details on the +available options. + +Try it for yourself: + +1. Save [the complete configuration][complete-config] +2. Run the Java agent with + `-Dotel.experimental.config.file=/path/to/otel-config.yaml` + +## Availability + +After reading about declarative configuration, you might be wondering where it +is available and how you can start using it. You can find guidance on how to get +started and which languages are supported in the +[documentation][declarative-docs]. As of the time of writing of this post, Java +is fully compliant and PHP, JavaScript and Go are partially compliant. To see +the latest status, check the [compliance matrix][compliance-matrix] or the +[language implementations tracking issue][tracking-issue]. + +### Java + +As described previously, declarative configuration in +[Java][java-declarative-config] is experimental but ready to use. Use the +example we discussed earlier to set up your new configuration. If you have +questions or feedback reach out in [`#otel-java`][slack-java] on the CNCF Slack. + +_Note to other language maintainers: It is useful to create a bridge module that +adapts declarative config settings and environment variables to a common +interface. For Java, this is the [Declarative Config Bridge][java-bridge]._ + +### JavaScript + +The implementation in the JavaScript SDK is currently under development. A new +package called [opentelemetry-configuration][js-package] has been created, and +it handles both environment variables and declarative configuration. With this +approach, the user doesn't need to change their instrumentation when they switch +between environment variables and configuration file, since the new package +handles it and returns the same configuration model for both cases. Currently, +this configuration package is being added to other instrumentation packages, so +they can take advantage of the declarative configuration. If you have questions, +reach out in [`#otel-js`][slack-js] on the CNCF Slack. + +### PHP + +The PHP implementation is partially compliant, and you can start using it by +[initializing from your config file][php-docs]. For help or feedback, reach out +in [`#otel-php`][slack-php] on the CNCF Slack. + +### Go + +Go has a [partial implementation][go-package] of declarative configuration. Each +supported schema version has its own corresponding package directory. For +example, importing `go.opentelemetry.io/contrib/otelconf/v0.3.0` gives you the +code that supports version 0.3.0 of the configuration schema. You can find all +available versions in the [package index][go-package-index]. If you have +questions on how to use it, reach out in [`#otel-go`][slack-go] on the CNCF +Slack. + +## The journey + +So why did it actually take us five years to ignore health check endpoints in +tracing? + +The journey to declarative configuration, and consecutively, the solution for +health check exclusion, highlights a core tenet of OpenTelemetry: building +sustainable solutions through rigorous specifications. + +From the outset, OpenTelemetry's reliance on environment variables, while +universally available, proved increasingly complex for advanced configurations. +New environment variables were eventually disallowed, creating a void that a +more robust solution needed to fill. + +The replacement, as we’ve presented in this blog post, is declarative +configuration. Crafting and agreeing upon the precise syntax and semantics was a +time-consuming, and sometimes exhausting, process. For example, we discussed +several proposals on how environment variables could be embedded until we came +up with the current solution of using +`${OTEL_EXPORTER_OTLP_ENDPOINT:-http://localhost:4318}`. + +This process serves as a powerful case study for how the OpenTelemetry community +operates. It's a testament to establishing consensus, fostering collaboration, +and the collective effort required to introduce significant new features and +drive their implementation across diverse projects. + +## What's next for declarative configuration? + +The journey of declarative configuration is far from over. Our current focus +involves a substantial effort to expand language support, which is crucial for +ensuring that developers, regardless of their preferred tools, can leverage the +benefits of a declarative approach. + +We are keenly interested in user feedback as we continue to develop and refine +these features. We encourage you to begin experimenting with the current +implementations and to actively communicate any missing functionalities, pain +points, or areas for improvement. This collaborative approach will help us +prioritize development efforts and ensure that the solutions we build truly meet +the needs of the community. You share your feedback or questions using the +channel [`#otel-config-file`][slack-config] from CNCF Slack. + +Beyond providing feedback, there are other ways to get involved and contribute +to the growth of declarative configuration. Each OpenTelemetry SDK has a +[Special Interest Groups (SIGs)][sigs] dedicated to its implementation. Joining +these SIGs offers a direct avenue to understand the current status of +development, participate in discussions, and identify opportunities to +contribute. Whether it's through code contributions, documentation enhancements, +or simply sharing your experiences, every contribution helps to advance the +declarative configuration ecosystem. Your active participation is key to +fostering a robust and versatile set of tools for modern application +development. + +We hope to hear from you! + +## Additional resources + +To learn more about the work going on for declarative configuration, here are +some additional resources to explore: + +- [Simplifying OpenTelemetry with Configuration - Alex Boten, Honeycomb & Jack + Berg, New Relic][yt-config] +- [Declarative configuration documentation](/docs/languages/sdk-configuration/declarative-configuration/) +- [Declarative configuration repository][declarative-repo] + +[drop-spans-issue]: + https://github.com/open-telemetry/opentelemetry-java-instrumentation/issues/1060 +[java-project]: https://github.com/orgs/open-telemetry/projects/151 +[migration-file]: + https://github.com/open-telemetry/opentelemetry-configuration/blob/main/examples/sdk-migration-config.yaml +[full-file]: + https://github.com/open-telemetry/opentelemetry-configuration/blob/main/examples/kitchen-sink.yaml +[java-sampler]: + https://github.com/open-telemetry/opentelemetry-java-contrib/tree/main/samplers +[complete-config]: + https://gist.github.com/zeitlinger/09585b1ab57c454f87e6dcb9a6f50a5c +[declarative-docs]: /docs/languages/sdk-configuration/declarative-configuration +[compliance-matrix]: + https://github.com/open-telemetry/opentelemetry-specification/blob/main/spec-compliance-matrix.md#declarative-configuration +[java-declarative-config]: /docs/zero-code/java/agent/declarative-configuration/ +[slack-java]: https://cloud-native.slack.com/archives/C014L2KCTE3 +[slack-js]: https://cloud-native.slack.com/archives/C01NL1GRPQR +[slack-php]: https://cloud-native.slack.com/archives/C01NFPCV44V +[slack-go]: https://cloud-native.slack.com/archives/C01NPAXACKT +[slack-config]: https://cloud-native.slack.com/archives/C0476L7UJT1 +[java-bridge]: + https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/main/declarative-config-bridge +[js-package]: + https://github.com/open-telemetry/opentelemetry-js/tree/main/experimental/packages/opentelemetry-configuration +[php-docs]: + https://github.com/open-telemetry/opentelemetry-php/tree/main/src/Config/SDK#initialization-from-configuration-file +[go-package]: + https://github.com/open-telemetry/opentelemetry-go-contrib/tree/main/otelconf +[go-package-index]: https://pkg.go.dev/go.opentelemetry.io/contrib/otelconf +[sigs]: + https://github.com/open-telemetry/community?tab=readme-ov-file#implementation-sigs +[yt-config]: https://www.youtube.com/watch?v=u6svjtGpXO4 +[declarative-repo]: + https://github.com/open-telemetry/opentelemetry-configuration +[list-not-supported]: + /docs/zero-code/java/agent/declarative-configuration/#not-yet-supported-features +[tracking-issue]: + https://github.com/open-telemetry/opentelemetry-configuration/issues/100 diff --git a/static/refcache.json b/static/refcache.json index fe66b5c10b2a..953c45cf3c70 100644 --- a/static/refcache.json +++ b/static/refcache.json @@ -623,6 +623,10 @@ "StatusCode": 206, "LastSeen": "2025-02-01T06:58:10.47602-05:00" }, + "https://cloud-native.slack.com/archives/C014L2KCTE3": { + "StatusCode": 200, + "LastSeen": "2025-10-10T02:38:02.796342331Z" + }, "https://cloud-native.slack.com/archives/C01LSCJBXDZ": { "StatusCode": 200, "LastSeen": "2025-02-01T07:12:45.385018-05:00" @@ -635,6 +639,10 @@ "StatusCode": 200, "LastSeen": "2025-02-02T10:59:14.907548-05:00" }, + "https://cloud-native.slack.com/archives/C01NFPCV44V": { + "StatusCode": 200, + "LastSeen": "2025-10-10T02:38:04.362391387Z" + }, "https://cloud-native.slack.com/archives/C01NL1GRPQR": { "StatusCode": 200, "LastSeen": "2025-03-20T10:14:45.444428061Z" @@ -691,6 +699,10 @@ "StatusCode": 200, "LastSeen": "2025-07-02T23:23:40.808529001Z" }, + "https://cloud-native.slack.com/archives/C0476L7UJT1": { + "StatusCode": 200, + "LastSeen": "2025-10-10T02:38:05.341071752Z" + }, "https://cloud-native.slack.com/archives/C05PXDFTCPJ": { "StatusCode": 200, "LastSeen": "2025-10-10T13:26:50.618385414Z" @@ -2939,6 +2951,10 @@ "StatusCode": 206, "LastSeen": "2025-03-11T12:45:30.424838382Z" }, + "https://gist.github.com/zeitlinger/09585b1ab57c454f87e6dcb9a6f50a5c": { + "StatusCode": 206, + "LastSeen": "2025-10-10T02:38:01.012653609Z" + }, "https://git-scm.com/": { "StatusCode": 206, "LastSeen": "2025-10-10T13:21:57.367473008Z" @@ -8119,6 +8135,10 @@ "StatusCode": 206, "LastSeen": "2025-02-18T15:49:02.655822-05:00" }, + "https://github.com/open-telemetry/opentelemetry-configuration/issues/100": { + "StatusCode": 206, + "LastSeen": "2025-10-10T18:59:39.511045863Z" + }, "https://github.com/open-telemetry/opentelemetry-configuration/issues/189": { "StatusCode": 206, "LastSeen": "2025-05-14T12:26:39.966343-04:00" @@ -8875,6 +8895,10 @@ "StatusCode": 206, "LastSeen": "2025-10-09T23:42:10.603333473Z" }, + "https://github.com/open-telemetry/opentelemetry-go-contrib/tree/main/otelconf": { + "StatusCode": 206, + "LastSeen": "2025-10-10T02:38:04.704689318Z" + }, "https://github.com/open-telemetry/opentelemetry-go-instrumentation": { "StatusCode": 206, "LastSeen": "2025-02-02T10:58:54.881145-05:00" @@ -9083,6 +9107,10 @@ "StatusCode": 206, "LastSeen": "2025-10-10T08:31:58.452011547Z" }, + "https://github.com/open-telemetry/opentelemetry-java-contrib/tree/main/samplers": { + "StatusCode": 206, + "LastSeen": "2025-10-10T02:37:58.373816778Z" + }, "https://github.com/open-telemetry/opentelemetry-java-docs/tree/main/log-appender": { "StatusCode": 206, "LastSeen": "2025-10-09T19:47:15.059421437Z" @@ -9183,6 +9211,10 @@ "StatusCode": 206, "LastSeen": "2025-09-11T16:06:24.460726-04:00" }, + "https://github.com/open-telemetry/opentelemetry-java-instrumentation/issues/1060": { + "StatusCode": 206, + "LastSeen": "2025-10-10T02:37:47.556276093Z" + }, "https://github.com/open-telemetry/opentelemetry-java-instrumentation/issues/7413": { "StatusCode": 206, "LastSeen": "2025-10-10T13:22:05.42927022Z" @@ -9195,6 +9227,10 @@ "StatusCode": 206, "LastSeen": "2025-10-08T13:20:31.336860094Z" }, + "https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/main/declarative-config-bridge": { + "StatusCode": 206, + "LastSeen": "2025-10-10T02:38:03.173832186Z" + }, "https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/main/examples/extension": { "StatusCode": 206, "LastSeen": "2025-10-10T10:25:50.044581176Z" @@ -10107,6 +10143,10 @@ "StatusCode": 206, "LastSeen": "2025-10-09T19:47:03.984033268Z" }, + "https://github.com/open-telemetry/opentelemetry-js/tree/main/experimental/packages/opentelemetry-configuration": { + "StatusCode": 206, + "LastSeen": "2025-10-10T02:38:03.571651418Z" + }, "https://github.com/open-telemetry/opentelemetry-js/tree/main/experimental/packages/opentelemetry-exporter-prometheus": { "StatusCode": 206, "LastSeen": "2025-10-09T19:52:23.921286433Z" @@ -10403,6 +10443,10 @@ "StatusCode": 206, "LastSeen": "2025-10-10T08:31:56.000593169Z" }, + "https://github.com/open-telemetry/opentelemetry-php/tree/main/src/Config/SDK#initialization-from-configuration-file": { + "StatusCode": 206, + "LastSeen": "2025-10-10T02:38:03.926822693Z" + }, "https://github.com/open-telemetry/opentelemetry-php/tree/main/src/Context": { "StatusCode": 206, "LastSeen": "2025-10-10T08:31:56.407424267Z" @@ -12655,6 +12699,10 @@ "StatusCode": 206, "LastSeen": "2025-09-25T05:59:26.340756427Z" }, + "https://github.com/orgs/open-telemetry/projects/151": { + "StatusCode": 206, + "LastSeen": "2025-10-10T02:37:52.212711083Z" + }, "https://github.com/orgs/open-telemetry/projects/38": { "StatusCode": 206, "LastSeen": "2025-10-10T10:30:46.229330866Z" @@ -16087,6 +16135,18 @@ "StatusCode": 200, "LastSeen": "2025-10-10T13:26:40.578136138Z" }, + "https://opentelemetry.io/docs/languages/sdk-configuration/declarative-configuration": { + "StatusCode": 206, + "LastSeen": "2025-10-10T02:38:02.095298974Z" + }, + "https://opentelemetry.io/docs/zero-code/java/agent/declarative-configuration/": { + "StatusCode": 206, + "LastSeen": "2025-10-10T02:38:02.276796461Z" + }, + "https://opentelemetry.io/docs/zero-code/java/agent/declarative-configuration/#not-yet-supported-features": { + "StatusCode": 206, + "LastSeen": "2025-10-10T02:37:55.178293826Z" + }, "https://opentracing.io": { "StatusCode": 206, "LastSeen": "2025-10-08T01:24:12.097492981Z" @@ -17371,6 +17431,10 @@ "StatusCode": 200, "LastSeen": "2025-02-06T02:28:12.345Z" }, + "https://pkg.go.dev/go.opentelemetry.io/contrib/otelconf": { + "StatusCode": 200, + "LastSeen": "2025-10-10T02:38:04.884771156Z" + }, "https://pkg.go.dev/go.opentelemetry.io/contrib/processors/minsev": { "StatusCode": 200, "LastSeen": "2025-03-18T14:59:55.049408358Z"