Introduce Cloud Logging Exporter (#169)

Add OpenTelemetry Signal Exporters for Cloud Logging

Register signal exporters for logs, metrics and traces,
that forward signals to SAP Cloud Logging, if an appropriate
service binding is found or do a non operation otherwise.
This allows developers to explicitly configure the export, e.g.
by `otel.logs.exporter=cloud-logging`. Additional exporters
can be added with comma-separated labels.

Signed-off-by: Karsten Schnitter <[email protected]>

Author: KarstenSchnitter

cf-java-logging-support-opentelemetry-agent-extension/README.md

@@ -1,14 +1,15 @@
 # OpenTelemetry Java Agent Extension for SAP Cloud Logging
 
 This module provides an extension for the [OpenTelemetry Java Agent](https://opentelemetry.io/docs/instrumentation/java/automatic/).
-The extension scans the service bindings of an application for SAP Cloud Logging.
+The extension scans the service bindings of an application for [SAP Cloud Logging](https://discovery-center.cloud.sap/serviceCatalog/cloud-logging).
 If such a binding is found, the OpenTelemetry Java Agent is configured to ship observability data to that service.
 Thus, this extension provides a convenient auto-instrumentation for Java applications running on SAP BTP.
 
-The extension provides two main features:
+The extension provides the following main features:
 
-* auto-configuration of the OpenTelemetry connection to SAP Cloud Logging
-* adding resource attributes to describe the CF application
+* additional exporters for logs, metrics and traces for [SAP Cloud Logging](https://discovery-center.cloud.sap/serviceCatalog/cloud-logging)
+* auto-configuration of the generic OpenTelemetry connection to [SAP Cloud Logging](https://discovery-center.cloud.sap/serviceCatalog/cloud-logging)
+* adding resource attributes describing the CF application
 
 See the section on [configuration](#configuration) for further details.
 
@@ -18,7 +19,7 @@ Any Java application can be instrumented with the OpenTelemetry Java Agent and t
 
 ```sh
 java -javaagent:/path/to/opentelemetry-javaagent-<version>.jar \
-     -Dotel.javaagent-extensions=/path/to/cf-java-logging-support-opentelemetry-agent-extension-<versions>.jar \
+     -Dotel.javaagent-extensions=/path/to/cf-java-logging-support-opentelemetry-agent-extension-<version>.jar \
      # your Java application command
 ```
 
@@ -37,27 +38,76 @@ java -javaagent:BOOT-INF/lib/opentelemetry-javaagent-<version>.jar \
 
 See the [example manifest](../sample-spring-boot/manifest-otel-javaagent.yml), how this translates into a deployment description.
 
-For the instrumentation to send observability data to SAP Cloud Logging, the application needs to be bound to a corresponding service instance.
-The service instance can be either managed or [user-provided](#using-user-provided-service-instances).
+Once the agent is attached to the JVM with the ectension in place, there are two ways, which can be used to send data to [SAP Cloud Logging](https://discovery-center.cloud.sap/serviceCatalog/cloud-logging):
+
+1. Use the `cloud-logging` exporters explicitly as provided by the extension.
+This can be achieved via system properties or environment variables:
+```sh
+-Dotel.logs.exporter=cloud-logging \
+-Dotel.metrics.exporter=cloud-logging \
+-Dotel.traces.exporter=cloud-logging
+
+#or
+
+export OTEL_LOGS_EXPORTER=cloud-logging
+export OTEL_METRICS_EXPORTER=cloud-logging
+export OTEL_TRACES_EXPORTER=cloud-logging
+java #...
+```
+
+2. Use the default `otlp` exporter with the provided default configuration from the extension:
+
+```sh
+-Dotel.logs.exporter=otlp \
+-Dotel.metrics.exporter=otlp # default value \
+-Dotel.traces.exporter=otlp # default value
+
+#or
 
-Note, that the OpenTelemetry Java Agent currently only sends traces and metrics by default.
-To enable logs, the additional property `-Dotel.logs.exporter=otlp` is required.
+export OTEL_LOGS_EXPORTER=otlp
+export OTEL_METRICS_EXPORTER=otlp # default value
+export OTEL_TRACES_EXPORTER=otlp # default value
+java #...
+```
+
+Note, that the OpenTelemetry Java Agent currently sends traces and metrics by default using the `otlp` exporter.
+That means, without any configuration the agent with the extension will forward metrics and traces to [SAP Cloud Logging](https://discovery-center.cloud.sap/serviceCatalog/cloud-logging).
+See TODO for the difference between `cloud-logging` and `otlp` exporters.
+The benefit of the `cloud-logging` exporter is, that it can be combined with a different configuration of the `otlp` exporter.
+
+For the instrumentation to send observability data to [SAP Cloud Logging](https://discovery-center.cloud.sap/serviceCatalog/cloud-logging), the application needs to be bound to a corresponding service instance.
+The service instance can be either managed or [user-provided](#using-user-provided-service-instances).
 
 ## Configuration
 
 The OpenTelemetry Java Agent supports a wide variety of [configuration options](https://opentelemetry.io/docs/instrumentation/java/automatic/agent-config/).
 As the extension provides configuration via SPI, all its configuration takes lower precedence than other configuration options for OpenTelemetry.
 Users can easily overwrite any setting using environment variables or system properties.
 
+### Using the Extension
+
+The extension needs to be started with the OpenTelemetry Java Agent as outlined in the [Quick Start Guide](#quickstart-guide).
+You need to enable shipping data either by using the `cloud-logging` exporters or relying on the `otlp` exporters for each signal type.
+Multiple different exporters can be configured with comma separation.
+Using the custom `cloud-logging` exporter will enable you, to use the default `otlp` exporter for different services.
+The extension will configure a default endpoint and credentials for the `otlp` endpoints, so no further configuration is required.
+
+Note, that the `cloud-logging` exporter is just a facade for the `otlp` exporter to allow configuration of multiple data sinks.
+There is no custom network client provided by this extension.
+
 ### Configuring the Extension
 
 The extension itself can be configured by specifying the following system properties:
 
 | Property | Default Value | Comment |
 |----------|---------------|---------|
-| `com.sap.otel.extension.cloud-logging.label` | `cloud-logging` | The label of the managed service binding to bind to. |
-| `com.sap.otel.extension.cloud-logging.tag` | `Cloud Logging` | The tag of any service binding (managed or user-provided) to bind to. |
-| `otel.javaagent.extension.sap.cf.resource.enabled` or `env(OTEL_JAVAAGENT_EXTENSION_SAP_CF_RESOURCE_ENABLED)` | `true` | Whether to add CF resource attributes to all events. |
+| `otel.javaagent.extension.sap.cf.binding.cloud-logging.label` or `com.sap.otel.extension.cloud-logging.label` | `cloud-logging` | The label of the managed service binding to bind to. |
+| `otel.javaagent.extension.sap.cf.binding.cloud-logging.tag` or `com.sap.otel.extension.cloud-logging.tag` | `Cloud Logging` | The tag of any service binding (managed or user-provided) to bind to. |
+| `otel.javaagent.extension.sap.cf.binding.user-provided.label` | `user-provided` | The label of a user-provided service binding to bind to. Note, this label is defined by the Cloud Foundry instance. |
+| `otel.javaagent.extension.sap.cf.resource.enabled` | `true` | Whether to add CF resource attributes to all events. |
+
+> The `otel.javaagent.extension.sap.*` properties are preferred over the `com.sap.otel.extension.*` properties, which are kept for compatibility. 
+Each `otel.javaagent.extension.sap.*` property can also be provided as environment variable `OTEL_JAVAAGENT_EXTENSION_SAP_*`.
 
 The extension will scan the environment variable `VCAP_SERVICES` for CF service bindings.
 User-provided bindings will take precedence over managed bindings of the configured label ("cloud-logging" by default).
@@ -90,7 +140,7 @@ The [OpenTelemetry Java Instrumentation project](https://github.com/open-telemet
 
 ## Using User-Provided Service Instances
 
-The extension provides support not only for managed service instance of SAP Cloud Logging but also for user-provided service instances.
+The extension provides support not only for managed service instance of [SAP Cloud Logging](https://discovery-center.cloud.sap/serviceCatalog/cloud-logging) but also for user-provided service instances.
 This helps to fine-tune the configuration, e.g. leave out or reconfigure the syslog drain.
 Furthermore, this helps on sharing service instances across CF orgs or landscapes.
 
@@ -103,7 +153,7 @@ The extension requires four fields in the user-provided service credentials and
 | `ingest-otlp-cert`| The mTLS client certificate in PEM format matching the client key. Line breaks as `\n`. |
 | `server-ca` | The trusted mTLS server certificate in PEM format. Line breaks as `\n`. |
 
-If you have a SAP Cloud Logging service key, you can generate the required JSON file with jq:
+If you have a [SAP Cloud Logging](https://discovery-center.cloud.sap/serviceCatalog/cloud-logging) service key, you can generate the required JSON file with jq:
 
 ```bash
 cf service-key cls test \
@@ -119,5 +169,17 @@ Using this file, you can create the required user-provided service:
 ```
 
 Note, that you can easily feed arbitrary credentials to the extension.
-It does not need to be SAP Cloud Logging.
+It does not need to be [SAP Cloud Logging](https://discovery-center.cloud.sap/serviceCatalog/cloud-logging).
 You can even change the tag using the configuration parameters of the extension.
+
+## Implementation Differences between Cloud-Logging and OTLP Exporter
+
+The `cloud-logging` exporter provided by this extension is a facade for the `OtlpGrpcExporter` provided by the OpenTelemetry Java Agent, just like the `otlp` exporter.
+The difference is just during the bootstrapping phase.
+The main differences are:
+
+* The `cloud-logging` exporter will send data to all found bindings to SAP Cloud Logging.
+The auto-instrumentation of the `otlp` exporter will just configure the first binding it finds priotizing user-provided services.
+* The `otlp` configuration will write the required certificates and keys to temporary files, which are deleted when the JVM is shut down. The `cloud-logging` exporter will keep the secrets in memory.
+* Since the `otlp` exporter is the default for traces and metrics, just attaching the extension and binding to Cloud Logging will result in metrics and traces being forwarded. 
+The `cloud-logging` exporter needs to be configured explictly as does the `otlp` exporter for logs.

cf-java-logging-support-opentelemetry-agent-extension/dependency-reduced-pom.xml

@@ -32,6 +32,10 @@
                 <excludes>
                   <exclude>io.opentelemetry</exclude>
                   <exclude>com.fasterxml.jackson.core</exclude>
+                  <exclude>com.squareup.okhttp3</exclude>
+                  <exclude>com.squareup.okio</exclude>
+                  <exclude>org.jetbrains.kotlin</exclude>
+                  <exclude>org.jetbrains</exclude>
                 </excludes>
               </artifactSet>
             </configuration>
@@ -58,7 +62,13 @@
       <groupId>io.opentelemetry</groupId>
       <artifactId>opentelemetry-sdk-extension-autoconfigure</artifactId>
       <version>1.31.0</version>
-      <scope>provided</scope>
+      <scope>compile</scope>
+    </dependency>
+    <dependency>
+      <groupId>io.opentelemetry</groupId>
+      <artifactId>opentelemetry-exporter-otlp</artifactId>
+      <version>1.31.0</version>
+      <scope>compile</scope>
     </dependency>
     <dependency>
       <groupId>org.slf4j</groupId>

cf-java-logging-support-opentelemetry-agent-extension/pom.xml

@@ -49,7 +49,10 @@
         <dependency>
             <groupId>io.opentelemetry</groupId>
             <artifactId>opentelemetry-sdk-extension-autoconfigure</artifactId>
-            <scope>provided</scope>
+        </dependency>
+        <dependency>
+            <groupId>io.opentelemetry</groupId>
+            <artifactId>opentelemetry-exporter-otlp</artifactId>
         </dependency>
         <dependency>
             <groupId>io.pivotal.cfenv</groupId>
@@ -83,6 +86,10 @@
                                 <excludes>
                                     <exclude>io.opentelemetry</exclude>
                                     <exclude>com.fasterxml.jackson.core</exclude>
+                                    <exclude>com.squareup.okhttp3</exclude>
+                                    <exclude>com.squareup.okio</exclude>
+                                    <exclude>org.jetbrains.kotlin</exclude>
+                                    <exclude>org.jetbrains</exclude>
                                 </excludes>
                             </artifactSet>
                         </configuration>

cf-java-logging-support-opentelemetry-agent-extension/src/main/java/com/sap/hcf/cf/logging/opentelemetry/agent/ext/CloudLoggingConfigurationCustomizerProvider.java

@@ -13,6 +13,8 @@ public class CloudLoggingConfigurationCustomizerProvider implements AutoConfigur
     public void customize(AutoConfigurationCustomizer autoConfiguration) {
         autoConfiguration
                 .addPropertiesSupplier(new CloudLoggingBindingPropertiesSupplier(cfEnv));
+
+        // ConfigurableLogRecordExporterProvider
     }
 
 }

cf-java-logging-support-opentelemetry-agent-extension/src/main/java/com/sap/hcf/cf/logging/opentelemetry/agent/ext/attributes/CloudFoundryResourceCustomizer.java

@@ -34,14 +34,16 @@ public Resource apply(Resource resource, ConfigProperties configProperties) {
         CfApplication cfApp = cfEnv.getApp();
         ResourceBuilder rb = Resource.builder();
         rb.put("service.name", cfApp.getApplicationName());
-        rb.put("sap.cf.source_id", getString(cfApp, "source_id"));
+        rb.put("sap.cf.source_id", cfApp.getApplicationId());
         rb.put("sap.cf.instance_id", cfApp.getInstanceIndex());
         rb.put("sap.cf.app_id", cfApp.getApplicationId());
         rb.put("sap.cf.app_name", cfApp.getApplicationName());
         rb.put("sap.cf.space_id", cfApp.getSpaceId());
         rb.put("sap.cf.space_name", cfApp.getSpaceName());
         rb.put("sap.cf.org_id", getString(cfApp, "organization_id"));
         rb.put("sap.cf.org_name", getString(cfApp, "organization_name"));
+        rb.put("sap.cf.process.id", getString(cfApp, "process_id"));
+        rb.put("sap.cf.process.type", getString(cfApp, "process_type"));
         return rb.build();
     }
 

cf-java-logging-support-opentelemetry-agent-extension/src/main/java/com/sap/hcf/cf/logging/opentelemetry/agent/ext/binding/CloudLoggingBindingPropertiesSupplier.java

@@ -1,5 +1,7 @@
 package com.sap.hcf.cf.logging.opentelemetry.agent.ext.binding;
 
+import io.opentelemetry.sdk.autoconfigure.spi.ConfigProperties;
+import io.opentelemetry.sdk.autoconfigure.spi.internal.DefaultConfigProperties;
 import io.pivotal.cfenv.core.CfEnv;
 import io.pivotal.cfenv.core.CfService;
 
@@ -17,18 +19,34 @@
 public class CloudLoggingBindingPropertiesSupplier implements Supplier<Map<String, String>> {
 
     private static final Logger LOG = Logger.getLogger(CloudLoggingBindingPropertiesSupplier.class.getName());
-    private static final String CLOUD_LOGGING_LABEL = System.getProperty("com.sap.otel.extension.cloud-logging.label", "cloud-logging");
-    private static final String CLOUD_LOGGING_TAG = System.getProperty("com.sap.otel.extension.cloud-logging.tag", "Cloud Logging");
-    private static final String USER_PROVIDED_LABEL = "user-provided";
-    public static final String OTLP_ENDPOINT = "ingest-otlp-endpoint";
-    public static final String OTLP_CLIENT_KEY = "ingest-otlp-key";
-    public static final String OTLP_CLIENT_CERT = "ingest-otlp-cert";
-    public static final String OTLP_SERVER_CERT = "server-ca";
+    private static final String OTLP_ENDPOINT = "ingest-otlp-endpoint";
+    private static final String OTLP_CLIENT_KEY = "ingest-otlp-key";
+    private static final String OTLP_CLIENT_CERT = "ingest-otlp-cert";
+    private static final String OTLP_SERVER_CERT = "server-ca";
 
-    private final CfEnv cfEnv;
+    private final CloudLoggingServicesProvider cloudLoggingServicesProvider;
 
     public CloudLoggingBindingPropertiesSupplier(CfEnv cfEnv) {
-        this.cfEnv = cfEnv;
+        Map<String, String> defaults = new HashMap<>();
+        defaults.put("com.sap.otel.extension.cloud-logging.label", "cloud-logging");
+        defaults.put("com.sap.otel.extension.cloud-logging.tag", "Cloud Logging");
+        defaults.put("otel.javaagent.extension.sap.cf.binding.user-provided.label", "user-provided");
+        ConfigProperties configProperties = DefaultConfigProperties.create(defaults);
+        this.cloudLoggingServicesProvider = new CloudLoggingServicesProvider(configProperties, cfEnv);
+    }
+
+    private static boolean isBlank(String text) {
+        return text == null || text.trim().isEmpty();
+    }
+
+    private static File writeFile(String prefix, String suffix, String content) throws IOException {
+        File file = File.createTempFile(prefix, suffix);
+        file.deleteOnExit();
+        try (FileWriter writer = new FileWriter(file)) {
+            writer.append(content);
+            LOG.fine("Created temporary file " + file.getAbsolutePath());
+        }
+        return file;
     }
 
     /**
@@ -41,10 +59,7 @@ public CloudLoggingBindingPropertiesSupplier(CfEnv cfEnv) {
      */
     @Override
     public Map<String, String> get() {
-        Stream<CfService> userProvided = cfEnv.findServicesByLabel(USER_PROVIDED_LABEL).stream();
-        Stream<CfService> managed = cfEnv.findServicesByLabel(CLOUD_LOGGING_LABEL).stream();
-        return Stream.concat(userProvided, managed)
-                .filter(svc -> svc.existsByTagIgnoreCase(CLOUD_LOGGING_TAG))
+        return cloudLoggingServicesProvider.get()
                 .findFirst()
                 .map(this::createEndpointConfiguration).orElseGet(Collections::emptyMap);
     }
@@ -94,18 +109,4 @@ private Map<String, String> createEndpointConfiguration(CfService svc) {
         }
     }
 
-    private static boolean isBlank(String text) {
-        return text == null || text.trim().isEmpty();
-    }
-
-    private static File writeFile(String prefix, String suffix, String content) throws IOException {
-        File file = File.createTempFile(prefix, suffix);
-        file.deleteOnExit();
-        try (FileWriter writer = new FileWriter(file)) {
-            writer.append(content);
-            LOG.fine("Created temporary file " + file.getAbsolutePath());
-        }
-        return file;
-    }
-
 }

cf-java-logging-support-opentelemetry-agent-extension/src/main/java/com/sap/hcf/cf/logging/opentelemetry/agent/ext/binding/CloudLoggingServicesProvider.java

@@ -0,0 +1,49 @@
+package com.sap.hcf.cf.logging.opentelemetry.agent.ext.binding;
+
+import io.opentelemetry.sdk.autoconfigure.spi.ConfigProperties;
+import io.pivotal.cfenv.core.CfEnv;
+import io.pivotal.cfenv.core.CfService;
+
+import java.util.List;
+import java.util.function.Supplier;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+public class CloudLoggingServicesProvider implements Supplier<Stream<CfService>> {
+
+    private static final String DEFAULT_USER_PROVIDED_LABEL = "user-provided";
+    private static final String DEFAULT_CLOUD_LOGGING_LABEL = "cloud-logging";
+    private static final String DEFAULT_CLOUD_LOGGING_TAG = "Cloud Logging";
+
+    private final List<CfService> services;
+
+    public CloudLoggingServicesProvider(ConfigProperties config, CfEnv cfEnv) {
+        String userProvidedLabel = getUserProvidedLabel(config);
+        String cloudLoggingLabel = getCloudLoggingLabel(config);
+        String cloudLoggingTag = getCloudLoggingTag(config);
+        List<CfService> userProvided = cfEnv.findServicesByLabel(userProvidedLabel);
+        List<CfService> managed = cfEnv.findServicesByLabel(cloudLoggingLabel);
+        this.services = Stream.concat(userProvided.stream(), managed.stream())
+                .filter(svc -> svc.existsByTagIgnoreCase(cloudLoggingTag))
+                .collect(Collectors.toList());
+    }
+
+    private String getUserProvidedLabel(ConfigProperties config) {
+        return config.getString("otel.javaagent.extension.sap.cf.binding.user-provided.label", DEFAULT_USER_PROVIDED_LABEL);
+    }
+
+    private String getCloudLoggingLabel(ConfigProperties config) {
+        String fromOwnProperties = System.getProperty("com.sap.otel.extension.cloud-logging.label", DEFAULT_CLOUD_LOGGING_LABEL);
+        return config.getString("otel.javaagent.extension.sap.cf.binding.cloud-logging.label", fromOwnProperties);
+    }
+
+    private String getCloudLoggingTag(ConfigProperties config) {
+        String fromOwnProperties = System.getProperty("com.sap.otel.extension.cloud-logging.tag", DEFAULT_CLOUD_LOGGING_TAG);
+        return config.getString("otel.javaagent.extension.sap.cf.binding.cloud-logging.tag", fromOwnProperties);
+    }
+
+    @Override
+    public Stream<CfService> get() {
+        return services.stream();
+    }
+}