Skip to content

As a library

Jump to bottom
Manpreet edited this page Jul 22, 2023 · 11 revisions

In this mode, you can download the JAR file from the release artifacts and import it as a dependency in your code. We are working on the process to publish this to Maven so that the dependency import is easier.

Contents

Pre-requisites

Input definitions

As mentioned in the User Guide, the first pre-requisite to using the tool is preparing the resource definition and at least one of metrics, logs or trace definition input files. If you are running a quick experiment or PoC, you may re-use the example definition files.
Once you have the input files ready, an instance of GeneratorInput can be initialized as follows for YAML input files:

GeneratorInput generatorInput = new GeneratorInput.YAMLFilesBuilder(RESOURCES_YAML)
    .withMetricDefinitionYAML(METRICS_YAML)
    .withLogDefinitionYAML(LOGS_YAML)
    .withTraceDefinitionYAML(TRACES_YAML)
    .build();

and for JSON input files:

GeneratorInput generatorInput = new GeneratorInput.JSONFilesBuilder(RESOURCES_JSON)
    .withMetricDefinitionJSON(METRICS_JSON)
    .withLogDefinitionJSON(LOGS_JSON)
    .withTraceDefinitionJSON(TRACES_JSON)
    .build();

There may be cases where you want to programmatically modify the content of these input files before they are consumed by the generator. To do this, you can deserialize them into the corresponding Resources, Metrics, Logs and Traces DTOs and after the modification call the DTOBuilder instead of the YAMLBuilder or JSONBuilder to initialize the GeneratorInput.

GeneratorInput generatorInput = new GeneratorInput.DTOBuilder(resources)
    .withMetricDefinitions(metrics)
    .withLogDefinitions(logs)
    .withTraceDefinitions(traces)
    .build();

AuthHandler and PayloadHandler

The test telemetry generator provides interfaces and a few basic implementations for those to post your data to a HTTP endpoint. The AuthHandler interface is to specify the authentication scheme used by the endpoint and only consists of a single method to provide the value for the HTTP Authorization header.

public interface AuthHandler {
    String getAuthString();
}

Currently we have:

  • NoAuthHandler which would lead to no Authorization header sent with the request
  • BasicAuthHandler available which accepts a username and a password We will soon add support for OAuth authorization which provides a Bearer token.

The other interface is the PayloadHandler which is responsible for posting the metrics, logs and traces packets to an endpoint. It is also fairly simple with a single method only returning a boolean indicating whether the payload was posted successfully.

public interface PayloadHandler {
    boolean postPayload(GeneratedMessageV3 message);
}

We have two implementations for the payload handler available - RESTPayloadHandler and GRPCPayloadHandler. Other than the endpoint details, both of these implementations need an implementation of AuthHandler which means that you can provide your own implementation and customize the way authentication is done.

Other than the AuthHandler object, the GRPCPayloadHandler needs the host and gRPCPort where you want to post the data. Similarly, the RESTPayloadHandler needs an endpoint URL along with an instance of AuthHandler. In cases where the URL path in the REST endpoint may be different for metrics, logs and traces, there are setters available for these which will append these URL paths to the base endpoint URL.
The generator needs an instance of PayloadHandler so you write your own implementation and provide an instance of the same.

Running the generator

Once you have the instances of GeneratorInput and PayloadHandler instance ready, the main class TelemetryGenerator can be initialized and the data generation process started as follows:

TelemetryGenerator telemetryGenerator = new TelemetryGenerator(generatorInput, payloadHandler);
telemetryGenerator.runGenerator();

Logging

We are using logback for printing out the log messages and by default the message level is INFO. The messages will be written to the STDOUT as well as in the logs/generator.log file. At the default INFO level, some high level information about the status of the data generation is provided. In case you are running into an issue or you simply want to see all the data that is being generated in OpenTelemetry format, simply set LOG_LEVEL=DEBUG in the environment.

Transport Storage

The main TelemetryGenerator class accepts a boolean to flag if you want all the OpenTelemetry payloads that are being posted to be stored along with the boolean status of whether posting of that particular payload to the specified endpoint was successful or not. These are stored in memory so if you do not need this, setting this to false while initializing the TelemetryGenerator is preferable. If being used, after the data generation is complete, the object storing these can be fetched as follows:

TelemetryGenerator telemetryGenerator = new TelemetryGenerator(generatorInput, payloadHandler, true);
telemetryGenerator.runGenerator();
TransportStorage transportStorage = telemetryGenerator.getTransportStorage();

The TransportStorage class has all the data stored in maps:

private ConcurrentMap<String, List<ExportMetricsServiceRequest>> storedMetricsPayloads;
private ConcurrentMap<String, Map<String, List<ExportLogsServiceRequest>>> storedLogsPayloads;
private ConcurrentMap<String, List<ExportTraceServiceRequest>> storedTracesPayloads;
private ConcurrentMap<String, List<Boolean>> metricsResponses;
private ConcurrentMap<String, Map<String, List<Boolean>>> logsResponses;
private ConcurrentMap<String, List<Boolean>> tracesResponses;

The keys for these are composed as follows:

  • The metrics are grouped by the reporting resource so that all the metrics for a certain resource type are in the same payload. Further, if the metrics have different payload count and/or payload frequency, they will be grouped separately into different threads. So the key for storing metrics payloads and the statuses of those payloads would be resource_type::payload_frequency::payload_count.
  • The logs are simply grouped by their name. In case you did not provide a name, the telemetry generator would have automatically set a name/id for it.
  • The traces are grouped by the name of the root span. However, if a high copy count was provided, they may be split into different threads and each of them will have a separate key like root_span::group::0, root_span::group::1 and so on.

Programmatic monitoring of data generation

By default, when the TelemetryGenerator.runGenerator() method is called, it will block the main thread. However, there might be cases where you want to start it in a separate thread and monitor the data generation status programmatically. To handle such cases, the GeneratorsMonitor class provides methods like getMetricsCurrentPayloadCount() to get the current number of metrics payloads that have been posted and getMetricsTotalPayloadCount() to get the total number of payloads to be posted as per the definitions. Similar methods are available for logs as well as traces. Also, the killGenerator() method helps to cleanly exit the data generation process.

User defined expressions

By default, the telemetry generator provides a wide array of expressions for generating attribute values, metric values, resource parent-child mapping and more. However, if using it as a dependency, there is a way to implement and provide your own expressions in the definition files. We first implement our own expression methods (example) and they have to be public and static. After that we can simply add a reference to the methods in the ResourceExpressionsJELProvider class or the MELTExpressionsJELProvider depending on if we are going to use the expression in resource definition or metric/log/trace definitions or both.

ResourceExpressionsJELProvider.addExpression("", "", "io.opentelemetry.contrib.generator.telemetry.jel.CustomExpressions", "geometricWith2");
MELTExpressionsJELProvider.addExpression("", "", "io.opentelemetry.contrib.generator.telemetry.jel.CustomExpressions", "geometricWith2");

This should be done before the generator is started. Various examples of how to do this are available in the tests. Also, reading more about the ELProcessor class in Jakarta Expressions Language API will be helpful.

We are continuously adding new features and support for some much needed functionalities. All such work is being tracked via the open issues. Please feel free to open an issue if something you'd like to see is not available.

Clone this wiki locally