-
Notifications
You must be signed in to change notification settings - Fork 25.6k
OTLP: determine target data stream based on attributes and scope name #133903
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
13 commits
Select commit
Hold shift + click to select a range
e766658
OTLP: data stream routing based on attributes and scope name
felixbarny 8a95e5d
Merge remote-tracking branch 'origin/main' into otlp-index-routing
felixbarny ec56b19
Address review comments
felixbarny 39fe446
[CI] Auto commit changes from spotless
dd0cf6f
Merge remote-tracking branch 'origin/main' into otlp-index-routing
felixbarny f811ab1
Serialize data stream object
felixbarny 6f196b1
[CI] Auto commit changes from spotless
4883108
Merge remote-tracking branch 'origin/main' into otlp-index-routing
felixbarny ec8ef6e
Address review comments
felixbarny 4147742
[CI] Auto commit changes from spotless
38b41d1
Merge remote-tracking branch 'origin/main' into otlp-index-routing
felixbarny 0a022b4
Merge remote-tracking branch 'felixbarny/otlp-index-routing' into otl…
felixbarny 051723c
[CI] Auto commit changes from spotless
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
158 changes: 158 additions & 0 deletions
158
.../otel-data/src/main/java/org/elasticsearch/xpack/oteldata/otlp/datapoint/TargetIndex.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,158 @@ | ||
| /* | ||
| * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one | ||
| * or more contributor license agreements. Licensed under the Elastic License | ||
| * 2.0; you may not use this file except in compliance with the Elastic License | ||
| * 2.0. | ||
| */ | ||
|
|
||
| package org.elasticsearch.xpack.oteldata.otlp.datapoint; | ||
|
|
||
| import io.opentelemetry.proto.common.v1.KeyValue; | ||
|
|
||
| import org.elasticsearch.cluster.metadata.DataStream; | ||
| import org.elasticsearch.core.Nullable; | ||
|
|
||
| import java.util.List; | ||
|
|
||
| /** | ||
| * Represents the target index for a data point, which can be either a specific index or a data stream. | ||
| * The index is determined based on attributes, scope name, and default values. | ||
| */ | ||
| public final class TargetIndex { | ||
|
|
||
| public static final String TYPE_METRICS = "metrics"; | ||
|
|
||
| private static final String ELASTICSEARCH_INDEX = "elasticsearch.index"; | ||
| private static final String DATA_STREAM_DATASET = "data_stream.dataset"; | ||
| private static final String DATA_STREAM_NAMESPACE = "data_stream.namespace"; | ||
| private static final String DEFAULT_DATASET = "generic"; | ||
| private static final String OTEL_DATASET_SUFFIX = ".otel"; | ||
| private static final String DEFAULT_NAMESPACE = "default"; | ||
| private static final TargetIndex DEFAULT_METRICS_TARGET = evaluate(TYPE_METRICS, List.of(), null, List.of(), List.of()); | ||
|
|
||
| private String index; | ||
| private String type; | ||
| private String dataset; | ||
| private String namespace; | ||
|
|
||
| public static TargetIndex defaultMetrics() { | ||
| return DEFAULT_METRICS_TARGET; | ||
| } | ||
|
|
||
| public static boolean isTargetIndexAttribute(String attributeKey) { | ||
| return attributeKey.equals(ELASTICSEARCH_INDEX) | ||
| || attributeKey.equals(DATA_STREAM_DATASET) | ||
| || attributeKey.equals(DATA_STREAM_NAMESPACE); | ||
| } | ||
|
|
||
| /** | ||
| * Determines the target index for a data point. | ||
| * | ||
| * @param type The data stream type (e.g., "metrics", "logs"). | ||
| * @param attributes The attributes associated with the data point. | ||
| * @param receiverName The name of the receiver, which may influence the dataset (receiver-based routing). | ||
| * @param scopeAttributes Attributes associated with the scope. | ||
| * @param resourceAttributes Attributes associated with the resource. | ||
| * @return A TargetIndex instance representing the target index for the data point. | ||
| */ | ||
| public static TargetIndex evaluate( | ||
| String type, | ||
| List<KeyValue> attributes, | ||
| @Nullable String receiverName, | ||
| List<KeyValue> scopeAttributes, | ||
| List<KeyValue> resourceAttributes | ||
| ) { | ||
| // Order: | ||
| // 1. elasticsearch.index from attributes, scope.attributes, resource.attributes | ||
| // 2. read data_stream.* from attributes, scope.attributes, resource.attributes | ||
| // 3. receiver-based routing based on scope.name | ||
| // 4. use default hardcoded data_stream.* (<type>-generic-default) | ||
| TargetIndex target = new TargetIndex(); | ||
| target.populateFrom(attributes); | ||
| target.populateFrom(scopeAttributes); | ||
| target.populateFrom(resourceAttributes); | ||
| if (target.index == null) { | ||
| target.type = type; | ||
| if (target.dataset == null && receiverName != null) { | ||
| target.dataset = receiverName; | ||
| } | ||
| target.dataset = DataStream.sanitizeDataset(target.dataset); | ||
| if (target.dataset == null) { | ||
| target.dataset = DEFAULT_DATASET; | ||
| } | ||
| // add otel suffix to match OTel index template | ||
| target.dataset = target.dataset + OTEL_DATASET_SUFFIX; | ||
| target.namespace = DataStream.sanitizeNamespace(target.namespace); | ||
|
|
||
| if (target.namespace == null) { | ||
| target.namespace = DEFAULT_NAMESPACE; | ||
| } | ||
| target.index = target.type + "-" + target.dataset + "-" + target.namespace; | ||
| } else { | ||
| target.type = null; | ||
| target.dataset = null; | ||
| target.namespace = null; | ||
| } | ||
| return target; | ||
| } | ||
|
|
||
| private TargetIndex() {} | ||
|
|
||
| private void populateFrom(List<KeyValue> attributes) { | ||
| if (isPopulated()) { | ||
| return; | ||
| } | ||
| for (int i = 0, size = attributes.size(); i < size; i++) { | ||
| KeyValue attr = attributes.get(i); | ||
| if (attr.getKey().equals(ELASTICSEARCH_INDEX)) { | ||
| index = attr.getValue().getStringValue(); | ||
felixbarny marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| } else if (dataset == null && attr.getKey().equals(DATA_STREAM_DATASET)) { | ||
| dataset = attr.getValue().getStringValue(); | ||
| } else if (namespace == null && attr.getKey().equals(DATA_STREAM_NAMESPACE)) { | ||
| namespace = attr.getValue().getStringValue(); | ||
| } | ||
| } | ||
| } | ||
|
|
||
| private boolean isPopulated() { | ||
| return (dataset != null && namespace != null) || index != null; | ||
| } | ||
|
|
||
| public boolean isDataStream() { | ||
| return type != null && dataset != null && namespace != null; | ||
kkrik-es marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| } | ||
|
|
||
| public String index() { | ||
| return index; | ||
| } | ||
|
|
||
| public String type() { | ||
| return type; | ||
| } | ||
|
|
||
| public String dataset() { | ||
| return dataset; | ||
| } | ||
|
|
||
| public String namespace() { | ||
| return namespace; | ||
| } | ||
|
|
||
| @Override | ||
| public String toString() { | ||
| return index; | ||
| } | ||
|
|
||
| @Override | ||
| public boolean equals(Object o) { | ||
| if (o == null || getClass() != o.getClass()) return false; | ||
|
|
||
| TargetIndex that = (TargetIndex) o; | ||
| return index.equals(that.index); | ||
| } | ||
|
|
||
| @Override | ||
| public int hashCode() { | ||
| return index.hashCode(); | ||
| } | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I wonder if these should be ordered e.g. lexicographically, so that naming is deterministic in the presence of multiple matching attributes.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorting may be pretty expensive. But I see what you mean. If both
elasticsearch.indexanddata_stream.*attributes are set, it depends on the ordering. We can probably fix that by defining a precedence and only checking forisPopulatedin the beginning of the method.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Wouldn't this still use the first match on
DATA_STREAM_DATASETorDATA_STREAM_NAMESPACE? These are user-provided, so the list order is arbitrary? Maybe a unittest will help here.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not sure I understand. The
DATA_STREAM_DATASETandDATA_STREAM_NAMESPACEattributes populate different properties. The order doesn't matter as we're iterating over all attributes.