-
Notifications
You must be signed in to change notification settings - Fork 936
Define an Entity Merge algorithm (from OTEP #264) #4768
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
f788395
321199d
5f7deeb
9b29ce2
53cbb18
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -13,6 +13,8 @@ weight: 2 | |
| <!-- toc --> | ||
|
|
||
| - [Identity](#identity) | ||
| - [Merging Resources](#merging-resources) | ||
| * [Merging Entities into a Resource](#merging-entities-into-a-resource) | ||
|
|
||
| <!-- tocstop --> | ||
|
|
||
|
|
@@ -47,3 +49,35 @@ different if one contains an entity not found in the other. | |
| Some resources include raw attributes in additon to Entities. Raw attributes are | ||
| considered identifying on a resource. That is, if the key-value pairs of | ||
| raw attributes are different, then you can assume the resource is different. | ||
|
|
||
| ## Merging Resources | ||
|
|
||
| Note: The current SDK specification outlines a [merge algorithm](sdk#merge). | ||
| This specification updates the algorithm to be compliant with entities. This | ||
| section will replace that section upon stabilization of entities. SDKs SHOULD | ||
| NOT update their merge algorithm until full Entity SDK support is provided. | ||
|
|
||
| Merging resources is an action of joining together the context of observation. | ||
| That is, we can look at the resource context for a signal and *expand* that | ||
| context to include more details (see | ||
| [telescoping identity](README.md#telescoping)). As such, a merge SHOULD preserve | ||
| any identity that already existed on a Resource while adding in new identifying | ||
| information or descriptive attributes. | ||
|
|
||
| ### Merging Entities into a Resource | ||
|
|
||
| We define the following algorithm for merging entities into an existing | ||
| resource. | ||
|
|
||
| - Construct a set of existing entities on the resource, `E`. | ||
| - For each entity, `new_entity`, in priority order (highest first), | ||
| do one of the following: | ||
| - If an entity `e'` exsits in `E` with the same entity type as `new_entity`: | ||
| - Perform a [Entity DataModel Merge](../entities/data-model.md#merging-of-entities), if applicable, otherwise ignore `new_entity`. | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
nit: maybe rephrase to "if they are mergeable"? If applicable confused me here.
Do we consider this an error (e.g. surfaced using the SDK error handler or logged as a warning in the collector)? This makes it sound like these cases should be silently ignored. |
||
| - Otherwise, add the entity `new_entity` to set `E` | ||
| - Update the Resource to use the set of entities `E`. | ||
| - If all entities within `E` have the same `schema_url`, set the | ||
| resources `schema_url` to match. | ||
| - Otherwise set the Resource `schema_url` blank. | ||
| - Remove any attribute from `Attributes` which exists in either the | ||
| description or identity of an entity in `E`. | ||
|
Comment on lines
+82
to
+83
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This step is unclear to me. Does it mean that any attributes of the
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It's more that they override them. Remember, in the datamodel, attributes are a separate thing from entities vs. in OTLP. So what this says, is in the datamodel, you remove it from the "loose" attributes, because it exists in the new entity's attributes. The new entity will provide a new value that lands in OTLP resource.
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. IMO this deserves a clarifying note, as I think most readers (for better or worse) are more familiar with OTLP than the data model document, and assume Note: For cases where Entity Attributes are encoded as references to Resource Attributes, as is the case for OTLP, this implies Entity Attributes take precedence over any Resource Attributes which are not associated with an Entity. Even though entities in the data model have their own set of attributes, I still agree that entity conflict resolution needs to be a part of the merge process. E.g. if two entities claim to have detected different values for the same key, it seems like that still needs to be resolved when merging entities into a resource. |
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How does an SDK determine the relative priority of entities?