[entities] contradiction between resource and entity attributes

[dyladan]

During the spec meeting yesterday we discussed OTEP 4485: Extending attributes to support complex values. During the discussion we talked about which signals may add support for complex attributes. Notably, resource does not and will not support complex attributes. Entity descriptive attributes, but not identifying attributes, already plan to support complex attributes as stated in the Entity Data Model.

This raises a contradiction because entity attributes are all currently transmitted as resource attributes. If an entity descriptive attribute is added with a complex value it cannot be transmitted unmodified as a part of the resource attributes while maintaining the existing definition of resource attribute values.

edit: OTEP 4485 has been updated to propose supporting complex attributes in resource, removing the contradiction if it is accepted and merged.

Below are some ways we may be able to move forward:

Allow complex attributes in resource [OTEP 4485]

Pros:

• Simple to understand
• All entity attributes included in resource

Cons:

• Some may view this as a breaking change
• Duplication of data causing higher transmission costs

Support complex attributes in resource via entity descriptive attributes only

1. Require SDK support for complex resource attributes (export pipeline).
2. DO NOT allow complex attributes via the existing resource detectors. The only way to add a complex resource attribute is to include it in the descriptive attributes of a detected entity.

pros:

• Avoids the problem of metrics with complex attribute identity while allowing complex descriptive attributes
• Any backends that do not support complex resource attributes can safely drop/stringify them without affecting identity
• Backwards compatible

cons:

• Some duplication of data with entity signal (maybe mitigated by potential configuration options to drop descriptive attributes)

Drop complex descriptive entity attributes, include them in new entity signal only

Not needed if the current OTEP 4485 proposal is accepted.

Pros:

• No breaking change to resource
• Save some bandwidth by dropping some complex, possibly large, attributes from every telemetry export

Cons:

• Some, but not all, entity attributes are missing from resource. It may be unclear to users why.

Same as above but instead of dropping, stringify. Include complex representation in entity signal

Pros:

• No breaking change to resource
• All entity attributes included in resource

Cons:

• Processing cost on both client and server
• Increased transmission cost
• Processing pipelines may modify one signal but not the other resulting in conflicts

Drop all descriptive entity attributes, include them in new entity signal only

Less urgent with OTEP 4485 but may still have value.

Pros:

• No breaking change to resource
• Save bandwidth by dropping all unnecessary attributes from every telemetry export (identity attributes required in order to associate telemetry with entity)

Cons:

• Requires more work on the backend
• Entity signal may arrive out of order with telemetry or not come at all.

---

edited 4/24 to add Support complex attributes in resource via entity descriptive attributes only

[trask]

Allow complex attributes in resource

We updated #4485 yesterday after discussions in spec and log sig meetings and this is now the current proposal.

[dyladan]

Edited the issue to reflect that

[dyladan]

We discussed this in the entities SIG today. Here's a short summary of the discussion:

1. We still definitely want to have complex descriptive entity attributes.
2. Complex attributes in identity are problematic, especially for metrics. Because entity will determine metric identity (similar to existing resource) this means it is a problem for us as well.
3. Complex attributes in resource are only problematic insofar as they contribute to identity. IF we can reasonably distinguish between identifying and descriptive resource attributes, we may be able to have complex resource attributes which are descriptive only. This is one of the things entities was conceived to address.
4. Dropping all descriptive attributes from the resource representation of entities works well only for those users who are fully bought in on entities including receiving the entities signal. May be a good configurable option, but is probably not a reasonable default behavior before entities is widely adopted.
5. We will likely have to do one of the intermediate proposals such as stringifying or dropping complex resource attributes, or only allowing them if they are descriptive.

[dyladan]

Another proposal came up on the call that I will lay out here:

Support complex attributes in resource via entity descriptive attributes only

1. Require SDK support for complex resource attributes (export pipeline).
2. DO NOT allow complex attributes via the existing resource detectors. The only way to add a complex resource attribute is to include it in the descriptive attributes of a detected entity.

pros:

• Avoids the problem of metrics with complex attribute identity while allowing complex descriptive attributes
• Any backends that do not support complex resource attributes can safely drop/stringify them without affecting identity
• Backwards compatible

cons:

• Some duplication of data with entity signal (maybe mitigated by potential configuration options to drop descriptive attributes)

[dyladan]

Closing this because the OTEP merged. The end result is that complex attributes are allowed everywhere but are discouraged in places where they affect metric identity (resources, metric attributes). We should apply the same logic to entities, which would mean discouraging their use in at least identifying entity attributes.