[EndGoal] [DoNotMerge] Extend the set of attribute value types

oteps/4485-extending-attributes-to-support-complex-values.md

@@ -94,7 +94,7 @@ extending the standard attributes provides a more seamless and user-friendly API
 
 Currently, the SDK specification has a clause that says extending
 the set of standard attribute would be
-[considered a breaking change](/specification/common/README.md#standard-attribute).
+[considered a breaking change](/specification/common/README.md#attribute).
 
 We believe that removing this clause and extending standard
 attributes can be done gracefully across the OpenTelemetry ecosystem

oteps/entities/0256-entities-data-model.md

@@ -139,7 +139,7 @@ MAY change over the lifetime of the entity. MAY be empty. These
 attributes are not part of entity's identity.
 <p>
 Follows <a
-href="../../specification/logs/data-model.md#type-any">any</a>
+href="https://github.com/open-telemetry/opentelemetry-specification/blob/v1.44.0/specification/logs/data-model.md#type-any">any</a>
 value definition in the OpenTelemetry spec - it can be a scalar value,
 byte array, an array or map of values. Arbitrary deep nesting of values
 for arrays and maps is allowed.
@@ -682,7 +682,7 @@ There are a couple of reasons:
 ### Attribute Data Type
 
 The data model requires the Attributes field to use the extended
-[any](../../specification/logs/data-model.md#type-any)
+[any](https://github.com/open-telemetry/opentelemetry-specification/blob/v1.44.0/specification/logs/data-model.md#type-any)
 attribute values, that allows more complex data types. This is different from the data
 type used by the Id field, which is more restricted in the shape.
 

specification/common/README.md

@@ -15,50 +15,81 @@ path_base_for_github_subdir:
 
 <!-- toc -->
 
+- [AnyValue](#anyvalue)
+- [map<string, AnyValue>](#mapstring-anyvalue)
 - [Attribute](#attribute)
-  * [Standard Attribute](#standard-attribute)
-  * [Attribute Limits](#attribute-limits)
-    + [Configurable Parameters](#configurable-parameters)
-    + [Exempt Entities](#exempt-entities)
-- [Attribute Collections](#attribute-collections)
+  * [Attribute Collections](#attribute-collections)
+- [Attribute Limits](#attribute-limits)
+  * [Configurable Parameters](#configurable-parameters)
+  * [Exempt Entities](#exempt-entities)
 
 <!-- tocstop -->
 
 </details>
 
-## Attribute
-
-<a id="attributes"></a>
+## AnyValue
 
-An `Attribute` is a key-value pair, which MUST have the following properties:
+`AnyValue` is either:
 
-- The attribute key MUST be a non-`null` and non-empty string.
-  - Case sensitivity of keys is preserved. Keys that differ in casing are treated as distinct keys.
-- The attribute value is either:
-  - A primitive type: string, boolean, double precision floating point (IEEE 754-1985) or signed 64 bit integer.
-  - An array of primitive type values. The array MUST be homogeneous,
-    i.e., it MUST NOT contain values of different types.
+- a primitive type: string, boolean, double precision floating point
+  (IEEE 754-1985), or signed 64 bit integer,
+- a homogeneous array of primitive type values. A homogeneous array MUST NOT
+  contain values of different types.
+- a byte array.
+- a heterogeneous array of `AnyValue`,
+- a [`map<string, AnyValue>`](#mapstring-anyvalue),
+- an empty value (e.g. `null`, `undefined` in JavaScript/TypeScript,
+  `None` in Python, `nil` in Go/Ruby, etc.).
 
-For protocols that do not natively support non-string values, non-string values SHOULD be represented as JSON-encoded strings.  For example, the expression `int64(100)` will be encoded as `100`, `float64(1.5)` will be encoded as `1.5`, and an empty array of any type will be encoded as `[]`.
+For protocols that do not natively support non-string values, non-string values
+SHOULD be represented as JSON-encoded strings. For example, the expression
+`int64(100)` will be encoded as `100`, `float64(1.5)` will be encoded as `1.5`,
+and an empty array of any type will be encoded as `[]`.
 
-Attribute values expressing a numerical value of zero, an empty string, or an
-empty array are considered meaningful and MUST be stored and passed on to
+AnyValues expressing an empty value, a numerical value of zero, an empty string,
+or an empty array are considered meaningful and MUST be stored and passed on to
 processors / exporters.
 
-Attribute values of `null` are not valid and attempting to set a `null` value is
-undefined behavior.
-
-`null` values SHOULD NOT be allowed in arrays. However, if it is impossible to
-make sure that no `null` values are accepted
+While `null` is a valid attribute value, its use within homogeneous arrays
+SHOULD generally be avoided unless language constraints make this impossible.
+However, if it is impossible to make sure that no `null` values are accepted
 (e.g. in languages that do not have appropriate compile-time type checking),
-`null` values within arrays MUST be preserved as-is (i.e., passed on to span
+`null` values within homogeneous arrays MUST be preserved as-is (i.e., passed on to
 processors / exporters as `null`). If exporters do not support exporting `null`
 values, they MAY replace those values by 0, `false`, or empty strings.
 This is required for map/dictionary structures represented as two arrays with
 indices that are kept in sync (e.g., two attributes `header_keys` and `header_values`,
 both containing an array of strings to represent a mapping
 `header_keys[i] -> header_values[i]`).
 
+## map<string, AnyValue>
+
+`map<string, AnyValue>` is a map of string keys to `AnyValue` values.
+The keys in the map are unique (duplicate keys are not allowed).
+
+Arbitrary deep nesting of values for arrays and maps is allowed (essentially
+allows to represent an equivalent of a JSON object).
+
+The representation of the map is language-dependent.
+
+The implementation MUST by default ensure that the exported maps contain only unique keys.
+
+The implementation MAY have an option to allow exporting maps with duplicate keys
+(e.g. for better performance).
+If such option is provided, it MUST be documented that for many receivers,
+handling of maps with duplicate keys is unpredictable and it is the users'
+responsibility to ensure keys are not duplicate.
+
+## Attribute
+
+<a id="attributes"></a>
+
+An `Attribute` is a key-value pair, which MUST have the following properties:
+
+- The attribute key MUST be a non-`null` and non-empty string.
+  - Case sensitivity of keys is preserved. Keys that differ in casing are treated as distinct keys.
+- The attribute value MUST be one of types defined in [AnyValue](#anyvalue).
+
 Attributes are equal when their keys and values are equal.
 
 See [Attribute Naming](https://github.com/open-telemetry/semantic-conventions/blob/main/docs/general/naming.md#attributes) for naming guidelines.
@@ -68,22 +99,52 @@ See [Requirement Level](https://github.com/open-telemetry/semantic-conventions/b
 See [this document](attribute-type-mapping.md) to find out how to map values obtained
 outside OpenTelemetry into OpenTelemetry attribute values.
 
-### Standard Attribute
+### Attribute Collections
 
-Attributes are used in various places throughout the OpenTelemetry data model.
-We designate the [previous attribute section](#attribute) as the standard
-attribute definition, in order to facilitate more intuitive and consistent API /
-SDK design.
+[Resources](../resource/sdk.md),
+[Instrumentation Scopes](instrumentation-scope.md),
+[Metric points](../metrics/data-model.md#metric-points),
+[Spans](../trace/api.md#set-attributes), Span
+[Events](../trace/api.md#add-events), Span
+[Links](../trace/api.md#link) and
+[Log Records](../logs/data-model.md),
+contain a collection of attributes.
 
-The standard attribute definition SHOULD be used to represent attributes in data
-modeling unless there is a strong justification to diverge. For example, the Log
-Data Model has an extended [attributes](../logs/data-model.md#field-attributes)
-definition allowing values of [type `Any`](../logs/data-model.md#type-any). This
-reflects that LogRecord attributes are expected to model data produced from
-external log APIs, which do not necessarily have the same value type
-restrictions as the standard attribute definition.
+Implementation MUST by default ensure that the exported attribute collections
+contain only unique keys. The enforcement of uniqueness may be performed
+in a variety of ways as it best fits the limitations of the particular
+implementation.
+
+Normally for the telemetry generated using OpenTelemetry SDKs the attribute
+key-value pairs are set via an API that either accepts a single key-value pair
+or a collection of key-value pairs. Setting an attribute with the same key as an
+existing attribute SHOULD overwrite the existing attribute's value. See for
+example Span's [SetAttribute](../trace/api.md#set-attributes) API.
 
-### Attribute Limits
+A typical implementation of [SetAttribute](../trace/api.md#set-attributes) API
+will enforce the uniqueness by overwriting any existing attribute values pending
+to be exported, so that when the Span is eventually exported the exporters see
+only unique attributes. The OTLP format in particular requires that exported
+Resources, Spans, Metric data points and Log Records contain only unique
+attributes.
+
+Some other implementations may use a streaming approach where every
+[SetAttribute](../trace/api.md#set-attributes) API call immediately results in
+that individual attribute value being exported using a streaming wire protocol.
+In such cases the enforcement of uniqueness will likely be the responsibility of
+the recipient of this data.
+
+Implementations MAY have an option to allow exporting attribute collections
+with duplicate keys (e.g. for better performance).
+If such option is provided, it MUST be documented that for many receivers,
+handling of maps with duplicate keys is unpredictable and it is the users'
+responsibility to ensure keys are not duplicate.
+
+Collection of attributes are equal when they contain the same attributes,
+irrespective of the order in which those elements appear
+(unordered collection equality).
+
+## Attribute Limits
 
 Execution of erroneous code can result in unintended attributes. If there are no
 limits placed on attributes, they can quickly exhaust available memory, resulting
@@ -99,12 +160,19 @@ If an SDK provides a way to:
   - if it is a string, if it exceeds that limit (counting any character in it as
     1), SDKs MUST truncate that value, so that its length is at most equal
     to the limit,
-  - if it is an array of strings, then apply the above rule to each of the
-    values separately,
+  - if it is a byte array, if it exceeds that limit (counting each byte as 1),
+    SDKs MUST truncate that value, so that its length is at most equal to the limit,
+  - if it is an array of [AnyValue](#anyvalue), then apply the limit to
+    each value within the array recursively,
+  - if it is a [`map<string, AnyValue>`](#mapstring-anyvalue), then apply the
+    limit to each value within the map recursively,
   - otherwise a value MUST NOT be truncated;
-- set a limit of unique attribute keys such that:
-  - for each unique attribute key, addition of which would result in exceeding
-    the limit, SDK MUST discard that key/value pair.
+- set an attribute count limit such that:
+  - if an attribute addition into an attribute collection would result
+    in exceeding the limit (counting each attribute in the collection as 1),
+    SDK MUST discard that attribute, so that the total number of attributes in
+    an attribute collection is at most equal to the limit;
+  - otherwise an attribute MUST NOT be discarded.
 
 There MAY be a log emitted to indicate to the user that an attribute was
 truncated or discarded. To prevent excessive logging, the log MUST NOT be
@@ -121,12 +189,12 @@ use the model-specific limit, if it isn't set, then the SDK MUST attempt to use
 the general limit. If neither are defined, then the SDK MUST try to use the
 model-specific limit default value, followed by the global limit default value.
 
-#### Configurable Parameters
+### Configurable Parameters
 
 * `AttributeCountLimit` (Default=128) - Maximum allowed attribute count per record;
-* `AttributeValueLengthLimit` (Default=Infinity) - Maximum allowed attribute value length;
+* `AttributeValueLengthLimit` (Default=Infinity) - Maximum allowed attribute value length (applies to string values and byte arrays);
 
-#### Exempt Entities
+### Exempt Entities
 
 Resource attributes SHOULD be exempt from the limits described above as resources
 are not susceptible to the scenarios (auto-instrumentation) that result in
@@ -139,40 +207,3 @@ attribute limits for Resources.
 Attributes, which belong to Metrics, are exempt from the limits described above
 at this time, as discussed in
 [Metrics Attribute Limits](../metrics/sdk.md#attribute-limits).
-
-## Attribute Collections
-
-[Resources](../resource/sdk.md),
-[Instrumentation Scopes](instrumentation-scope.md),
-[Metric points](../metrics/data-model.md#metric-points),
-[Spans](../trace/api.md#set-attributes), Span
-[Events](../trace/api.md#add-events), Span
-[Links](../trace/api.md#link) and
-[Log Records](../logs/data-model.md) may contain a collection of attributes. The
-keys in each such collection are unique, i.e. there MUST NOT exist more than one
-key-value pair with the same key. The enforcement of uniqueness may be performed
-in a variety of ways as it best fits the limitations of the particular
-implementation.
-
-Normally for the telemetry generated using OpenTelemetry SDKs the attribute
-key-value pairs are set via an API that either accepts a single key-value pair
-or a collection of key-value pairs. Setting an attribute with the same key as an
-existing attribute SHOULD overwrite the existing attribute's value. See for
-example Span's [SetAttribute](../trace/api.md#set-attributes) API.
-
-A typical implementation of [SetAttribute](../trace/api.md#set-attributes) API
-will enforce the uniqueness by overwriting any existing attribute values pending
-to be exported, so that when the Span is eventually exported the exporters see
-only unique attributes. The OTLP format in particular requires that exported
-Resources, Spans, Metric data points and Log Records contain only unique
-attributes.
-
-Some other implementations may use a streaming approach where every
-[SetAttribute](../trace/api.md#set-attributes) API call immediately results in
-that individual attribute value being exported using a streaming wire protocol.
-In such cases the enforcement of uniqueness will likely be the responsibility of
-the recipient of this data.
-
-Collection of attributes are equal when they contain the same attributes,
-irrespective of the order in which those elements appear
-(unordered collection equality).

specification/common/attribute-type-mapping.md

@@ -13,6 +13,7 @@ linkTitle: Mapping to AnyValue
 
 - [Converting to AnyValue](#converting-to-anyvalue)
   * [Primitive Values](#primitive-values)
+    + [Empty Value](#empty-value)
     + [Integer Values](#integer-values)
     + [Enumerations](#enumerations)
     + [Floating Point Values](#floating-point-values)
@@ -54,6 +55,12 @@ follow the rules described below.
 
 ### Primitive Values
 
+#### Empty Value
+
+Empty values MUST be converted to AnyValue with no
+[value](https://github.com/open-telemetry/opentelemetry-proto/blob/38b5b9b6e5257c6500a843f7fdacf89dd95833e8/opentelemetry/proto/common/v1/common.proto#L28-L30)
+field being set.
+
 #### Integer Values
 
 Integer values which are within the range of 64 bit signed numbers

specification/entities/data-model.md

@@ -48,8 +48,8 @@ physical format and encoding of how entity data is recorded).
 | Field        | Type                                   | Description     |
 |--------------|----------------------------------------|-----------------|
 | Type         | string                                 | Defines the type of the entity. MUST not change during the lifetime of the entity. For example: "service" or "host". This field is required and MUST not be empty for valid entities. |
-| Id           | map<string, standard attribute value>  | Attributes that identify the entity.<p>MUST not change during the lifetime of the entity. The Id must contain at least one attribute.<p>Follows OpenTelemetry [Standard attribute definition](../common/README.md#standard-attribute). SHOULD follow OpenTelemetry [semantic conventions](https://github.com/open-telemetry/semantic-conventions) for attributes. |
-| Description  | map<string, any>                       | Descriptive (non-identifying) attributes of the entity.<p>MAY change over the lifetime of the entity. MAY be empty. These attributes are not part of entity's identity.<p>Follows [any](../logs/data-model.md#type-any) value definition in the OpenTelemetry spec. Arbitrary deep nesting of values for arrays and maps is allowed.<p>SHOULD follow OpenTelemetry [semantic conventions](https://github.com/open-telemetry/semantic-conventions/blob/main/docs/README.md) for attributes. |
+| Id           | map<string, attribute value>  | Attributes that identify the entity.<p>MUST not change during the lifetime of the entity. The Id must contain at least one attribute.<p>Follows OpenTelemetry [attribute definition](../common/README.md#attribute). SHOULD follow OpenTelemetry [semantic conventions](https://github.com/open-telemetry/semantic-conventions) for attributes. |
+| Description  | map<string, attribute value>                       | Descriptive (non-identifying) attributes of the entity.<p>MAY change over the lifetime of the entity. MAY be empty. These attributes are not part of entity's identity.<p>Follows OpenTelemetry [attribute definition](../common/README.md#attribute). SHOULD follow OpenTelemetry [semantic conventions](https://github.com/open-telemetry/semantic-conventions/blob/main/docs/README.md) for attributes. |
 
 ## Minimally Sufficient Identity
 

specification/logs/api.md

@@ -124,14 +124,6 @@ The API MUST accept the following parameters:
 - [Attributes](./data-model.md#field-attributes) (optional)
 - [Event Name](./data-model.md#field-eventname) (optional)
 
-**Status**: [Development](../document-status.md)
-
-The API SHOULD provide functionality for users to convert
-[Standard Attributes](../common/README.md#standard-attribute)
-so they can be used, or directly accept them, in the log signal.
-This allows the reuse of [Standard Attributes](../common/README.md#standard-attribute)
-across signals.
-
 ### Enabled
 
 To help users avoid performing computationally expensive operations when