From a8306557e9cf84ad2f410079a778b276775daedf Mon Sep 17 00:00:00 2001 From: Ted Young Date: Mon, 2 Dec 2024 12:43:42 -0800 Subject: [PATCH 01/19] OTEP: Define ResourceProvider --- oteps/0000-resource-provider.md | 272 ++++++++++++++++++++++++++++++++ 1 file changed, 272 insertions(+) create mode 100644 oteps/0000-resource-provider.md diff --git a/oteps/0000-resource-provider.md b/oteps/0000-resource-provider.md new file mode 100644 index 00000000000..374ad528f80 --- /dev/null +++ b/oteps/0000-resource-provider.md @@ -0,0 +1,272 @@ +# Resource Provider + +Define a mechanism for updating the resources associated with an application. + +## Motivation + +Resources were originally defined as immutable. For the common cases related to +server-side application development, because the lifespan for most resources associated +with server-side applications either match or outlive the lifespan of the application. + +However, it turns out that not all swans are white, and some resources utilized +by applications change while the application is still running. This is especially +true in client-side applications running in the browser and on mobile devices. + +Examples of resources whose availability may change include networking (wifi, cellular, none), +application state (foreground, background, sleeping), and session management (sessions +starting and ending without the application being shut down or the browser being +refreshed). + +Tracking these resource changes are critical. Without them, it would be impossible +to segment the telemetry correctly. The lifespan of a session is a far more important +segmentation than the lifespan of an application instance, as the application lifespan +is often somewhat arbitrary. The performance of an application when it is foregrounded +cannot be understood when the foregrounded telemetry is mixed with backgrounded +telemetry. Failure modes may exist when network availability drops due to a switch +in networking – how an application performs when it has access to wifi vs when it +does not is a critical distinction. + +## Explanation + +Resources are managed via a ResourceProvider. Setting an attribute on a ResourceProvider +will cause that attribute value to be included in any future requests for the resource +object managed by the provider. Programs such as the SDK can listen for resource +changes and respond accordingly. + +## Internal details + +### ResourceListener + +A ResourceListener is a function that takes a resource reference as a parameter. +Resource listeners SHOULD NOT be required to be thread safe. + +### ResourceProvider + +#### NewResourceProvider([resource]) ResourceProvider + +NewResourceProvider instantiates an implementation of the ResourceProvider interface. + +The ResourceProvider interface has the following methods. + +#### MergeResource(resource) + +MergeResource creates a new resource, representing the union of the resource parameter +and the resource contained within the Provider. The ResourceProvider is updated +to hold a reference to the merged resource, and all OnChange listeners are called +with this new resource. + +#### GetResource() Resource + +GetResource returns a reference to the current resource held by the ResourceProvider. + +#### OnChange(resourceListener) + +Registers a ResourceListener to be called when MergeResource updates the resource. + +#### Implementation Notes + +For multithreaded systems, a lock SHOULD be used to queue all calls to MergeResource. + +The resource reference held by the ResourceProvider SHOULD be updated atomically, +so that calls to GetResource do not require a lock. + +Calls to listeners SHOULD be serialized, to avoid thread safety issues and ensure that +callbacks are processed in the right order. + +### SDK Changes + +NewTraceProvider, NewMetricsProvider, and NewLoggerProvider now take a ResourceProvider +as a parameter. How SDKs handle resource changes is listed under [open questions](#open-questions). + +## Trade-offs and mitigations + +This change should be fully backwards compatible, with one potential exception: +fingerprinting. It is possible that an analysis tool which accepts OTLP may identify +individual services by creating an identifier by hashing all of the resource attributes. + +In practice, the only implementation we have discovered that does this is the OpenTelemetry +Go SDK. But the Go SDK is not a backend; all analysis tools have a specific concept +of identity that is fulfilled by a specific subject of resource attributes. + +Since we control the Go SDK, we can develop a path forward specific to that particular +library. That path should be identified before this OTEP is accepted. + +Beyond fingerprinting, there are no destabilizing changes because the resources +that we have already declared "immutable" match the lifespan of the application +and have no reason to be updated. Developers are not going to start messing with +the service.id arbitrarily just because they can, and resource detectors solve the +problem of accidentally starting the application while async resources are still +being fetched. + +## Prior art and alternatives + +An alternative to updating resources would be to create span, metrics, and log +processors which attach these resource attributes to every instance of every +span and log. + +There are two problems to this approach. One is that the duplication of attributes +is very inefficient. This is a problem on clients, which have limited network +bandwidth and processing power. This problem is compounded by a lack of support +for gzip and other compression algorithms on the browser. + +Second, and perhaps more important, is that this approach does not match our data +model. These application states are global; they are not specific to any particular +transaction or event. Span attributes should record information specific to that +particular operation, log record attributes should record information specific to +that particular event. The correct place in our data model model for attributes +that identify the application, describe the environment the application is running +in, and describe the resources available to that application should be modelled +as resource attributes. + +## Open questions + +The primary open question – which must be resolved before this OTEP is accepted – +is how to handle spans that bridge a change in resources. + +For example, a long running background operation may span more than one session. +Networking may change from wifi to a cellular connection at any time, a user might +log in at any time, the application might be backgrounded at any time. + +Simply put, how should the SDK handle spans that have already started when a resource +changes? What about the logs that are associated with that span? EntityState can be +used to record the exact moment when these values change. But resources need to act +as search indexes and metric dimensions. For those situations, we only get to pick +one value. + +The simplest implementation is for the BatchProcessor to listen for resource changes, +and to flush the current batch whenever a change occurs. The old batch gets the old +resource, the new batch gets the new resource. This would be easy to implement, +but is it actually what we want? Either as part of this OTEP or as a quick follow +up, we need to define the expected behavior for the BatchProcessor when it is listening +for resource changes. + +## FAQ + +### Is there some distinction between "identifying resources" and "updatable resources"? + +Surprising as it may be, there is no direct correlation between an attribute being +"identifying" and that attribute matching the lifespan of an application. + +Some resources are used to identify a service instance – `service.name`, `service.instance.id`, etc. +These resources naturally match the lifespan of the service instance. An "immutability requirement" +is not necessary in this case because there is no reason to ever update these values. + +Other resources are used to identify other critical lifespans, and these values +do change. For example, the user's identity may change as the user logs in and out +of the application. And multiple sessions may start and end over the lifespan of +an application. + +Therefore there is no need to conflate "identifying" with "immutable." Telemetry +simply models reality. If we change a resource attribute that is not supposed to +change, that is an implementation error. If we don't change a resource attribute +when the application state changes, that is also an implementation error. With the +correct tools these errors are unlikely, as it is very obvious when these individual +attributes should and shouldn't change. + +### Why were resources immutable in the first place? + +Use of the term "immutable" points at the real reason this requirement was initially +added to the specification.When an application initially boots up, gathering some resources +require async operations that may take time to acquire. The start of the application +must be delayed until all of these resources are resolved, otherwise the initial +batches of telemetry would be poorly indexed. This initial telemetry is critical +and too valuable to lose due to a late loading of certain resources. + +A convenient cudgel with which to beat developers into doing the right thing is +to make the resource object "immutable" by not providing an update function. This +makes it impossible to late load resources and helps to avoid this mistake when +installing OpenTelemetry in an application. + +However, OpenTelemetry has since developed a resource detector pattern that gives +developers the tools they need to cleanly resolve all initial resources before +application start. This is a sufficient solution for the bootstrapping problem; +at this point in OpenTelemetry's development adding an update function to resources +would not cause issues in this regard. + +## Example Usage + +Pseudocode example of a ResourceProvider in use. The resource provider is loaded with all available permanent resources, then passed to a TraceProvider. The ResourceProvider is also passed to a session manager, which updates an ephemeral resource in the background. + +``` +var resources = {“service.name” = “example-service”}; + +// Example of a deny list validator. +var validator = NewDenyListValidator(PERMANENT_RESOURCE_KEYS); + +// The ResourceProvider is initialized with +// a dictionary of resources and a validator. +var resourceProvider = NewResourceProvider(resources, validator); + +// The resourceProvider can be passed to resource detectors +// to populate async resources. +DetectResources(resourceProvider); + +// The TraceProvider now takes a ResourceProvider. +// The TraceProvider calls Freeze on the ResourceProvider. +// After this point, it is no longer possible to update or add +// additional permanent resources. +var traceProvider = NewTraceProvider(resourceProvider); + +// Whenever the SessionManager starts a new session +// it updates the ResourceProvider with a new session id. +sessionManager.OnChange( + func(sessionID){ + resourceProvider.SetAttribute(“session.id”, sessionID); + } +); + +``` + +## Example Implementation + +Pseudocode examples for a possible Validator and ResourceProvider implementation. Attention is placed on making the ResourceProvider thread safe, without introducing any locking or synchronization overhead to `GetResource`, which is the only ResourceProvider method on the hot path for OpenTelemetry instrumentation. + +``` +// Example of a thread-safe ResourceProvider +class ResourceProvider{ + + *Resource resource + Lock lock + listeners [func(resource){}] // a list of callback functions + + GetResource(){ + return this.resource; + } + + OnChange(listener) { + this.lock.Acquire(); + + listeners.append(listener) + + this.lock.Release(); + } + + // MergeResource essentially has the same implementation as SetAttribute. + MergeResource(resource){ + this.lock.Acquire(); + + var mergedResource = this.resource.Merge(resource) + + // safely change the resource reference without blocking + AtomicSwap(this.resource, mergedResource) + + // calling listeners inside of the lock ensures that the listeners do not fire + // out of order or get called simultaneously by multiple threads, but would + // also allow a poorly implemented listener to block the ResourceProvider. + for (listener in listeners) { + listener(mergedResource) + } + + this.lock.Release(); + + } + + FreezePermanent(){ + this.lock.Acquire(); + + this.isFrozen = true; + + this.lock.Release(); + } +} +``` From c8d53016556ba2d9f36ca3c6f2f9675557463ed5 Mon Sep 17 00:00:00 2001 From: Ted Young Date: Mon, 2 Dec 2024 12:49:28 -0800 Subject: [PATCH 02/19] add pr number to filename --- oteps/{0000-resource-provider.md => 4316-resource-provider.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename oteps/{0000-resource-provider.md => 4316-resource-provider.md} (100%) diff --git a/oteps/0000-resource-provider.md b/oteps/4316-resource-provider.md similarity index 100% rename from oteps/0000-resource-provider.md rename to oteps/4316-resource-provider.md From a3cb621190c7124a10bd5652f5e0b2206db56d91 Mon Sep 17 00:00:00 2001 From: Ted Young Date: Wed, 11 Dec 2024 16:54:19 -0800 Subject: [PATCH 03/19] formatting --- oteps/4316-resource-provider.md | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/oteps/4316-resource-provider.md b/oteps/4316-resource-provider.md index 374ad528f80..e5f80fdae3b 100644 --- a/oteps/4316-resource-provider.md +++ b/oteps/4316-resource-provider.md @@ -185,7 +185,10 @@ would not cause issues in this regard. ## Example Usage -Pseudocode example of a ResourceProvider in use. The resource provider is loaded with all available permanent resources, then passed to a TraceProvider. The ResourceProvider is also passed to a session manager, which updates an ephemeral resource in the background. +Pseudocode example of a ResourceProvider in use. The resource provider is loaded +with all available permanent resources, then passed to a TraceProvider. The +ResourceProvider is also passed to a session manager, which updates an ephemeral +resource in the background. ``` var resources = {“service.name” = “example-service”}; @@ -219,7 +222,10 @@ sessionManager.OnChange( ## Example Implementation -Pseudocode examples for a possible Validator and ResourceProvider implementation. Attention is placed on making the ResourceProvider thread safe, without introducing any locking or synchronization overhead to `GetResource`, which is the only ResourceProvider method on the hot path for OpenTelemetry instrumentation. +Pseudocode examples for a possible Validator and ResourceProvider implementation. +Attention is placed on making the ResourceProvider thread safe, without introducing +any locking or synchronization overhead to `GetResource`, which is the only +ResourceProvider method on the hot path for OpenTelemetry instrumentation. ``` // Example of a thread-safe ResourceProvider From 2fd655c68c86b1c60e081db79a130edda5bf528e Mon Sep 17 00:00:00 2001 From: Ted Young Date: Wed, 11 Dec 2024 18:57:07 -0800 Subject: [PATCH 04/19] Add entities to ResourceProvider --- oteps/4316-resource-provider.md | 68 +++++++++++++++++++++++++-------- 1 file changed, 53 insertions(+), 15 deletions(-) diff --git a/oteps/4316-resource-provider.md b/oteps/4316-resource-provider.md index e5f80fdae3b..903c51a9c04 100644 --- a/oteps/4316-resource-provider.md +++ b/oteps/4316-resource-provider.md @@ -35,33 +35,71 @@ changes and respond accordingly. ## Internal details -### ResourceListener +### EntityListener -A ResourceListener is a function that takes a resource reference as a parameter. -Resource listeners SHOULD NOT be required to be thread safe. +An EntityListener MUST provide the following operations + +- On EntityState +- On EntityDelete + +#### On EntityState + +`On EntityState` MUST accept the following parameters: + +* `EntityState`: represents the entity that has changed. +* `Resource`: represents the entire set of resources after the entity changes + have been applied. + +#### On EntityDelete + +`On EntityDelete` MUST accept the following parameters: + +* `EntityDelete`: represents the entity that has been deleted. +* `Resource`: represents the entire set of resources after the entity + has been deleted. ### ResourceProvider -#### NewResourceProvider([resource]) ResourceProvider +#### ResourceProvider creation + +Creation of a ResourceProvider MUST accept the following parameters: + +* `Entities`: a list of entities. + +Internally, the entities MUST be merged in the order provided to create the initial +resource. + +#### Update Entity + +`Update Entity` replaces the resource attributes associated with an entity. + +Update Entity MUST accept the following parameters: + +* `ID`: the ID of the Entity being updated. +* `attributes`: the new set of attributes associated with the entity. + +Internally, `Update Entity` MUST trigger the `On EntityState` operation for all +registered `EntityListeners`. + +#### Delete Entity -NewResourceProvider instantiates an implementation of the ResourceProvider interface. +`Delete Entity` replaces the resource attributes associated with an entity. -The ResourceProvider interface has the following methods. +Update Entity MUST accept the following parameters: -#### MergeResource(resource) +* `ID`: the ID of the Entity being updated. -MergeResource creates a new resource, representing the union of the resource parameter -and the resource contained within the Provider. The ResourceProvider is updated -to hold a reference to the merged resource, and all OnChange listeners are called -with this new resource. +Internally, `Update Entity` MUST trigger the `On EntityState` operation for all +registered `EntityListeners`. -#### GetResource() Resource +#### Get Resource -GetResource returns a reference to the current resource held by the ResourceProvider. +`Get Resource` MUST return a reference to the current resource held by the ResourceProvider. -#### OnChange(resourceListener) +#### On Change -Registers a ResourceListener to be called when MergeResource updates the resource. +`On Change` registers an `EntityListener` to be called every time an entity is updated +or deleted. #### Implementation Notes From a1d44613a8eeea4f0abf64eacdf706772aa04218 Mon Sep 17 00:00:00 2001 From: Ted Young Date: Wed, 11 Dec 2024 19:07:02 -0800 Subject: [PATCH 05/19] clarify locking --- oteps/4316-resource-provider.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/oteps/4316-resource-provider.md b/oteps/4316-resource-provider.md index 903c51a9c04..d3296744c7f 100644 --- a/oteps/4316-resource-provider.md +++ b/oteps/4316-resource-provider.md @@ -103,13 +103,14 @@ or deleted. #### Implementation Notes -For multithreaded systems, a lock SHOULD be used to queue all calls to MergeResource. +For multithreaded systems, a lock SHOULD be used to queue all calls to `UpdateEntity` +and `DeleteEntity`. This is to help avoid inconsistent reads and writes. The resource reference held by the ResourceProvider SHOULD be updated atomically, -so that calls to GetResource do not require a lock. +so that calls to `GetResource` do not require a lock. -Calls to listeners SHOULD be serialized, to avoid thread safety issues and ensure that -callbacks are processed in the right order. +Calls to EntityListeners SHOULD be serialized, to avoid thread safety issues and +ensure that callbacks are processed in the right order. ### SDK Changes From 844ffa8c4e1e8d452d3f89a34c6f3df019ed4481 Mon Sep 17 00:00:00 2001 From: Ted Young Date: Wed, 11 Dec 2024 19:36:02 -0800 Subject: [PATCH 06/19] updated explanation --- oteps/4316-resource-provider.md | 25 ++++++++++++++++++------- 1 file changed, 18 insertions(+), 7 deletions(-) diff --git a/oteps/4316-resource-provider.md b/oteps/4316-resource-provider.md index d3296744c7f..9afaa4d1399 100644 --- a/oteps/4316-resource-provider.md +++ b/oteps/4316-resource-provider.md @@ -28,19 +28,30 @@ does not is a critical distinction. ## Explanation -Resources are managed via a ResourceProvider. Setting an attribute on a ResourceProvider -will cause that attribute value to be included in any future requests for the resource -object managed by the provider. Programs such as the SDK can listen for resource -changes and respond accordingly. +Changes to resources and entities are managed via a ResourceProvider. When the resources +represented by an entity change, the telemetry system records these changes by updating +the entity managed by the ResourceProvider. These changes are then propagated to the +rest of the telemetry system via EntityListeners that have been registered with the +ResourceProvider. + +The loose coupling provided by a ResourceProvider allows each subsystem to focus +on their various responsibilities without having to be directly aware of each other. +For a highly extensible cross-cutting concern such as OpenTelemetry, this loose +coupling is a valuable feature. ## Internal details +Like the other Providers used in OpenTelemetry, the ResourceProvider MUST allow +for alternative implementations. This means that the ResourceProvider API and +the ResourceProvider implementation we provide MUST be loosely coupled, following +the same API/SDK pattern used everywhere in OpenTelemetry. + ### EntityListener -An EntityListener MUST provide the following operations +An EntityListener MUST provide the following operations: -- On EntityState -- On EntityDelete +- `On EntityState` +- `On EntityDelete` #### On EntityState From cc9877fd8944311b954a52f5b1118c22877f65cc Mon Sep 17 00:00:00 2001 From: Ted Young Date: Wed, 11 Dec 2024 20:12:52 -0800 Subject: [PATCH 07/19] update example --- oteps/4316-resource-provider.md | 79 ++++++++++++++++++++++++++------- 1 file changed, 64 insertions(+), 15 deletions(-) diff --git a/oteps/4316-resource-provider.md b/oteps/4316-resource-provider.md index 9afaa4d1399..34b637fb205 100644 --- a/oteps/4316-resource-provider.md +++ b/oteps/4316-resource-provider.md @@ -277,52 +277,101 @@ Attention is placed on making the ResourceProvider thread safe, without introduc any locking or synchronization overhead to `GetResource`, which is the only ResourceProvider method on the hot path for OpenTelemetry instrumentation. -``` +```php // Example of a thread-safe ResourceProvider class ResourceProvider{ *Resource resource Lock lock - listeners [func(resource){}] // a list of callback functions + Map[string:Entity] entities // a map of Entities using that uses entity IDs as keys + Array[EntityListener] listeners - GetResource(){ + GetResource() Resource { return this.resource; } - OnChange(listener) { + OnChange(EntityListener listener) { this.lock.Acquire(); - listeners.append(listener) + listeners.Append(listener); this.lock.Release(); } - // MergeResource essentially has the same implementation as SetAttribute. - MergeResource(resource){ + UpdateEntity(string ID, Map[Attribute] attributes){ this.lock.Acquire(); - var mergedResource = this.resource.Merge(resource) + // Acquire the correct entity based on ID + var entity = this.entities[ID] + + // If there is no entity, log the error and return. This follows the pattern + // of not returning errors in the OpenTelemetry API. + if(!entity) { + LogError(EntityNotFound); + this.lock.Release(); + return; + } + + // Update the attributes on the entity. + entity.attributes.Merge(attributes); + + // Update the attributes on the resource. + var mergedResource = this.resource.Merge(attributes); // safely change the resource reference without blocking - AtomicSwap(this.resource, mergedResource) + AtomicSwap(this.resource, mergedResource); // calling listeners inside of the lock ensures that the listeners do not fire // out of order or get called simultaneously by multiple threads, but would // also allow a poorly implemented listener to block the ResourceProvider. - for (listener in listeners) { - listener(mergedResource) + for (listener in this.listeners) { + // create an EntityState event from the entity + var entityState = entity.EntityState(); + listener.OnEntityState(entityState, mergedResource); } this.lock.Release(); - } - FreezePermanent(){ + DeleteEntity(string ID, Map[Attribute] attributes){ this.lock.Acquire(); - this.isFrozen = true; + // Acquire the correct entity based on ID + var entity = this.entities[ID] + + // If there is no entity, log the error and return. This follows the pattern + // of not returning errors in the OpenTelemetry API. + if (!entity) { + LogError(EntityNotFound); + this.lock.Release(); + return; + } + + // remove the entity from the map + this.entities.Delete(ID); + + // Get the attributes to delete from the resource + var keys = new Array(); + for (entity.attributes as name:value) { + keys.Push(name) + } + + // Delete the attributes + var mergedResource = this.resource.Delete(keys); - this.lock.Release(); + // safely change the resource reference without blocking + AtomicSwap(this.resource, mergedResource); + + // calling listeners inside of the lock ensures that the listeners do not fire + // out of order or get called simultaneously by multiple threads, but would + // also allow a poorly implemented listener to block the ResourceProvider. + for (listener in this.listeners) { + // create an EntityDelete event from the entity + var entityDelete = entity.EntityDelete(); + listener.OnEntityDelete(entityDelete, mergedResource); + } + + this.lock.Release(); } } ``` From 3cd7641b4209e054ef38f61118934538a368845b Mon Sep 17 00:00:00 2001 From: Ted Young Date: Wed, 11 Dec 2024 20:56:59 -0800 Subject: [PATCH 08/19] reformat ResourceProvider description --- oteps/4316-resource-provider.md | 29 +++++++++++++++++------------ 1 file changed, 17 insertions(+), 12 deletions(-) diff --git a/oteps/4316-resource-provider.md b/oteps/4316-resource-provider.md index 34b637fb205..6506ef5b56c 100644 --- a/oteps/4316-resource-provider.md +++ b/oteps/4316-resource-provider.md @@ -71,6 +71,22 @@ An EntityListener MUST provide the following operations: ### ResourceProvider +A `ResourceProvider` MUST provide the following operations: + +* `Update Entity` +* `Delete Entity` +* `Get Resource` +* `On Change` + +For multithreaded systems, a lock SHOULD be used to queue all calls to `UpdateEntity` +and `DeleteEntity`. This is to help avoid inconsistent reads and writes. + +The resource reference held by the ResourceProvider SHOULD be updated atomically, +so that calls to `GetResource` do not require a lock. + +Calls to EntityListeners SHOULD be serialized, to avoid thread safety issues and +ensure that callbacks are processed in the right order. + #### ResourceProvider creation Creation of a ResourceProvider MUST accept the following parameters: @@ -112,17 +128,6 @@ registered `EntityListeners`. `On Change` registers an `EntityListener` to be called every time an entity is updated or deleted. -#### Implementation Notes - -For multithreaded systems, a lock SHOULD be used to queue all calls to `UpdateEntity` -and `DeleteEntity`. This is to help avoid inconsistent reads and writes. - -The resource reference held by the ResourceProvider SHOULD be updated atomically, -so that calls to `GetResource` do not require a lock. - -Calls to EntityListeners SHOULD be serialized, to avoid thread safety issues and -ensure that callbacks are processed in the right order. - ### SDK Changes NewTraceProvider, NewMetricsProvider, and NewLoggerProvider now take a ResourceProvider @@ -216,7 +221,7 @@ attributes should and shouldn't change. ### Why were resources immutable in the first place? Use of the term "immutable" points at the real reason this requirement was initially -added to the specification.When an application initially boots up, gathering some resources +added to the specification. When an application initially boots up, gathering some resources require async operations that may take time to acquire. The start of the application must be delayed until all of these resources are resolved, otherwise the initial batches of telemetry would be poorly indexed. This initial telemetry is critical From 7e24b05c151bd857fbef26d8e189b2e80562568b Mon Sep 17 00:00:00 2001 From: Ted Young Date: Wed, 18 Dec 2024 14:17:01 -0800 Subject: [PATCH 09/19] resource must be entirely regenerated --- oteps/4316-resource-provider.md | 30 +++++++++++++++--------------- 1 file changed, 15 insertions(+), 15 deletions(-) diff --git a/oteps/4316-resource-provider.md b/oteps/4316-resource-provider.md index 6506ef5b56c..605e9f37d5f 100644 --- a/oteps/4316-resource-provider.md +++ b/oteps/4316-resource-provider.md @@ -105,7 +105,10 @@ Update Entity MUST accept the following parameters: * `ID`: the ID of the Entity being updated. * `attributes`: the new set of attributes associated with the entity. -Internally, `Update Entity` MUST trigger the `On EntityState` operation for all +After an entity is updated, a new resource object MUST be generated by merging +the list of entities together in order. + +`Update Entity` MUST trigger the `On EntityState` operation for all registered `EntityListeners`. #### Delete Entity @@ -116,7 +119,10 @@ Update Entity MUST accept the following parameters: * `ID`: the ID of the Entity being updated. -Internally, `Update Entity` MUST trigger the `On EntityState` operation for all +After an entity is deleted, a new resource object MUST be generated by merging +the list of entities together in order. + +`Delete Entity` MUST trigger the `On EntityState` operation for all registered `EntityListeners`. #### Get Resource @@ -288,7 +294,7 @@ class ResourceProvider{ *Resource resource Lock lock - Map[string:Entity] entities // a map of Entities using that uses entity IDs as keys + OrderedMap[string:Entity] entities // an ordered map of Entities that uses entity IDs as keys Array[EntityListener] listeners GetResource() Resource { @@ -317,11 +323,11 @@ class ResourceProvider{ return; } - // Update the attributes on the entity. - entity.attributes.Merge(attributes); + // Replace the attributes on the entity. + entity.attributes = attributes; - // Update the attributes on the resource. - var mergedResource = this.resource.Merge(attributes); + // create a new resource + var mergedResource = NewResource(this.entities); // safely change the resource reference without blocking AtomicSwap(this.resource, mergedResource); @@ -354,15 +360,9 @@ class ResourceProvider{ // remove the entity from the map this.entities.Delete(ID); - - // Get the attributes to delete from the resource - var keys = new Array(); - for (entity.attributes as name:value) { - keys.Push(name) - } - // Delete the attributes - var mergedResource = this.resource.Delete(keys); + // create a new resource + var mergedResource = NewResource(this.entities); // safely change the resource reference without blocking AtomicSwap(this.resource, mergedResource); From 6e1d24da6d7599c80c12fbbd4e708cc28b06a577 Mon Sep 17 00:00:00 2001 From: Ted Young Date: Wed, 18 Dec 2024 14:23:49 -0800 Subject: [PATCH 10/19] add Add Entity method --- oteps/4316-resource-provider.md | 43 +++++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+) diff --git a/oteps/4316-resource-provider.md b/oteps/4316-resource-provider.md index 605e9f37d5f..4f2dfeb2779 100644 --- a/oteps/4316-resource-provider.md +++ b/oteps/4316-resource-provider.md @@ -96,6 +96,22 @@ Creation of a ResourceProvider MUST accept the following parameters: Internally, the entities MUST be merged in the order provided to create the initial resource. +#### Add Entity + +`Add Entity` appends a new entity on to the end of the list of entities. + +Add Entity MUST accept the following parameters: + +* `ID`: the ID of the Entity being created. +* `name`: the name of the Entity being created. +* `attributes`: the set of attributes associated with the entity. + +After an entity is created, it MUST be appended to the list of current entities. +A new resource object MUST be generated by merging the list of entities together in order. + +`Add Entity` MUST trigger the `On EntityState` operation for all +registered `EntityListeners`. + #### Update Entity `Update Entity` replaces the resource attributes associated with an entity. @@ -309,6 +325,33 @@ class ResourceProvider{ this.lock.Release(); } + AddEntity(string ID, string name, Map[Attribute] attributes){ + this.lock.Acquire(); + + // Acquire the correct entity based on ID + var entity = NewEntity(ID, name, attributes); + + // Append the entity to the end of the OrderedMap and set the key to the ID + this.entities.Append(ID, entity); + + // create a new resource + var mergedResource = NewResource(this.entities); + + // safely change the resource reference without blocking + AtomicSwap(this.resource, mergedResource); + + // calling listeners inside of the lock ensures that the listeners do not fire + // out of order or get called simultaneously by multiple threads, but would + // also allow a poorly implemented listener to block the ResourceProvider. + for (listener in this.listeners) { + // create an EntityState event from the entity + var entityState = entity.EntityState(); + listener.OnEntityState(entityState, mergedResource); + } + + this.lock.Release(); + } + UpdateEntity(string ID, Map[Attribute] attributes){ this.lock.Acquire(); From 3d01a5fd7e846bfb43e0c47ecdae31a8517f22f4 Mon Sep 17 00:00:00 2001 From: Ted Young Date: Wed, 18 Dec 2024 14:26:03 -0800 Subject: [PATCH 11/19] remove example usage for now --- oteps/4316-resource-provider.md | 35 +-------------------------------- 1 file changed, 1 insertion(+), 34 deletions(-) diff --git a/oteps/4316-resource-provider.md b/oteps/4316-resource-provider.md index 4f2dfeb2779..1fe6111e46f 100644 --- a/oteps/4316-resource-provider.md +++ b/oteps/4316-resource-provider.md @@ -262,40 +262,7 @@ would not cause issues in this regard. ## Example Usage -Pseudocode example of a ResourceProvider in use. The resource provider is loaded -with all available permanent resources, then passed to a TraceProvider. The -ResourceProvider is also passed to a session manager, which updates an ephemeral -resource in the background. - -``` -var resources = {“service.name” = “example-service”}; - -// Example of a deny list validator. -var validator = NewDenyListValidator(PERMANENT_RESOURCE_KEYS); - -// The ResourceProvider is initialized with -// a dictionary of resources and a validator. -var resourceProvider = NewResourceProvider(resources, validator); - -// The resourceProvider can be passed to resource detectors -// to populate async resources. -DetectResources(resourceProvider); - -// The TraceProvider now takes a ResourceProvider. -// The TraceProvider calls Freeze on the ResourceProvider. -// After this point, it is no longer possible to update or add -// additional permanent resources. -var traceProvider = NewTraceProvider(resourceProvider); - -// Whenever the SessionManager starts a new session -// it updates the ResourceProvider with a new session id. -sessionManager.OnChange( - func(sessionID){ - resourceProvider.SetAttribute(“session.id”, sessionID); - } -); - -``` +DRAFT ## Example Implementation From 47d549e2fd63d7d59bdbde42615350895c4a88fb Mon Sep 17 00:00:00 2001 From: Ted Young Date: Wed, 18 Dec 2024 14:28:35 -0800 Subject: [PATCH 12/19] move entitystate creation out of callback loop --- oteps/4316-resource-provider.md | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/oteps/4316-resource-provider.md b/oteps/4316-resource-provider.md index 1fe6111e46f..8996bb859f1 100644 --- a/oteps/4316-resource-provider.md +++ b/oteps/4316-resource-provider.md @@ -307,12 +307,13 @@ class ResourceProvider{ // safely change the resource reference without blocking AtomicSwap(this.resource, mergedResource); + // create an EntityState event from the entity + var entityState = entity.EntityState(); + // calling listeners inside of the lock ensures that the listeners do not fire // out of order or get called simultaneously by multiple threads, but would // also allow a poorly implemented listener to block the ResourceProvider. for (listener in this.listeners) { - // create an EntityState event from the entity - var entityState = entity.EntityState(); listener.OnEntityState(entityState, mergedResource); } @@ -342,12 +343,13 @@ class ResourceProvider{ // safely change the resource reference without blocking AtomicSwap(this.resource, mergedResource); + // create an EntityState event from the entity + var entityState = entity.EntityState(); + // calling listeners inside of the lock ensures that the listeners do not fire // out of order or get called simultaneously by multiple threads, but would // also allow a poorly implemented listener to block the ResourceProvider. for (listener in this.listeners) { - // create an EntityState event from the entity - var entityState = entity.EntityState(); listener.OnEntityState(entityState, mergedResource); } @@ -377,12 +379,13 @@ class ResourceProvider{ // safely change the resource reference without blocking AtomicSwap(this.resource, mergedResource); + // create an EntityDelete event from the entity + var entityDelete = entity.EntityDelete(); + // calling listeners inside of the lock ensures that the listeners do not fire // out of order or get called simultaneously by multiple threads, but would // also allow a poorly implemented listener to block the ResourceProvider. for (listener in this.listeners) { - // create an EntityDelete event from the entity - var entityDelete = entity.EntityDelete(); listener.OnEntityDelete(entityDelete, mergedResource); } From b134f1b468109d40e4ca7aece8d4231909a83706 Mon Sep 17 00:00:00 2001 From: Ted Young Date: Wed, 18 Dec 2024 14:47:18 -0800 Subject: [PATCH 13/19] service.id -> service.instance.id --- oteps/4316-resource-provider.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/oteps/4316-resource-provider.md b/oteps/4316-resource-provider.md index 8996bb859f1..bf09c269197 100644 --- a/oteps/4316-resource-provider.md +++ b/oteps/4316-resource-provider.md @@ -171,9 +171,9 @@ library. That path should be identified before this OTEP is accepted. Beyond fingerprinting, there are no destabilizing changes because the resources that we have already declared "immutable" match the lifespan of the application and have no reason to be updated. Developers are not going to start messing with -the service.id arbitrarily just because they can, and resource detectors solve the -problem of accidentally starting the application while async resources are still -being fetched. +the `service.instance.id` resource arbitrarily just because they can, and resource +detectors solve the problem of accidentally starting the application while async +resources are still being fetched. ## Prior art and alternatives From babedd0b3c8bdda6233ae7115eb7c3dadf8d929e Mon Sep 17 00:00:00 2001 From: Ted Young Date: Wed, 29 Jan 2025 14:06:52 -0800 Subject: [PATCH 14/19] Thrashing should not be managed --- oteps/4316-resource-provider.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/oteps/4316-resource-provider.md b/oteps/4316-resource-provider.md index bf09c269197..cd3ec6f0050 100644 --- a/oteps/4316-resource-provider.md +++ b/oteps/4316-resource-provider.md @@ -197,6 +197,8 @@ as resource attributes. ## Open questions +### How should spans be batched when they bridge a resource change? + The primary open question – which must be resolved before this OTEP is accepted – is how to handle spans that bridge a change in resources. @@ -217,6 +219,12 @@ but is it actually what we want? Either as part of this OTEP or as a quick follo up, we need to define the expected behavior for the BatchProcessor when it is listening for resource changes. +### What if a resource thrashes? + +Because entity changes can now update resources, and those updates may trigger batch and flushing operations within the SDK, it is possible for rapid changes to resources to create a cascade of SDK operations that may lead to performance issues. + +Investigating the practical implications of this problem has not led to any examples of resources that can be expected to exhibit this behavior. For the time being, backoff or other thrash mitigation strategies are left out of this proposal. If a specific resource does end up requiring some form of throttling, the resource detector that updates that particular resource should manage this issue. If a common strategy for throttling emerges, it can be added to the ResourceProvider at a later date. + ## FAQ ### Is there some distinction between "identifying resources" and "updatable resources"? From 6f76983335d2ce0a33554b9c239053f400feb9ad Mon Sep 17 00:00:00 2001 From: Ted Young Date: Wed, 29 Jan 2025 14:09:57 -0800 Subject: [PATCH 15/19] rename ResourceProvider to EntityProvider --- oteps/4316-resource-provider.md | 50 ++++++++++++++++----------------- 1 file changed, 25 insertions(+), 25 deletions(-) diff --git a/oteps/4316-resource-provider.md b/oteps/4316-resource-provider.md index cd3ec6f0050..bfcb38794e0 100644 --- a/oteps/4316-resource-provider.md +++ b/oteps/4316-resource-provider.md @@ -1,6 +1,6 @@ -# Resource Provider +# Entity Provider -Define a mechanism for updating the resources associated with an application. +Define a mechanism for updating the entities and resources associated with an application. ## Motivation @@ -28,22 +28,22 @@ does not is a critical distinction. ## Explanation -Changes to resources and entities are managed via a ResourceProvider. When the resources +Changes to resources and entities are managed via a EntityProvider. When the resources represented by an entity change, the telemetry system records these changes by updating -the entity managed by the ResourceProvider. These changes are then propagated to the +the entity managed by the EntityProvider. These changes are then propagated to the rest of the telemetry system via EntityListeners that have been registered with the -ResourceProvider. +EntityProvider. -The loose coupling provided by a ResourceProvider allows each subsystem to focus +The loose coupling provided by a EntityProvider allows each subsystem to focus on their various responsibilities without having to be directly aware of each other. For a highly extensible cross-cutting concern such as OpenTelemetry, this loose coupling is a valuable feature. ## Internal details -Like the other Providers used in OpenTelemetry, the ResourceProvider MUST allow -for alternative implementations. This means that the ResourceProvider API and -the ResourceProvider implementation we provide MUST be loosely coupled, following +Like the other Providers used in OpenTelemetry, the EntityProvider MUST allow +for alternative implementations. This means that the EntityProvider API and +the EntityProvider implementation we provide MUST be loosely coupled, following the same API/SDK pattern used everywhere in OpenTelemetry. ### EntityListener @@ -69,9 +69,9 @@ An EntityListener MUST provide the following operations: * `Resource`: represents the entire set of resources after the entity has been deleted. -### ResourceProvider +### EntityProvider -A `ResourceProvider` MUST provide the following operations: +A `EntityProvider` MUST provide the following operations: * `Update Entity` * `Delete Entity` @@ -81,15 +81,15 @@ A `ResourceProvider` MUST provide the following operations: For multithreaded systems, a lock SHOULD be used to queue all calls to `UpdateEntity` and `DeleteEntity`. This is to help avoid inconsistent reads and writes. -The resource reference held by the ResourceProvider SHOULD be updated atomically, +The resource reference held by the EntityProvider SHOULD be updated atomically, so that calls to `GetResource` do not require a lock. Calls to EntityListeners SHOULD be serialized, to avoid thread safety issues and ensure that callbacks are processed in the right order. -#### ResourceProvider creation +#### EntityProvider creation -Creation of a ResourceProvider MUST accept the following parameters: +Creation of a EntityProvider MUST accept the following parameters: * `Entities`: a list of entities. @@ -143,7 +143,7 @@ registered `EntityListeners`. #### Get Resource -`Get Resource` MUST return a reference to the current resource held by the ResourceProvider. +`Get Resource` MUST return a reference to the current resource held by the EntityProvider. #### On Change @@ -152,7 +152,7 @@ or deleted. ### SDK Changes -NewTraceProvider, NewMetricsProvider, and NewLoggerProvider now take a ResourceProvider +NewTraceProvider, NewMetricsProvider, and NewLoggerProvider now take a EntityProvider as a parameter. How SDKs handle resource changes is listed under [open questions](#open-questions). ## Trade-offs and mitigations @@ -223,7 +223,7 @@ for resource changes. Because entity changes can now update resources, and those updates may trigger batch and flushing operations within the SDK, it is possible for rapid changes to resources to create a cascade of SDK operations that may lead to performance issues. -Investigating the practical implications of this problem has not led to any examples of resources that can be expected to exhibit this behavior. For the time being, backoff or other thrash mitigation strategies are left out of this proposal. If a specific resource does end up requiring some form of throttling, the resource detector that updates that particular resource should manage this issue. If a common strategy for throttling emerges, it can be added to the ResourceProvider at a later date. +Investigating the practical implications of this problem has not led to any examples of resources that can be expected to exhibit this behavior. For the time being, backoff or other thrash mitigation strategies are left out of this proposal. If a specific resource does end up requiring some form of throttling, the resource detector that updates that particular resource should manage this issue. If a common strategy for throttling emerges, it can be added to the EntityProvider at a later date. ## FAQ @@ -274,14 +274,14 @@ DRAFT ## Example Implementation -Pseudocode examples for a possible Validator and ResourceProvider implementation. -Attention is placed on making the ResourceProvider thread safe, without introducing +Pseudocode examples for a possible Validator and EntityProvider implementation. +Attention is placed on making the EntityProvider thread safe, without introducing any locking or synchronization overhead to `GetResource`, which is the only -ResourceProvider method on the hot path for OpenTelemetry instrumentation. +EntityProvider method on the hot path for OpenTelemetry instrumentation. ```php -// Example of a thread-safe ResourceProvider -class ResourceProvider{ +// Example of a thread-safe EntityProvider +class EntityProvider{ *Resource resource Lock lock @@ -320,7 +320,7 @@ class ResourceProvider{ // calling listeners inside of the lock ensures that the listeners do not fire // out of order or get called simultaneously by multiple threads, but would - // also allow a poorly implemented listener to block the ResourceProvider. + // also allow a poorly implemented listener to block the EntityProvider. for (listener in this.listeners) { listener.OnEntityState(entityState, mergedResource); } @@ -356,7 +356,7 @@ class ResourceProvider{ // calling listeners inside of the lock ensures that the listeners do not fire // out of order or get called simultaneously by multiple threads, but would - // also allow a poorly implemented listener to block the ResourceProvider. + // also allow a poorly implemented listener to block the EntityProvider. for (listener in this.listeners) { listener.OnEntityState(entityState, mergedResource); } @@ -392,7 +392,7 @@ class ResourceProvider{ // calling listeners inside of the lock ensures that the listeners do not fire // out of order or get called simultaneously by multiple threads, but would - // also allow a poorly implemented listener to block the ResourceProvider. + // also allow a poorly implemented listener to block the EntityProvider. for (listener in this.listeners) { listener.OnEntityDelete(entityDelete, mergedResource); } From 1d55c30174807f1f3f03e10ecc2b2457a5a30c44 Mon Sep 17 00:00:00 2001 From: Josh Suereth Date: Thu, 17 Jul 2025 10:58:03 -0400 Subject: [PATCH 16/19] Update OTEP 4316 to include justification for entity API. --- oteps/4316-resource-provider.md | 168 +++++++++++++++++++++++++++----- 1 file changed, 145 insertions(+), 23 deletions(-) diff --git a/oteps/4316-resource-provider.md b/oteps/4316-resource-provider.md index bfcb38794e0..5ed15d204a1 100644 --- a/oteps/4316-resource-provider.md +++ b/oteps/4316-resource-provider.md @@ -26,9 +26,22 @@ telemetry. Failure modes may exist when network availability drops due to a swit in networking – how an application performs when it has access to wifi vs when it does not is a critical distinction. +Additionally, the notion of "Resource Detectors", while specified, is mostly +left to SDKs to figure out the details and remains a bit of a wild west for +non-SDK implementors. Most SDKs provide extension mechanisms for resource +detection, and these extensions look awkwardly like other insturmentation, but +without the benefit of a clear API and stability guarantees. + +Even within Semantic conventions, it has been problematic defining conventions +for resource and sorting out code generation capabilities for providing these +attributes. Going forward, we'd like to provide a clear, stable mechanism for +instrumentation to provide resource attributes, with clear a "initialization" +phase to continue to ensure consistent, reliable observability with existing +Resource use cases. + ## Explanation -Changes to resources and entities are managed via a EntityProvider. When the resources +Changes to resources and entities are managed via an EntityProvider. When the resources represented by an entity change, the telemetry system records these changes by updating the entity managed by the EntityProvider. These changes are then propagated to the rest of the telemetry system via EntityListeners that have been registered with the @@ -39,7 +52,66 @@ on their various responsibilities without having to be directly aware of each ot For a highly extensible cross-cutting concern such as OpenTelemetry, this loose coupling is a valuable feature. -## Internal details +Additionally, an explicit initialization phase is added for SDK components, +where EntityProvider while provide a clear signal when initialization across +Resource detection has completed prior to reporting signals. + +## API Details + +A new `EntityProvider` API is added, which allows reporting resource Entity +values. + +### EntityProvider + +The `EntityProvider` API MUST provide the following operations: + +* `Add or Update Entity` +* `Replace Entity` +* `Delete Entity` + +#### Add or Update Entity + +`Add or Update Entity` appends a new entity on to the end of the list of +entities. If the Entity already exists, then the description of the entity +is updated. + +Add or Update Entity MUST accept the following parameters: + +* `type`: the type of the Entity being created. +* `ID`: the set of attributes which identify the entity. +* `description`: the set of attributes which describe the entity. +* `schema_url` (optional): The Schema URL that should be recorded for entity. + +If the incoming Entity's `type` is not found in the current list of entities, +then the new Entity is added to the list. + +If the incoming Entity conflicts with an existing entity, it is ignored. + +Otherwise, the description of the Entity is updated with the incoming +description. + +#### Replace Entity + +`Replace Entity` replaces the resource attributes associated with an entity. + +Replace Entity MUST accept the following parameters: + +* `type`: the type of the Entity being created. +* `ID`: the set of attributes which identify the entity. +* `description`: the set of attributes which describe the entity. +* `schema_url` (optional): The Schema URL that should be recorded for entity. + +This is equivalent to an (optional) "delete" then "addOrUpdate" for an entity. + +#### Delete Entity + +`Delete Entity` removes the resource attributes associated with an entity. + +Delete Entity MUST accept the following parameters: + +* `type`: the type of the Entity being removed. + +## SDK details Like the other Providers used in OpenTelemetry, the EntityProvider MUST allow for alternative implementations. This means that the EntityProvider API and @@ -50,9 +122,18 @@ the same API/SDK pattern used everywhere in OpenTelemetry. An EntityListener MUST provide the following operations: +- `On ResourceInitialize` - `On EntityState` - `On EntityDelete` +#### On ResourceInitialize + +`On EntityState` MUST accept the following parameters: + +* `Resource`: represents the entire set of resources after initalize. + +This operation MUST only be called once per EntityProvider. + #### On EntityState `On EntityState` MUST accept the following parameters: @@ -73,13 +154,15 @@ An EntityListener MUST provide the following operations: A `EntityProvider` MUST provide the following operations: -* `Update Entity` +* `Add or Update Entity` +* `Replace Entity` * `Delete Entity` * `Get Resource` * `On Change` -For multithreaded systems, a lock SHOULD be used to queue all calls to `UpdateEntity` -and `DeleteEntity`. This is to help avoid inconsistent reads and writes. +For multithreaded systems, a lock SHOULD be used to queue all calls to +`AddorUpdateEntity`, `UpdateEntity` and `DeleteEntity`. +This is to help avoid inconsistent reads and writes. The resource reference held by the EntityProvider SHOULD be updated atomically, so that calls to `GetResource` do not require a lock. @@ -96,36 +179,70 @@ Creation of a EntityProvider MUST accept the following parameters: Internally, the entities MUST be merged in the order provided to create the initial resource. -#### Add Entity +#### Add or Update Entity -`Add Entity` appends a new entity on to the end of the list of entities. +`Add or Update Entity` appends a new entity on to the end of the list of +entities. If the Entity already exists, then the description of the entity +is updated. -Add Entity MUST accept the following parameters: +Add or Update Entity MUST accept the following parameters: -* `ID`: the ID of the Entity being created. -* `name`: the name of the Entity being created. -* `attributes`: the set of attributes associated with the entity. +* `type`: the type of the Entity being created. +* `ID`: the set of attributes which identify the entity. +* `description`: the set of attributes which describe the entity. +* `schema_url` (optional): The Schema URL that should be recorded for entity. + +If the incoming Entity's `type` is not found in the current list of entities, +then the new Entity is added to the list. After an entity is created, it MUST be appended to the list of current entities. -A new resource object MUST be generated by merging the list of entities together in order. +A new resource object MUST be generated by merging the list of entities together +in order. + +If the incoming Entity's `type` parameter matches an existing Entity in +the current list AND the `ID` attributes for these entities are different, then +the SDK MUST ignore the new entity. + +If the incoming Entity's `type` parameter matches an existing Entity in the +current list AND the `ID` attributes for these entities are the same but +the `schema_url` is different, then the SDK MUST ignore the new entity. + +If the incoming Entity's `type` parameter matches an existing Entity in the +current list AND the `ID` attributes for these entities are the same AND the +`schema_url` is the same, then the SDK MUST add any new attributes found in the +`description` to the existing entity. Any new `description` attributes with the +same keys as existing `description` attributes SHOULD replace previous values. `Add Entity` MUST trigger the `On EntityState` operation for all registered `EntityListeners`. -#### Update Entity +#### Replace Entity -`Update Entity` replaces the resource attributes associated with an entity. +`Replace Entity` replaces the resource attributes associated with an entity. -Update Entity MUST accept the following parameters: +Replace Entity MUST accept the following parameters: -* `ID`: the ID of the Entity being updated. -* `attributes`: the new set of attributes associated with the entity. +* `type`: the type of the Entity being created. +* `ID`: the set of attributes which identify the entity. +* `description`: the set of attributes which describe the entity. +* `schema_url` (optional): The Schema URL that should be recorded for entity. -After an entity is updated, a new resource object MUST be generated by merging -the list of entities together in order. +If an existing Entity with the same `type` already exists, then `Replace Entity` +MUST trigger an `On EntityDelete` operation for all registered `EntityListener`s +with the previous state of the Entity. The Resource reported MUST NOT include +the removed Entity. -`Update Entity` MUST trigger the `On EntityState` operation for all -registered `EntityListeners`. +A new entity should be created and added to the list of entities. After the +entity is created, it MUST be appended to the list of current entities. A new +resource object MUST be generated by merging the list of entities together +in order. + +`Replace Entity` MUST trigger the `On EntityState` operation for all +registered `EntityListener`s. + +NOTE: `EntityListener`s SHOULD receive an `On EntityDelete` operation when +a previous entity was removed before receiving an `On EntityUpdate` for the +new entity. #### Delete Entity @@ -133,7 +250,7 @@ registered `EntityListeners`. Update Entity MUST accept the following parameters: -* `ID`: the ID of the Entity being updated. +* `type`: the type of the Entity being removed. After an entity is deleted, a new resource object MUST be generated by merging the list of entities together in order. @@ -143,13 +260,18 @@ registered `EntityListeners`. #### Get Resource -`Get Resource` MUST return a reference to the current resource held by the EntityProvider. +`Get Resource` MUST return a reference to the current resource held by the +EntityProvider. #### On Change `On Change` registers an `EntityListener` to be called every time an entity is updated or deleted. +If the `EntityProvider` is already initialized, then it MUST call +`On ResourceInitialize` immediately with the current resource held by the +EntityProvider. + ### SDK Changes NewTraceProvider, NewMetricsProvider, and NewLoggerProvider now take a EntityProvider From 2de7ab247ba1c376f43d5ad9fad49894e0cc082e Mon Sep 17 00:00:00 2001 From: Josh Suereth Date: Mon, 21 Jul 2025 10:23:14 -0400 Subject: [PATCH 17/19] Add clarifications on use cases and focus of the otep. --- oteps/4316-resource-provider.md | 37 +++++++++++++++++++++++++++++---- 1 file changed, 33 insertions(+), 4 deletions(-) diff --git a/oteps/4316-resource-provider.md b/oteps/4316-resource-provider.md index 5ed15d204a1..4013070e015 100644 --- a/oteps/4316-resource-provider.md +++ b/oteps/4316-resource-provider.md @@ -56,17 +56,39 @@ Additionally, an explicit initialization phase is added for SDK components, where EntityProvider while provide a clear signal when initialization across Resource detection has completed prior to reporting signals. +## High Level Details + +This OTEP proposes two main changes to Resource within OpenTelemetry: + +- The creation of an API which can be used to report entity changes where + the lifetime of the entity does not match the lifetime of the SDK. +- An explicit initialization phase to the SDK, which allows for coordination + between resource detection and signal providers. + ## API Details A new `EntityProvider` API is added, which allows reporting resource Entity values. +The API provides three primary user cases: + +- Instrumentation can provide entity status in a one-time fashion, which is + used to identify the resource at startup of the SDK. +- Instrumentation can provide an entity in a scoped manner (add, then delete) + when it can attach directly to the Entities lifecycle. For example, reporting + [Activity](https://developer.android.com/guide/components/activities/activity-lifecycle) + status in Android. +- Instrumentation can watch for entity changes and replace or update the status + of the entity in the SDK. For example, updating session / page status when + a page comes back from inactive. + + ### EntityProvider The `EntityProvider` API MUST provide the following operations: * `Add or Update Entity` -* `Replace Entity` +* `Add or Replace Entity` * `Delete Entity` #### Add or Update Entity @@ -90,11 +112,11 @@ If the incoming Entity conflicts with an existing entity, it is ignored. Otherwise, the description of the Entity is updated with the incoming description. -#### Replace Entity +#### Add or Replace Entity -`Replace Entity` replaces the resource attributes associated with an entity. +`Add or Replace Entity` adds replaces the resource attributes associated with an entity. -Replace Entity MUST accept the following parameters: +Add or Replace Entity MUST accept the following parameters: * `type`: the type of the Entity being created. * `ID`: the set of attributes which identify the entity. @@ -172,6 +194,13 @@ ensure that callbacks are processed in the right order. #### EntityProvider creation +TODO - Rework this section. + +- We want to allow instrumentation to provide entities at startup. We don't want + a different API for managing lifecycle changes of entities vs. startup if + we can help it. +- We want to defer `GetResource` calls until this has completed, ideally. + Creation of a EntityProvider MUST accept the following parameters: * `Entities`: a list of entities. From 5c34ccd1d1eb64b6b7a44788b48184f0dc9bf70c Mon Sep 17 00:00:00 2001 From: Josh Suereth Date: Mon, 25 Aug 2025 10:24:25 -0400 Subject: [PATCH 18/19] Update description from prototype. --- oteps/4316-resource-provider.md | 139 ++++++++++++++++++-------------- 1 file changed, 78 insertions(+), 61 deletions(-) diff --git a/oteps/4316-resource-provider.md b/oteps/4316-resource-provider.md index 4013070e015..01a48729e9b 100644 --- a/oteps/4316-resource-provider.md +++ b/oteps/4316-resource-provider.md @@ -150,7 +150,7 @@ An EntityListener MUST provide the following operations: #### On ResourceInitialize -`On EntityState` MUST accept the following parameters: +`On ResourceInitialize` MUST accept the following parameters: * `Resource`: represents the entire set of resources after initalize. @@ -174,17 +174,15 @@ This operation MUST only be called once per EntityProvider. ### EntityProvider -A `EntityProvider` MUST provide the following operations: +In addition to providing the `EntityProvider` API, an `EntityProvider` MUST +provide the following operations in the SDK: -* `Add or Update Entity` -* `Replace Entity` -* `Delete Entity` -* `Get Resource` * `On Change` +* (optional) `GetResource` -For multithreaded systems, a lock SHOULD be used to queue all calls to -`AddorUpdateEntity`, `UpdateEntity` and `DeleteEntity`. -This is to help avoid inconsistent reads and writes. +For multithreaded systems, all calls to `AddorUpdateEntity`, `UpdateEntity` and +`DeleteEntity` SHOULD avoid inconsistent reads and writes using appropriate +concurrency mechanisms by treating each method as an atomic operation. The resource reference held by the EntityProvider SHOULD be updated atomically, so that calls to `GetResource` do not require a lock. @@ -192,79 +190,88 @@ so that calls to `GetResource` do not require a lock. Calls to EntityListeners SHOULD be serialized, to avoid thread safety issues and ensure that callbacks are processed in the right order. -#### EntityProvider creation +EntityProvider has two states: -TODO - Rework this section. +- *Resource detection*: The provider is initializing, and waiting for + ResourceDetectors to complete. No events will be fired. +- *Initialized*: A complete resource is available and operations will fire + listener events. -- We want to allow instrumentation to provide entities at startup. We don't want - a different API for managing lifecycle changes of entities vs. startup if - we can help it. -- We want to defer `GetResource` calls until this has completed, ideally. +#### EntityProvider creation Creation of a EntityProvider MUST accept the following parameters: -* `Entities`: a list of entities. +* `detectors`: a list of ResourceDetectors. +* (optional) `initialization_timeout`: A timeout for when to abandon slow + ResourceDetectors and consider a resource initialized. -Internally, the entities MUST be merged in the order provided to create the initial -resource. +EntityProvider MUST allow initial resource detection during its creation. This +initialization SHOULD not block other SDK providers from initializing (e.g. +MeterProvider, TracerProvider). -#### Add or Update Entity +EntityProvider SHOULD provide the `EntityProvider` API to `ResourceDetector`s +during resource detection phase. -`Add or Update Entity` appends a new entity on to the end of the list of -entities. If the Entity already exists, then the description of the entity -is updated. +An EntityProvider MAY allow customizable concurrency behavior, e.g. using a +separate thread for EntityListener events. -Add or Update Entity MUST accept the following parameters: +Internally, the entities discovered via resource detection MUST be merged in +the order provided to create the initial resource. -* `type`: the type of the Entity being created. -* `ID`: the set of attributes which identify the entity. -* `description`: the set of attributes which describe the entity. -* `schema_url` (optional): The Schema URL that should be recorded for entity. +During resource detection, EntityProvider MUST NOT fire any EntityListener +events. -If the incoming Entity's `type` is not found in the current list of entities, -then the new Entity is added to the list. +Upon completion of the resource detection phase, EntityProvider MUST fire +an `On ResourceInitialize` event to all EntityListeners. + +Any calls to `GetResource` operation, if provided, SHOULD block until +resource detection is completed. + +Upon failure for resource detection to complete within a timeout, a resource +SHOULD be constructed with available completed detection, `GetResource` +operations MUST be unblocked and `On ResourceInitialize` event MUST be fired +to all EntityListeners. + +#### Add or Update Entity + +The `Add or Update Entity` operation MUST match the API definition. -After an entity is created, it MUST be appended to the list of current entities. +The behavior MUST match the following: + + - If the incoming Entity's `type` is not found in the current list of entities, +then the new Entity is added to the list. + - After an entity is created, it MUST be appended to the list of current entities. A new resource object MUST be generated by merging the list of entities together in order. - -If the incoming Entity's `type` parameter matches an existing Entity in + - If the incoming Entity's `type` parameter matches an existing Entity in the current list AND the `ID` attributes for these entities are different, then the SDK MUST ignore the new entity. - -If the incoming Entity's `type` parameter matches an existing Entity in the + - If the incoming Entity's `type` parameter matches an existing Entity in the current list AND the `ID` attributes for these entities are the same but the `schema_url` is different, then the SDK MUST ignore the new entity. - -If the incoming Entity's `type` parameter matches an existing Entity in the + - If the incoming Entity's `type` parameter matches an existing Entity in the current list AND the `ID` attributes for these entities are the same AND the `schema_url` is the same, then the SDK MUST add any new attributes found in the `description` to the existing entity. Any new `description` attributes with the same keys as existing `description` attributes SHOULD replace previous values. -`Add Entity` MUST trigger the `On EntityState` operation for all +`Add or Update Entity` MUST trigger the `On EntityState` operation for all registered `EntityListeners`. -#### Replace Entity - -`Replace Entity` replaces the resource attributes associated with an entity. - -Replace Entity MUST accept the following parameters: +#### Add or Replace Entity -* `type`: the type of the Entity being created. -* `ID`: the set of attributes which identify the entity. -* `description`: the set of attributes which describe the entity. -* `schema_url` (optional): The Schema URL that should be recorded for entity. +The `Add or Replace Entity` operation MUST match the API definition. -If an existing Entity with the same `type` already exists, then `Replace Entity` -MUST trigger an `On EntityDelete` operation for all registered `EntityListener`s -with the previous state of the Entity. The Resource reported MUST NOT include -the removed Entity. +The behavior MUST match the following: -A new entity should be created and added to the list of entities. After the -entity is created, it MUST be appended to the list of current entities. A new -resource object MUST be generated by merging the list of entities together -in order. +- If an existing Entity with the same `type` already exists, then + `Add or Replace Entity` MUST not trigger an `On EntityDelete` operation. + Instead, a single `On EntityUpdate` event will be sent with fully updated + resource, and the new entity. +- A new entity should be created and added to the list of entities. After the + entity is created, it MUST be appended to the list of current entities. A new + resource object MUST be generated by merging the list of entities together + in order. `Replace Entity` MUST trigger the `On EntityState` operation for all registered `EntityListener`s. @@ -275,11 +282,7 @@ new entity. #### Delete Entity -`Delete Entity` replaces the resource attributes associated with an entity. - -Update Entity MUST accept the following parameters: - -* `type`: the type of the Entity being removed. +The `Delete Entity` operation MUST accept the parameters defined in the API. After an entity is deleted, a new resource object MUST be generated by merging the list of entities together in order. @@ -292,6 +295,9 @@ registered `EntityListeners`. `Get Resource` MUST return a reference to the current resource held by the EntityProvider. +This operation MAY block when EntityProvider is in a resource detection state, +but MUST NOT block indefinitely. + #### On Change `On Change` registers an `EntityListener` to be called every time an entity is updated @@ -303,8 +309,19 @@ EntityProvider. ### SDK Changes -NewTraceProvider, NewMetricsProvider, and NewLoggerProvider now take a EntityProvider -as a parameter. How SDKs handle resource changes is listed under [open questions](#open-questions). +All other signal providers (TracerProvider, MeterProvider, LoggerProvider, etc) +MUST be updated to use EntityProvider to obtain `Resource`. This SHOULD be done +through either using `EntityListener` interface. This MAY be done via a +`Get Resource` operation, although this limits how the SDK can handle resource +changes. + +How SDKs handle resource changes is listed under [open questions](#open-questions). + +### SDK Helpers + +SDKs which do not provder a `Get Resource` operation MAY provide a +`LatestResource` component which acts as an `EntityListener` and provides a +mechanism to remember and retrieve the latest resource. ## Trade-offs and mitigations From d1f216f2f7dfab5c441e4ab8d6e8fe97bc37875c Mon Sep 17 00:00:00 2001 From: Josh Suereth Date: Mon, 25 Aug 2025 11:54:49 -0400 Subject: [PATCH 19/19] Fix lints. --- oteps/4316-resource-provider.md | 55 ++++++++++++++++++++++----------- 1 file changed, 37 insertions(+), 18 deletions(-) diff --git a/oteps/4316-resource-provider.md b/oteps/4316-resource-provider.md index 01a48729e9b..932cb0802c5 100644 --- a/oteps/4316-resource-provider.md +++ b/oteps/4316-resource-provider.md @@ -82,7 +82,6 @@ The API provides three primary user cases: of the entity in the SDK. For example, updating session / page status when a page comes back from inactive. - ### EntityProvider The `EntityProvider` API MUST provide the following operations: @@ -152,7 +151,7 @@ An EntityListener MUST provide the following operations: `On ResourceInitialize` MUST accept the following parameters: -* `Resource`: represents the entire set of resources after initalize. +* `Resource`: represents the entire set of resources after initialize. This operation MUST only be called once per EntityProvider. @@ -238,22 +237,22 @@ The `Add or Update Entity` operation MUST match the API definition. The behavior MUST match the following: - - If the incoming Entity's `type` is not found in the current list of entities, -then the new Entity is added to the list. - - After an entity is created, it MUST be appended to the list of current entities. -A new resource object MUST be generated by merging the list of entities together -in order. - - If the incoming Entity's `type` parameter matches an existing Entity in -the current list AND the `ID` attributes for these entities are different, then -the SDK MUST ignore the new entity. - - If the incoming Entity's `type` parameter matches an existing Entity in the -current list AND the `ID` attributes for these entities are the same but -the `schema_url` is different, then the SDK MUST ignore the new entity. - - If the incoming Entity's `type` parameter matches an existing Entity in the -current list AND the `ID` attributes for these entities are the same AND the -`schema_url` is the same, then the SDK MUST add any new attributes found in the -`description` to the existing entity. Any new `description` attributes with the -same keys as existing `description` attributes SHOULD replace previous values. +- If the incoming Entity's `type` is not found in the current list of entities, + then the new Entity is added to the list. +- After an entity is created, it MUST be appended to the list of current entities. + A new resource object MUST be generated by merging the list of entities together + in order. +- If the incoming Entity's `type` parameter matches an existing Entity in + the current list AND the `ID` attributes for these entities are different, then + the SDK MUST ignore the new entity. +- If the incoming Entity's `type` parameter matches an existing Entity in the + current list AND the `ID` attributes for these entities are the same but + the `schema_url` is different, then the SDK MUST ignore the new entity. +- If the incoming Entity's `type` parameter matches an existing Entity in the + current list AND the `ID` attributes for these entities are the same AND the + `schema_url` is the same, then the SDK MUST add any new attributes found in the + `description` to the existing entity. Any new `description` attributes with the + same keys as existing `description` attributes SHOULD replace previous values. `Add or Update Entity` MUST trigger the `On EntityState` operation for all registered `EntityListeners`. @@ -307,6 +306,26 @@ If the `EntityProvider` is already initialized, then it MUST call `On ResourceInitialize` immediately with the current resource held by the EntityProvider. +### ResourceDetector + +A `ResourceDetector` concept is updated. + +ResourceDetector MUST provide the following operations: + +- `Detect entities` + +#### Detect Entities operation + +The Detect Entities Operation SHOULD accept an `EntityProvider` where +detected entities will be reported. + +The Detect Entities Operation SHOULD allow asynchronous or long-running +exeuction, e.g. issuing an HTTP request to a local service for identity, reading +from disk, etc. + +The DetectEntities Operation SHOULD return a status to indicate completion or +failure. + ### SDK Changes All other signal providers (TracerProvider, MeterProvider, LoggerProvider, etc)