OpenTelemetry exporter plugin, and built-in plugin provision by vigoo · Pull Request #3014 · golemcloud/golem

vigoo · 2026-03-18T08:19:04Z

Resolves #1613
Resolves #1614

Introduces a way to have built-in oplog processor plugins:

they are stored in the plugins dir of this repo, compiled with golem-cli
we store the compiled wasm in the repo too, so cargo install works for golem/golem-cli
they are embedded in the registry service binary
the built-in plugins are automatically deployed to the system account
every new environment automatically gets grants to use them

Adds the first such built-in plugin, the OpenTelemetry exporter:

supports traces, logs and metrics
tested with dockerized otel-collector and jaeger

Follow-up: #3015

…d the entries

vigoo · 2026-03-18T09:32:24Z

One thing I forgot to check / implement: how do we provision changed / updated built-in plugins when deploying to prod (but keeping the live db)

vigoo · 2026-03-18T10:23:48Z

One thing I forgot to check / implement: how do we provision changed / updated built-in plugins when deploying to prod (but keeping the live db)

The single line change I've pushed that makes plugins versioned by the golem version fixes this, I think

mschuwalow · 2026-03-18T13:42:07Z

golem-registry-service/src/services/builtin_plugin_provisioner.rs

+        }
+    };
+
+    // 3. Upload "otlp-exporter" component, or update it if it already exists


We should use the component / wasm hash here to prevent updating the component on each startup of each instance

Yeah that broke in the last commit, previously it was supposed to be named by the version (but that has other issues), thanks

mschuwalow · 2026-03-18T13:46:47Z

golem-worker-executor/src/services/oplog/plugin.rs

            .await
        {
            Ok(()) => {
+                // Enqueue succeeded — immediately confirm


Does the receiver side protects against receiving the same batch multiple times?

Yes, all the "exactly once semantics" changes were done in a separate PR before this. Here I just added this extra logging of process failures back to the caller agent's event stream.
(For double send, it's solved by having deterministic oplog processor worker name + determinstic idempotency keys)

mschuwalow · 2026-03-18T13:49:00Z

golem-registry-service/src/bootstrap/mod.rs


        let auth_service = Arc::new(AuthService::new(repos.account_repo.clone()));

+        let root_account_id = config


Let's use a dedicated account for this without special roles. I don't want to leak the admin account id to users

By leaking, you mean that they can see the ID because of the plugin grants?

yes, exactly

mschuwalow · 2026-03-18T13:51:32Z

golem-registry-service/src/services/builtin_plugin_provisioner.rs

+    let apps = application_service
+        .list_in_account(root_account_id, &auth)
+        .await
+        .map_err(|e| anyhow::anyhow!("Failed to list applications: {e}"))?;
+
+    for app_entry in &apps {
+        let envs = match environment_service
+            .list_in_application(app_entry.id, &auth)
+            .await
+        {
+            Ok(envs) => envs,
+            Err(e) => {
+                tracing::warn!(
+                    "Failed to list environments for app {}: {e}",
+                    app_entry.name
+                );
+                continue;
+            }
+        };
+
+        for env_entry in &envs {
+            match environment_plugin_grant_service
+                .create(
+                    env_entry.id,
+                    EnvironmentPluginGrantCreation {
+                        plugin_registration_id: plugin.id,
+                    },
+                    &auth,
+                )
+                .await
+            {
+                Ok(_) => {}
+                Err(EnvironmentPluginGrantError::GrantForPluginAlreadyExists) => {}
+                Err(e) => {
+                    tracing::warn!("Failed to grant plugin to env {}: {e}", env_entry.name);
+                }
+            }
+        }
+    }


This block only grants access to environments in the root account's applications, which is not useful.

There should not be any applications or environments in that account (outside the one used for the plugin definition)

there supposed to be grants created for the built-in plugins whenever someone creates a new environment. i'll double check and remove this unnecessary part

mschuwalow · 2026-03-18T14:03:06Z

golem-registry-service/src/services/environment.rs

+        let plugin_grants: Vec<EnvironmentPluginGrantRecord> = builtin_plugins
+            .into_iter()
+            .map(|p| {
+                EnvironmentPluginGrantRecord::creation(


EnvironmentPluginGrants can be managed by users and deleted as long as they have access to the environment. I don't think we should allow deletion of builtin plugins from environments.
Additionally, these grants will show up in the api and cli for users.

I think builtin plugins want to have special handling and not reuse exactly the same mechanism as user plugins. A couple of options that I think make sense:

Do not give builtin plugins an id at all, but rather have an enum for builtin plugins. Whereever we can refer to a plugin we allow doing it either by id or by selecting one of the builtin ones

Store them in the db with an id like now, but either have a dedicated table for them or extend the current table such that they can live outside of the normal account / application / environment hierarchy. All queries that retrieve plugins can join with the buitin plugins. In that case we do not have an automatically generated account for them at all and can use the system account id wherever we do want to display an owner (clearly marking them as system).

Alternatively, if we do keep the current system managed account + grant structure, we should protect the automatically created grants from deletion and make them more descriptive for users.

I would rather keep it simple - that's why I wanted to build it on top of the existing things. They are not special in any way other than we want them to be available for everyone without explicitly installing them.

I think if we move them to another root account as you asked, and maybe disallow deleting the grants, we have the desired UX and we have minimal complexity on the implementation side (for me the provision step and auto-granting the plugins feels very nice and scoped)

That sounds good to me 👍

mschuwalow · 2026-03-19T16:39:05Z

golem-registry-service/src/config.rs

+                id: AccountId(uuid!("adb2694f-cd9f-425d-905d-ca2888c9c5de")),
+                name: "Builtin Plugin Owner".to_string(),
+                email: AccountEmail("builtin-plugin-owner@golem.cloud".to_string()),
+                token: TokenSecret::trusted("32d6072d-64e9-4a4a-b8f9-fadf68bb446b".to_string()),


I would prefer adjusting this so we don't create a token for this account, as it should never be used interactively. But that can easily be done in a followup.

mschuwalow · 2026-03-19T16:39:44Z

golem-registry-service/src/config.rs

+                name: "Builtin Plugin Owner".to_string(),
+                email: AccountEmail("builtin-plugin-owner@golem.cloud".to_string()),
+                token: TokenSecret::trusted("32d6072d-64e9-4a4a-b8f9-fadf68bb446b".to_string()),
+                role: AccountRole::BuiltinPluginOwner,


Not sure about having a dedicated "marker" role. So far the roles were actually used for permission checks, but I'm not strongly against it.

mschuwalow · 2026-03-19T16:40:10Z

golem-registry-service/src/config.rs


+#[derive(Clone, Debug, Serialize, Deserialize, Default)]
+pub struct BuiltinPluginsConfig {
+    pub enabled: bool,


Enum would be more consistent

will do in a next pr

vigoo added 30 commits March 6, 2026 14:07

Deterministic idempotency key

00a8adf

oplog forwarding unit tests

d306753

Fixed existing oplog-processor tests

31d1e7b

More oplog processor tests and supporting code

8d516e4

Merge branch 'main' into oplog-processor-improvements

439cb81

OplogProcessorCheckpoint oplog entry

fc29fc1

Tracking oplog processor checkpoints in agent status record

ea3b061

New get-last-processed-index method to oplog processor interface

8b37023

ForwardingOplog live path improvements

f153438

Fix

f207339

Periodic locality recovery

ff8d135

Logging skill

ba8d473

Oplog processor test component updates

c3eebac

Some more tests

244c7f9

Test fix

99cf05d

Updated testing skill and idempotency_key check fix

343467f

Fixes

35a54f9

Fix how locality recovery determines that the target already processe…

e53ca89

…d the entries

Accept pending state too

2d22248

Clippy & format

39f148e

Merge branch 'main' into oplog-processor-improvements

bb7ae53

Missing file

7daa7a2

TS SDK fix

f21e432

Updated openapi

1ffe62a

Fix build-components.sh

13e564b

Rebuilt test components

3422c18

Merge branch 'main' into oplog-processor-improvements

660631d

Merge branch 'main' into oplog-processor-improvements

c812784

Fix

cb27ea8

Clippy

00c3200

Use golem_version as plugin version

e74a2a2

vigoo added 6 commits March 18, 2026 11:24

Merge branch 'main' into otlp-plugin

954a99e

Merge branch 'main' into otlp-plugin

d5963af

Merge branch 'main' into otlp-plugin

0fe611a

Tagged the new plugin and otlp tests and running them in a new CI job

6b580bc

Fixes

d2d606e

Merge branch 'main' into otlp-plugin

0c77e13

mschuwalow reviewed Mar 18, 2026

View reviewed changes

vigoo added 16 commits March 18, 2026 17:34

Remove unnecessary step from the provisioning

2feeed9

Trying to fix the docker issue on CI

2f7a5f8

Separate account for built-in plugins

bebc1f7

Hash based component check in provision

f36d3de

Cleanup the provision logic

9ecd9bb

Do not allow deleting built-in plugin grants

f0ded8b

Fix

7489df0

Fixes

2e45d11

Regenerated plugins

2991a4b

Rebuilt wasms

193db9d

Rebuilt wasm

e75620a

Oplog processor fixes

d239588

Rebuilt wasms

1a2e9d3

Merge branch 'main' into otlp-plugin

c415a50

Fix flaky test

3aedd2f

Flaky test fix

c30f8dc

mschuwalow approved these changes Mar 19, 2026

View reviewed changes

vigoo merged commit 3e2695a into main Mar 19, 2026
82 of 84 checks passed

vigoo deleted the otlp-plugin branch March 19, 2026 17:36

github-actions bot locked and limited conversation to collaborators Mar 19, 2026


		let auth_service = Arc::new(AuthService::new(repos.account_repo.clone()));

		let root_account_id = config

Conversation

vigoo commented Mar 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vigoo commented Mar 18, 2026

Uh oh!

vigoo commented Mar 18, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mschuwalow Mar 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

vigoo commented Mar 18, 2026 •

edited

Loading

mschuwalow Mar 18, 2026 •

edited

Loading