diff --git a/.agents/skills/adding-dependencies/SKILL.md b/.agents/skills/adding-dependencies/SKILL.md new file mode 100644 index 0000000000..938553e589 --- /dev/null +++ b/.agents/skills/adding-dependencies/SKILL.md @@ -0,0 +1,69 @@ +--- +name: adding-dependencies +description: "Adding or updating crate dependencies in the Golem workspace. Use when adding a new Rust dependency, changing dependency versions, or configuring dependency features." +--- + +# Adding Dependencies + +All crate dependencies in the Golem workspace are centrally managed. Versions and default features are specified **once** in the root `Cargo.toml` under `[workspace.dependencies]`, and workspace members reference them with `{ workspace = true }`. + +## Adding a New Dependency + +### Step 1: Add to root workspace Cargo.toml + +Add the dependency under `[workspace.dependencies]` in the root `Cargo.toml`, specifying the version and any default features: + +```toml +# Simple version +my-crate = "1.2.3" + +# With features +my-crate = { version = "1.2.3", features = ["feature1", "feature2"] } + +# With default-features disabled +my-crate = { version = "1.2.3", default-features = false } +``` + +Keep entries **alphabetically sorted** within the section. Internal workspace crates are listed first (with `path`), followed by external dependencies. + +### Step 2: Reference from workspace member + +In the member crate's `Cargo.toml`, add the dependency using `workspace = true`: + +```toml +[dependencies] +my-crate = { workspace = true } + +# To add extra features beyond what the workspace specifies +my-crate = { workspace = true, features = ["extra-feature"] } + +# To make it optional +my-crate = { workspace = true, optional = true } +``` + +**Never** specify a version directly in a member crate's `Cargo.toml`. Always use `{ workspace = true }`. + +The same pattern applies to `[dev-dependencies]` and `[build-dependencies]`. + +### Step 3: Verify + +```shell +cargo build -p # Build the specific crate +cargo make build # Full workspace build +``` + +## Updating a Dependency Version + +Change the version **only** in the root `Cargo.toml` under `[workspace.dependencies]`. All workspace members automatically pick up the new version. + +## Pinned and Patched Dependencies + +Some dependencies use exact versions (`=x.y.z`) to ensure compatibility. Check the `[patch.crates-io]` section in the root `Cargo.toml` for git-overridden crates (e.g., `wasmtime`). When updating patched dependencies, both the version under `[workspace.dependencies]` and the corresponding `[patch.crates-io]` entry must be updated together. + +## Checklist + +1. Version specified in root `Cargo.toml` under `[workspace.dependencies]` +2. Member crate references it with `{ workspace = true }` +3. No version numbers in member crate `Cargo.toml` files +4. Entry is alphabetically sorted in the workspace dependencies list +5. `cargo make build` succeeds diff --git a/.agents/skills/debugging-hanging-tests/SKILL.md b/.agents/skills/debugging-hanging-tests/SKILL.md new file mode 100644 index 0000000000..99fa30f74a --- /dev/null +++ b/.agents/skills/debugging-hanging-tests/SKILL.md @@ -0,0 +1,89 @@ +--- +name: debugging-hanging-tests +description: "Diagnosing and fixing hanging worker executor or integration tests. Use when a test hangs indefinitely, times out, or appears stuck during execution." +--- + +# Debugging Hanging Tests + +Worker executor and integration tests can hang indefinitely due to `unimplemented!()` panics in async tasks, deadlocks, missing shard assignments, or other async runtime issues. This skill provides a systematic workflow for diagnosing and resolving these hangs. + +## Common Causes + +| Cause | Symptom | +|-------|---------| +| `unimplemented!()` panic in async task | Test hangs after a log line mentioning the unimplemented feature | +| Deadlock | Test hangs with no further log output | +| Missing shard assignment | Worker never starts executing | +| Channel sender dropped | Receiver awaits forever with no error | +| Infinite retry loop | Repeated log lines with the same error | + +## Step 1: Add a Timeout + +Add a `#[timeout]` attribute so the test fails with a clear error instead of hanging forever: + +```rust +use test_r::test; +use test_r::timeout; + +#[test] +#[timeout("30s")] +async fn my_hanging_test() { + // ... +} +``` + +Choose a timeout generous enough for normal execution but short enough to fail quickly when hung (30s–60s for most tests, up to 120s for complex integration tests). + +## Step 2: Capture Full Output + +Run the test with `--nocapture` and save **all output** to a file. The root cause often appears far before the point where the test hangs: + +```shell +cargo test -p -- --nocapture > tmp/test_output.txt 2>&1 +``` + +**Important:** Always redirect to a file. The output can be thousands of lines, and the relevant error may be near the beginning while the hang occurs at the end. + +## Step 3: Search for Root Cause + +Search the saved output file for these patterns, in order of likelihood: + +```shell +grep -n "unimplemented" tmp/test_output.txt +grep -n "panic" tmp/test_output.txt +grep -n "ERROR" tmp/test_output.txt +grep -n "WARN" tmp/test_output.txt +``` + +### What to look for + +- **`not yet implemented`** or **`unimplemented`**: An async task hit an unimplemented code path and panicked. The panic is silently swallowed by the async runtime, causing the caller to await forever. +- **`panic`**: Similar to above — a panic in a spawned task won't propagate to the test. +- **`ERROR` with retry**: A service call failing repeatedly, causing an infinite retry loop. +- **Repeated identical log lines**: Indicates a retry loop or polling cycle that never succeeds. + +## Step 4: Fix the Root Cause + +### If caused by `unimplemented!()` +Implement the missing functionality, or if it's a test-only issue, provide a stub/mock. + +### If caused by a deadlock +Look for: +- Multiple `lock()` calls on the same mutex in nested scopes +- `await` while holding a lock guard +- Circular lock dependencies between tasks + +### If caused by missing shard assignment +Check that the test setup properly initializes the shard manager and assigns shards before starting workers. + +### If caused by a dropped sender +Ensure all channel senders are kept alive for the duration the receiver needs them. Check for early returns or error paths that drop the sender. + +## Checklist + +1. `#[timeout("30s")]` added to the hanging test +2. Test run with `--nocapture`, output saved to file +3. Output searched for `unimplemented`, `panic`, `ERROR` +4. Root cause identified and fixed +5. Test passes within the timeout +6. Remove the `#[timeout]` if it was only added for debugging (or keep it as a safety net) diff --git a/.agents/skills/modifying-http-endpoints/SKILL.md b/.agents/skills/modifying-http-endpoints/SKILL.md new file mode 100644 index 0000000000..4483ca4023 --- /dev/null +++ b/.agents/skills/modifying-http-endpoints/SKILL.md @@ -0,0 +1,101 @@ +--- +name: modifying-http-endpoints +description: "Adding or modifying HTTP REST API endpoints in Golem services. Use when creating new endpoints, changing existing API routes, or updating request/response types for the Golem REST API." +--- + +# Modifying HTTP Endpoints + +## Framework + +Golem uses **Poem** with **poem-openapi** for REST API endpoints. Endpoints are defined as methods on API structs annotated with `#[OpenApi]` and `#[oai]`. + +## Where Endpoints Live + +- **Worker service**: `golem-worker-service/src/api/` — worker lifecycle, invocation, oplog +- **Registry service**: `golem-registry-service/src/api/` — components, environments, deployments, plugins, accounts + +Each service has an `api/mod.rs` that defines an `Apis` type tuple and a `make_open_api_service` function combining all API structs. + +## Adding a New Endpoint + +### 1. Define the endpoint method + +Add a method to the appropriate API struct (e.g., `WorkerApi`, `ComponentsApi`): + +```rust +#[oai( + path = "/:component_id/workers/:worker_name/my-action", + method = "post", + operation_id = "my_action" +)] +async fn my_action( + &self, + component_id: Path, + worker_name: Path, + request: Json, + token: GolemSecurityScheme, +) -> Result> { + // ... +} +``` + +### 2. If adding a new API struct + +1. Create a new file in the service's `api/` directory +2. Define a struct and impl block with `#[OpenApi(prefix_path = "/v1/...", tag = ApiTags::...)]` +3. Add it to the `Apis` type tuple in `api/mod.rs` +4. Instantiate it in `make_open_api_service` + +### 3. Request/response types + +- Define types in `golem-common/src/model/` with `poem_openapi::Object` derive +- If the type is used in the generated client, add it to the type mapping in `golem-client/build.rs` + +## After Modifying Endpoints + +After any endpoint change, you **must** regenerate and rebuild: + +### Step 1: Regenerate OpenAPI specs + +```shell +cargo make generate-openapi +``` + +This builds the services, dumps their OpenAPI YAML, merges them, and stores the result in `openapi/`. + +### Step 2: Clean and rebuild golem-client + +The `golem-client` crate auto-generates its code from the OpenAPI spec at build time via `build.rs`. After regenerating the specs: + +```shell +cargo clean -p golem-client +cargo build -p golem-client +``` + +The clean step is necessary because the build script uses `rerun-if-changed` on the YAML file, but cargo may cache stale generated code. + +### Step 3: If new types are used in the client + +Add type mappings in `golem-client/build.rs` to the `gen()` call's type replacement list. This maps OpenAPI schema names to existing Rust types from `golem-common` or `golem-wasm`. + +### Step 4: Build and verify + +```shell +cargo make build +``` + +Then run the appropriate tests: + +- HTTP API tests: `cargo make api-tests-http` +- gRPC API tests: `cargo make api-tests-grpc` + +## Checklist + +1. Endpoint method added with `#[oai]` annotation +2. New API struct registered in `api/mod.rs` `Apis` tuple and `make_open_api_service` (if applicable) +3. Request/response types defined in `golem-common` with `poem_openapi::Object` +4. Type mappings added in `golem-client/build.rs` (if applicable) +5. `cargo make generate-openapi` run +6. `cargo clean -p golem-client && cargo build -p golem-client` run +7. `cargo make build` succeeds +8. `cargo make fix` run before PR diff --git a/.agents/skills/modifying-service-configs/SKILL.md b/.agents/skills/modifying-service-configs/SKILL.md new file mode 100644 index 0000000000..033ad33677 --- /dev/null +++ b/.agents/skills/modifying-service-configs/SKILL.md @@ -0,0 +1,80 @@ +--- +name: modifying-service-configs +description: "Modifying service configuration types or defaults. Use when changing config structs, adding config fields, or updating default values for any Golem service." +--- + +# Modifying Service Configs + +Golem services use a configuration system built on [Figment](https://github.com/SergioBenitez/Figment) via a custom `ConfigLoader`. Configuration defaults are serialized to TOML and env-var reference files that are checked into the repository and validated in CI. + +## How Configuration Works + +Each service has a configuration struct that implements: +- `Default` — provides default values +- `Serialize` / `Deserialize` — for TOML and env-var serialization +- `SafeDisplay` — for logging without exposing secrets + +Services load config by merging (in order): defaults → TOML file → environment variables. + +## Service Config Locations + +| Service | Config struct | File | +|---------|--------------|------| +| Worker Executor | `GolemConfig` | `golem-worker-executor/src/services/golem_config.rs` | +| Worker Service | `WorkerServiceConfig` | `golem-worker-service/src/config.rs` | +| Registry Service | `RegistryServiceConfig` | `golem-registry-service/src/config.rs` | +| Shard Manager | `ShardManagerConfig` | `golem-shard-manager/src/shard_manager_config.rs` | +| Compilation Service | `ServerConfig` | `golem-component-compilation-service/src/config.rs` | + +The all-in-one `golem` binary has its own merged config that combines multiple service configs. + +## Modifying a Config + +### Step 1: Edit the config struct + +Add, remove, or modify fields in the appropriate config struct. Update the `Default` implementation if default values change. + +### Step 2: Regenerate config files + +```shell +cargo make generate-configs +``` + +This builds the service binaries and runs them with `--dump-config-default-toml` and `--dump-config-default-env-var` flags, producing reference files that reflect the current `Default` implementation. + +### Step 3: Verify + +```shell +cargo make build +``` + +### Step 4: Check configs match + +CI runs `cargo make check-configs` which regenerates configs and diffs them against committed files. If this fails, you forgot to run `cargo make generate-configs`. + +## Adding a New Config Field + +1. Add the field to the config struct with a `serde` attribute if needed +2. Set its default value in the `Default` impl +3. Run `cargo make generate-configs` to update reference files +4. If the field requires a new environment variable, the env-var mapping is derived automatically from the field path + +## Removing a Config Field + +1. Remove the field from the struct and `Default` impl +2. Run `cargo make generate-configs` +3. Check for any code that references the removed field + +## Nested Config Types + +Many config structs compose sub-configs (e.g., `GolemConfig` contains `WorkersServiceConfig`, `BlobStoreServiceConfig`, etc.). When modifying a sub-config type that's shared across services, regenerate configs for all affected services — `cargo make generate-configs` handles this automatically. + +## Checklist + +1. Config struct modified with appropriate `serde` attributes +2. `Default` implementation updated +3. `cargo make generate-configs` run +4. Generated TOML and env-var files committed +5. `cargo make build` succeeds +6. `cargo make check-configs` passes (CI validation) +7. `cargo make fix` run before PR diff --git a/.agents/skills/modifying-test-components/SKILL.md b/.agents/skills/modifying-test-components/SKILL.md new file mode 100644 index 0000000000..45f7d216bb --- /dev/null +++ b/.agents/skills/modifying-test-components/SKILL.md @@ -0,0 +1,99 @@ +--- +name: modifying-test-components +description: "Building or modifying test WASM components in test-components/. Use when a test component needs to be rebuilt, a new test component is needed, or SDK changes require downstream test component rebuilds." +--- + +# Modifying Test Components + +Worker executor tests and integration tests use pre-compiled WASM files from `test-components/`. These are checked into the repository as binary artifacts. + +## Key Rules + +1. **Do not rebuild test components unless necessary.** Use existing compiled WASM files. +2. **Only rebuild if the test component has its own `AGENTS.md`** with build instructions. If it doesn't, the component cannot be rebuilt by you. +3. **SDK changes require rebuilding dependent test components.** If you modify `sdks/rust/` or `sdks/ts/`, you must rebuild any test components that use the changed SDK. + +## When to Rebuild + +| Change | Action Required | +|--------|----------------| +| Modifying a test component's source code | Rebuild that component (if it has an AGENTS.md) | +| Modifying `sdks/rust/` (golem-rust) | Rebuild Rust test components that depend on it | +| Modifying `sdks/ts/` (golem-ts-sdk) | Rebuild agent template WASM first, then TS test components | +| Modifying WIT interfaces | Run `cargo make wit`, then rebuild affected components | +| No source changes | Do not rebuild | + +## Rebuilding a Test Component + +### Step 1: Check for build instructions + +```shell +cat test-components//AGENTS.md +``` + +If no `AGENTS.md` exists, **stop** — you cannot rebuild this component. + +### Step 2: Follow the component's AGENTS.md + +Each component's `AGENTS.md` contains specific build instructions. Follow them exactly. + +## TS SDK Change Rebuild Chain + +When modifying the TypeScript SDK, you must follow this exact rebuild order: + +### 1. Build the TS SDK packages + +```shell +cd sdks/ts +npx pnpm install +npx pnpm run build +``` + +### 2. Rebuild the agent template WASM + +This step is **required** before rebuilding any TS test components. The agent template embeds the SDK runtime. + +```shell +cd sdks/ts +npx pnpm run build-agent-template +``` + +**Requires `cargo-component` v0.21.1** — see `sdks/ts/AGENTS.md` for installation. + +### 3. Rebuild affected TS test components + +Follow each component's `AGENTS.md` for specific instructions. + +## Rust SDK Change Rebuild Chain + +When modifying the Rust SDK: + +### 1. Build the Rust SDK + +```shell +cargo build -p golem-rust +cargo build -p golem-rust-macro +``` + +### 2. Rebuild affected Rust test components + +Follow each component's `AGENTS.md` for specific instructions. Rust test components typically use `cargo component build` with a local path dependency on `golem-rust`. + +## Finding Test Components + +Test components live in `test-components/`. To find which ones have build instructions: + +```shell +ls test-components/*/AGENTS.md +``` + +To find which test components depend on a specific SDK, check their `Cargo.toml` (Rust) or `package.json` (TS) for SDK references. + +## Checklist + +1. Confirmed the component has an `AGENTS.md` with build instructions +2. If SDK was changed: rebuilt SDK first +3. If TS SDK was changed: rebuilt agent template WASM before components +4. Followed the component's specific `AGENTS.md` build instructions +5. Committed the rebuilt WASM binary +6. Ran the relevant tests to verify (`cargo make worker-executor-tests` or `cargo make integration-tests`) diff --git a/.agents/skills/modifying-wit-interfaces/SKILL.md b/.agents/skills/modifying-wit-interfaces/SKILL.md new file mode 100644 index 0000000000..07486bab58 --- /dev/null +++ b/.agents/skills/modifying-wit-interfaces/SKILL.md @@ -0,0 +1,121 @@ +--- +name: modifying-wit-interfaces +description: "Adding or modifying WIT (WebAssembly Interface Types) interfaces. Use when changing .wit files, updating WIT dependencies, or working with component interfaces." +--- + +# Modifying WIT Interfaces + +Golem uses WIT (WebAssembly Interface Types) to define component interfaces. WIT files are centrally managed and synchronized across multiple sub-projects. + +## Directory Structure + +### Central WIT directory + +The root `wit/` directory is the source of truth: + +``` +wit/ +├── host.wit # Core Golem host interface +├── deps.toml # WIT dependency declarations +├── deps.lock # Locked dependency versions +└── deps/ # Fetched WIT dependencies + ├── io/ + ├── clocks/ + ├── golem-1.x/ + ├── golem-core/ + ├── golem-agent/ + └── logging/ +``` + +### Synchronized copies + +WIT files are copied to these sub-projects by `cargo make wit`: + +| Target | WIT deps copied | +|--------|----------------| +| `golem-wasm/wit/deps/` | io, clocks, golem-1.x, golem-core | +| `golem-common/wit/deps/` | io, clocks, golem-1.x, golem-core, golem-agent | +| `cli/golem-cli/wit/deps/` | clocks, io, golem-1.x, golem-core, golem-agent, logging | +| `sdks/rust/golem-rust/wit/deps/` | All deps + golem-ai | +| `sdks/ts/wit/deps/` | All deps + golem-ai | +| `test-components/oplog-processor/wit/deps/` | All deps | + +**Never manually edit** files in any `wit/deps/` directory. They are overwritten by `cargo make wit`. + +## Modifying an Existing WIT Interface + +### Step 1: Edit the WIT file + +Edit the relevant `.wit` file in the root `wit/` directory (e.g., `wit/host.wit` or a file under `wit/deps/`). + +If editing a dependency managed by `deps.toml`, you may need to update the dependency version in `wit/deps.toml` first. + +### Step 2: Synchronize WIT across sub-projects + +```shell +cargo make wit +``` + +This removes all `wit/deps/` directories in sub-projects and re-copies the correct subset from the root. + +### Step 3: Verify synchronization + +```shell +cargo make check-wit +``` + +This runs `cargo make wit` and then checks `git diff` to ensure the committed WIT files match what the sync produces. If this fails in CI, you forgot to run `cargo make wit`. + +### Step 4: Build and verify + +```shell +cargo make build +``` + +WIT changes affect generated bindings in multiple crates. A full build ensures all bindings are regenerated correctly. + +## Adding a New WIT Dependency + +### Step 1: Add to deps.toml + +Edit `wit/deps.toml` to add the new dependency. + +### Step 2: Fetch and sync + +```shell +cargo make wit +``` + +### Step 3: Update sync tasks if needed + +If the new dependency needs to be available in specific sub-projects, edit `Makefile.toml` to add the copy step in the appropriate `wit-*` task (e.g., `wit-golem-wasm`, `wit-golem-common`, `wit-golem-cli`, `wit-sdks`, `wit-test-components`). + +## Downstream Impact + +WIT changes can have wide-reaching effects: + +| What changed | What needs rebuilding | +|---|---| +| Core interfaces (`golem-1.x`, `golem-core`) | Everything: services, SDKs, test components | +| Agent interfaces (`golem-agent`) | golem-common, CLI, SDKs, agent test components | +| SDK-only interfaces (`golem-ai`) | SDKs only | +| Host interface (`host.wit`) | Worker executor, services | + +### SDK rebuild chain + +If WIT changes affect SDK interfaces: + +1. **Rust SDK**: Rebuild `golem-rust` (bindings are generated via `wit_bindgen::generate!`) +2. **TS SDK**: Rebuild packages (`npx pnpm run build` in `sdks/ts/`), then rebuild agent template WASM (`npx pnpm run build-agent-template`) +3. **Test components**: Rebuild any test components that use the changed interfaces (see their `AGENTS.md`) + +## Checklist + +1. WIT file edited in root `wit/` directory (not in a `wit/deps/` copy) +2. `cargo make wit` run to synchronize +3. `cargo make check-wit` passes +4. `Makefile.toml` sync tasks updated if a new dependency was added +5. `cargo make build` succeeds +6. SDKs rebuilt if SDK interfaces changed +7. Test components rebuilt if their interfaces changed +8. `cargo make fix` run before PR diff --git a/.agents/skills/pre-pr-checklist/SKILL.md b/.agents/skills/pre-pr-checklist/SKILL.md new file mode 100644 index 0000000000..e18c95e152 --- /dev/null +++ b/.agents/skills/pre-pr-checklist/SKILL.md @@ -0,0 +1,89 @@ +--- +name: pre-pr-checklist +description: "Final checks before submitting a pull request. Use when preparing to create a PR, to ensure formatting, linting, and the correct tests have been run." +--- + +# Pre-PR Checklist + +Run through this checklist before creating a pull request. It ensures code quality gates pass and the right tests have been executed. + +## Step 1: Run cargo make fix + +**Always required.** This runs `rustfmt` and `clippy` with automatic fixes: + +```shell +cargo make fix +``` + +Address any remaining warnings or errors that couldn't be auto-fixed. + +## Step 2: Run the Right Tests + +Choose tests based on what you changed. **Do not run `cargo make test`** — it runs everything and takes a very long time. + +| What Changed | Test Command | +|---|---| +| Core logic, shared types, utilities | `cargo make unit-tests` | +| Worker executor functionality | `cargo make worker-executor-tests` | +| Service integration | `cargo make integration-tests` | +| CLI changes | `cargo make cli-tests` | +| HTTP API endpoints | `cargo make api-tests-http` | +| gRPC API endpoints | `cargo make api-tests-grpc` | +| Rust SDK (`sdks/rust/`) | `cargo test -p golem-rust` + `cargo make worker-executor-tests` | +| TypeScript SDK (`sdks/ts/`) | `npx pnpm run test` (in `sdks/ts/`) + `cargo make cli-tests` | + +If your change spans multiple areas, run multiple test suites. + +### Worker executor test groups + +For faster iteration, worker executor tests can be run by group: + +```shell +cargo make worker-executor-tests-group1 +cargo make worker-executor-tests-group2 +cargo make worker-executor-tests-misc +``` + +## Step 3: Regenerate Artifacts (if applicable) + +| What Changed | Regeneration Command | +|---|---| +| HTTP API endpoints | `cargo make generate-openapi` then `cargo clean -p golem-client && cargo build -p golem-client` | +| Service config structs/defaults | `cargo make generate-configs` | +| WIT interfaces | `cargo make wit` | +| TS SDK runtime code | `npx pnpm run build-agent-template` (in `sdks/ts/`) | + +## Step 4: Verify Build + +```shell +cargo make build +``` + +## Step 5: Review Staged Files + +Only stage files directly related to your change: + +```shell +git diff --stat # Review unstaged changes +git add # Stage only relevant files +``` + +**Never use `git add -A` or `git add .`** — they may include unrelated changes from concurrent work. + +## Quick Reference + +Minimum steps for any PR: + +```shell +cargo make fix # Format + lint +cargo make build # Full build +# Run appropriate tests from the table above +``` + +## Checklist + +1. [ ] `cargo make fix` run — no remaining warnings +2. [ ] Correct test suite(s) run — all pass +3. [ ] Artifacts regenerated if applicable (OpenAPI, configs, WIT, agent template) +4. [ ] `cargo make build` succeeds +5. [ ] Only relevant files staged for commit diff --git a/.agents/skills/sdk-development/SKILL.md b/.agents/skills/sdk-development/SKILL.md new file mode 100644 index 0000000000..d64480f8ff --- /dev/null +++ b/.agents/skills/sdk-development/SKILL.md @@ -0,0 +1,158 @@ +--- +name: sdk-development +description: "Working on the Rust or TypeScript SDKs in sdks/. Use when modifying SDK code, adding SDK features, or testing SDK changes with the main Golem platform." +--- + +# SDK Development + +The SDKs in `sdks/` are **not part of the main build flow** (`cargo make build` does not build them). Each SDK has its own build system and conventions. + +## Rust SDK (`sdks/rust/`) + +### Crates + +- `golem-rust` — Runtime API wrappers (transactions, durability, agentic framework, value conversions) +- `golem-rust-macro` — Procedural macros (`#[derive(IntoValue)]`, `#[agent_definition]`, etc.) + +### Building + +```shell +cd sdks/rust +cargo build -p golem-rust +cargo build -p golem-rust-macro +``` + +### Testing + +Tests use `test-r`. Each test file must have `test_r::enable!();` at the top. + +```shell +cargo test -p golem-rust +cargo test -p golem-rust --features export_golem_agentic # Agent tests +``` + +### Testing with the main platform + +```shell +# From repository root +cargo make worker-executor-tests +``` + +### Testing with golem-cli + +Set `GOLEM_RUST_PATH` to use local SDK in generated applications: + +```shell +export GOLEM_RUST_PATH=/path/to/golem/sdks/rust/golem-rust +golem-cli app new my-test-app +``` + +### Code style + +```shell +cargo fmt +cargo clippy +``` + +## TypeScript SDK (`sdks/ts/`) + +### Prerequisites + +- Node.js +- pnpm (managed via `packageManager` field) +- `wasm-rquickjs-cli`: `cargo install wasm-rquickjs-cli --version ` (check `WASM_RQUICKJS_VERSION` in `.github/workflows/ci.yaml`) +- `cargo-component` v0.21.1 (exact version required for agent template builds) + +### Packages + +Build order matters: `golem-ts-types-core` → `golem-ts-typegen` → `golem-ts-sdk` + +### Building + +```shell +cd sdks/ts +npx pnpm install +npx pnpm run build +``` + +### Testing + +```shell +npx pnpm run test +cd packages/golem-ts-sdk && pnpm run test # Specific package +``` + +### Agent template WASM + +The agent template WASM embeds the SDK runtime. You **must** rebuild it when: + +- `wasm-rquickjs-cli` is updated +- WIT dependencies change +- SDK runtime code changes (`baseAgent.ts`, `index.ts`, `resolvedAgent.ts`) + +```shell +cargo install cargo-component --version 0.21.1 +npx pnpm run build-agent-template +``` + +Running `pnpm run build` alone is **not sufficient** — it only updates the JS bundle, not the pre-compiled WASM that TS components use. + +### Testing with the main platform + +```shell +# From repository root +cargo make cli-tests +``` + +### Testing with golem-cli + +```shell +export GOLEM_TS_PACKAGES_PATH=/path/to/golem/sdks/ts/packages +npx pnpm install && npx pnpm run build # Build first! +golem-cli app new my-test-app +``` + +### Code style + +```shell +npx pnpm run lint +npx pnpm run format +``` + +## Downstream Rebuild Requirements + +SDK changes can require rebuilding test components. This is the most common source of errors. + +### Rust SDK change → test components + +1. Build `golem-rust` / `golem-rust-macro` +2. Find Rust test components depending on the SDK: check `test-components/*/Cargo.toml` for `golem-rust` references +3. Rebuild each affected component following its `AGENTS.md` + +### TS SDK change → test components + +1. Build TS SDK packages (`npx pnpm run build` in `sdks/ts/`) +2. Rebuild agent template WASM (`npx pnpm run build-agent-template` in `sdks/ts/`) +3. Find TS test components depending on the SDK +4. Rebuild each affected component following its `AGENTS.md` + +**The agent template rebuild step is critical and easily forgotten.** + +## WIT Dependencies + +Both SDKs have WIT files synced from the root `wit/` directory. **Never manually edit** `wit/deps/` in either SDK. + +```shell +# From repository root +cargo make wit +``` + +## Checklist + +1. SDK code modified +2. SDK builds successfully +3. SDK tests pass +4. Agent template rebuilt (if TS SDK runtime code changed) +5. Dependent test components rebuilt (if any) +6. Platform tests pass (`cargo make worker-executor-tests` for Rust SDK, `cargo make cli-tests` for TS SDK) +7. Code formatted and linted diff --git a/.github/workflows/ci.yaml b/.github/workflows/ci.yaml index 2b1f9ae7bc..b5a72e10e1 100644 --- a/.github/workflows/ci.yaml +++ b/.github/workflows/ci.yaml @@ -116,9 +116,10 @@ jobs: - name: Setup Rust run: rustup update stable --no-self-update && rustup default stable - uses: davidB/rust-cargo-make@v1 + - uses: cargo-bins/cargo-binstall@main - name: Install cargo-test-r - run: cargo install -f --locked --git https://github.com/vigoo/test-r --branch cargo-test-r cargo-test-r # TODO: use cargo-binstall + run: cargo binstall --force --locked cargo-test-r@2.2.2 # - name: Restore binary archive # uses: actions/download-artifact@v4 # with: @@ -165,6 +166,7 @@ jobs: # pull-request-report: true flaky-report: true collapse-large-reports: true + exit-on-no-files: true env: GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} @@ -189,6 +191,7 @@ jobs: - name: Setup Rust run: rustup update stable --no-self-update && rustup default stable - uses: davidB/rust-cargo-make@v1 + - uses: cargo-bins/cargo-binstall@main - name: Setup Redis uses: shogo82148/actions-setup-redis@v1.35.1 with: @@ -196,7 +199,7 @@ jobs: auto-start: false - name: Install cargo-test-r - run: cargo install -f --locked --git https://github.com/vigoo/test-r --branch cargo-test-r cargo-test-r # TODO: use cargo-binstall + run: cargo binstall --force --locked cargo-test-r@2.2.2 # - name: Restore binary archive # uses: actions/download-artifact@v4 # with: @@ -235,6 +238,7 @@ jobs: # pull-request-report: true flaky-report: true collapse-large-reports: true + exit-on-no-files: true env: GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} @@ -255,8 +259,6 @@ jobs: description: "IT #3" - name: integration-tests-group4 description: "IT #4" - - name: api-tests - description: "API tests" - name: cli-integration-tests-group1 description: "CLI tests #1" - name: cli-integration-tests-group2 @@ -281,7 +283,6 @@ jobs: auto-start: false - uses: cargo-bins/cargo-binstall@main - if: startsWith(matrix.group.name, 'cli-integration-tests') - name: Install cargo-component if: startsWith(matrix.group.name, 'cli-integration-tests') run: cargo binstall --force cargo-component@0.21.1 @@ -301,7 +302,7 @@ jobs: run: cargo install wasm-rquickjs-cli --version $WASM_RQUICKJS_VERSION - name: Install cargo-test-r - run: cargo install -f --locked --git https://github.com/vigoo/test-r --branch cargo-test-r cargo-test-r # TODO: use cargo-binstall + run: cargo binstall --force --locked cargo-test-r@2.2.2 # - name: Restore binary archive # uses: actions/download-artifact@v4 # with: @@ -346,6 +347,7 @@ jobs: # pull-request-report: true flaky-report: true collapse-large-reports: true + exit-on-no-files: true env: GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} diff --git a/AGENTS.md b/AGENTS.md index dddb0a0e03..374a8563b8 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -22,7 +22,7 @@ cargo build -p # Build specific crate Always run `cargo make build` before starting work to ensure all dependencies are compiled. -**Note:** The SDKs in `sdks/` are not part of the main build flow. When working on SDKs, follow the specific instructions in `sdks/rust/AGENTS.md` or `sdks/ts/AGENTS.md`. +**Note:** The SDKs in `sdks/` are not part of the main build flow. Load the `sdk-development` skill when working on SDKs. ## Testing @@ -37,8 +37,6 @@ fn my_test() { } ``` -Choose the appropriate test command based on what you're changing: - **Do not run `cargo make test`** - it runs all tests and takes a very long time. Instead, choose the appropriate test command: | Change Type | Test Command | @@ -47,8 +45,6 @@ Choose the appropriate test command based on what you're changing: | Worker executor functionality | `cargo make worker-executor-tests` | | Service integration | `cargo make integration-tests` | | CLI changes | `cargo make cli-tests` | -| API changes (HTTP) | `cargo make api-tests-http` | -| API changes (gRPC) | `cargo make api-tests-grpc` | **Whenever tests are modified, always run the affected tests to verify they still pass before considering the task complete.** @@ -57,39 +53,33 @@ For specific tests during development: cargo test -p -- --report-time ``` -Worker executor tests are grouped for parallel CI execution: -```shell -cargo make worker-executor-tests-group1 # Run specific group -``` - ## Test Components Worker executor tests and integration tests use pre-compiled WASM files from the `test-components/` directory. These are checked into the repository and **rebuilding them is not automated**. Do not attempt to rebuild test components - use the existing compiled WASM files, EXCEPT if the test component itself has an AGENTS.md file with instructions of how to do so. -**Important:** When modifying SDK code (`sdks/rust/` or `sdks/ts/`), you must rebuild any test components that depend on the changed SDK. For TS SDK changes, you must first rebuild the agent template wasm (`npx pnpm run build-agent-template` in `sdks/ts/`) before rebuilding TS test components. See each test component's `AGENTS.md` for build instructions. +Load the `modifying-test-components` skill when rebuilding is needed. ## Running Locally -Build and run the all-in-one `golem` binary from `cli/golem`: - ```shell cargo build -p golem # Build the golem binary ./target/debug/golem # Run locally ``` -Or build everything together with `cargo make build` and run the same binary. - -## Code Generation +## Skills -When modifying REST API endpoints: -```shell -cargo make generate-openapi # Regenerate OpenAPI specs -``` +Load these skills for guided workflows on complex tasks: -When modifying service configuration types: -```shell -cargo make generate-configs # Regenerate config files -``` +| Skill | When to Use | +|-------|-------------| +| `modifying-http-endpoints` | Adding or modifying REST API endpoints (covers OpenAPI regeneration, golem-client rebuild, type mappings) | +| `adding-dependencies` | Adding or updating crate dependencies (covers workspace dependency management, versioning, features) | +| `debugging-hanging-tests` | Diagnosing worker executor or integration tests that hang indefinitely | +| `modifying-test-components` | Building or modifying test WASM components, or rebuilding after SDK changes | +| `modifying-wit-interfaces` | Adding or modifying WIT interfaces and synchronizing across sub-projects | +| `modifying-service-configs` | Changing service configuration structs, defaults, or adding new config fields | +| `sdk-development` | Working on the Rust or TypeScript SDKs in `sdks/` | +| `pre-pr-checklist` | Final checks before submitting a pull request | ## Before Submitting a PR @@ -98,7 +88,7 @@ cargo make generate-configs # Regenerate config files cargo make fix ``` -This runs `rustfmt` and `clippy` with automatic fixes. Address any remaining warnings or errors. +This runs `rustfmt` and `clippy` with automatic fixes. Load `pre-pr-checklist` skill for the full workflow. ## Code Style @@ -107,13 +97,9 @@ This runs `rustfmt` and `clippy` with automatic fixes. Address any remaining war - Use existing libraries and utilities from the codebase - Security: Never expose or log secrets/keys -## WIT Dependencies +## Dependency Management -When working with WIT interfaces: -```shell -cargo make wit # Fetch WIT dependencies -cargo make check-wit # Verify WIT dependencies are up-to-date -``` +All crate dependencies must have their versions specified in the root workspace `Cargo.toml` under `[workspace.dependencies]`. Workspace members must reference them using `x = { workspace = true }` in their own `Cargo.toml` rather than specifying versions directly. ## Debugging Tests @@ -122,17 +108,14 @@ Use `--nocapture` when debugging tests to allow debugger attachment: cargo test -p -- --nocapture ``` -**Handling hanging tests:** Worker executor and integration tests can hang indefinitely (e.g., due to `unimplemented!()` panics in async tasks, deadlocks, or missing shard assignments). To debug a hanging test: - -1. Add a `#[timeout("30s")]` attribute (from `test_r::timeout`) so the test fails instead of hanging forever -2. Run with `--nocapture` to capture all log output -3. Save the **full output** to a file for analysis (the relevant error may appear far before the hang point) - +**Always save test output to a file** when running worker executor tests, integration tests, or CLI tests. These tests are slow and produce potentially thousands of lines of logs. Never pipe output directly to `grep`, `head`, `tail`, etc. — if you need to examine different parts of the output, you would have to re-run the entire slow test. Instead: ```shell -cargo test -p -- --nocapture > tmp/test_output.txt 2>&1 +cargo test -p -- --nocapture > tmp/test_output.txt 2>&1 +# Then search/inspect the saved file as needed +grep -n "pattern" tmp/test_output.txt ``` -Then search the saved output for `ERROR`, `panic`, or `unimplemented` to find the root cause. +**Handling hanging tests:** Load the `debugging-hanging-tests` skill for a step-by-step workflow. ## Project Structure diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 2a8a00ff95..fd9182a0ed 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -103,15 +103,6 @@ To run all CLI tests use ```shell cargo make cli-tests ``` - -#### Running API tests -The **API tests** test the REST and gRPC APIs separately: - -```shell -cargo make api-tests-http # HTTP API tests -cargo make api-tests-grpc # gRPC API tests -``` - #### Running sharding tests For sharding-related tests with file logging: diff --git a/Cargo.lock b/Cargo.lock index 39a5bb690e..9287803516 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1085,6 +1085,17 @@ dependencies = [ "backtrace", ] +[[package]] +name = "backtrace-on-stack-overflow" +version = "0.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7fd2d70527f3737a1ad17355e260706c1badebabd1fa06a7a053407380df841b" +dependencies = [ + "backtrace", + "libc", + "nix 0.23.2", +] + [[package]] name = "base16ct" version = "0.1.1" @@ -4299,6 +4310,7 @@ dependencies = [ "golem-wasm", "humantime-serde", "itertools 0.14.0", + "log", "prometheus", "redis", "reqwest", @@ -4317,6 +4329,7 @@ dependencies = [ "tokio-tungstenite 0.25.0", "tonic 0.14.2", "tracing", + "tracing-core", "tryhard", "url", "uuid", @@ -5350,6 +5363,7 @@ dependencies = [ "anyhow", "async-trait", "axum", + "backtrace-on-stack-overflow", "base64 0.22.1", "blake3", "bytes 1.11.0", @@ -5400,7 +5414,7 @@ version = "0.9.7" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "189d0897e4cbe8c75efedf3502c18c887b05046e59d28404d4d8e46cbc4d1e86" dependencies = [ - "memoffset", + "memoffset 0.9.1", ] [[package]] @@ -6126,6 +6140,15 @@ dependencies = [ "rustix 1.1.2", ] +[[package]] +name = "memoffset" +version = "0.6.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5aa361d4faea93603064a027415f07bd8e1d5c88c9fbf68bf56a285428fd79ce" +dependencies = [ + "autocfg", +] + [[package]] name = "memoffset" version = "0.9.1" @@ -6342,6 +6365,19 @@ dependencies = [ "smallvec", ] +[[package]] +name = "nix" +version = "0.23.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8f3790c00a0150112de0f4cd161e3d7fc4b2d8a5542ffc35f099a2562aecb35c" +dependencies = [ + "bitflags 1.3.2", + "cc", + "cfg-if", + "libc", + "memoffset 0.6.5", +] + [[package]] name = "nix" version = "0.29.0" @@ -6352,7 +6388,7 @@ dependencies = [ "cfg-if", "cfg_aliases", "libc", - "memoffset", + "memoffset 0.9.1", ] [[package]] @@ -8960,7 +8996,7 @@ checksum = "32a58fa8a7ccff2aec4f39cc45bf5f985cec7125ab271cf681c279fd00192b49" dependencies = [ "countme", "hashbrown 0.14.5", - "memoffset", + "memoffset 0.9.1", "rustc-hash 1.1.0", "text-size", ] diff --git a/Cargo.toml b/Cargo.toml index 72b046209d..fd33992d7e 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -70,6 +70,7 @@ aws-sdk-acm = "1" aws-sdk-elasticloadbalancingv2 = "1" aws-sdk-route53 = "1" aws-sdk-s3 = "1.65.0" +backtrace-on-stack-overflow = "0.3.0" axum = { version = "0.8", features = ["multipart"] } axum-jrpc = "0.9.0" base64 = "0.22.1" @@ -257,6 +258,7 @@ tonic-prost-build = "0.14.2" tonic-reflection = "0.14.2" tonic-tracing-opentelemetry = "0.30.0" tracing = { version = "0.1.41", features = ["log"] } +tracing-core = "0.1" tracing-futures = "0.2.5" tracing-opentelemetry = "0.31.0" tracing-serde = "0.2.0" diff --git a/Makefile.toml b/Makefile.toml index 608abc976b..303075a7b0 100644 --- a/Makefile.toml +++ b/Makefile.toml @@ -534,26 +534,6 @@ cargo-test-r run \ -- --nocapture --test-threads=1 ''' -[tasks.api-tests] -description = "Runs all API tests" -dependencies = ["api-tests-http", "api-tests-grpc"] - -[tasks.api-tests-http] -description = "Runs API HTTP tests only" -dependencies = ["build-bins-non-ci"] -env = { "RUST_LOG" = "info", "RUST_BACKTRACE" = "1", "GOLEM_CLIENT_PROTOCOL" = "http", "QUIET" = "true" } -script = ''' -cargo-test-r run --package integration-tests --test api -- --nocapture --report-time $JUNIT_OPTS -''' - -[tasks.api-tests-grpc] -description = "Runs API GRPC tests only" -dependencies = ["build-bins-non-ci"] -env = { "RUST_LOG" = "info", "RUST_BACKTRACE" = "1", "GOLEM_CLIENT_PROTOCOL" = "grpc", "QUIET" = "true" } -script = ''' -cargo-test-r run --package integration-tests --test api :tag: -- --nocapture --report-time $JUNIT_OPTS -''' - [tasks.registry-repo-coverage] description = "Run registry service tests with covarage report for repository sources" script = ''' diff --git a/cli/golem-cli/src/bridge_gen/rust/mod.rs b/cli/golem-cli/src/bridge_gen/rust/mod.rs index 135151a145..ee1d4ace75 100644 --- a/cli/golem-cli/src/bridge_gen/rust/mod.rs +++ b/cli/golem-cli/src/bridge_gen/rust/mod.rs @@ -288,6 +288,7 @@ impl RustBridgeGenerator { mode, schedule_at, idempotency_key: None, + deployment_revision: None, }, ) .await?; diff --git a/cli/golem-cli/src/command_handler/worker/mod.rs b/cli/golem-cli/src/command_handler/worker/mod.rs index 78f4ffb11e..57e3033f8c 100644 --- a/cli/golem-cli/src/command_handler/worker/mod.rs +++ b/cli/golem-cli/src/command_handler/worker/mod.rs @@ -460,6 +460,7 @@ impl WorkerCommandHandler { mode, schedule_at, idempotency_key: Some(idempotency_key.value.clone()), + deployment_revision: None, }; let clients = self.ctx.golem_clients().await?; diff --git a/cli/golem-cli/src/model/component.rs b/cli/golem-cli/src/model/component.rs index 73348db2fe..97c9c5b430 100644 --- a/cli/golem-cli/src/model/component.rs +++ b/cli/golem-cli/src/model/component.rs @@ -15,7 +15,6 @@ use crate::model::environment::ResolvedEnvironmentIdentity; use crate::model::wave::function_wave_compatible; use crate::model::worker::WorkerName; -use anyhow::{anyhow, bail}; use chrono::{DateTime, Utc}; use golem_common::model::agent::wit_naming::ToWitNaming; use golem_common::model::agent::{ @@ -26,7 +25,6 @@ use golem_common::model::component::{ }; use golem_common::model::component::{ComponentName, InitialComponentFile}; -use golem_common::model::component_metadata::{ParsedFunctionName, ParsedFunctionSite}; use golem_common::model::environment::EnvironmentId; use golem_common::model::trim_date::TrimDateTime; use golem_wasm::analysis::wave::DisplayNamedFunc; @@ -146,33 +144,14 @@ impl ComponentView { pub fn new(show_sensitive: bool, show_exports_for_rib: bool, value: ComponentDto) -> Self { let exports = { - if value.metadata.is_agent() { - if show_exports_for_rib { - let agent_types = value - .metadata - .agent_types() - .iter() - .map(|a| a.to_wit_naming()) - .collect::>(); - - show_exported_agents(&agent_types, true, true) - } else { - value - .metadata - .agent_types() - .iter() - .flat_map(|agent| { - show_exported_functions( - value.metadata.exports(), - true, - agent_interface_name(&value, &agent.wrapper_type_name()).as_deref(), - ) - }) - .collect() - } - } else { - show_exported_functions(value.metadata.exports(), true, None) - } + let agent_types = value + .metadata + .agent_types() + .iter() + .map(|a| a.to_wit_naming()) + .collect::>(); + + show_exported_agents(&agent_types, true, true) }; ComponentView { @@ -501,64 +480,6 @@ pub fn format_function_name(prefix: Option<&str>, name: &str) -> String { } } -fn resolve_function<'t>( - component: &'t ComponentDto, - function: &str, -) -> anyhow::Result<(&'t AnalysedFunction, ParsedFunctionName)> { - let parsed = ParsedFunctionName::parse(function).map_err(|err| anyhow!(err))?; - let mut functions = Vec::new(); - - for export in component.metadata.exports() { - match export { - AnalysedExport::Instance(interface) => { - if matches!(parsed.site().interface_name(), Some(name) if name == interface.name) { - for function in &interface.functions { - if parsed.function().function_name() == function.name { - functions.push(function); - } - } - } - } - AnalysedExport::Function(ref f @ AnalysedFunction { name, .. }) => { - if parsed.site() == &ParsedFunctionSite::Global - && &parsed.function().function_name() == name - { - functions.push(f); - } - } - } - } - - if functions.len() > 1 { - bail!( - "Multiple function results with the same name ({}) declared", - function - ) - } else if let Some(func) = functions.first() { - Ok((func, parsed)) - } else { - bail!("Can't find function ({}) in component", function) - } -} - -pub fn function_result_types<'t>( - component: &'t ComponentDto, - function: &str, -) -> anyhow::Result> { - let (func, _) = resolve_function(component, function)?; - - Ok(func.result.iter().map(|r| &r.typ).collect()) -} - -pub fn function_params_types<'t>( - component: &'t ComponentDto, - function: &str, -) -> anyhow::Result> { - let (func, _parsed) = resolve_function(component, function)?; - - Ok(func.parameters.iter().map(|r| &r.typ).collect()) -} - pub fn agent_interface_name(component: &ComponentDto, agent_type_name: &str) -> Option { match ( component.metadata.root_package_name(), diff --git a/golem-api-grpc/proto/golem/registry/v1/registry_service.proto b/golem-api-grpc/proto/golem/registry/v1/registry_service.proto index 9df93e5f08..e75a33390f 100644 --- a/golem-api-grpc/proto/golem/registry/v1/registry_service.proto +++ b/golem-api-grpc/proto/golem/registry/v1/registry_service.proto @@ -42,6 +42,7 @@ service RegistryService { rpc GetAllAgentTypes (GetAllAgentTypesRequest) returns (GetAllAgentTypesResponse); rpc GetAgentType (GetAgentTypeRequest) returns (GetAgentTypeResponse); rpc ResolveLatestAgentTypeByNames (ResolveLatestAgentTypeByNamesRequest) returns (ResolveLatestAgentTypeByNamesResponse); + rpc ResolveAgentTypeAtDeployment (ResolveAgentTypeAtDeploymentRequest) returns (ResolveAgentTypeAtDeploymentResponse); // current deployment agents/routes rpc GetActiveRoutesForDomain (GetActiveRoutesForDomainRequest) returns (GetActiveRoutesForDomainResponse); @@ -265,6 +266,25 @@ message ResolveLatestAgentTypeByNamesSuccessResponse { RegisteredAgentType agent_type = 1; } +message ResolveAgentTypeAtDeploymentRequest { + golem.common.AccountId accountId = 1; + string app_name = 2; + string environment_name = 3; + string agent_type_name = 4; + uint64 deployment_revision = 5; +} + +message ResolveAgentTypeAtDeploymentResponse { + oneof result { + ResolveAgentTypeAtDeploymentSuccessResponse success = 1; + RegistryServiceError error = 2; + } +} + +message ResolveAgentTypeAtDeploymentSuccessResponse { + RegisteredAgentType agent_type = 1; +} + message GetActiveRoutesForDomainRequest { string domain = 1; } diff --git a/golem-common/src/model/component_metadata.rs b/golem-common/src/model/component_metadata.rs index 4afca1b6ed..34acf6cb27 100644 --- a/golem-common/src/model/component_metadata.rs +++ b/golem-common/src/model/component_metadata.rs @@ -15,7 +15,7 @@ use crate::model::agent::wit_naming::ToWitNaming; use crate::model::agent::{AgentType, AgentTypeName}; use crate::model::base64::Base64; -use crate::{virtual_exports, SafeDisplay}; +use crate::SafeDisplay; use golem_wasm::analysis::wit_parser::WitAnalysisContext; use golem_wasm::analysis::{AnalysedExport, AnalysedFunction, AnalysisFailure}; use golem_wasm::analysis::{ @@ -74,10 +74,6 @@ impl ComponentMetadata { } } - pub fn exports(&self) -> &[AnalysedExport] { - &self.data.exports - } - pub fn producers(&self) -> &[Producers] { &self.data.producers } @@ -753,7 +749,6 @@ impl RawComponentMetadata { } add_resource_drops(&mut exports); - add_virtual_exports(&mut exports); let memories = wit_analysis .linear_memories() @@ -991,17 +986,6 @@ fn drop_from_constructor_or_method(fun: &AnalysedFunction) -> AnalysedFunction { } } -fn add_virtual_exports(exports: &mut Vec) { - // Some interfaces like the golem/http:incoming-handler do not exist on the component - // but are dynamically created by the worker executor based on other existing interfaces. - - if virtual_exports::http_incoming_handler::implements_required_interfaces(exports) { - exports.extend(vec![ - virtual_exports::http_incoming_handler::ANALYZED_EXPORT.clone(), - ]); - }; -} - mod protobuf { use crate::model::base64::Base64; use crate::model::component_metadata::{ diff --git a/golem-common/src/model/oplog/matcher.rs b/golem-common/src/model/oplog/matcher.rs index 4be08d1f44..374a30c9bb 100644 --- a/golem-common/src/model/oplog/matcher.rs +++ b/golem-common/src/model/oplog/matcher.rs @@ -108,6 +108,7 @@ impl PublicOplogEntry { } PublicOplogEntry::AgentInvocationStarted(params) => { Self::string_match("agentinvocationstarted", &[], query_path, query) + || Self::string_match("invoke", &[], query_path, query) || Self::string_match("agent-invocation-started", &[], query_path, query) || match ¶ms.invocation { PublicAgentInvocation::AgentInitialization(inv_params) => { @@ -159,6 +160,7 @@ impl PublicOplogEntry { } PublicOplogEntry::AgentInvocationFinished(params) => { Self::string_match("agentinvocationfinished", &[], query_path, query) + || Self::string_match("invoke", &[], query_path, query) || Self::string_match("agent-invocation-finished", &[], query_path, query) || Self::string_match(¶ms.consumed_fuel.to_string(), &[], query_path, query) } @@ -198,8 +200,9 @@ impl PublicOplogEntry { || Self::string_match("end-remote-write", &[], query_path, query) } PublicOplogEntry::PendingAgentInvocation(params) => { - Self::string_match("pendingworkerinvocation", &[], query_path, query) - || Self::string_match("pending-worker-invocation", &[], query_path, query) + Self::string_match("pendingagentinvocation", &[], query_path, query) + || Self::string_match("invoke", &[], query_path, query) + || Self::string_match("pending-agent-invocation", &[], query_path, query) || match ¶ms.invocation { PublicAgentInvocation::AgentInitialization(params) => { Self::string_match("agent-initialization", &[], query_path, query) diff --git a/golem-common/src/tracing.rs b/golem-common/src/tracing.rs index 093a7bcd96..a804980233 100644 --- a/golem-common/src/tracing.rs +++ b/golem-common/src/tracing.rs @@ -52,6 +52,7 @@ pub struct OutputConfig { pub json: bool, pub json_flatten: bool, pub json_flatten_span: bool, + pub json_source_location: bool, pub ansi: bool, pub compact: bool, pub pretty: bool, @@ -71,6 +72,7 @@ impl OutputConfig { json: false, json_flatten: true, json_flatten_span: true, + json_source_location: false, ansi: false, compact: false, pretty: false, @@ -86,6 +88,7 @@ impl OutputConfig { json: false, json_flatten: true, json_flatten_span: true, + json_source_location: false, ansi: true, compact: false, pretty: false, @@ -101,6 +104,7 @@ impl OutputConfig { json: true, json_flatten: false, json_flatten_span: false, + json_source_location: false, ansi: false, compact: false, pretty: false, @@ -116,6 +120,7 @@ impl OutputConfig { json: true, json_flatten: true, json_flatten_span: false, + json_source_location: false, ansi: false, compact: false, pretty: false, @@ -131,6 +136,7 @@ impl OutputConfig { json: true, json_flatten: true, json_flatten_span: true, + json_source_location: false, ansi: false, compact: false, pretty: false, @@ -160,6 +166,9 @@ impl SafeDisplay for OutputConfig { if self.json_flatten_span { flags.push("json_flatten_span"); } + if self.json_source_location { + flags.push("json_source_location"); + } if self.pretty { flags.push("pretty"); } @@ -401,6 +410,7 @@ pub mod directive { warn("wasmtime_environ"), warn("wit_parser"), warn("golem_client"), + warn("bollard"), ] } } @@ -623,6 +633,8 @@ where tracing_subscriber::fmt::layer() .json() .flatten_event(config.json_flatten) + .with_file(config.json_source_location) + .with_line_number(config.json_source_location) .with_span_events(span_events) .with_writer(writer) .with_filter(filter) diff --git a/golem-component-compilation-service/config/component-compilation-service.sample.env b/golem-component-compilation-service/config/component-compilation-service.sample.env index d0f72b3849..7a8497d96b 100644 --- a/golem-component-compilation-service/config/component-compilation-service.sample.env +++ b/golem-component-compilation-service/config/component-compilation-service.sample.env @@ -29,6 +29,7 @@ GOLEM__TRACING__FILE__ENABLED=false GOLEM__TRACING__FILE__JSON=true GOLEM__TRACING__FILE__JSON_FLATTEN=true GOLEM__TRACING__FILE__JSON_FLATTEN_SPAN=true +GOLEM__TRACING__FILE__JSON_SOURCE_LOCATION=false GOLEM__TRACING__FILE__PRETTY=false GOLEM__TRACING__FILE__SPAN_EVENTS_ACTIVE=false GOLEM__TRACING__FILE__SPAN_EVENTS_FULL=false @@ -43,6 +44,7 @@ GOLEM__TRACING__STDERR__ENABLED=false GOLEM__TRACING__STDERR__JSON=false GOLEM__TRACING__STDERR__JSON_FLATTEN=false GOLEM__TRACING__STDERR__JSON_FLATTEN_SPAN=false +GOLEM__TRACING__STDERR__JSON_SOURCE_LOCATION=false GOLEM__TRACING__STDERR__PRETTY=false GOLEM__TRACING__STDERR__SPAN_EVENTS_ACTIVE=false GOLEM__TRACING__STDERR__SPAN_EVENTS_FULL=false @@ -53,6 +55,7 @@ GOLEM__TRACING__STDOUT__ENABLED=true GOLEM__TRACING__STDOUT__JSON=false GOLEM__TRACING__STDOUT__JSON_FLATTEN=true GOLEM__TRACING__STDOUT__JSON_FLATTEN_SPAN=true +GOLEM__TRACING__STDOUT__JSON_SOURCE_LOCATION=false GOLEM__TRACING__STDOUT__PRETTY=false GOLEM__TRACING__STDOUT__SPAN_EVENTS_ACTIVE=false GOLEM__TRACING__STDOUT__SPAN_EVENTS_FULL=false @@ -104,6 +107,7 @@ GOLEM__TRACING__FILE__ENABLED=false GOLEM__TRACING__FILE__JSON=true GOLEM__TRACING__FILE__JSON_FLATTEN=true GOLEM__TRACING__FILE__JSON_FLATTEN_SPAN=true +GOLEM__TRACING__FILE__JSON_SOURCE_LOCATION=false GOLEM__TRACING__FILE__PRETTY=false GOLEM__TRACING__FILE__SPAN_EVENTS_ACTIVE=false GOLEM__TRACING__FILE__SPAN_EVENTS_FULL=false @@ -118,6 +122,7 @@ GOLEM__TRACING__STDERR__ENABLED=false GOLEM__TRACING__STDERR__JSON=false GOLEM__TRACING__STDERR__JSON_FLATTEN=false GOLEM__TRACING__STDERR__JSON_FLATTEN_SPAN=false +GOLEM__TRACING__STDERR__JSON_SOURCE_LOCATION=false GOLEM__TRACING__STDERR__PRETTY=false GOLEM__TRACING__STDERR__SPAN_EVENTS_ACTIVE=false GOLEM__TRACING__STDERR__SPAN_EVENTS_FULL=false @@ -128,6 +133,7 @@ GOLEM__TRACING__STDOUT__ENABLED=true GOLEM__TRACING__STDOUT__JSON=false GOLEM__TRACING__STDOUT__JSON_FLATTEN=true GOLEM__TRACING__STDOUT__JSON_FLATTEN_SPAN=true +GOLEM__TRACING__STDOUT__JSON_SOURCE_LOCATION=false GOLEM__TRACING__STDOUT__PRETTY=false GOLEM__TRACING__STDOUT__SPAN_EVENTS_ACTIVE=false GOLEM__TRACING__STDOUT__SPAN_EVENTS_FULL=false diff --git a/golem-component-compilation-service/config/component-compilation-service.toml b/golem-component-compilation-service/config/component-compilation-service.toml index f1f01e0230..c71b7799eb 100644 --- a/golem-component-compilation-service/config/component-compilation-service.toml +++ b/golem-component-compilation-service/config/component-compilation-service.toml @@ -57,6 +57,7 @@ enabled = false json = true json_flatten = true json_flatten_span = true +json_source_location = false pretty = false span_events_active = false span_events_full = false @@ -75,6 +76,7 @@ enabled = false json = false json_flatten = false json_flatten_span = false +json_source_location = false pretty = false span_events_active = false span_events_full = false @@ -87,6 +89,7 @@ enabled = true json = false json_flatten = true json_flatten_span = true +json_source_location = false pretty = false span_events_active = false span_events_full = false @@ -166,6 +169,7 @@ without_time = false # json = true # json_flatten = true # json_flatten_span = true +# json_source_location = false # pretty = false # span_events_active = false # span_events_full = false @@ -184,6 +188,7 @@ without_time = false # json = false # json_flatten = false # json_flatten_span = false +# json_source_location = false # pretty = false # span_events_active = false # span_events_full = false @@ -196,6 +201,7 @@ without_time = false # json = false # json_flatten = true # json_flatten_span = true +# json_source_location = false # pretty = false # span_events_active = false # span_events_full = false diff --git a/golem-debugging-service/config/debug-worker-executor.sample.env b/golem-debugging-service/config/debug-worker-executor.sample.env index 7528d9800d..eb36f17cc5 100644 --- a/golem-debugging-service/config/debug-worker-executor.sample.env +++ b/golem-debugging-service/config/debug-worker-executor.sample.env @@ -107,6 +107,7 @@ GOLEM__TRACING__FILE__ENABLED=false GOLEM__TRACING__FILE__JSON=true GOLEM__TRACING__FILE__JSON_FLATTEN=true GOLEM__TRACING__FILE__JSON_FLATTEN_SPAN=true +GOLEM__TRACING__FILE__JSON_SOURCE_LOCATION=false GOLEM__TRACING__FILE__PRETTY=false GOLEM__TRACING__FILE__SPAN_EVENTS_ACTIVE=false GOLEM__TRACING__FILE__SPAN_EVENTS_FULL=false @@ -121,6 +122,7 @@ GOLEM__TRACING__STDERR__ENABLED=false GOLEM__TRACING__STDERR__JSON=false GOLEM__TRACING__STDERR__JSON_FLATTEN=false GOLEM__TRACING__STDERR__JSON_FLATTEN_SPAN=false +GOLEM__TRACING__STDERR__JSON_SOURCE_LOCATION=false GOLEM__TRACING__STDERR__PRETTY=false GOLEM__TRACING__STDERR__SPAN_EVENTS_ACTIVE=false GOLEM__TRACING__STDERR__SPAN_EVENTS_FULL=false @@ -131,6 +133,7 @@ GOLEM__TRACING__STDOUT__ENABLED=true GOLEM__TRACING__STDOUT__JSON=false GOLEM__TRACING__STDOUT__JSON_FLATTEN=true GOLEM__TRACING__STDOUT__JSON_FLATTEN_SPAN=true +GOLEM__TRACING__STDOUT__JSON_SOURCE_LOCATION=false GOLEM__TRACING__STDOUT__PRETTY=false GOLEM__TRACING__STDOUT__SPAN_EVENTS_ACTIVE=false GOLEM__TRACING__STDOUT__SPAN_EVENTS_FULL=false @@ -245,6 +248,7 @@ GOLEM__TRACING__FILE__ENABLED=false GOLEM__TRACING__FILE__JSON=true GOLEM__TRACING__FILE__JSON_FLATTEN=true GOLEM__TRACING__FILE__JSON_FLATTEN_SPAN=true +GOLEM__TRACING__FILE__JSON_SOURCE_LOCATION=false GOLEM__TRACING__FILE__PRETTY=false GOLEM__TRACING__FILE__SPAN_EVENTS_ACTIVE=false GOLEM__TRACING__FILE__SPAN_EVENTS_FULL=false @@ -259,6 +263,7 @@ GOLEM__TRACING__STDERR__ENABLED=false GOLEM__TRACING__STDERR__JSON=false GOLEM__TRACING__STDERR__JSON_FLATTEN=false GOLEM__TRACING__STDERR__JSON_FLATTEN_SPAN=false +GOLEM__TRACING__STDERR__JSON_SOURCE_LOCATION=false GOLEM__TRACING__STDERR__PRETTY=false GOLEM__TRACING__STDERR__SPAN_EVENTS_ACTIVE=false GOLEM__TRACING__STDERR__SPAN_EVENTS_FULL=false @@ -269,6 +274,7 @@ GOLEM__TRACING__STDOUT__ENABLED=true GOLEM__TRACING__STDOUT__JSON=false GOLEM__TRACING__STDOUT__JSON_FLATTEN=true GOLEM__TRACING__STDOUT__JSON_FLATTEN_SPAN=true +GOLEM__TRACING__STDOUT__JSON_SOURCE_LOCATION=false GOLEM__TRACING__STDOUT__PRETTY=false GOLEM__TRACING__STDOUT__SPAN_EVENTS_ACTIVE=false GOLEM__TRACING__STDOUT__SPAN_EVENTS_FULL=false diff --git a/golem-debugging-service/config/debug-worker-executor.toml b/golem-debugging-service/config/debug-worker-executor.toml index bc7fd89286..87aa542a42 100644 --- a/golem-debugging-service/config/debug-worker-executor.toml +++ b/golem-debugging-service/config/debug-worker-executor.toml @@ -179,6 +179,7 @@ enabled = false json = true json_flatten = true json_flatten_span = true +json_source_location = false pretty = false span_events_active = false span_events_full = false @@ -197,6 +198,7 @@ enabled = false json = false json_flatten = false json_flatten_span = false +json_source_location = false pretty = false span_events_active = false span_events_full = false @@ -209,6 +211,7 @@ enabled = true json = false json_flatten = true json_flatten_span = true +json_source_location = false pretty = false span_events_active = false span_events_full = false @@ -396,6 +399,7 @@ without_time = false # json = true # json_flatten = true # json_flatten_span = true +# json_source_location = false # pretty = false # span_events_active = false # span_events_full = false @@ -414,6 +418,7 @@ without_time = false # json = false # json_flatten = false # json_flatten_span = false +# json_source_location = false # pretty = false # span_events_active = false # span_events_full = false @@ -426,6 +431,7 @@ without_time = false # json = false # json_flatten = true # json_flatten_span = true +# json_source_location = false # pretty = false # span_events_active = false # span_events_full = false diff --git a/golem-registry-service/config/registry-service.sample.env b/golem-registry-service/config/registry-service.sample.env index 7ac668fd8c..72c31d0304 100644 --- a/golem-registry-service/config/registry-service.sample.env +++ b/golem-registry-service/config/registry-service.sample.env @@ -64,6 +64,7 @@ GOLEM__TRACING__FILE__ENABLED=false GOLEM__TRACING__FILE__JSON=true GOLEM__TRACING__FILE__JSON_FLATTEN=true GOLEM__TRACING__FILE__JSON_FLATTEN_SPAN=true +GOLEM__TRACING__FILE__JSON_SOURCE_LOCATION=false GOLEM__TRACING__FILE__PRETTY=false GOLEM__TRACING__FILE__SPAN_EVENTS_ACTIVE=false GOLEM__TRACING__FILE__SPAN_EVENTS_FULL=false @@ -78,6 +79,7 @@ GOLEM__TRACING__STDERR__ENABLED=false GOLEM__TRACING__STDERR__JSON=false GOLEM__TRACING__STDERR__JSON_FLATTEN=false GOLEM__TRACING__STDERR__JSON_FLATTEN_SPAN=false +GOLEM__TRACING__STDERR__JSON_SOURCE_LOCATION=false GOLEM__TRACING__STDERR__PRETTY=false GOLEM__TRACING__STDERR__SPAN_EVENTS_ACTIVE=false GOLEM__TRACING__STDERR__SPAN_EVENTS_FULL=false @@ -88,6 +90,7 @@ GOLEM__TRACING__STDOUT__ENABLED=true GOLEM__TRACING__STDOUT__JSON=false GOLEM__TRACING__STDOUT__JSON_FLATTEN=true GOLEM__TRACING__STDOUT__JSON_FLATTEN_SPAN=true +GOLEM__TRACING__STDOUT__JSON_SOURCE_LOCATION=false GOLEM__TRACING__STDOUT__PRETTY=false GOLEM__TRACING__STDOUT__SPAN_EVENTS_ACTIVE=false GOLEM__TRACING__STDOUT__SPAN_EVENTS_FULL=false diff --git a/golem-registry-service/config/registry-service.toml b/golem-registry-service/config/registry-service.toml index a89ae9a246..ee13d4425b 100644 --- a/golem-registry-service/config/registry-service.toml +++ b/golem-registry-service/config/registry-service.toml @@ -106,6 +106,7 @@ enabled = false json = true json_flatten = true json_flatten_span = true +json_source_location = false pretty = false span_events_active = false span_events_full = false @@ -124,6 +125,7 @@ enabled = false json = false json_flatten = false json_flatten_span = false +json_source_location = false pretty = false span_events_active = false span_events_full = false @@ -136,6 +138,7 @@ enabled = true json = false json_flatten = true json_flatten_span = true +json_source_location = false pretty = false span_events_active = false span_events_full = false diff --git a/golem-registry-service/src/grpc/api_impl.rs b/golem-registry-service/src/grpc/api_impl.rs index 835b133591..2a4ffc4d2e 100644 --- a/golem-registry-service/src/grpc/api_impl.rs +++ b/golem-registry-service/src/grpc/api_impl.rs @@ -40,23 +40,27 @@ use golem_api_grpc::proto::golem::registry::v1::{ GetDeployedComponentMetadataRequest, GetDeployedComponentMetadataResponse, GetDeployedComponentMetadataSuccessResponse, GetResourceLimitsRequest, GetResourceLimitsResponse, GetResourceLimitsSuccessResponse, RegistryServiceError, - ResolveComponentRequest, ResolveComponentResponse, ResolveComponentSuccessResponse, - ResolveLatestAgentTypeByNamesRequest, ResolveLatestAgentTypeByNamesResponse, - ResolveLatestAgentTypeByNamesSuccessResponse, UpdateWorkerConnectionLimitRequest, - UpdateWorkerConnectionLimitResponse, UpdateWorkerLimitRequest, UpdateWorkerLimitResponse, - authenticate_token_response, batch_update_fuel_usage_response, download_component_response, + ResolveAgentTypeAtDeploymentRequest, ResolveAgentTypeAtDeploymentResponse, + ResolveAgentTypeAtDeploymentSuccessResponse, ResolveComponentRequest, ResolveComponentResponse, + ResolveComponentSuccessResponse, ResolveLatestAgentTypeByNamesRequest, + ResolveLatestAgentTypeByNamesResponse, ResolveLatestAgentTypeByNamesSuccessResponse, + UpdateWorkerConnectionLimitRequest, UpdateWorkerConnectionLimitResponse, + UpdateWorkerLimitRequest, UpdateWorkerLimitResponse, authenticate_token_response, + batch_update_fuel_usage_response, download_component_response, get_active_routes_for_domain_response, get_agent_deployments_response, get_agent_type_response, get_all_agent_types_response, get_all_deployed_component_revisions_response, get_auth_details_for_environment_response, get_component_metadata_response, get_deployed_component_metadata_response, get_resource_limits_response, registry_service_error, - resolve_component_response, resolve_latest_agent_type_by_names_response, - update_worker_connection_limit_response, update_worker_limit_response, + resolve_agent_type_at_deployment_response, resolve_component_response, + resolve_latest_agent_type_by_names_response, update_worker_connection_limit_response, + update_worker_limit_response, }; use golem_common::model::account::AccountId; use golem_common::model::agent::{AgentTypeName, RegisteredAgentType}; use golem_common::model::application::{ApplicationId, ApplicationName}; use golem_common::model::auth::TokenSecret; use golem_common::model::component::{ComponentDto, ComponentId, ComponentRevision}; +use golem_common::model::deployment::DeploymentRevision; use golem_common::model::domain_registration::Domain; use golem_common::model::environment::{EnvironmentId, EnvironmentName}; use golem_common::recorded_grpc_api_request; @@ -449,6 +453,37 @@ impl RegistryServiceGrpcApi { agent_type: Some(RegisteredAgentType::from(agent_type).into()), }) } + + async fn resolve_agent_type_at_deployment_internal( + &self, + request: ResolveAgentTypeAtDeploymentRequest, + ) -> Result { + let account_id = request + .account_id + .ok_or("missing account_id field")? + .try_into()?; + let app_name = ApplicationName(request.app_name); + let environment_name = EnvironmentName(request.environment_name); + let agent_type_name = AgentTypeName(request.agent_type_name); + let deployment_revision = + DeploymentRevision::try_from(request.deployment_revision).map_err(|e| e.to_string())?; + + let agent_type = self + .deployment_service + .get_agent_type_by_names_at_deployment( + account_id, + &app_name, + &environment_name, + &agent_type_name, + deployment_revision, + &AuthCtx::System, + ) + .await?; + + Ok(ResolveAgentTypeAtDeploymentSuccessResponse { + agent_type: Some(RegisteredAgentType::from(agent_type).into()), + }) + } } #[async_trait] @@ -826,6 +861,35 @@ impl golem_api_grpc::proto::golem::registry::v1::registry_service_server::Regist })) } + async fn resolve_agent_type_at_deployment( + &self, + request: Request, + ) -> Result, Status> { + let request = request.into_inner(); + let record = recorded_grpc_api_request!( + "resolve_agent_type_at_deployment", + account_id = proto_account_id_string(&request.account_id), + app_name = &request.app_name, + environment_name = &request.environment_name, + agent_type_name = &request.agent_type_name, + deployment_revision = request.deployment_revision, + ); + + let response = match self + .resolve_agent_type_at_deployment_internal(request) + .instrument(record.span.clone()) + .await + .apply(|r| record.result(r)) + { + Ok(result) => resolve_agent_type_at_deployment_response::Result::Success(result), + Err(error) => resolve_agent_type_at_deployment_response::Result::Error(error.into()), + }; + + Ok(Response::new(ResolveAgentTypeAtDeploymentResponse { + result: Some(response), + })) + } + async fn get_active_routes_for_domain( &self, request: Request, diff --git a/golem-registry-service/src/services/component/write.rs b/golem-registry-service/src/services/component/write.rs index 5de8001463..57e6d03a0c 100644 --- a/golem-registry-service/src/services/component/write.rs +++ b/golem-registry-service/src/services/component/write.rs @@ -686,7 +686,6 @@ impl ComponentWriteService { debug!( environment_id = %environment_id, - exports = ?finalized_revision.metadata.exports(), "Finalized component", ); diff --git a/golem-registry-service/src/services/deployment/read.rs b/golem-registry-service/src/services/deployment/read.rs index b3244dcc15..c3a9321032 100644 --- a/golem-registry-service/src/services/deployment/read.rs +++ b/golem-registry-service/src/services/deployment/read.rs @@ -389,4 +389,45 @@ impl DeploymentService { self.get_deployed_agent_type(environment.id, agent_type_name) .await } + + pub async fn get_agent_type_by_names_at_deployment( + &self, + account_id: AccountId, + app_name: &ApplicationName, + environment_name: &EnvironmentName, + agent_type_name: &AgentTypeName, + deployment_revision: DeploymentRevision, + auth: &AuthCtx, + ) -> Result { + let application = self + .application_service + .get_in_account(account_id, app_name, auth) + .await?; + + let environment = self + .environment_service + .get_in_application(application.id, environment_name, auth) + .await?; + + // Validate that the deployment revision exists in this environment + self.deployment_repo + .get_deployment_revision(environment.id.0, deployment_revision.into()) + .await? + .ok_or(DeploymentError::DeploymentNotFound(deployment_revision))?; + + let agent_type = self + .deployment_repo + .get_deployment_agent_type( + environment.id.0, + deployment_revision.into(), + &agent_type_name.0, + ) + .await? + .ok_or(DeploymentError::AgentTypeNotFound( + agent_type_name.0.clone(), + ))? + .try_into()?; + + Ok(agent_type) + } } diff --git a/golem-registry-service/src/services/plugin_registration.rs b/golem-registry-service/src/services/plugin_registration.rs index 535cddb346..12c15c4942 100644 --- a/golem-registry-service/src/services/plugin_registration.rs +++ b/golem-registry-service/src/services/plugin_registration.rs @@ -26,12 +26,8 @@ use golem_service_base::model::auth::AccountAction; use golem_service_base::model::auth::{AuthCtx, AuthorizationError}; use golem_service_base::model::plugin_registration::{PluginRegistration, PluginSpec}; use golem_service_base::repo::RepoError; -use golem_wasm::analysis::AnalysedExport; use std::sync::Arc; -const OPLOG_PROCESSOR_INTERFACE: &str = "golem:api/oplog-processor"; -const OPLOG_PROCESSOR_VERSION_PREFIX: &str = "1."; - #[derive(Debug, thiserror::Error)] pub enum PluginRegistrationError { #[error("Registered plugin not found for id {0}")] @@ -236,9 +232,10 @@ impl PluginRegistrationService { let implements_oplog_processor_interface = component .metadata - .exports() - .iter() - .any(is_valid_oplog_processor_implementation); + .oplog_processor() + .ok() + .flatten() + .is_some(); if !implements_oplog_processor_interface { return Err(PluginRegistrationError::OplogProcessorComponentDoesNotExist); @@ -247,19 +244,3 @@ impl PluginRegistrationService { Ok(()) } } - -fn is_valid_oplog_processor_implementation(analyzed_export: &AnalysedExport) -> bool { - match analyzed_export { - AnalysedExport::Instance(inner) => { - let parts = inner.name.split("@").collect::>(); - - parts.len() == 2 && { - let interface_name = parts[0]; - let version = parts[1]; - interface_name == OPLOG_PROCESSOR_INTERFACE - && version.starts_with(OPLOG_PROCESSOR_VERSION_PREFIX) - } - } - _ => false, - } -} diff --git a/golem-service-base/src/clients/registry.rs b/golem-service-base/src/clients/registry.rs index 4c5d290e12..70fdeac020 100644 --- a/golem-service-base/src/clients/registry.rs +++ b/golem-service-base/src/clients/registry.rs @@ -24,15 +24,17 @@ use golem_api_grpc::proto::golem::registry::v1::{ GetActiveRoutesForDomainRequest, GetAgentDeploymentsRequest, GetAgentTypeRequest, GetAllAgentTypesRequest, GetAllDeployedComponentRevisionsRequest, GetAuthDetailsForEnvironmentRequest, GetComponentMetadataRequest, - GetDeployedComponentMetadataRequest, GetResourceLimitsRequest, ResolveComponentRequest, + GetDeployedComponentMetadataRequest, GetResourceLimitsRequest, + ResolveAgentTypeAtDeploymentRequest, ResolveComponentRequest, UpdateWorkerConnectionLimitRequest, UpdateWorkerLimitRequest, authenticate_token_response, batch_update_fuel_usage_response, download_component_response, get_active_routes_for_domain_response, get_agent_deployments_response, get_agent_type_response, get_all_agent_types_response, get_all_deployed_component_revisions_response, get_auth_details_for_environment_response, get_component_metadata_response, get_deployed_component_metadata_response, get_resource_limits_response, - resolve_component_response, resolve_latest_agent_type_by_names_response, - update_worker_connection_limit_response, update_worker_limit_response, + resolve_agent_type_at_deployment_response, resolve_component_response, + resolve_latest_agent_type_by_names_response, update_worker_connection_limit_response, + update_worker_limit_response, }; use golem_common::config::{ConfigExample, HasConfigExamples}; use golem_common::model::WorkerId; @@ -42,6 +44,7 @@ use golem_common::model::application::{ApplicationId, ApplicationName}; use golem_common::model::auth::TokenSecret; use golem_common::model::component::ComponentDto; use golem_common::model::component::{ComponentId, ComponentRevision}; +use golem_common::model::deployment::DeploymentRevision; use golem_common::model::domain_registration::Domain; use golem_common::model::environment::{EnvironmentId, EnvironmentName}; use golem_common::{IntoAnyhow, SafeDisplay, grpc_uri}; @@ -156,6 +159,15 @@ pub trait RegistryService: Send + Sync { agent_type_name: &AgentTypeName, ) -> Result; + async fn resolve_agent_type_at_deployment( + &self, + account_id: &AccountId, + app_name: &ApplicationName, + environment_name: &EnvironmentName, + agent_type_name: &AgentTypeName, + deployment_revision: DeploymentRevision, + ) -> Result; + async fn get_active_routes_for_domain( &self, domain: &Domain, @@ -679,6 +691,43 @@ impl RegistryService for GrpcRegistryService { } } + async fn resolve_agent_type_at_deployment( + &self, + account_id: &AccountId, + app_name: &ApplicationName, + environment_name: &EnvironmentName, + agent_type_name: &AgentTypeName, + deployment_revision: DeploymentRevision, + ) -> Result { + let response = self + .client + .call("resolve_agent_type_at_deployment", move |client| { + let request = ResolveAgentTypeAtDeploymentRequest { + account_id: Some((*account_id).into()), + app_name: app_name.0.clone(), + environment_name: environment_name.0.clone(), + agent_type_name: agent_type_name.0.clone(), + deployment_revision: deployment_revision.get(), + }; + Box::pin(client.resolve_agent_type_at_deployment(request)) + }) + .await? + .into_inner(); + + match response.result { + None => Err(RegistryServiceError::empty_response()), + Some(resolve_agent_type_at_deployment_response::Result::Success(payload)) => { + Ok(payload + .agent_type + .ok_or("missing agent_type field")? + .try_into()?) + } + Some(resolve_agent_type_at_deployment_response::Result::Error(error)) => { + Err(error.into()) + } + } + } + async fn get_active_routes_for_domain( &self, domain: &Domain, diff --git a/golem-shard-manager/config/shard-manager.sample.env b/golem-shard-manager/config/shard-manager.sample.env index b94ec20e0e..8523a74b48 100644 --- a/golem-shard-manager/config/shard-manager.sample.env +++ b/golem-shard-manager/config/shard-manager.sample.env @@ -33,6 +33,7 @@ GOLEM__TRACING__FILE__ENABLED=false GOLEM__TRACING__FILE__JSON=true GOLEM__TRACING__FILE__JSON_FLATTEN=true GOLEM__TRACING__FILE__JSON_FLATTEN_SPAN=true +GOLEM__TRACING__FILE__JSON_SOURCE_LOCATION=false GOLEM__TRACING__FILE__PRETTY=false GOLEM__TRACING__FILE__SPAN_EVENTS_ACTIVE=false GOLEM__TRACING__FILE__SPAN_EVENTS_FULL=false @@ -47,6 +48,7 @@ GOLEM__TRACING__STDERR__ENABLED=false GOLEM__TRACING__STDERR__JSON=false GOLEM__TRACING__STDERR__JSON_FLATTEN=false GOLEM__TRACING__STDERR__JSON_FLATTEN_SPAN=false +GOLEM__TRACING__STDERR__JSON_SOURCE_LOCATION=false GOLEM__TRACING__STDERR__PRETTY=false GOLEM__TRACING__STDERR__SPAN_EVENTS_ACTIVE=false GOLEM__TRACING__STDERR__SPAN_EVENTS_FULL=false @@ -57,6 +59,7 @@ GOLEM__TRACING__STDOUT__ENABLED=true GOLEM__TRACING__STDOUT__JSON=false GOLEM__TRACING__STDOUT__JSON_FLATTEN=true GOLEM__TRACING__STDOUT__JSON_FLATTEN_SPAN=true +GOLEM__TRACING__STDOUT__JSON_SOURCE_LOCATION=false GOLEM__TRACING__STDOUT__PRETTY=false GOLEM__TRACING__STDOUT__SPAN_EVENTS_ACTIVE=false GOLEM__TRACING__STDOUT__SPAN_EVENTS_FULL=false @@ -113,6 +116,7 @@ GOLEM__TRACING__FILE__ENABLED=false GOLEM__TRACING__FILE__JSON=true GOLEM__TRACING__FILE__JSON_FLATTEN=true GOLEM__TRACING__FILE__JSON_FLATTEN_SPAN=true +GOLEM__TRACING__FILE__JSON_SOURCE_LOCATION=false GOLEM__TRACING__FILE__PRETTY=false GOLEM__TRACING__FILE__SPAN_EVENTS_ACTIVE=false GOLEM__TRACING__FILE__SPAN_EVENTS_FULL=false @@ -127,6 +131,7 @@ GOLEM__TRACING__STDERR__ENABLED=false GOLEM__TRACING__STDERR__JSON=false GOLEM__TRACING__STDERR__JSON_FLATTEN=false GOLEM__TRACING__STDERR__JSON_FLATTEN_SPAN=false +GOLEM__TRACING__STDERR__JSON_SOURCE_LOCATION=false GOLEM__TRACING__STDERR__PRETTY=false GOLEM__TRACING__STDERR__SPAN_EVENTS_ACTIVE=false GOLEM__TRACING__STDERR__SPAN_EVENTS_FULL=false @@ -137,6 +142,7 @@ GOLEM__TRACING__STDOUT__ENABLED=true GOLEM__TRACING__STDOUT__JSON=false GOLEM__TRACING__STDOUT__JSON_FLATTEN=true GOLEM__TRACING__STDOUT__JSON_FLATTEN_SPAN=true +GOLEM__TRACING__STDOUT__JSON_SOURCE_LOCATION=false GOLEM__TRACING__STDOUT__PRETTY=false GOLEM__TRACING__STDOUT__SPAN_EVENTS_ACTIVE=false GOLEM__TRACING__STDOUT__SPAN_EVENTS_FULL=false diff --git a/golem-shard-manager/config/shard-manager.toml b/golem-shard-manager/config/shard-manager.toml index f2ba9fe48b..e6932c8ec8 100644 --- a/golem-shard-manager/config/shard-manager.toml +++ b/golem-shard-manager/config/shard-manager.toml @@ -51,6 +51,7 @@ enabled = false json = true json_flatten = true json_flatten_span = true +json_source_location = false pretty = false span_events_active = false span_events_full = false @@ -69,6 +70,7 @@ enabled = false json = false json_flatten = false json_flatten_span = false +json_source_location = false pretty = false span_events_active = false span_events_full = false @@ -81,6 +83,7 @@ enabled = true json = false json_flatten = true json_flatten_span = true +json_source_location = false pretty = false span_events_active = false span_events_full = false @@ -166,6 +169,7 @@ type = "Disabled" # json = true # json_flatten = true # json_flatten_span = true +# json_source_location = false # pretty = false # span_events_active = false # span_events_full = false @@ -184,6 +188,7 @@ type = "Disabled" # json = false # json_flatten = false # json_flatten_span = false +# json_source_location = false # pretty = false # span_events_active = false # span_events_full = false @@ -196,6 +201,7 @@ type = "Disabled" # json = false # json_flatten = true # json_flatten_span = true +# json_source_location = false # pretty = false # span_events_active = false # span_events_full = false diff --git a/golem-test-framework/Cargo.toml b/golem-test-framework/Cargo.toml index db5fe2f806..bde445cecb 100644 --- a/golem-test-framework/Cargo.toml +++ b/golem-test-framework/Cargo.toml @@ -31,6 +31,7 @@ colored = { workspace = true } futures = { workspace = true } humantime-serde = { workspace = true } itertools = { workspace = true } +log = { workspace = true } prometheus = { workspace = true } redis = { workspace = true } reqwest = { workspace = true } @@ -48,6 +49,7 @@ tokio-stream = { workspace = true } tokio-tungstenite = { workspace = true, features = ["native-tls"] } tonic = { workspace = true } tracing = { workspace = true } +tracing-core = { workspace = true } tryhard = { workspace = true } url = { workspace = true } uuid = { workspace = true } diff --git a/golem-test-framework/src/components/dynamic_span.rs b/golem-test-framework/src/components/dynamic_span.rs new file mode 100644 index 0000000000..d5ee6a54a6 --- /dev/null +++ b/golem-test-framework/src/components/dynamic_span.rs @@ -0,0 +1,209 @@ +// Copyright 2024-2025 Golem Cloud +// +// Licensed under the Golem Source License v1.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://license.golem.cloud/LICENSE +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +use std::collections::HashMap; +use std::sync::{Mutex, OnceLock}; +use tracing::Span; +use tracing_core::callsite::{DefaultCallsite, Identifier}; +use tracing_core::field::FieldSet; +use tracing_core::metadata::Kind; +use tracing_core::{Callsite, Interest, Level, Metadata}; + +#[derive(Default)] +struct DynCallsite { + delegate: OnceLock, +} + +impl Callsite for DynCallsite { + fn set_interest(&self, interest: Interest) { + if let Some(d) = self.delegate.get() { + d.set_interest(interest); + } + } + + fn metadata(&self) -> &Metadata<'_> { + self.delegate + .get() + .expect("DynCallsite not initialized") + .metadata() + } +} + +static INTERN_CACHE: OnceLock>> = OnceLock::new(); + +fn intern(s: &str) -> &'static str { + let mut map = INTERN_CACHE + .get_or_init(|| Mutex::new(HashMap::new())) + .lock() + .unwrap(); + if let Some(leaked) = map.get(s) { + return leaked; + } + let leaked: &'static str = Box::leak(s.to_string().into_boxed_str()); + map.insert(s.to_string(), leaked); + leaked +} + +fn new_callsite_and_meta( + name: &str, + target: &str, + level: Level, + field_names: &[&str], + kind: Kind, + file: Option<&str>, + line: Option, +) -> &'static Metadata<'static> { + let static_field_names: Vec<&'static str> = field_names.iter().map(|n| intern(n)).collect(); + let static_field_names_slice: &'static [&'static str] = + Box::leak(static_field_names.into_boxed_slice()); + + let callsite: &'static DynCallsite = Box::leak(Box::::default()); + let meta: &'static Metadata<'static> = Box::leak(Box::new(Metadata::new( + intern(name), + intern(target), + level, + file.map(intern), + line, + None, + FieldSet::new(static_field_names_slice, Identifier(callsite)), + kind, + ))); + + callsite + .delegate + .set(DefaultCallsite::new(meta)) + .expect("DynCallsite already initialized"); + + meta +} + +// --------------- Span support --------------- + +static SPAN_CACHE: OnceLock>>> = OnceLock::new(); + +fn span_cache() -> &'static Mutex>> { + SPAN_CACHE.get_or_init(|| Mutex::new(HashMap::new())) +} + +fn span_cache_key(target: &str, name: &str, field_names: &mut Vec<&str>) -> String { + field_names.sort(); + if field_names.is_empty() { + format!("{target}:{name}") + } else { + format!("{target}:{}:{}", name, field_names.join(",")) + } +} + +fn get_or_create_span_meta( + target: &str, + name: &str, + field_names: &[&str], + cache_key: &str, +) -> &'static Metadata<'static> { + let mut map = span_cache().lock().unwrap(); + if let Some(meta) = map.get(cache_key) { + return meta; + } + + let meta = new_callsite_and_meta( + name, + target, + Level::TRACE, + field_names, + Kind::SPAN, + None, + None, + ); + map.insert(cache_key.to_string(), meta); + meta +} + +pub fn make_span(target: &str, name: &str, fields: &[(String, String)]) -> Span { + let mut field_names: Vec<&str> = fields.iter().map(|(k, _)| k.as_str()).collect(); + let cache_key = span_cache_key(target, name, &mut field_names); + let meta = get_or_create_span_meta(target, name, &field_names, &cache_key); + + let span = Span::new(meta, &meta.fields().value_set(&[])); + for (k, v) in fields { + span.record(k.as_str(), v.as_str()); + } + span +} + +// --------------- Event support --------------- + +static EVENT_CACHE: OnceLock>>> = OnceLock::new(); + +fn event_cache() -> &'static Mutex>> { + EVENT_CACHE.get_or_init(|| Mutex::new(HashMap::new())) +} + +fn level_key(level: Level) -> &'static str { + match level { + Level::TRACE => "T", + Level::DEBUG => "D", + Level::INFO => "I", + Level::WARN => "W", + Level::ERROR => "E", + } +} + +fn get_or_create_event_meta( + target: &str, + level: Level, + file: Option<&str>, + line: Option, +) -> &'static Metadata<'static> { + let cache_key = format!( + "{}:{}:{}:{}", + target, + level_key(level), + file.unwrap_or(""), + line.unwrap_or(0) + ); + let mut map = event_cache().lock().unwrap(); + if let Some(meta) = map.get(&cache_key) { + return meta; + } + + let meta = new_callsite_and_meta( + "child_process_event", + target, + level, + &["message"], + Kind::EVENT, + file, + line, + ); + map.insert(cache_key, meta); + meta +} + +pub fn dispatch_event( + target: &str, + level: Level, + message: &str, + file: Option<&str>, + line: Option, +) { + let meta = get_or_create_event_meta(target, level, file, line); + let message_field = meta.fields().field("message").unwrap(); + tracing_core::Event::dispatch( + meta, + &meta.fields().value_set(&[( + &message_field, + Some(&message as &dyn tracing_core::field::Value), + )]), + ); +} diff --git a/golem-test-framework/src/components/mod.rs b/golem-test-framework/src/components/mod.rs index e9e4c0fb9d..c3990e444f 100644 --- a/golem-test-framework/src/components/mod.rs +++ b/golem-test-framework/src/components/mod.rs @@ -30,6 +30,7 @@ use url::Url; pub mod blob_storage; pub mod component_compilation_service; mod docker; +mod dynamic_span; pub mod rdb; pub mod redis; pub mod redis_monitor; @@ -66,12 +67,12 @@ impl ChildProcessLogger { let stdout_handle = std::thread::spawn(move || { let reader = BufReader::new(stdout); for line in reader.lines() { - match out_level { - Level::TRACE => trace!("{} {}", prefix_clone, line.unwrap()), - Level::DEBUG => debug!("{} {}", prefix_clone, line.unwrap()), - Level::INFO => info!("{} {}", prefix_clone, line.unwrap()), - Level::WARN => warn!("{} {}", prefix_clone, line.unwrap()), - Level::ERROR => error!("{} {}", prefix_clone, line.unwrap()), + match line { + Ok(line) => relay_line(&prefix_clone, &line, out_level), + Err(e) => { + warn!("{} failed to read stdout: {e}", prefix_clone); + break; + } } } }); @@ -80,12 +81,12 @@ impl ChildProcessLogger { let stderr_handle = std::thread::spawn(move || { let reader = BufReader::new(stderr); for line in reader.lines() { - match err_level { - Level::TRACE => trace!("{} {}", prefix_clone, line.unwrap()), - Level::DEBUG => debug!("{} {}", prefix_clone, line.unwrap()), - Level::INFO => info!("{} {}", prefix_clone, line.unwrap()), - Level::WARN => warn!("{} {}", prefix_clone, line.unwrap()), - Level::ERROR => error!("{} {}", prefix_clone, line.unwrap()), + match line { + Ok(line) => relay_line(&prefix_clone, &line, err_level), + Err(e) => { + warn!("{} failed to read stderr: {e}", prefix_clone); + break; + } } } }); @@ -97,6 +98,132 @@ impl ChildProcessLogger { } } +fn relay_line(prefix: &str, line: &str, fallback_level: Level) { + let Ok(obj) = serde_json::from_str::(line) else { + emit_at_level(fallback_level, prefix, line); + return; + }; + + let level = obj + .get("level") + .and_then(|v| v.as_str()) + .and_then(parse_tracing_level) + .unwrap_or(fallback_level); + + let target = obj + .get("target") + .and_then(|v| v.as_str()) + .unwrap_or("child_process"); + + let file = obj.get("filename").and_then(|v| v.as_str()); + + let line = obj + .get("line_number") + .and_then(|v| v.as_u64()) + .map(|n| n as u32); + + let message = obj + .get("fields") + .and_then(|f| f.get("message")) + .and_then(|v| v.as_str()) + .or_else(|| obj.get("message").and_then(|v| v.as_str())) + .unwrap_or(""); + + let span_infos = parse_span_infos(&obj); + // Create and enter each span before creating the next, so each becomes + // a child of the previous one (proper nesting). + let _entered: Vec = span_infos + .iter() + .map(|(name, fields)| dynamic_span::make_span(prefix, name, fields).entered()) + .collect(); + + let event_fields = + format_kv_fields(obj.get("fields").and_then(|f| f.as_object()), &["message"]); + + let msg = if event_fields.is_empty() { + format!("{prefix} {message}") + } else { + format!("{prefix} {message} {event_fields}") + }; + dynamic_span::dispatch_event(target, level, &msg, file, line); +} + +fn parse_span_infos(obj: &serde_json::Value) -> Vec<(String, Vec<(String, String)>)> { + obj.get("spans") + .and_then(|s| s.as_array()) + .map(|arr| { + arr.iter() + .filter_map(|span_obj| { + let name = span_obj.get("name")?.as_str()?.to_string(); + let fields = parse_kv_fields(span_obj.as_object(), &["name"]); + Some((name, fields)) + }) + .collect() + }) + .unwrap_or_default() +} + +fn parse_kv_fields( + map: Option<&serde_json::Map>, + skip: &[&str], +) -> Vec<(String, String)> { + let Some(map) = map else { + return Vec::new(); + }; + map.iter() + .filter(|(k, _)| !skip.contains(&k.as_str())) + .map(|(k, v)| { + let val = match v { + serde_json::Value::String(s) => s.clone(), + other => other.to_string(), + }; + (k.clone(), val) + }) + .collect() +} + +fn format_kv_fields( + map: Option<&serde_json::Map>, + skip: &[&str], +) -> String { + let pairs = parse_kv_fields(map, skip); + if pairs.is_empty() { + String::new() + } else { + pairs + .iter() + .map(|(k, v)| format!("{k}={v}")) + .collect::>() + .join(" ") + } +} + +fn emit_at_level(level: Level, prefix: &str, line: &str) { + match level { + Level::TRACE => trace!("{prefix} {line}"), + Level::DEBUG => debug!("{prefix} {line}"), + Level::INFO => info!("{prefix} {line}"), + Level::WARN => warn!("{prefix} {line}"), + Level::ERROR => error!("{prefix} {line}"), + } +} + +fn parse_tracing_level(s: &str) -> Option { + let s = s.trim(); + let s = s + .strip_prefix("Level(") + .and_then(|s| s.strip_suffix(')')) + .unwrap_or(s); + match s.to_uppercase().as_str() { + "TRACE" => Some(Level::TRACE), + "DEBUG" => Some(Level::DEBUG), + "INFO" => Some(Level::INFO), + "WARN" => Some(Level::WARN), + "ERROR" => Some(Level::ERROR), + _ => None, + } +} + pub async fn wait_for_startup_grpc(host: &str, grpc_port: u16, name: &str, timeout: Duration) { info!( "Waiting for {name} (GRPC) start on host {host}:{grpc_port}, timeout: {}s", @@ -219,6 +346,17 @@ impl EnvVarBuilder { .with_rust_back_log() .with_tracing_from_env() .with("GOLEM__TRACING__STDOUT__ANSI", "false".to_string()) + .with("GOLEM__TRACING__STDOUT__ENABLED", "true".to_string()) + .with("GOLEM__TRACING__STDOUT__JSON", "true".to_string()) + .with("GOLEM__TRACING__STDOUT__JSON_FLATTEN", "false".to_string()) + .with( + "GOLEM__TRACING__STDOUT__JSON_FLATTEN_SPAN", + "false".to_string(), + ) + .with( + "GOLEM__TRACING__STDOUT__JSON_SOURCE_LOCATION", + "true".to_string(), + ) } fn with(mut self, name: &str, value: String) -> Self { diff --git a/golem-test-framework/src/components/service/mod.rs b/golem-test-framework/src/components/service/mod.rs index 3d090bbdca..6f599cdacf 100644 --- a/golem-test-framework/src/components/service/mod.rs +++ b/golem-test-framework/src/components/service/mod.rs @@ -29,6 +29,11 @@ fn env_vars(env_vars: HashMap, verbosity: Level) -> HashMap = diff --git a/golem-test-framework/src/config/dsl_impl.rs b/golem-test-framework/src/config/dsl_impl.rs index 0f5bb3dd46..9375aac143 100644 --- a/golem-test-framework/src/config/dsl_impl.rs +++ b/golem-test-framework/src/config/dsl_impl.rs @@ -45,6 +45,7 @@ use golem_common::model::component::{ ComponentDto, ComponentFileOptions, ComponentFilePath, ComponentId, ComponentName, ComponentRevision, PluginInstallation, }; +use golem_common::model::deployment::DeploymentRevision; use golem_common::model::environment::{ Environment, EnvironmentCreation, EnvironmentId, EnvironmentName, }; @@ -67,7 +68,7 @@ use tokio_tungstenite::tungstenite::client::IntoClientRequest; use tokio_tungstenite::tungstenite::protocol::frame::Payload; use tokio_tungstenite::tungstenite::Message; use tokio_tungstenite::{Connector, MaybeTlsStream, WebSocketStream}; -use tracing::debug; +use tracing::{debug, trace}; use uuid::Uuid; pub struct NameResolutionCache { @@ -156,6 +157,7 @@ pub struct TestUserContext { pub token: TokenSecret, pub auto_deploy_enabled: bool, pub name_cache: Arc, + pub last_deployments: Arc>>, } impl TestUserContext { @@ -228,6 +230,8 @@ impl TestDsl for TestUserContext { let agent_types = extract_agent_types(&source_path, false, true).await?; + trace!("Agent types in component {component_name}:\n{agent_types:#?}"); + let component = client .create_component( &environment_id.0, @@ -245,8 +249,11 @@ impl TestDsl for TestUserContext { .await?; if self.auto_deploy_enabled { - // deploy environment to make component visible - self.deploy_environment(&component.environment_id).await?; + let deployment = self.deploy_environment(&component.environment_id).await?; + self.last_deployments + .write() + .unwrap() + .insert(component.environment_id, deployment.revision); } Ok(component) @@ -344,8 +351,11 @@ impl TestDsl for TestUserContext { .await?; if self.auto_deploy_enabled { - // deploy environment to make component visible - self.deploy_environment(&component.environment_id).await?; + let deployment = self.deploy_environment(&component.environment_id).await?; + self.last_deployments + .write() + .unwrap() + .insert(component.environment_id, deployment.revision); } Ok(component) @@ -415,6 +425,7 @@ impl TestDsl for TestUserContext { mode: golem_client::model::AgentInvocationMode::Schedule, schedule_at: None, idempotency_key: None, + deployment_revision: None, }, ) .await @@ -427,6 +438,7 @@ impl TestDsl for TestUserContext { component: &ComponentDto, agent_id: &AgentId, idempotency_key: Option<&IdempotencyKey>, + deployment_revision: Option, method_name: &str, params: DataValue, ) -> anyhow::Result { @@ -463,6 +475,7 @@ impl TestDsl for TestUserContext { mode: golem_client::model::AgentInvocationMode::Await, schedule_at: None, idempotency_key: None, + deployment_revision: deployment_revision.map(i64::from), }, ) .await @@ -928,6 +941,20 @@ impl TestDslExtended for TestUserContext { Ok((application, environment)) } + + fn get_last_deployment_revision( + &self, + environment_id: &EnvironmentId, + ) -> anyhow::Result { + self.last_deployments + .read() + .unwrap() + .get(environment_id) + .copied() + .ok_or_else(|| { + anyhow!("No deployment revision recorded for environment {environment_id}") + }) + } } struct HttpWorkerLogEventStream { diff --git a/golem-test-framework/src/config/mod.rs b/golem-test-framework/src/config/mod.rs index 699f0ba75b..4a934bc6da 100644 --- a/golem-test-framework/src/config/mod.rs +++ b/golem-test-framework/src/config/mod.rs @@ -32,6 +32,7 @@ use golem_common::model::account::{AccountCreation, AccountEmail}; use golem_common::model::auth::AccountRole; use golem_service_base::service::initial_component_files::InitialComponentFilesService; use golem_service_base::storage::blob::BlobStorage; +use std::collections::HashMap; use std::path::Path; use std::sync::Arc; use uuid::Uuid; @@ -67,6 +68,7 @@ pub trait TestDependencies: Send + Sync + Clone { deps: self.clone(), auto_deploy_enabled: true, name_cache: Arc::new(NameResolutionCache::new()), + last_deployments: Arc::new(std::sync::RwLock::new(HashMap::new())), } } @@ -104,6 +106,7 @@ pub trait TestDependencies: Send + Sync + Clone { deps: self.clone(), auto_deploy_enabled: true, name_cache: Arc::new(NameResolutionCache::new()), + last_deployments: Arc::new(std::sync::RwLock::new(HashMap::new())), }) } @@ -151,6 +154,7 @@ pub trait TestDependencies: Send + Sync + Clone { deps: self.clone(), auto_deploy_enabled: true, name_cache: Arc::new(NameResolutionCache::new()), + last_deployments: Arc::new(std::sync::RwLock::new(HashMap::new())), }) } diff --git a/golem-test-framework/src/dsl/mod.rs b/golem-test-framework/src/dsl/mod.rs index ad20d453bf..eaba325ca8 100644 --- a/golem-test-framework/src/dsl/mod.rs +++ b/golem-test-framework/src/dsl/mod.rs @@ -37,7 +37,9 @@ use golem_common::model::component::{ PluginInstallation, }; use golem_common::model::component_metadata::RawComponentMetadata; -use golem_common::model::deployment::{CurrentDeployment, DeploymentCreation, DeploymentVersion}; +use golem_common::model::deployment::{ + CurrentDeployment, DeploymentCreation, DeploymentRevision, DeploymentVersion, +}; use golem_common::model::domain_registration::{Domain, DomainRegistrationCreation}; use golem_common::model::environment::{Environment, EnvironmentId}; use golem_common::model::environment_plugin_grant::EnvironmentPluginGrantId; @@ -254,10 +256,29 @@ pub trait TestDsl { method_name: &str, params: DataValue, ) -> anyhow::Result { - self.invoke_and_await_agent_impl(component, agent_id, None, method_name, params) + self.invoke_and_await_agent_impl(component, agent_id, None, None, method_name, params) .await } + async fn invoke_and_await_agent_at_deployment( + &self, + component: &ComponentDto, + agent_id: &AgentId, + deployment_revision: DeploymentRevision, + method_name: &str, + params: DataValue, + ) -> anyhow::Result { + self.invoke_and_await_agent_impl( + component, + agent_id, + None, + Some(deployment_revision), + method_name, + params, + ) + .await + } + async fn invoke_and_await_agent_with_key( &self, component: &ComponentDto, @@ -270,6 +291,7 @@ pub trait TestDsl { component, agent_id, Some(idempotency_key), + None, method_name, params, ) @@ -281,6 +303,7 @@ pub trait TestDsl { component: &ComponentDto, agent_id: &AgentId, idempotency_key: Option<&IdempotencyKey>, + deployment_revision: Option, method_name: &str, params: DataValue, ) -> anyhow::Result; @@ -594,6 +617,11 @@ pub trait TestDslExtended: TestDsl { Ok(deployment) } + + fn get_last_deployment_revision( + &self, + environment_id: &EnvironmentId, + ) -> anyhow::Result; } pub struct StoreComponentBuilder<'a, Dsl: TestDsl + ?Sized> { diff --git a/golem-worker-executor-test-utils/src/dsl_impl.rs b/golem-worker-executor-test-utils/src/dsl_impl.rs index 6068cbbb20..ecb66e0d10 100644 --- a/golem-worker-executor-test-utils/src/dsl_impl.rs +++ b/golem-worker-executor-test-utils/src/dsl_impl.rs @@ -33,6 +33,7 @@ use golem_common::model::component::{ ComponentDto, ComponentFilePath, ComponentId, ComponentName, ComponentRevision, InitialComponentFile, PluginInstallation, }; +use golem_common::model::deployment::DeploymentRevision; use golem_common::model::environment::EnvironmentId; use golem_common::model::oplog::{PublicOplogEntry, PublicOplogEntryWithIndex}; use golem_common::model::worker::RevertWorkerTarget; @@ -347,6 +348,7 @@ impl TestDsl for TestWorkerExecutor { component: &ComponentDto, agent_id: &AgentId, idempotency_key: Option<&IdempotencyKey>, + _deployment_revision: Option, method_name: &str, params: DataValue, ) -> anyhow::Result { diff --git a/golem-worker-executor/config/worker-executor.sample.env b/golem-worker-executor/config/worker-executor.sample.env index 614a9312aa..131b3cf279 100644 --- a/golem-worker-executor/config/worker-executor.sample.env +++ b/golem-worker-executor/config/worker-executor.sample.env @@ -123,6 +123,7 @@ GOLEM__TRACING__FILE__ENABLED=false GOLEM__TRACING__FILE__JSON=true GOLEM__TRACING__FILE__JSON_FLATTEN=true GOLEM__TRACING__FILE__JSON_FLATTEN_SPAN=true +GOLEM__TRACING__FILE__JSON_SOURCE_LOCATION=false GOLEM__TRACING__FILE__PRETTY=false GOLEM__TRACING__FILE__SPAN_EVENTS_ACTIVE=false GOLEM__TRACING__FILE__SPAN_EVENTS_FULL=false @@ -137,6 +138,7 @@ GOLEM__TRACING__STDERR__ENABLED=false GOLEM__TRACING__STDERR__JSON=false GOLEM__TRACING__STDERR__JSON_FLATTEN=false GOLEM__TRACING__STDERR__JSON_FLATTEN_SPAN=false +GOLEM__TRACING__STDERR__JSON_SOURCE_LOCATION=false GOLEM__TRACING__STDERR__PRETTY=false GOLEM__TRACING__STDERR__SPAN_EVENTS_ACTIVE=false GOLEM__TRACING__STDERR__SPAN_EVENTS_FULL=false @@ -147,6 +149,7 @@ GOLEM__TRACING__STDOUT__ENABLED=true GOLEM__TRACING__STDOUT__JSON=false GOLEM__TRACING__STDOUT__JSON_FLATTEN=true GOLEM__TRACING__STDOUT__JSON_FLATTEN_SPAN=true +GOLEM__TRACING__STDOUT__JSON_SOURCE_LOCATION=false GOLEM__TRACING__STDOUT__PRETTY=false GOLEM__TRACING__STDOUT__SPAN_EVENTS_ACTIVE=false GOLEM__TRACING__STDOUT__SPAN_EVENTS_FULL=false @@ -278,6 +281,7 @@ GOLEM__TRACING__FILE__ENABLED=false GOLEM__TRACING__FILE__JSON=true GOLEM__TRACING__FILE__JSON_FLATTEN=true GOLEM__TRACING__FILE__JSON_FLATTEN_SPAN=true +GOLEM__TRACING__FILE__JSON_SOURCE_LOCATION=false GOLEM__TRACING__FILE__PRETTY=false GOLEM__TRACING__FILE__SPAN_EVENTS_ACTIVE=false GOLEM__TRACING__FILE__SPAN_EVENTS_FULL=false @@ -292,6 +296,7 @@ GOLEM__TRACING__STDERR__ENABLED=false GOLEM__TRACING__STDERR__JSON=false GOLEM__TRACING__STDERR__JSON_FLATTEN=false GOLEM__TRACING__STDERR__JSON_FLATTEN_SPAN=false +GOLEM__TRACING__STDERR__JSON_SOURCE_LOCATION=false GOLEM__TRACING__STDERR__PRETTY=false GOLEM__TRACING__STDERR__SPAN_EVENTS_ACTIVE=false GOLEM__TRACING__STDERR__SPAN_EVENTS_FULL=false @@ -302,6 +307,7 @@ GOLEM__TRACING__STDOUT__ENABLED=true GOLEM__TRACING__STDOUT__JSON=false GOLEM__TRACING__STDOUT__JSON_FLATTEN=true GOLEM__TRACING__STDOUT__JSON_FLATTEN_SPAN=true +GOLEM__TRACING__STDOUT__JSON_SOURCE_LOCATION=false GOLEM__TRACING__STDOUT__PRETTY=false GOLEM__TRACING__STDOUT__SPAN_EVENTS_ACTIVE=false GOLEM__TRACING__STDOUT__SPAN_EVENTS_FULL=false @@ -418,6 +424,7 @@ GOLEM__TRACING__FILE__ENABLED=false GOLEM__TRACING__FILE__JSON=true GOLEM__TRACING__FILE__JSON_FLATTEN=true GOLEM__TRACING__FILE__JSON_FLATTEN_SPAN=true +GOLEM__TRACING__FILE__JSON_SOURCE_LOCATION=false GOLEM__TRACING__FILE__PRETTY=false GOLEM__TRACING__FILE__SPAN_EVENTS_ACTIVE=false GOLEM__TRACING__FILE__SPAN_EVENTS_FULL=false @@ -432,6 +439,7 @@ GOLEM__TRACING__STDERR__ENABLED=false GOLEM__TRACING__STDERR__JSON=false GOLEM__TRACING__STDERR__JSON_FLATTEN=false GOLEM__TRACING__STDERR__JSON_FLATTEN_SPAN=false +GOLEM__TRACING__STDERR__JSON_SOURCE_LOCATION=false GOLEM__TRACING__STDERR__PRETTY=false GOLEM__TRACING__STDERR__SPAN_EVENTS_ACTIVE=false GOLEM__TRACING__STDERR__SPAN_EVENTS_FULL=false @@ -442,6 +450,7 @@ GOLEM__TRACING__STDOUT__ENABLED=true GOLEM__TRACING__STDOUT__JSON=false GOLEM__TRACING__STDOUT__JSON_FLATTEN=true GOLEM__TRACING__STDOUT__JSON_FLATTEN_SPAN=true +GOLEM__TRACING__STDOUT__JSON_SOURCE_LOCATION=false GOLEM__TRACING__STDOUT__PRETTY=false GOLEM__TRACING__STDOUT__SPAN_EVENTS_ACTIVE=false GOLEM__TRACING__STDOUT__SPAN_EVENTS_FULL=false diff --git a/golem-worker-executor/config/worker-executor.toml b/golem-worker-executor/config/worker-executor.toml index 62c0905803..f8ced05e3e 100644 --- a/golem-worker-executor/config/worker-executor.toml +++ b/golem-worker-executor/config/worker-executor.toml @@ -213,6 +213,7 @@ enabled = false json = true json_flatten = true json_flatten_span = true +json_source_location = false pretty = false span_events_active = false span_events_full = false @@ -231,6 +232,7 @@ enabled = false json = false json_flatten = false json_flatten_span = false +json_source_location = false pretty = false span_events_active = false span_events_full = false @@ -243,6 +245,7 @@ enabled = true json = false json_flatten = true json_flatten_span = true +json_source_location = false pretty = false span_events_active = false span_events_full = false @@ -456,6 +459,7 @@ without_time = false # json = true # json_flatten = true # json_flatten_span = true +# json_source_location = false # pretty = false # span_events_active = false # span_events_full = false @@ -474,6 +478,7 @@ without_time = false # json = false # json_flatten = false # json_flatten_span = false +# json_source_location = false # pretty = false # span_events_active = false # span_events_full = false @@ -486,6 +491,7 @@ without_time = false # json = false # json_flatten = true # json_flatten_span = true +# json_source_location = false # pretty = false # span_events_active = false # span_events_full = false @@ -692,6 +698,7 @@ without_time = false # json = true # json_flatten = true # json_flatten_span = true +# json_source_location = false # pretty = false # span_events_active = false # span_events_full = false @@ -710,6 +717,7 @@ without_time = false # json = false # json_flatten = false # json_flatten_span = false +# json_source_location = false # pretty = false # span_events_active = false # span_events_full = false @@ -722,6 +730,7 @@ without_time = false # json = false # json_flatten = true # json_flatten_span = true +# json_source_location = false # pretty = false # span_events_active = false # span_events_full = false diff --git a/golem-worker-executor/tests/revert.rs b/golem-worker-executor/tests/revert.rs index 207f0d414f..8f55ba0dfd 100644 --- a/golem-worker-executor/tests/revert.rs +++ b/golem-worker-executor/tests/revert.rs @@ -16,7 +16,7 @@ use crate::Tracing; use golem_common::model::component::ComponentRevision; use golem_common::model::oplog::PublicOplogEntry; use golem_common::model::worker::{RevertLastInvocations, RevertToOplogIndex, RevertWorkerTarget}; -use golem_common::model::OplogIndex; +use golem_common::model::{OplogIndex, WorkerStatus}; use golem_common::{agent_id, data_value}; use golem_test_framework::dsl::{update_counts, TestDsl}; use golem_wasm::Value; @@ -285,6 +285,15 @@ async fn revert_auto_update( .await?; executor.log_output(&worker_id).await?; + // Wait for the worker to finish initialization before reading the oplog + executor + .wait_for_status( + &worker_id, + WorkerStatus::Idle, + std::time::Duration::from_secs(10), + ) + .await?; + let updated_component = executor .update_component(&component.id, "it_agent_update_v2_release") .await?; diff --git a/golem-worker-service/config/worker-service.sample.env b/golem-worker-service/config/worker-service.sample.env index 4795041abe..2a913db2f8 100644 --- a/golem-worker-service/config/worker-service.sample.env +++ b/golem-worker-service/config/worker-service.sample.env @@ -63,6 +63,7 @@ GOLEM__TRACING__FILE__ENABLED=false GOLEM__TRACING__FILE__JSON=true GOLEM__TRACING__FILE__JSON_FLATTEN=true GOLEM__TRACING__FILE__JSON_FLATTEN_SPAN=true +GOLEM__TRACING__FILE__JSON_SOURCE_LOCATION=false GOLEM__TRACING__FILE__PRETTY=false GOLEM__TRACING__FILE__SPAN_EVENTS_ACTIVE=false GOLEM__TRACING__FILE__SPAN_EVENTS_FULL=false @@ -77,6 +78,7 @@ GOLEM__TRACING__STDERR__ENABLED=false GOLEM__TRACING__STDERR__JSON=false GOLEM__TRACING__STDERR__JSON_FLATTEN=false GOLEM__TRACING__STDERR__JSON_FLATTEN_SPAN=false +GOLEM__TRACING__STDERR__JSON_SOURCE_LOCATION=false GOLEM__TRACING__STDERR__PRETTY=false GOLEM__TRACING__STDERR__SPAN_EVENTS_ACTIVE=false GOLEM__TRACING__STDERR__SPAN_EVENTS_FULL=false @@ -87,6 +89,7 @@ GOLEM__TRACING__STDOUT__ENABLED=true GOLEM__TRACING__STDOUT__JSON=false GOLEM__TRACING__STDOUT__JSON_FLATTEN=true GOLEM__TRACING__STDOUT__JSON_FLATTEN_SPAN=true +GOLEM__TRACING__STDOUT__JSON_SOURCE_LOCATION=false GOLEM__TRACING__STDOUT__PRETTY=false GOLEM__TRACING__STDOUT__SPAN_EVENTS_ACTIVE=false GOLEM__TRACING__STDOUT__SPAN_EVENTS_FULL=false diff --git a/golem-worker-service/config/worker-service.toml b/golem-worker-service/config/worker-service.toml index 4a16c8ee39..6c143e1182 100644 --- a/golem-worker-service/config/worker-service.toml +++ b/golem-worker-service/config/worker-service.toml @@ -97,6 +97,7 @@ enabled = false json = true json_flatten = true json_flatten_span = true +json_source_location = false pretty = false span_events_active = false span_events_full = false @@ -115,6 +116,7 @@ enabled = false json = false json_flatten = false json_flatten_span = false +json_source_location = false pretty = false span_events_active = false span_events_full = false @@ -127,6 +129,7 @@ enabled = true json = false json_flatten = true json_flatten_span = true +json_source_location = false pretty = false span_events_active = false span_events_full = false diff --git a/golem-worker-service/src/api/agents.rs b/golem-worker-service/src/api/agents.rs index 381494605a..3ddbb857dc 100644 --- a/golem-worker-service/src/api/agents.rs +++ b/golem-worker-service/src/api/agents.rs @@ -11,7 +11,6 @@ use golem_service_base::api_tags::ApiTags; use golem_service_base::model::auth::GolemSecurityScheme; use poem_openapi::param::Header; use poem_openapi::payload::Json; -use poem_openapi::types::Type; use poem_openapi_derive::{Enum, Object, OpenApi}; use serde::{Deserialize, Serialize}; use std::sync::Arc; @@ -43,7 +42,7 @@ impl AgentsApi { ) -> Result> { let auth = self.auth_service.authenticate_token(token.secret()).await?; - if request.idempotency_key.is_empty() { + if request.idempotency_key.is_none() { request.idempotency_key = idempotency_key.0; } @@ -89,6 +88,7 @@ pub struct AgentInvocationRequest { pub mode: AgentInvocationMode, pub schedule_at: Option>, pub idempotency_key: Option, + pub deployment_revision: Option, } #[derive(Debug, Clone, Serialize, Deserialize, Object)] diff --git a/golem-worker-service/src/service/worker/service.rs b/golem-worker-service/src/service/worker/service.rs index 91527d3174..a16c2d04f8 100644 --- a/golem-worker-service/src/service/worker/service.rs +++ b/golem-worker-service/src/service/worker/service.rs @@ -28,6 +28,7 @@ use golem_common::model::agent::{ use golem_common::model::component::{ ComponentDto, ComponentFilePath, ComponentId, ComponentRevision, PluginPriority, }; +use golem_common::model::deployment::DeploymentRevision; use golem_common::model::oplog::OplogCursor; use golem_common::model::oplog::OplogIndex; use golem_common::model::worker::WorkerUpdateMode; @@ -750,20 +751,43 @@ impl WorkerService { request: AgentInvocationRequest, auth: AuthCtx, ) -> WorkerResult { - let registered_agent_type = self - .registry_service - .resolve_latest_agent_type_by_names( - &auth.account_id(), - &request.app_name, - &request.env_name, - &request.agent_type_name, - ) - .await?; + let registered_agent_type = match request.deployment_revision { + Some(rev) => { + let rev_u64 = u64::try_from(rev).map_err(|_| { + WorkerServiceError::Internal(format!( + "Invalid deployment revision (must be non-negative): {rev}" + )) + })?; + let deployment_revision = DeploymentRevision::new(rev_u64).map_err(|e| { + WorkerServiceError::Internal(format!("Invalid deployment revision: {e}")) + })?; + self.registry_service + .resolve_agent_type_at_deployment( + &auth.account_id(), + &request.app_name, + &request.env_name, + &request.agent_type_name, + deployment_revision, + ) + .await? + } + None => { + self.registry_service + .resolve_latest_agent_type_by_names( + &auth.account_id(), + &request.app_name, + &request.env_name, + &request.agent_type_name, + ) + .await? + } + }; + let component_id = registered_agent_type.implemented_by.component_id; let component_metadata = self .component_service .get_revision( - registered_agent_type.implemented_by.component_id, + component_id, registered_agent_type.implemented_by.component_revision, ) .await?; diff --git a/integration-tests/Cargo.toml b/integration-tests/Cargo.toml index 499a02fb55..0bd93cbeb4 100644 --- a/integration-tests/Cargo.toml +++ b/integration-tests/Cargo.toml @@ -42,6 +42,7 @@ uuid = { workspace = true, features = ["v4"] } serde_yaml = { workspace = true } [dev-dependencies] +backtrace-on-stack-overflow = { workspace = true } goldenfile = { workspace = true } test-r = { workspace = true } wac-graph = { workspace = true } diff --git a/integration-tests/tests/fork.rs b/integration-tests/tests/fork.rs index 0689d85811..80a98ece87 100644 --- a/integration-tests/tests/fork.rs +++ b/integration-tests/tests/fork.rs @@ -687,13 +687,13 @@ async fn fork_self(deps: &EnvBasedTestDependencies, _tracing: &Tracing) -> anyho async move { let route = Router::new() .route( - "/fork-test/step1/:name/:input", + "/fork-test/step1/{name}/{input}", get(move |args: Path<(String, String)>| async move { Json(format!("{}-{}", args.0 .0, args.0 .1)) }), ) .route( - "/fork-test/step2/:name/:fork/:phantom_id", + "/fork-test/step2/{name}/{fork}/{phantom_id}", get(move |args: Path<(String, String, String)>| { let fork_phantom_id_tx = fork_phantom_id_tx.clone(); async move { diff --git a/integration-tests/tests/lib.rs b/integration-tests/tests/lib.rs index f37f92a8bc..f837127bb2 100644 --- a/integration-tests/tests/lib.rs +++ b/integration-tests/tests/lib.rs @@ -31,8 +31,9 @@ pub struct Tracing; impl Tracing { pub fn init() -> Self { + unsafe { backtrace_on_stack_overflow::enable() }; init_tracing_with_default_debug_env_filter( - &TracingConfig::test("integration-tests").with_env_overrides(), + &TracingConfig::test_pretty_without_time("integration-tests").with_env_overrides(), ); Self } diff --git a/integration-tests/tests/sharding.rs b/integration-tests/tests/sharding.rs index 17b794dc95..8242534222 100644 --- a/integration-tests/tests/sharding.rs +++ b/integration-tests/tests/sharding.rs @@ -40,6 +40,7 @@ mod tests { impl Tracing { pub fn init() -> Self { + unsafe { backtrace_on_stack_overflow::enable() }; init_tracing_with_default_debug_env_filter( &TracingConfig::test("sharding-tests").with_env_overrides(), ); diff --git a/integration-tests/tests/worker.rs b/integration-tests/tests/worker.rs index c0befa37c9..a99d7e4330 100644 --- a/integration-tests/tests/worker.rs +++ b/integration-tests/tests/worker.rs @@ -1612,6 +1612,8 @@ async fn agent_update_constructor_signature( .store() .await?; + let first_deployment_revision = user.get_last_deployment_revision(&env.id)?; + let agent1_id = agent_id!("counter-agent", "agent1"); let agent1 = user.start_agent(&component.id, agent1_id.clone()).await?; let result1a = user @@ -1651,7 +1653,13 @@ async fn agent_update_constructor_signature( // Still able to call both agents let result3a = user - .invoke_and_await_agent(&component, &agent1_id, "increment", data_value!()) + .invoke_and_await_agent_at_deployment( + &component, + &agent1_id, + first_deployment_revision, + "increment", + data_value!(), + ) .await?; let result4a = user @@ -1663,7 +1671,13 @@ async fn agent_update_constructor_signature( // Still able to do RPC let result3b = user - .invoke_and_await_agent(&component, &old_singleton_id, "call", data_value!("agent1")) + .invoke_and_await_agent_at_deployment( + &component, + &old_singleton_id, + first_deployment_revision, + "call", + data_value!("agent1"), + ) .await?; assert_eq!(result3b.into_return_value(), Some(Value::U32(4))); diff --git a/openapi/golem-service.yaml b/openapi/golem-service.yaml index 5782bfe8d9..339a11c143 100644 --- a/openapi/golem-service.yaml +++ b/openapi/golem-service.yaml @@ -7812,6 +7812,9 @@ components: format: date-time idempotencyKey: type: string + deploymentRevision: + type: integer + format: int64 required: - appName - envName diff --git a/openapi/golem-worker-service.yaml b/openapi/golem-worker-service.yaml index 3d2cf38e73..5882f1a04a 100644 --- a/openapi/golem-worker-service.yaml +++ b/openapi/golem-worker-service.yaml @@ -1754,6 +1754,9 @@ components: format: date-time idempotencyKey: type: string + deploymentRevision: + type: integer + format: int64 AgentInvocationResult: type: object title: AgentInvocationResult diff --git a/sdks/rust/golem-rust/src/value_and_type/mod.rs b/sdks/rust/golem-rust/src/value_and_type/mod.rs index a68c5526c2..b63564f3e8 100644 --- a/sdks/rust/golem-rust/src/value_and_type/mod.rs +++ b/sdks/rust/golem-rust/src/value_and_type/mod.rs @@ -1778,8 +1778,8 @@ impl IntoValue for AgentId { fn add_to_type_builder(builder: T) -> T::Result { let builder = builder.record(Some("agent-id".to_string()), Some("golem".to_string())); - let builder = ComponentId::add_to_type_builder(builder.field("component_id")); - let builder = String::add_to_type_builder(builder.field("agent_id")); + let builder = ComponentId::add_to_type_builder(builder.field("component-id")); + let builder = String::add_to_type_builder(builder.field("agent-id")); builder.finish() } } diff --git a/test-components/golem_it_agent_rpc_rust_release.wasm b/test-components/golem_it_agent_rpc_rust_release.wasm index 996ced33ba..ea9e7f3280 100644 Binary files a/test-components/golem_it_agent_rpc_rust_release.wasm and b/test-components/golem_it_agent_rpc_rust_release.wasm differ diff --git a/test-components/golem_it_host_api_tests_release.wasm b/test-components/golem_it_host_api_tests_release.wasm index f07f334aae..109169b491 100644 Binary files a/test-components/golem_it_host_api_tests_release.wasm and b/test-components/golem_it_host_api_tests_release.wasm differ