gRPC and OpenAPI Integration

[jdegoes]

This specification defines a new feature whereby Golem workers will be able to interact with gRPC and OpenAPI services in a type-safe way, similar to how Golem workers interact with other workers in a type-safe way.

This feature will greatly expand the number of applications that can be built and deployed on Golem and simplify usability of existing Golem applications currently written using lower-level libraries, such as HTTP client libraries.

When fully implemented, Golem users will be able to simply add new gRPC and OpenAPI dependencies, and then immediately begin interacting with them in a type-safe way, through automatically generated WIT definitions that structure the interacts with the remote services, and through automatically and dynamically implemented stubs that provide implementation of these WIT interfaces and which communicate to the underlying gRPC / OpenAPI dependencies.

Background

In Golem's current worker-to-worker communication, WIT interfaces describe the public interface of components, with wasm-rpc handling the transformation of these interfaces to support the addressing of specific workers. For external service communication, we need similar but stateless transformations from Protobuf and OpenAPI specifications to WIT.

The current process for worker-to-worker communication is as follows:

1. The user is writing a component A, which depends on a component B. The user adds component B as a dependency of component A, by editing the Golem application manifest file (a YAML file read by golem-cli).
2. The public interface of component B is described by WIT (WASM Interface Types).
3. golem-cli, using the wasm-rpc project, can generate a transformed WIT file that is the stateful version of the original WIT exported by component B. This stateful aspect allows developers to target specific workers, which is necessary for worker-to-worker communication--but which is not generally necessary for gRPC or OpenAPI, because these protocols are stateless.
4. wasm-rpc generates and compiles stubs for RPC, which implement the transformed (stateful) WIT, and require Golem host functions for performing the actual RPC. Note that this step is in the process of being simplified, and shortly, the stubs will be dynamically added on the server side, in the worker-executor, using a custom linker, and this model of dynamic stub generation must be used for both gRPC and OpenAPI support.
5. The user writes component A, invoking the transformed WIT, and linking against the stubs, to eliminate the requirement on the transformed WIT. This step is disappearing as worker-to-worker transitions to the server-side, dynamically generated stubs; it will not be necessary for gRPC or OpenAPI support.
6. The worker-executor provides the Golem host functions necessary for RPC.

The process for supporting gRPC and OpenAPI will proceed in a similar fashion:

1. The user is writing a component A, which depends on gRPC defined by Protobuf B and OpenAPI API defined by schema C. The user adds a reference to B as one dependency of type grpc, and a reference to C as another dependency of type openapi, by editing the Golem application manifest file.
2. golem-cli, using an extended and enhanced wasm-rpc , generates a WIT file for B, corresponding to an idiomatic encoding of the gRPC services into WIT; and another WIT file for C, corresponding to an idiomatic encoding of the OpenAPI API into WIT.
3. The user writes component A, invoking the generated WIT for B and C as necessary to access the functionality of API B and API C.
4. The worker-executor, when executing instances of component A, dynamically stubs the WIT interfaces and links them to the instances, allowing instances to interact with the gRPC service and the OpenAPI API.

In supporting both gRPC and OpenAPI, there are several major challenges:

1. For an arbitrary gRPC or OpenAPI spec, programmatically generating an equivalent WIT that is "WIT idiomatic", and which does not look or feel like it was programmatically generated. This is much more challenging for OpenAPI schemas than it is for Protobuf.
2. Dynamically adding stubs in the worker-executor corresponding to the generated WIT. These stubs will be implemented in Rust and programmatically execute gRPC and OpenAPI invocations using appropriate metadata.
3. Ensuring durable execution of the API calls at the level of the individual calls, using the same mechanisms that the worker-executor is already using to make WASI calls durable (essentially, branching off record/play back mode and reading from / writing to the oplog).

Beyond these major challenges, there are several other essential features of the implementation to consider:

• Capturing the protobuf and OpenAPI schemas during component creation and update. These will be captured by golem-cli, the command-line interface for Golem.
• Modifying the Golem application manifest file (YAML) parser and schema to accept new types of dependencies, including the protobuf files and OpenAPI files. Currently, the only type of dependency supported by the parser and schema is wasm-rpc, for describing worker-to-worker communication.
• Modifying the component creation and update REST APIs to accept information on the new types of dependencies.
• Storing the protobufs and OpenAPI files during component creation and update; or more precisely, storing a structured representation that has enough information for the worker-executor to generate and add the dynamic stubs.
• Accessing the protobuf and OpenAPI structured information from inside the worker-executor, so when a worker is launched, they have what they need to dynamically add the gRPC / HTTP stubs.

Input Formats

• Protocol Buffers v3 (proto3) with gRPC service definitions
• OpenAPI 3.0.x YAML/JSON specifications

Common Requirements

Package Translation

• Must generate WIT package name and version
• For gRPC: Use proto package name
• For OpenAPI: Use sanitized info.title
• Version must come from:
  • gRPC: Configuration input
  • OpenAPI: info.version

Type Mappings

Protobuf and OpenAPI types must be mapped into their most precise WIT types. A sketch of a few possible mappings for Protobuf is shown below:

oneof         -> variant
message       -> record
string        -> string 
int32         -> s32 
int64         -> s64 
uint32        -> u32 
uint64        -> u64 
float         -> float32 
double        -> float64 
bool/boolean  -> bool 
repeated T    -> list<T> 
optional T    -> option<T>

Note that OpenAPI schemas may embed JSON schemas, which can contain patterns. To the extent possible and reasonable, patterns should be converted into validations that occur in the stubs, to ensure that where possible, local and highly descriptive errors are produced--rather than relying on a remote service to produce a useful error in response to some kind of schema validation failure.

Error Handling

The following sketch of an error type could inform design of a generalized error type. The actual error type used in the implementation must be at least as capable and as expressive as the provided one.

variant error {  
  unauthorized { message: string }, 
  not-found { resource: string, id: string }, 
  validation-error { fields: list<string> }, 
  rate-limited { retry-after: u32 }, 
  server-error { message: string }, 
}

Authentication Types

The following sketches of authentication types could inform design of generalized authentication types. The actual authentication types used in the implementation must be at least as capable and as expressive as these sketches:

record bearer-auth {
  token: string, scheme: string, 
}
record basic-auth {
  username: string, password: string, 
}
record api-key-auth {
  key: string, 
}

gRPC-Specific Requirements

Message Translation

• Each message type must become a WIT record
• Names must be converted to kebab-case
• Must preserve all fields and their relationships
• Nested messages must become separate records

Service Translation

• Each service must become a WIT interface
• Each RPC method must become a WIT function
• Must return result<response-type, error> for WIT-idiomatic error handling

Example:

message  GetUserRequest  {   string user_id =  1; }

Must become:

record get-user-request {  user-id: string, }

The numerical order of fields in the protobuf MUST correspond to the linear order of fields in WIT.

OpenAPI-Specific Requirements

Schema Translation

• Each components.schema must become a WIT record
• Required fields must be non-optional
• Must preserve all relationships between types

Inline Type Translation

• All anonymous/inline schema definitions must be converted to named WIT types
• Names could be synthesized systematically using the following rules (or similar):
  1. For request bodies: {path}-{method}-request-body
  2. For response bodies: {path}-{method}-response-body
  3. For array items: {parent-type}-item
  4. For nested objects: {parent-type}-{field-name}
  5. For parameters: {path}-{method}-params

Example:

paths:
  /users: 
    post: 
      requestBody: 
        content: 
          application/json: 
            schema: 
              type: object
              properties: 
                name: 
                  type: string 
                addresses: 
                  type: array 
                items: 
                  type: object
                  properties: 
                    street: string 
                    city: string

Could generate something like:

record users-post-request-body {
  name: string,
  addresses: list<users-post-request-body-addresses-item>, 
}
record users-post-request-body-addresses-item {
  street: string, 
  city: string,
}

The exact naming scheme may vary, but must be:

• Deterministic
• Generate valid WIT identifiers
• Maintain clear relationship to source structure
• Avoid name collisions
• Create traceable mappings for worker-executor's use

Path Translation

• Must support paths to any number of levels deep
• Each base resource path must generate interface(s)
• Must group related operations logically
• Must handle path/query parameters correctly

Header Handling

Must specially handle these headers:

Authorization     -> auth field in request 
ETag              -> version field in response 
If-Match          -> expected-version in request 
Last-Modified     -> last-updated in response

All other headers must be mechanically translated to kebab-case option fields.

Resource Function Generation Rules

The following sketches of resource interfaces could inform design of generalized resource handling for REST APIs. The actual interfaces used in the implementation must be at least as capable and as expressive as these sketches:

interface resource-collection {
  list: func(params: list-params) -> result<list-response, error>; 
  create: func(params: create-params) -> result<item, error>; 
}
interface resource-item {
  get: func(id: string) -> result<item, error>; 
  update: func(id: string, params: update-params) -> result<item, error>; 
  delete: func(id: string) -> result<unit, error>; 
}

Naming Conventions

Conflict Resolution

Potential algorithm for resolving name conflicts:

1. Appending type suffix (_record, _params, _result, etc.)
2. If still conflicting, error out requiring manual resolution

Generated Names

• Must be valid WIT identifiers
• Must use kebab-case
• Should not exceed 64 characters
• Must prefix reserved words with %

Dynamic Stub Requirements

Dynamic stubs must be added to the worker-executor for all of the gRPC and OpenAPI dependencies. These stubs must, of course, match the type signatures of the generated WITs.

Durability must be ensured using the same mechanism that is used to provide durability for WASI inside the worker-executor (namely, direct interaction with the mode, whether record or playback, and the worker oplog).

Testing Requirements

Automated tests are required at the following levels:

• Unit. Unit testing should ensure correctness of each part of the system developed.
• Integration. Integration should ensure that when the components are composed (e.g. golem-cli with wasm-rpc), they function together as specified.
• System. End-to-end system tests should verify the correctness of the entire system, all the way from golem-cli to the worker-executor.

In particular, there must exist automated tests against REAL gRPC and OpenAPI APIs, which verify both the transformed WIT that is generated, and also which actually invoke the APIs from within a test worker, to verify that the dynamically added stubs in the worker-executor are working correctly. Durability tests must be added to verify durability of any gRPC or HTTP interacts that happen from within the generated stubs.

The solution will be tested against the following resources:

OpenAPI:

• https://github.com/cloudflare/api-schemas
• https://github.com/openai/openai-openapi
• https://github.com/github/rest-api-description

gRPC:

• https://grpcb.in/
• https://cloud.google.com/text-to-speech/docs/reference/rpc

We have several additional OpenAPI and gRPC resources we will be "surprise testing" against to ensure sufficient scope and quality.

Acceptance

We will accept the first solution which, in our sole opinion and estimation:

1. Demonstrates deep understanding of the purpose of this feature, even, if necessary, extending or changing parts of the specification in order to achieve the end goal in a way which is compatible with how worker-to-worker communication currently works, which is highly friendly to developers, and which does not sacrifice type safety, performance, or other characteristics important for production adoption of the feature.
2. Demonstrates high-quality, best practices in the Rust implementation and associated tests. Code should be modular, well-organized, strongly typed, free of duplication, and demonstrate the best of what Rust has to offer; with consistent conventions as utilized by the best parts of the existing code base.
3. Demonstrates a systematic and well-thought out approach to testing, having ample unit tests, integration tests, and system tests, which collectively verify end-to-end usage scenarios, including interacting with real-world gRPC and OpenAPI APIs from within Golem workers, in a type-safe way.
4. Has been updated with latest changes coming from our head branch.
5. Is repeatably passing the CI build.
6. Is sufficiently well-documented to enable other developers to build off the solution, and end-developers to use the solution in their own Golem applications.

[jdegoes]

Example for OpenAPI

openapi: '3.0.3'
info:
  title: Todo REST API
  version: '1.0.0'
  description: A RESTful API for managing todo items

servers:
  - url: /api/v1
    description: Base API path

components:
  schemas:
    Todo:
      type: object
      properties:
        id:
          type: string
          format: uuid
          readOnly: true
        title:
          type: string
          minLength: 1
          maxLength: 200
        description:
          type: string
          maxLength: 2000
        completed:
          type: boolean
          default: false
        dueDate:
          type: string
          format: date-time
        userId:
          type: string
        createdAt:
          type: string
          format: date-time
          readOnly: true
        updatedAt:
          type: string
          format: date-time
          readOnly: true
      required:
        - id
        - title
        - completed
        - userId
        - createdAt
        - updatedAt

    TodoCreate:
      type: object
      properties:
        title:
          type: string
          minLength: 1
          maxLength: 200
        description:
          type: string
          maxLength: 2000
        dueDate:
          type: string
          format: date-time
        userId:
          type: string
      required:
        - title
        - userId

    TodoUpdate:
      type: object
      properties:
        title:
          type: string
          minLength: 1
          maxLength: 200
        description:
          type: string
          maxLength: 2000
        completed:
          type: boolean
        dueDate:
          type: string
          format: date-time
      minProperties: 1

    TodoList:
      type: object
      properties:
        data:
          type: array
          items:
            $ref: '#/components/schemas/Todo'
        metadata:
          type: object
          properties:
            total:
              type: integer
              minimum: 0
            limit:
              type: integer
              minimum: 1
            offset:
              type: integer
              minimum: 0
          required:
            - total
            - limit
            - offset
      required:
        - data
        - metadata

    TodoResponse:
      type: object
      properties:
        data:
          $ref: '#/components/schemas/Todo'
      required:
        - data

    Error:
      type: object
      properties:
        error:
          type: object
          properties:
            code:
              type: string
              enum: 
                - VALIDATION_ERROR
                - UNAUTHORIZED
                - FORBIDDEN
                - NOT_FOUND
                - RATE_LIMIT_EXCEEDED
                - INTERNAL_ERROR
            message:
              type: string
            details:
              type: array
              items:
                type: object
                properties:
                  field:
                    type: string
                  message:
                    type: string
                required:
                  - field
                  - message
          required:
            - code
            - message

  parameters:
    TodoId:
      name: todoId
      in: path
      required: true
      schema:
        type: string
        format: uuid
    UserId:
      name: userId
      in: query
      required: true
      schema:
        type: string
    Status:
      name: status
      in: query
      schema:
        type: string
        enum: [active, completed]
    Limit:
      name: limit
      in: query
      schema:
        type: integer
        minimum: 1
        maximum: 100
        default: 50
    Offset:
      name: offset
      in: query
      schema:
        type: integer
        minimum: 0
        default: 0
    SortBy:
      name: sortBy
      in: query
      schema:
        type: string
        enum: [createdAt, dueDate]
        default: createdAt
    SortOrder:
      name: sortOrder
      in: query
      schema:
        type: string
        enum: [asc, desc]
        default: desc

  securitySchemes:
    bearerAuth:
      type: http
      scheme: bearer
      bearerFormat: JWT

security:
  - bearerAuth: []

paths:
  /todos:
    get:
      summary: List todos
      parameters:
        - $ref: '#/components/parameters/UserId'
        - $ref: '#/components/parameters/Status'
        - $ref: '#/components/parameters/Limit'
        - $ref: '#/components/parameters/Offset'
        - $ref: '#/components/parameters/SortBy'
        - $ref: '#/components/parameters/SortOrder'
      responses:
        '200':
          description: Successfully retrieved todos
          headers:
            ETag:
              schema:
                type: string
            Last-Modified:
              schema:
                type: string
            X-Request-ID:
              schema:
                type: string
            X-RateLimit-Limit:
              schema:
                type: integer
            X-RateLimit-Remaining:
              schema:
                type: integer
            X-RateLimit-Reset:
              schema:
                type: integer
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/TodoList'
        '400':
          description: Invalid parameters
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Error'
        '401':
          description: Unauthorized
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Error'
        '429':
          description: Too many requests
          headers:
            Retry-After:
              schema:
                type: integer
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Error'
        '500':
          description: Internal server error
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Error'

    post:
      summary: Create a new todo
      requestBody:
        required: true
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/TodoCreate'
      responses:
        '201':
          description: Todo created successfully
          headers:
            Location:
              schema:
                type: string
                format: uri
            ETag:
              schema:
                type: string
            X-Request-ID:
              schema:
                type: string
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/TodoResponse'
        '400':
          description: Invalid input
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Error'
        '401':
          description: Unauthorized
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Error'
        '500':
          description: Internal server error
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Error'

  /todos/{todoId}:
    parameters:
      - $ref: '#/components/parameters/TodoId'
    
    get:
      summary: Get a specific todo
      responses:
        '200':
          description: Successfully retrieved todo
          headers:
            ETag:
              schema:
                type: string
            Last-Modified:
              schema:
                type: string
            X-Request-ID:
              schema:
                type: string
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/TodoResponse'
        '401':
          description: Unauthorized
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Error'
        '404':
          description: Todo not found
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Error'
        '500':
          description: Internal server error
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Error'

    patch:
      summary: Update a todo
      requestBody:
        required: true
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/TodoUpdate'
      responses:
        '200':
          description: Todo updated successfully
          headers:
            ETag:
              schema:
                type: string
            X-Request-ID:
              schema:
                type: string
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/TodoResponse'
        '400':
          description: Invalid input
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Error'
        '401':
          description: Unauthorized
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Error'
        '404':
          description: Todo not found
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Error'
        '500':
          description: Internal server error
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Error'

    delete:
      summary: Delete a todo
      responses:
        '204':
          description: Todo deleted successfully
          headers:
            X-Request-ID:
              schema:
                type: string
        '401':
          description: Unauthorized
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Error'
        '404':
          description: Todo not found
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Error'
        '500':
          description: Internal server error
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Error'

package api:todos@1.0.0;

// Common types used across the API
interface types {
    // Authentication types derived from OpenAPI security schemes
    record bearer-token {
        token: string,
    }

    // Core resource type
    record todo {
        id: string,
        title: string,
        description: option<string>,
        completed: bool,
        due-date: option<string>,
        user-id: string,
        created-at: string,
        updated-at: string,
    }

    variant error {
        unauthorized,
        not-found,
        validation-error(list<string>),
        rate-limited { retry-after: u32 },
        server-error,
    }
}

interface todos-collection {
    use types.{todo, bearer-token, error};

    // Request/response types with domain-appropriate fields
    record list-request {
        auth: bearer-token,
        user-id: string,
        status: option<string>,
        limit: option<u32>,
        %offset: option<u32>,
    }

    record list-response {
        items: list<todo>,
        total: u32,
        limit: u32,
        %offset: u32,
        version: string,  // From ETag
        last-updated: option<string>,  // From Last-Modified
    }

    record create-request {
        auth: bearer-token,
        title: string,
        description: option<string>,
        due-date: option<string>,
        user-id: string,
    }

    // Note: Response includes versioning info directly in return type
    list: func(request: list-request) -> result<list-response, error>;
    
    create: func(request: create-request) -> result<todo, error>;
}

interface todos-resource {
    use types.{todo, bearer-token, error};

    record update-request {
        auth: bearer-token,
        title: option<string>,
        description: option<string>,
        completed: option<bool>,
        due-date: option<string>,
        expected-version: option<string>,  // If-Match header
    }

    get: func(id: string, auth: bearer-token) -> result<todo, error>;
    
    update: func(id: string, request: update-request) -> result<todo, error>;
    
    delete: func(id: string, auth: bearer-token) -> result<unit, error>;
}

world todos-api {
    export todos-collection;
    export todos-resource;
}

[jdegoes]

Example for gRPC

syntax = "proto3";
package core.todo.v1;

option go_package = "github.com/koblas/grpc-todo/protos";

message TodoListRequest { string user_id = 1; }

message TodoAddRequest {
  string user_id = 1;
  string task = 2;
}

message TodoDeleteRequest {
  string user_id = 1;
  string id = 2;
}

message TodoObject {
  string user_id = 1;
  string id = 2;
  string task = 3;
}

message TodoChangeEvent {
  string idemponcy_id = 1;
  TodoObject current = 3;
  TodoObject original = 4;
}

message TodoAddResponse { TodoObject todo = 1; }
message TodoListResponse { repeated TodoObject todos = 1; }
message TodoDeleteResponse { string message = 1; }

service TodoService {
  rpc TodoAdd(TodoAddRequest) returns (TodoAddResponse);
  rpc TodoDelete(TodoDeleteRequest) returns (TodoDeleteResponse);
  rpc TodoList(TodoListRequest) returns (TodoListResponse);
}

package core:todo@1.0.0;

interface types {
    record todo-object {
        user-id: string,
        id: string,
        task: string,
    }

    record todo-change-event {
        idempotency-id: string,
        current: todo-object,
        original: todo-object,
    }

    record todo-list-request {
        user-id: string,
    }

    record todo-add-request {
        user-id: string,
        task: string,
    }

    record todo-delete-request {
        user-id: string,
        id: string,
    }

    record todo-add-response {
        todo: todo-object,
    }

    record todo-list-response {
        todos: list<todo-object>,
    }

    record todo-delete-response {
        message: string,
    }

    variant todo-error {
        not-found,
        unauthorized,
        invalid-input,
        internal-error,
    }
}

interface todo-service {
    use types.{
        todo-add-request, 
        todo-add-response,
        todo-delete-request,
        todo-delete-response,
        todo-list-request,
        todo-list-response,
        todo-error,
    };

    todo-add: func(request: todo-add-request) -> result<todo-add-response, todo-error>;
    todo-delete: func(request: todo-delete-request) -> result<todo-delete-response, todo-error>;
    todo-list: func(request: todo-list-request) -> result<todo-list-response, todo-error>;
}

[algora-pbc]

💎 $10,000 bounty • Golem Cloud

Steps to solve:

1. Start working: Comment /attempt #1201 with your implementation plan
2. Submit work: Create a pull request including /claim #1201 in the PR body to claim the bounty
3. Receive payment: 100% of the bounty is received 2-5 days post-reward. Make sure you are eligible for payouts

❗ Important guidelines:

• To claim a bounty, you need to provide a short demo video of your changes in your pull request
• If anything is unclear, ask for clarification before starting as this will help avoid potential rework
• Low quality AI PRs will not receive review and will be closed
• Do not ask to be assigned unless you've contributed before

Thank you for contributing to golemcloud/golem!

Attempt	Started (UTC)	Solution	Actions
🟢 @RafaelJohn9	Jan 01, 2025, 07:37:56 PM	WIP
🟢 @gerred	Jan 02, 2025, 11:33:18 PM	WIP
🟢 @SaikiranSurapalli17	Aug 06, 2025, 05:25:17 PM	WIP
🟢 @abhishek12201	Mar 07, 2025, 04:08:38 PM	WIP
🔴 @SAIKIRANSURAPALLI	Jan 08, 2025, 10:15:23 AM	WIP
🟢 @Abdul-Samad-75	Feb 10, 2025, 09:36:33 AM	WIP
🟢 @fjkiani	Aug 13, 2025, 04:26:54 AM	#1906	Reward
🟢 @Rumixyz	Feb 14, 2025, 11:34:37 AM	WIP
🟢 @zelosleone	Dec 18, 2024, 08:49:27 PM	WIP
🟢 @uurl	Dec 18, 2024, 11:52:46 PM	WIP
🔴 @amankrx	Dec 19, 2024, 04:23:05 AM	WIP
🟢 @sreehariX	Dec 19, 2024, 10:30:38 AM	WIP
🟢 @DevendraSingh-1	Dec 19, 2024, 04:09:51 PM	WIP
🟢 @ManasMahanand	Dec 20, 2024, 03:56:18 AM	WIP
🟢 @promisingcoder	Jun 26, 2025, 03:29:15 PM	WIP
🟢 @	Jan 27, 2025, 01:58:13 PM	WIP
🟢 @Nanashi-lab	Mar 28, 2025, 05:40:43 AM	WIP
🟢 @webbdays	Mar 29, 2025, 03:45:48 AM	WIP
🟢 @itsparser	Jan 30, 2025, 08:36:10 AM	WIP
🟢 @kapilreddy	Jan 30, 2025, 09:42:30 AM	WIP
🟢 @Aditya-PS-05	Jul 30, 2025, 09:48:20 AM	WIP

[zelosleone]

/attempt #1201

Options

• Cancel my attempt

[zelosleone]

By the way, I am already finishing the OpenAPI Export Integration from #1178 should be finished within few hours...

[radumarias]

Would you be interested In something like gRPC (because it's performant and eficient) + Cap'n Proto (for zero copy as it doesn't have any serialization) + http/3 (because it's QUIC)?

https://docs.google.com/document/d/1Ru5UlOz-4dz9ors3FKEg2Ac-KxKVqI_wR0EitECp-mo/edit?usp=drivesdk

You could use protobuf generators to obtain structs and classes in supported languages, and if we see the other end using protobuf we will serialize as such and vice- versa. Also we will have converters from protobuf to Cap’n a Proto end vice versa. So anyone having gRPC services could easily migrate to this. That could do it in phases as we will be interoperable with protobuf.

Using HTTP/3 with QUIC for gRPC with Cap’n Proto and tonic crate brings several potential advantages, especially in scenarios requiring high-performance data exchange and improved reliability over modern networks.

Motivation and key advantages:
Challenge: there are no well-known production-ready implementations for HTTP/3 in Rust
Faster transfers
Faster connection time
Reduced latency
Reduced downtime
High throughput
Multiplexing without head-of-line blocking
Particularly beneficial for short-lived connections or high-frequency RPC calls
QUIC retransmits lost packets faster than TCP, reducing delays
QUIC allows connections to migrate between network interfaces (e.g., Wi-Fi to cellular) without disconnecting
QUIC includes TLS 1.3 at the transport layer, so all HTTP/3 connections are encrypted by design
HTTP/3 allows prioritizing certain streams, optimizing resource utilization for high-performance data pipelines
QUIC supports an unlimited number of concurrent streams without degrading performance
Cap’n Proto offers zero-copy as it doesn’t have any serialization

[uurl]

/attempt #1201

Options

• Cancel my attempt

[amankrx]

/attempt #1201

Options

• Cancel my attempt

[vigoo]

Please do not start working on the worker executor part of this until #1150 is done (which is work in progress at the moment).
There are many other parts of the tasks that can be worked on in the mean time.

[ilyakharlamov]

Note that there is no real need for a separate OpenAPI definition, as gRPC supports standard REST-JSON transcoding defined within gRPC itself.
So the universal gRPC-OpenAPI definition would look like this:

service TodoService {
  rpc TodoAdd(TodoAddRequest) returns (TodoAddResponse) {
      option (google.api.http) = {
      post: "/components/schemas/Todo/{user_id}"
    };
  };
  rpc TodoDelete(TodoDeleteRequest) returns (TodoDeleteResponse) {
      option (google.api.http) = {
      delete: "/components/schemas/Todo/{user_id}"
    };
  };
  rpc TodoList(TodoListRequest) returns (TodoListResponse) {
      option (google.api.http) = {
      get: "/components/schemas/Todo/{user_id}"
    };
  };
}

This approach is supported by many servers, such as Armeria, Microsoft ASP.NET, Google Cloud and gRPC-gateway

This is something that can be considered.

[sreehariX]

/attempt #1201

Options

• Cancel my attempt

[vigoo]

@ilyakharlamov the idea is that you have an OpenAPI spec for something you want to call from a Golem worker, and you get a generated typed-safe way to interact with it, just by providing this spec (which may be originated from a 3rd party). The REST-JSON mapping in gRPC in your example is not helping with this - someone would have to manually create this gRPC definition from the OpenAPI spec. It's actually making more work, if we want to support this as well, which we did not consider earlier. (As it would be a third way to describe a target API)

[dev3ndr4]

Implementation Plan for gRPC and OpenAPI Integration in Golem

Objective:

Integrate gRPC and OpenAPI support into Golem, enabling type-safe interactions between Golem workers and external services via Protobuf and OpenAPI schemas. The implementation will allow users to easily add gRPC and OpenAPI dependencies and interact with them in a type-safe manner through generated WIT (WASM Interface Types) definitions and dynamic stubs.

---

Key Steps in Implementation:

1. Extend Golem CLI for New Dependency Types:

   • Modify Golem CLI to support adding gRPC (Protobuf files) and OpenAPI (YAML/JSON schemas) dependencies to the Golem application manifest file.
   • Allow users to specify these dependencies for worker components.

2. Protobuf-to-WIT Translator:

   • Implement a tool to convert gRPC Protobuf service definitions and message types into WIT type definitions.
   • Translate Protobuf message types into WIT record, oneof types into WIT variant, and repeated fields into WIT list.

3. OpenAPI-to-WIT Translator:

   • Develop a parser to convert OpenAPI schemas (3.0.x) into WIT types, including request bodies, response bodies, and parameters.
   • Inline types in OpenAPI schemas will be converted into separate named WIT types to preserve structure.

4. Dynamic Stub Generation in Worker-Executor:

   • Modify the worker-executor to dynamically generate and inject stubs for gRPC and OpenAPI services at runtime.
   • These stubs will map to the generated WIT interfaces and allow workers to make API calls to remote services using appropriate gRPC and HTTP invocations.

5. Authentication and Error Handling:

   • Implement mechanisms to handle authentication types (Bearer, Basic, API Key) within WIT.
   • Create a unified error-handling system to map remote service errors (e.g., unauthorized, validation errors) to WIT-idiomatic error types.

---

Expected Outcome:

By the end of this implementation, Golem workers will be able to interact with remote gRPC and OpenAPI services in a type-safe way using automatically generated WIT definitions and dynamically linked stubs.

---

This is the high-level overview of the implementation. If approved, I will proceed with the detailed design and development.

---

/attempt #1201

Options

• Cancel my attempt

[ManasMahanand]

/attempt #1201

Options

• Cancel my attempt