gRPC and OpenAPI Integration

[jdegoes]

This specification defines a new feature whereby Golem workers will be able to interact with gRPC and OpenAPI services in a type-safe way, similar to how Golem workers interact with other workers in a type-safe way.

This feature will greatly expand the number of applications that can be built and deployed on Golem and simplify usability of existing Golem applications currently written using lower-level libraries, such as HTTP client libraries.

When fully implemented, Golem users will be able to simply add new gRPC and OpenAPI dependencies, and then immediately begin interacting with them in a type-safe way, through automatically generated WIT definitions that structure the interacts with the remote services, and through automatically and dynamically implemented stubs that provide implementation of these WIT interfaces and which communicate to the underlying gRPC / OpenAPI dependencies.

Background

In Golem's current worker-to-worker communication, WIT interfaces describe the public interface of components, with wasm-rpc handling the transformation of these interfaces to support the addressing of specific workers. For external service communication, we need similar but stateless transformations from Protobuf and OpenAPI specifications to WIT.

The current process for worker-to-worker communication is as follows:

1. The user is writing a component A, which depends on a component B. The user adds component B as a dependency of component A, by editing the Golem application manifest file (a YAML file read by golem-cli).
2. The public interface of component B is described by WIT (WASM Interface Types).
3. golem-cli, using the wasm-rpc project, can generate a transformed WIT file that is the stateful version of the original WIT exported by component B. This stateful aspect allows developers to target specific workers, which is necessary for worker-to-worker communication--but which is not generally necessary for gRPC or OpenAPI, because these protocols are stateless.
4. wasm-rpc generates and compiles stubs for RPC, which implement the transformed (stateful) WIT, and require Golem host functions for performing the actual RPC. Note that this step is in the process of being simplified, and shortly, the stubs will be dynamically added on the server side, in the worker-executor, using a custom linker, and this model of dynamic stub generation must be used for both gRPC and OpenAPI support.
5. The user writes component A, invoking the transformed WIT, and linking against the stubs, to eliminate the requirement on the transformed WIT. This step is disappearing as worker-to-worker transitions to the server-side, dynamically generated stubs; it will not be necessary for gRPC or OpenAPI support.
6. The worker-executor provides the Golem host functions necessary for RPC.

The process for supporting gRPC and OpenAPI will proceed in a similar fashion:

1. The user is writing a component A, which depends on gRPC defined by Protobuf B and OpenAPI API defined by schema C. The user adds a reference to B as one dependency of type grpc, and a reference to C as another dependency of type openapi, by editing the Golem application manifest file.
2. golem-cli, using an extended and enhanced wasm-rpc , generates a WIT file for B, corresponding to an idiomatic encoding of the gRPC services into WIT; and another WIT file for C, corresponding to an idiomatic encoding of the OpenAPI API into WIT.
3. The user writes component A, invoking the generated WIT for B and C as necessary to access the functionality of API B and API C.
4. The worker-executor, when executing instances of component A, dynamically stubs the WIT interfaces and links them to the instances, allowing instances to interact with the gRPC service and the OpenAPI API.

In supporting both gRPC and OpenAPI, there are several major challenges:

1. For an arbitrary gRPC or OpenAPI spec, programmatically generating an equivalent WIT that is "WIT idiomatic", and which does not look or feel like it was programmatically generated. This is much more challenging for OpenAPI schemas than it is for Protobuf.
2. Dynamically adding stubs in the worker-executor corresponding to the generated WIT. These stubs will be implemented in Rust and programmatically execute gRPC and OpenAPI invocations using appropriate metadata.
3. Ensuring durable execution of the API calls at the level of the individual calls, using the same mechanisms that the worker-executor is already using to make WASI calls durable (essentially, branching off record/play back mode and reading from / writing to the oplog).

Beyond these major challenges, there are several other essential features of the implementation to consider:

• Capturing the protobuf and OpenAPI schemas during component creation and update. These will be captured by golem-cli, the command-line interface for Golem.
• Modifying the Golem application manifest file (YAML) parser and schema to accept new types of dependencies, including the protobuf files and OpenAPI files. Currently, the only type of dependency supported by the parser and schema is wasm-rpc, for describing worker-to-worker communication.
• Modifying the component creation and update REST APIs to accept information on the new types of dependencies.
• Storing the protobufs and OpenAPI files during component creation and update; or more precisely, storing a structured representation that has enough information for the worker-executor to generate and add the dynamic stubs.
• Accessing the protobuf and OpenAPI structured information from inside the worker-executor, so when a worker is launched, they have what they need to dynamically add the gRPC / HTTP stubs.

Input Formats

• Protocol Buffers v3 (proto3) with gRPC service definitions
• OpenAPI 3.0.x YAML/JSON specifications

Common Requirements

Package Translation

• Must generate WIT package name and version
• For gRPC: Use proto package name
• For OpenAPI: Use sanitized info.title
• Version must come from:
  • gRPC: Configuration input
  • OpenAPI: info.version

Type Mappings

Protobuf and OpenAPI types must be mapped into their most precise WIT types. A sketch of a few possible mappings for Protobuf is shown below:

oneof         -> variant
message       -> record
string        -> string 
int32         -> s32 
int64         -> s64 
uint32        -> u32 
uint64        -> u64 
float         -> float32 
double        -> float64 
bool/boolean  -> bool 
repeated T    -> list<T> 
optional T    -> option<T>

Note that OpenAPI schemas may embed JSON schemas, which can contain patterns. To the extent possible and reasonable, patterns should be converted into validations that occur in the stubs, to ensure that where possible, local and highly descriptive errors are produced--rather than relying on a remote service to produce a useful error in response to some kind of schema validation failure.

Error Handling

The following sketch of an error type could inform design of a generalized error type. The actual error type used in the implementation must be at least as capable and as expressive as the provided one.

variant error {  
  unauthorized { message: string }, 
  not-found { resource: string, id: string }, 
  validation-error { fields: list<string> }, 
  rate-limited { retry-after: u32 }, 
  server-error { message: string }, 
}

Authentication Types

The following sketches of authentication types could inform design of generalized authentication types. The actual authentication types used in the implementation must be at least as capable and as expressive as these sketches:

record bearer-auth {
  token: string, scheme: string, 
}
record basic-auth {
  username: string, password: string, 
}
record api-key-auth {
  key: string, 
}

gRPC-Specific Requirements

Message Translation

• Each message type must become a WIT record
• Names must be converted to kebab-case
• Must preserve all fields and their relationships
• Nested messages must become separate records

Service Translation

• Each service must become a WIT interface
• Each RPC method must become a WIT function
• Must return result<response-type, error> for WIT-idiomatic error handling

Example:

message  GetUserRequest  {   string user_id =  1; }

Must become:

record get-user-request {  user-id: string, }

The numerical order of fields in the protobuf MUST correspond to the linear order of fields in WIT.

OpenAPI-Specific Requirements

Schema Translation

• Each components.schema must become a WIT record
• Required fields must be non-optional
• Must preserve all relationships between types

Inline Type Translation

• All anonymous/inline schema definitions must be converted to named WIT types
• Names could be synthesized systematically using the following rules (or similar):
  1. For request bodies: {path}-{method}-request-body
  2. For response bodies: {path}-{method}-response-body
  3. For array items: {parent-type}-item
  4. For nested objects: {parent-type}-{field-name}
  5. For parameters: {path}-{method}-params

Example:

paths:
  /users: 
    post: 
      requestBody: 
        content: 
          application/json: 
            schema: 
              type: object
              properties: 
                name: 
                  type: string 
                addresses: 
                  type: array 
                items: 
                  type: object
                  properties: 
                    street: string 
                    city: string

Could generate something like:

record users-post-request-body {
  name: string,
  addresses: list<users-post-request-body-addresses-item>, 
}
record users-post-request-body-addresses-item {
  street: string, 
  city: string,
}

The exact naming scheme may vary, but must be:

• Deterministic
• Generate valid WIT identifiers
• Maintain clear relationship to source structure
• Avoid name collisions
• Create traceable mappings for worker-executor's use

Path Translation

• Must support paths to any number of levels deep
• Each base resource path must generate interface(s)
• Must group related operations logically
• Must handle path/query parameters correctly

Header Handling

Must specially handle these headers:

Authorization     -> auth field in request 
ETag              -> version field in response 
If-Match          -> expected-version in request 
Last-Modified     -> last-updated in response

All other headers must be mechanically translated to kebab-case option fields.

Resource Function Generation Rules

The following sketches of resource interfaces could inform design of generalized resource handling for REST APIs. The actual interfaces used in the implementation must be at least as capable and as expressive as these sketches:

interface resource-collection {
  list: func(params: list-params) -> result<list-response, error>; 
  create: func(params: create-params) -> result<item, error>; 
}
interface resource-item {
  get: func(id: string) -> result<item, error>; 
  update: func(id: string, params: update-params) -> result<item, error>; 
  delete: func(id: string) -> result<unit, error>; 
}

Naming Conventions

Conflict Resolution

Potential algorithm for resolving name conflicts:

1. Appending type suffix (_record, _params, _result, etc.)
2. If still conflicting, error out requiring manual resolution

Generated Names

• Must be valid WIT identifiers
• Must use kebab-case
• Should not exceed 64 characters
• Must prefix reserved words with %

Dynamic Stub Requirements

Dynamic stubs must be added to the worker-executor for all of the gRPC and OpenAPI dependencies. These stubs must, of course, match the type signatures of the generated WITs.

Durability must be ensured using the same mechanism that is used to provide durability for WASI inside the worker-executor (namely, direct interaction with the mode, whether record or playback, and the worker oplog).

Testing Requirements

Automated tests are required at the following levels:

• Unit. Unit testing should ensure correctness of each part of the system developed.
• Integration. Integration should ensure that when the components are composed (e.g. golem-cli with wasm-rpc), they function together as specified.
• System. End-to-end system tests should verify the correctness of the entire system, all the way from golem-cli to the worker-executor.

In particular, there must exist automated tests against REAL gRPC and OpenAPI APIs, which verify both the transformed WIT that is generated, and also which actually invoke the APIs from within a test worker, to verify that the dynamically added stubs in the worker-executor are working correctly. Durability tests must be added to verify durability of any gRPC or HTTP interacts that happen from within the generated stubs.

The solution will be tested against the following resources:

OpenAPI:

• https://github.com/cloudflare/api-schemas
• https://github.com/openai/openai-openapi
• https://github.com/github/rest-api-description

gRPC:

• https://grpcb.in/
• https://cloud.google.com/text-to-speech/docs/reference/rpc

We have several additional OpenAPI and gRPC resources we will be "surprise testing" against to ensure sufficient scope and quality.

Acceptance

We will accept the first solution which, in our sole opinion and estimation:

1. Demonstrates deep understanding of the purpose of this feature, even, if necessary, extending or changing parts of the specification in order to achieve the end goal in a way which is compatible with how worker-to-worker communication currently works, which is highly friendly to developers, and which does not sacrifice type safety, performance, or other characteristics important for production adoption of the feature.
2. Demonstrates high-quality, best practices in the Rust implementation and associated tests. Code should be modular, well-organized, strongly typed, free of duplication, and demonstrate the best of what Rust has to offer; with consistent conventions as utilized by the best parts of the existing code base.
3. Demonstrates a systematic and well-thought out approach to testing, having ample unit tests, integration tests, and system tests, which collectively verify end-to-end usage scenarios, including interacting with real-world gRPC and OpenAPI APIs from within Golem workers, in a type-safe way.
4. Has been updated with latest changes coming from our head branch.
5. Is repeatably passing the CI build.
6. Is sufficiently well-documented to enable other developers to build off the solution, and end-developers to use the solution in their own Golem applications.

[jdegoes]

Example for OpenAPI

openapi: '3.0.3'
info:
  title: Todo REST API
  version: '1.0.0'
  description: A RESTful API for managing todo items

servers:
  - url: /api/v1
    description: Base API path

components:
  schemas:
    Todo:
      type: object
      properties:
        id:
          type: string
          format: uuid
          readOnly: true
        title:
          type: string
          minLength: 1
          maxLength: 200
        description:
          type: string
          maxLength: 2000
        completed:
          type: boolean
          default: false
        dueDate:
          type: string
          format: date-time
        userId:
          type: string
        createdAt:
          type: string
          format: date-time
          readOnly: true
        updatedAt:
          type: string
          format: date-time
          readOnly: true
      required:
        - id
        - title
        - completed
        - userId
        - createdAt
        - updatedAt

    TodoCreate:
      type: object
      properties:
        title:
          type: string
          minLength: 1
          maxLength: 200
        description:
          type: string
          maxLength: 2000
        dueDate:
          type: string
          format: date-time
        userId:
          type: string
      required:
        - title
        - userId

    TodoUpdate:
      type: object
      properties:
        title:
          type: string
          minLength: 1
          maxLength: 200
        description:
          type: string
          maxLength: 2000
        completed:
          type: boolean
        dueDate:
          type: string
          format: date-time
      minProperties: 1

    TodoList:
      type: object
      properties:
        data:
          type: array
          items:
            $ref: '#/components/schemas/Todo'
        metadata:
          type: object
          properties:
            total:
              type: integer
              minimum: 0
            limit:
              type: integer
              minimum: 1
            offset:
              type: integer
              minimum: 0
          required:
            - total
            - limit
            - offset
      required:
        - data
        - metadata

    TodoResponse:
      type: object
      properties:
        data:
          $ref: '#/components/schemas/Todo'
      required:
        - data

    Error:
      type: object
      properties:
        error:
          type: object
          properties:
            code:
              type: string
              enum: 
                - VALIDATION_ERROR
                - UNAUTHORIZED
                - FORBIDDEN
                - NOT_FOUND
                - RATE_LIMIT_EXCEEDED
                - INTERNAL_ERROR
            message:
              type: string
            details:
              type: array
              items:
                type: object
                properties:
                  field:
                    type: string
                  message:
                    type: string
                required:
                  - field
                  - message
          required:
            - code
            - message

  parameters:
    TodoId:
      name: todoId
      in: path
      required: true
      schema:
        type: string
        format: uuid
    UserId:
      name: userId
      in: query
      required: true
      schema:
        type: string
    Status:
      name: status
      in: query
      schema:
        type: string
        enum: [active, completed]
    Limit:
      name: limit
      in: query
      schema:
        type: integer
        minimum: 1
        maximum: 100
        default: 50
    Offset:
      name: offset
      in: query
      schema:
        type: integer
        minimum: 0
        default: 0
    SortBy:
      name: sortBy
      in: query
      schema:
        type: string
        enum: [createdAt, dueDate]
        default: createdAt
    SortOrder:
      name: sortOrder
      in: query
      schema:
        type: string
        enum: [asc, desc]
        default: desc

  securitySchemes:
    bearerAuth:
      type: http
      scheme: bearer
      bearerFormat: JWT

security:
  - bearerAuth: []

paths:
  /todos:
    get:
      summary: List todos
      parameters:
        - $ref: '#/components/parameters/UserId'
        - $ref: '#/components/parameters/Status'
        - $ref: '#/components/parameters/Limit'
        - $ref: '#/components/parameters/Offset'
        - $ref: '#/components/parameters/SortBy'
        - $ref: '#/components/parameters/SortOrder'
      responses:
        '200':
          description: Successfully retrieved todos
          headers:
            ETag:
              schema:
                type: string
            Last-Modified:
              schema:
                type: string
            X-Request-ID:
              schema:
                type: string
            X-RateLimit-Limit:
              schema:
                type: integer
            X-RateLimit-Remaining:
              schema:
                type: integer
            X-RateLimit-Reset:
              schema:
                type: integer
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/TodoList'
        '400':
          description: Invalid parameters
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Error'
        '401':
          description: Unauthorized
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Error'
        '429':
          description: Too many requests
          headers:
            Retry-After:
              schema:
                type: integer
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Error'
        '500':
          description: Internal server error
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Error'

    post:
      summary: Create a new todo
      requestBody:
        required: true
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/TodoCreate'
      responses:
        '201':
          description: Todo created successfully
          headers:
            Location:
              schema:
                type: string
                format: uri
            ETag:
              schema:
                type: string
            X-Request-ID:
              schema:
                type: string
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/TodoResponse'
        '400':
          description: Invalid input
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Error'
        '401':
          description: Unauthorized
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Error'
        '500':
          description: Internal server error
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Error'

  /todos/{todoId}:
    parameters:
      - $ref: '#/components/parameters/TodoId'
    
    get:
      summary: Get a specific todo
      responses:
        '200':
          description: Successfully retrieved todo
          headers:
            ETag:
              schema:
                type: string
            Last-Modified:
              schema:
                type: string
            X-Request-ID:
              schema:
                type: string
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/TodoResponse'
        '401':
          description: Unauthorized
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Error'
        '404':
          description: Todo not found
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Error'
        '500':
          description: Internal server error
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Error'

    patch:
      summary: Update a todo
      requestBody:
        required: true
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/TodoUpdate'
      responses:
        '200':
          description: Todo updated successfully
          headers:
            ETag:
              schema:
                type: string
            X-Request-ID:
              schema:
                type: string
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/TodoResponse'
        '400':
          description: Invalid input
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Error'
        '401':
          description: Unauthorized
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Error'
        '404':
          description: Todo not found
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Error'
        '500':
          description: Internal server error
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Error'

    delete:
      summary: Delete a todo
      responses:
        '204':
          description: Todo deleted successfully
          headers:
            X-Request-ID:
              schema:
                type: string
        '401':
          description: Unauthorized
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Error'
        '404':
          description: Todo not found
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Error'
        '500':
          description: Internal server error
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Error'

package api:todos@1.0.0;

// Common types used across the API
interface types {
    // Authentication types derived from OpenAPI security schemes
    record bearer-token {
        token: string,
    }

    // Core resource type
    record todo {
        id: string,
        title: string,
        description: option<string>,
        completed: bool,
        due-date: option<string>,
        user-id: string,
        created-at: string,
        updated-at: string,
    }

    variant error {
        unauthorized,
        not-found,
        validation-error(list<string>),
        rate-limited { retry-after: u32 },
        server-error,
    }
}

interface todos-collection {
    use types.{todo, bearer-token, error};

    // Request/response types with domain-appropriate fields
    record list-request {
        auth: bearer-token,
        user-id: string,
        status: option<string>,
        limit: option<u32>,
        %offset: option<u32>,
    }

    record list-response {
        items: list<todo>,
        total: u32,
        limit: u32,
        %offset: u32,
        version: string,  // From ETag
        last-updated: option<string>,  // From Last-Modified
    }

    record create-request {
        auth: bearer-token,
        title: string,
        description: option<string>,
        due-date: option<string>,
        user-id: string,
    }

    // Note: Response includes versioning info directly in return type
    list: func(request: list-request) -> result<list-response, error>;
    
    create: func(request: create-request) -> result<todo, error>;
}

interface todos-resource {
    use types.{todo, bearer-token, error};

    record update-request {
        auth: bearer-token,
        title: option<string>,
        description: option<string>,
        completed: option<bool>,
        due-date: option<string>,
        expected-version: option<string>,  // If-Match header
    }

    get: func(id: string, auth: bearer-token) -> result<todo, error>;
    
    update: func(id: string, request: update-request) -> result<todo, error>;
    
    delete: func(id: string, auth: bearer-token) -> result<unit, error>;
}

world todos-api {
    export todos-collection;
    export todos-resource;
}

[jdegoes]

Example for gRPC

syntax = "proto3";
package core.todo.v1;

option go_package = "github.com/koblas/grpc-todo/protos";

message TodoListRequest { string user_id = 1; }

message TodoAddRequest {
  string user_id = 1;
  string task = 2;
}

message TodoDeleteRequest {
  string user_id = 1;
  string id = 2;
}

message TodoObject {
  string user_id = 1;
  string id = 2;
  string task = 3;
}

message TodoChangeEvent {
  string idemponcy_id = 1;
  TodoObject current = 3;
  TodoObject original = 4;
}

message TodoAddResponse { TodoObject todo = 1; }
message TodoListResponse { repeated TodoObject todos = 1; }
message TodoDeleteResponse { string message = 1; }

service TodoService {
  rpc TodoAdd(TodoAddRequest) returns (TodoAddResponse);
  rpc TodoDelete(TodoDeleteRequest) returns (TodoDeleteResponse);
  rpc TodoList(TodoListRequest) returns (TodoListResponse);
}

package core:todo@1.0.0;

interface types {
    record todo-object {
        user-id: string,
        id: string,
        task: string,
    }

    record todo-change-event {
        idempotency-id: string,
        current: todo-object,
        original: todo-object,
    }

    record todo-list-request {
        user-id: string,
    }

    record todo-add-request {
        user-id: string,
        task: string,
    }

    record todo-delete-request {
        user-id: string,
        id: string,
    }

    record todo-add-response {
        todo: todo-object,
    }

    record todo-list-response {
        todos: list<todo-object>,
    }

    record todo-delete-response {
        message: string,
    }

    variant todo-error {
        not-found,
        unauthorized,
        invalid-input,
        internal-error,
    }
}

interface todo-service {
    use types.{
        todo-add-request, 
        todo-add-response,
        todo-delete-request,
        todo-delete-response,
        todo-list-request,
        todo-list-response,
        todo-error,
    };

    todo-add: func(request: todo-add-request) -> result<todo-add-response, todo-error>;
    todo-delete: func(request: todo-delete-request) -> result<todo-delete-response, todo-error>;
    todo-list: func(request: todo-list-request) -> result<todo-list-response, todo-error>;
}

[jdegoes]

/bounty $25000

[algora-pbc]

💎 $25,000 bounty • Golem Cloud

Steps to solve:

1. Start working: Comment /attempt #1201 with your implementation plan
2. Submit work: Create a pull request including /claim #1201 in the PR body to claim the bounty
3. Receive payment: 100% of the bounty is received 2-5 days post-reward. Make sure you are eligible for payouts

Thank you for contributing to golemcloud/golem!

Add a bounty • Share on socials

Attempt	Started (GMT+0)	Solution
🟢 @zelosleone	Dec 18, 2024, 8:49:27 PM	WIP
🟢 @uurl	Dec 18, 2024, 11:52:46 PM	WIP
🔴 @amankrx	Dec 19, 2024, 4:23:05 AM	WIP
🟢 @sreehariX	Dec 19, 2024, 10:30:38 AM	WIP
🟢 @DevendraSingh-1	Dec 19, 2024, 4:09:51 PM	WIP
🟢 @ManasMahanand	Dec 20, 2024, 3:56:18 AM	WIP
🟢 @RafaelJohn9	Jan 1, 2025, 7:37:56 PM	WIP
🟢 @gerred	Jan 2, 2025, 11:33:18 PM	WIP
🟢 @SAIKIRANSURAPALLI	Jan 8, 2025, 10:15:23 AM	WIP
🟢 @BenraouaneSoufiane	Jan 27, 2025, 1:58:13 PM	WIP
🟢 @itsparser	Jan 30, 2025, 8:36:10 AM	WIP
🟢 @kapilreddy	Jan 30, 2025, 9:42:30 AM	WIP
🟢 @Abdul-Samad-75	Feb 10, 2025, 9:36:33 AM	WIP
🟢 @Rumixyz	Feb 14, 2025, 11:34:37 AM	WIP

[zelosleone]

/attempt #1201

Options

• Cancel my attempt

[zelosleone]

By the way, I am already finishing the OpenAPI Export Integration from #1178 should be finished within few hours...

[radumarias]

Would you be interested In something like gRPC (because it's performant and eficient) + Cap'n Proto (for zero copy as it doesn't have any serialization) + http/3 (because it's QUIC)?

https://docs.google.com/document/d/1Ru5UlOz-4dz9ors3FKEg2Ac-KxKVqI_wR0EitECp-mo/edit?usp=drivesdk

You could use protobuf generators to obtain structs and classes in supported languages, and if we see the other end using protobuf we will serialize as such and vice- versa. Also we will have converters from protobuf to Cap’n a Proto end vice versa. So anyone having gRPC services could easily migrate to this. That could do it in phases as we will be interoperable with protobuf.

Using HTTP/3 with QUIC for gRPC with Cap’n Proto and tonic crate brings several potential advantages, especially in scenarios requiring high-performance data exchange and improved reliability over modern networks.

Motivation and key advantages:
Challenge: there are no well-known production-ready implementations for HTTP/3 in Rust
Faster transfers
Faster connection time
Reduced latency
Reduced downtime
High throughput
Multiplexing without head-of-line blocking
Particularly beneficial for short-lived connections or high-frequency RPC calls
QUIC retransmits lost packets faster than TCP, reducing delays
QUIC allows connections to migrate between network interfaces (e.g., Wi-Fi to cellular) without disconnecting
QUIC includes TLS 1.3 at the transport layer, so all HTTP/3 connections are encrypted by design
HTTP/3 allows prioritizing certain streams, optimizing resource utilization for high-performance data pipelines
QUIC supports an unlimited number of concurrent streams without degrading performance
Cap’n Proto offers zero-copy as it doesn’t have any serialization

[uurl]

/attempt #1201

Options

• Cancel my attempt

[amankrx]

/attempt #1201

Options

• Cancel my attempt

[vigoo]

Please do not start working on the worker executor part of this until #1150 is done (which is work in progress at the moment).
There are many other parts of the tasks that can be worked on in the mean time.

[ilyakharlamov]

Note that there is no real need for a separate OpenAPI definition, as gRPC supports standard REST-JSON transcoding defined within gRPC itself.
So the universal gRPC-OpenAPI definition would look like this:

service TodoService {
  rpc TodoAdd(TodoAddRequest) returns (TodoAddResponse) {
      option (google.api.http) = {
      post: "/components/schemas/Todo/{user_id}"
    };
  };
  rpc TodoDelete(TodoDeleteRequest) returns (TodoDeleteResponse) {
      option (google.api.http) = {
      delete: "/components/schemas/Todo/{user_id}"
    };
  };
  rpc TodoList(TodoListRequest) returns (TodoListResponse) {
      option (google.api.http) = {
      get: "/components/schemas/Todo/{user_id}"
    };
  };
}

This approach is supported by many servers, such as Armeria, Microsoft ASP.NET, Google Cloud and gRPC-gateway

This is something that can be considered.

[sreehariX]

/attempt #1201

Options

• Cancel my attempt

[vigoo]

@ilyakharlamov the idea is that you have an OpenAPI spec for something you want to call from a Golem worker, and you get a generated typed-safe way to interact with it, just by providing this spec (which may be originated from a 3rd party). The REST-JSON mapping in gRPC in your example is not helping with this - someone would have to manually create this gRPC definition from the OpenAPI spec. It's actually making more work, if we want to support this as well, which we did not consider earlier. (As it would be a third way to describe a target API)

[DevendraSingh-1]

Implementation Plan for gRPC and OpenAPI Integration in Golem

Objective:

Integrate gRPC and OpenAPI support into Golem, enabling type-safe interactions between Golem workers and external services via Protobuf and OpenAPI schemas. The implementation will allow users to easily add gRPC and OpenAPI dependencies and interact with them in a type-safe manner through generated WIT (WASM Interface Types) definitions and dynamic stubs.

---

Key Steps in Implementation:

1. Extend Golem CLI for New Dependency Types:

   • Modify Golem CLI to support adding gRPC (Protobuf files) and OpenAPI (YAML/JSON schemas) dependencies to the Golem application manifest file.
   • Allow users to specify these dependencies for worker components.

2. Protobuf-to-WIT Translator:

   • Implement a tool to convert gRPC Protobuf service definitions and message types into WIT type definitions.
   • Translate Protobuf message types into WIT record, oneof types into WIT variant, and repeated fields into WIT list.

3. OpenAPI-to-WIT Translator:

   • Develop a parser to convert OpenAPI schemas (3.0.x) into WIT types, including request bodies, response bodies, and parameters.
   • Inline types in OpenAPI schemas will be converted into separate named WIT types to preserve structure.

4. Dynamic Stub Generation in Worker-Executor:

   • Modify the worker-executor to dynamically generate and inject stubs for gRPC and OpenAPI services at runtime.
   • These stubs will map to the generated WIT interfaces and allow workers to make API calls to remote services using appropriate gRPC and HTTP invocations.

5. Authentication and Error Handling:

   • Implement mechanisms to handle authentication types (Bearer, Basic, API Key) within WIT.
   • Create a unified error-handling system to map remote service errors (e.g., unauthorized, validation errors) to WIT-idiomatic error types.

---

Expected Outcome:

By the end of this implementation, Golem workers will be able to interact with remote gRPC and OpenAPI services in a type-safe way using automatically generated WIT definitions and dynamically linked stubs.

---

This is the high-level overview of the implementation. If approved, I will proceed with the detailed design and development.

---

/attempt #1201

Options

• Cancel my attempt

[ManasMahanand]

/attempt #1201

Options

• Cancel my attempt

[ssddOnTop]

and in the grpc example comment

the file

...

service TodoService {
  rpc TodoAdd(TodoAddRequest) returns (TodoAddResponse);
  rpc TodoDelete(TodoDeleteRequest) returns (TodoDeleteResponse);
  rpc TodoList(TodoListRequest) returns (TodoListResponse);
}

...
interface todo-service {
...
    todo-add: func(request: todo-add-request) -> result<todo-add-response, todo-error>;
    todo-delete: func(request: todo-delete-request) -> result<todo-delete-response, todo-error>;
    todo-list: func(request: todo-list-request) -> result<todo-list-response, todo-error>;
}

do not contain any error type.. but the WIT contains todo-error

are we supposed to generate the error type in some predefined format or search for error keyword in available types?

@jdegoes @vigoo

ps:

if possible.. it would be easier to communicate via Discord.. I've asked the same question there (https://discord.com/channels/1134448700572319785/1291767182082048040/1320553191762427904)

[jdegoes]

@ssddOnTop That looks like a mistake. For gRPC, I think the best we can do is to reflect protocol-level errors.

[PiyushChandra17]

hey hey and hey i would love to work on this issue, also if anyone can collaborate on this work that would be great

[RafaelJohn9]

/attempt #1201

• Research and define the architecture.
• Implement pull mechanism.
• Implement push mechanism and authentication.
• Write tests and documentation.

Options

• Cancel my attempt

[gerred]

/attempt #1201

Options

• Cancel my attempt

[SAIKIRANSURAPALLI]

/attempt #1201

Options

• Cancel my attempt

[BenraouaneSoufiane]

/attempt #1201

Options

• Cancel my attempt

[itsparser]

/attempt #1201

Options

• Cancel my attempt

[kapilreddy]

/attempt #1201

Options

• Cancel my attempt

[jdegoes]

As this bounty has been open for 2 months without progress, we are considering withdrawing this bounty. If anyone has completed significant work in progress of this bounty and intends to complete the work in a reasonable timeframe, then please let us know within the next 24 hours!

[gerred]

Hey @jdegoes! I actually do a LOT with gRPC and OpenAPI, but I've been in the process of a move. I'm absolutely on this - just finally getting settled. Actually, this is in line with some other OSS I'm planning on working on, so very valuable for me to complete as well from an OSS lab perspective. I'm happy to set a reasonable timeline - I'm funemployed through this week and can put my attention to it if it's blocking you.

And WASM scheduling - this is oddly right in my wheelhouse, but just been slammed for time!

[ayewo]

@jdegoes

Perhaps you could garner more interest in the bounty by offering multiple payments, instead of 1 payment e.g. 3 payments of $15k, $10k and $5k each for 1st, 2nd and 3rd place respectively, instead of one $25k payment?

Right now, with only 1 payment on offer, the bounty terms highly favor bounty hunters that learn about this task as soon as it is posted and start working on it immediately because they have time on their hands (read: university students).

I imagine you are looking to reward experience and/or skill versus pure grit, so hoping you will consider this feedback for future bounties of this nature.

[jdegoes]

@gerred I would say there are really three main challenges:

1. One is just getting your head around what Golem is supposed to do and how the repository is put together. If you have already seen talks on Golem or built something on Golem, you are a big step ahead of the average contributor.
2. Another is figuring out how to dynamically create and add host functions in the worker executor. This happens right now for WASM-based RPC (our own RPC for worker-to-worker communication). It's been implemented and de-risked, and there is code to follow, but still understanding this will take some effort.
3. The final challenge, and this is perhaps the biggest one, is mapping OpenAPI to wit in a fashion that's as clean and idiomatic as possible. Such a mapping is not responsible for bugs in the OpenAPI spec, but there are two real challenges:
   1. First, OpenAPI is not a small spec and there are lots of ways of saying the same thing.
   2. Second, in practice, many APIs are extremely sloppily made, and figuring out a single algorithm that can result in "good" WIT-based API for "beautiful" REST APIs but which still works (albeit awkwardly) for "bad" APIs is challenging.

Finally, I suppose, one additional challenge is that however one maps the OpenAPI/gRPC to WIT must be replicated again in the worker executor, in a different form -- that is, stubs in the host must be created which implement the WIT. So really the same knowledge is being used in two different places (one to generate WIT, the other to generate an implementation that conforms to the WIT and calls the gRPC/OpenAPI API). In my experience this introduces a bit of a mess trying to keep them in sync exactly.

[jdegoes]

@gerred Related:

• I don't think the durability is what you need to focus on or that it will be challenging at all. It will be pretty much copy/paste of durability for existing host functions.
• gRPC is really easy because it has a good mapping to WIT.

[gerred]

@jdegoes That makes sense, I think we're on the same page and was omitting a bit because I'm catching up this morning. I've built out similar platforms and was really glad to see where WIT is. Selfishly, I want to have MCP tools running on this thing as well in the vein of mcp.run.

This all make sense, I can stick to that timeline of at least getting to a draft state and then we can close the implementation gap. Agreed re: the syncing.

I'll join the community as well, I've been avoiding WASM for a little bit but this was a great excuse to enter back in.

[Keshi]

I think this is probably a good idea, or maybe have some other collaborative process happen. I was excited to get to work on this at the beginning but I abandoned my effort when I saw that someone else seemed to have jumped on this enthusiastically, as I didn't want to develop a whole solution just to have someone beat me to the punch and end up with nothing for the time spent.

@jdegoes

Perhaps you could garner more interest in the bounty by offering multiple payments, instead of 1 payment e.g. 3 payments of $15k, $10k and $5k each for 1st, 2nd and 3rd place respectively, instead of one $25k payment?

Right now, with only 1 payment on offer, the bounty terms highly favor bounty hunters that learn about this task as soon as it is posted and start working on it immediately because they have time on their hands (read: university students).

I imagine you are looking to reward experience and/or skill versus pure grit, so hoping you will consider this feedback for future bounties of this nature.

[kapilreddy]

I have been working on this from last week. I am focusing on translating a Protobuf to WIT. I intend to get this done by end of this week. But, I am not sure of rest of the work required to integrate the translation into golem-cli workflows.

I am currently not focusing on OpenAPI. The reason being the spec is not small.

[Rumixyz]

/attempt #1201

Options

• Cancel my attempt

[webbdays]

@jdegoes
the idomatic wit is ok. looks good.
how the user of openapi in one of the golem component knows which of the function parameters is what for?
namely query params, request body etc...?
should we generate doc for that? i think the user who uses this will need additional time to understand this conversion the golem might have done and how its implemented inside the func using those func params?

[webbdays]

user i mean
developer using the the integration to call the exported fucntions which map to openapi endpoints.

[webbdays]

we can assume first n-1 parameters of functions as query params
nth parameter will be to send request body.

[webbdays]

for n-1 params we still need to document which are for query params and which are for path params.

[Aditya-Choudhry]

/request can i start working on this
can you provide what you need
golemcloud
Golem Cloud
golemcloud/golem#1201

[jdegoes]

@gerred @kapilreddy Any update??

[kapilreddy]

@jdegoes Hello. Apologies for the delay on replying here.

grpc to WIT conversion is in pretty good shape. I have tested it with https://grpcb.in/
I'll be adding more tests this week.

Currently I am trying to figure out how to update wasm-rpc

But, before I do that. I wanted to try out stub generation.
I am following this,
https://learn.golem.cloud/docs/rust-language-guide/rpc#set-up-the-golem-wasm-rpc-stub-generator
golem-cli v1.1.16 doesn't seem to support stubgen command

[CapuchaRojo]

/request 1201 Golem/GOLEMCLOUD [CapuchaRojo)] [Tech]( Python, Rust)
/split [Aditya-Choudhry]

Options

[jdegoes]

@kapilreddy Please note that a lot of things changed in the implementation: the stubs are now added dynamically in worker-executor. Some of the docs have yet to be updated.

I note this in the ticket description. You can see how stubs are added for wasm-rpc in the worker executor and copy the same approach (involving using wasmtime types).

The gRPC communication will happen in the worker executor through these dynamically added stubs.

[webbdays]

Hi,
Struck at Deserialize and serialize

resp json to rust type

rust bindings.rs generated from wit is not implementing this trait.
Please provide any suggestions.
Thanks.

[webbdays]

but impl deserialize trait dynamically from wit and openapi is bit challenging. possible but time confusing right.

[webbdays]

found this
-d, --additional_derive_attribute <ADDITIONAL_DERIVE_ATTRIBUTES>
Additional derive attributes to add to generated types. If using in a CLI, this flag can be specified multiple times to add multiple attributes.

      These derive attributes will be added to any generated structs or enums

[jdegoes]

@webbdays It seems you are trying to generate Rust that matches the WIT, but code generation is not on the table for this issue, because a newer approach has been discovered that yields superior usability. See the PR linked in the issue description, which was merged.

[webbdays]

@webbdays It seems you are trying to generate Rust that matches the WIT, but code generation is not on the table for this issue, because a newer approach has been discovered that yields superior usability. See the PR linked in the issue description, which was merged.

ok. Thanks for pointing that out.

[kapilreddy]

General approach

• Translate gRPC protobuf to WIT
• Add support for a new type 'grpc' in static-wasm-rpc and generate WIT ref code
• Add integration System tests

I should be ready with these changes by 3rd March.

I can pickup OpenAPI integration next. I had some ideas on how to phase the Open API changes. I'll document it later once I am near finishing gRPC changes.

Some references that I am using,

• Guide to setting up worker to worker communication This is outdated now due to [Stubless RPC] (Implement the dynamic linked stubless RPC #1150)
• Good example of App with worker to worker communication
• Test suite for stubless RPC
• How to string together different components

[aditya-sphereoutsourcing]

@kapilreddy I’ve been following the discussions and your approach to translating gRPC protobuf to WIT, adding grpc support in static-wasm-rpc, and integrating system tests. I'd love to contribute and assist in any way possible.

If there are specific tasks I can take on or areas where you need help, please let me know. Also, I'm interested in the OpenAPI integration you mentioned—I'd be happy to discuss potential phases and contribute to that as well.

Looking forward to collaborating! 🚀

[webbdays]

Hi @everyone @jdegoes,
example

endpoint1 has 200 -> returntype1, 202 -> returntype2 , 2xx -> rtype3 so...on ,,,,,,,,....
similar for error codes

do we need to use variant in wit for this case success case also? result<success, error>
bit challenging in case of 2xx.

[kapilreddy]

Changes

• Implement translation from gRPC protobuf to WIT
  - [ ] Add support for new 'grpc' type in static-wasm-rpc and generate WIT
  Implementation Clarification
  I've determined that adding a new dependency type isn't necessary. Instead, I've created a pathway to generate WIT directly from gRPC proto files, as shown in this configuration example:

  app:component-a:
    sourceProto: "test.proto"
    generatedWit: wit-generated
    componentWasm: dist/component-a.wasm
    sourceWit: "wit"

• Enhanced wasm-rpc-stubgen build process by adding a gen-wit step before gen-rpc

• This scans all components for protoSource entries in golem.yaml

• Converts identified protobuf files into WIT

• Note: Uses a modified ApplicationContext implementation since standard initialization requires all WIT files to be present first (feedback on this approach welcome)

• golem-cli app build will now automatically create wit for your component or components in an app if the field sourceProto is present along-with other wit related fields

• Fix a bug in the app subcommand CLI for wasm-rpc-stubgen (may already be fixed in main branch - will verify)

• Added tests to verify correct translation from gRPC proto to WIT

• System tests to verify correct application composition

• Integration tests for end-to-end verification including worker-executor
  Since there is a lot of work done regarding integration tests in main golem repo. Any guidelines on how to reuse them would be helpful.

• Update documentation in golem-cli

• Update documentation in golemcloud docs

Next update expected by March 7th.

[kapilreddy]

@aditya-sphereoutsourcing You can reach out to me on discord. I am mostly finished with grpc work. But we can work together on something else.

[jdegoes]

@kapilreddy

I want to split this bounty into the gRPC and OpenAPI work separately.

This is because gRPC is relatively straightforward and can be done more quickly than OpenAPI, which will have a lot of edge cases and require a lot of testing.

It looks like you are the only person working on this, so are you amenable to such a split?

[jdegoes]

I've determined that adding a new dependency type isn't necessary. Instead, I've created a pathway to generate WIT directly from gRPC proto files, as shown in this configuration example:

This is not really how I want this implemented. Instead, since we already have dependencies, including the notion of type (e.g. wasm-rpc), I want one type of dependency to be gRPC. Note that one project could have dependencies on many separate protobuf files.

[kapilreddy]

@jdegoes Thank you for the clarification on the expectation of the change. @vigoo Had also mentioned this on discord.
I'll post next update on this by 7th March. Though the update will have corrected implementation instead of integration and system tests.

re: Bounty split: Yes I am good to go ahead with a split.

[webbdays]

@jdegoes
I am also on track.

1. In middle of OpenApi to wit and meta info generation. need to work with types duplicates yet.
2. wit representing part in rust and serializing to wit is done.
3. Grpc also i have some research into it. understood it seems straight forward. but yet to start by me.
4. Found a way to dynamically make request to api using dynamic meta info stored. (using current impl able to reading/parse params values and feeding back. from and into done. making the req using meta info is pending which is a small func to work.
5. Implemented most of it.
6. pending handling resource handle.(like to pass initial configuration for usable by api requests. like obj instance for client) need to read and understand the code more.

I am positive enough that i can finish it. 💯 😅

[jdegoes]

@kapilreddy @webbdays

Translating from proto to WIT is the easy part of this project.

Then you need to dynamically implement this WIT in the worker-executor, so that components that use a proto-generated WIT are able to actually use those functions at runtime.

This involves writing some dynamic linking code in worker-executor that does generic gRPC requests/response, packing and unpacking types as appropriate for the generated WIT model.

If you don't understand what I am talking about here, then I'm afraid you have zero chance of solving the ticket.