Skip to content

Add support for vLLMs new gRPC interface #548

@wseaton

Description

@wseaton

In vLLM 0.14.0+, there is now first-class support for a gRPC set of endpoints. It would be great to have the ability to have guidellm optionally use this interface instead of HTTP/S for traffic, I would expect this to be a mutually exclusive mode.

Link to proto: https://github.com/vllm-project/vllm/blob/69d09fdd6cb66fdfc92f2f2388c4899dfd93d0ad/vllm/grpc/vllm_engine.proto#L2

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions