Add support for KV Cache for /chat/completions API #180

Open

Milestone

opened

Relevant for:

KV events
Prefill time calculations
Prometheus metrics

Metadata

Assignees

No one assigned

Labels

No labels

No labels

Type

Projects

No projects

Milestone

Release v0.6.0No due date

Relationships

None yet

Development

No branches or pull requests