Skip to content

Prefix caching possible also for structured output. Readme update.#335

Open
viktorpusTT wants to merge 1 commit intodevfrom
viktorpus/70b-galaxy-prefix-caching
Open

Prefix caching possible also for structured output. Readme update.#335
viktorpusTT wants to merge 1 commit intodevfrom
viktorpus/70b-galaxy-prefix-caching

Conversation

@viktorpusTT
Copy link

@viktorpusTT viktorpusTT commented Mar 2, 2026

Purpose

As a followup to tenstorrent/tt-metal#35904, this PR:

  • Adds support for random-prefix-len arg to structured outputs benchmark, to make this more param-compatible with vllm bench serve.
  • Updates readme.

Test Plan

Test Result

https://github.com/tenstorrent/tt-metal/actions/runs/22580104243/job/65410407537


Essential Elements of an Effective PR Description Checklist
  • The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
  • The test plan, such as providing test command.
  • The test results, such as pasting the results comparison before and after, or e2e results
  • (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
  • (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants