Skip to content

[LLM Runner] Add LogitProcessor interface and pipeline #19471

@kirklandsign

Description

@kirklandsign

Add a LogitProcessor abstract interface (process(float* logits, int32_t vocab_size)) and wire a processor chain into TextTokenGenerator::generate() between step() and logits_to_token(). No behavior change when no processors registered.

cc @larryliu0820 @mergennachin @cccclai @helunwencser @jackzhxng

Metadata

Metadata

Assignees

Labels

module: llmIssues related to LLM examples and apps, and to the extensions/llm/ codetriagedThis issue has been looked at a team member, and triaged and prioritized into an appropriate module

Type

No type

Projects

Status

To triage

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions