Skip to content

Make token and stat callback optional in multimodal runner #13664

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 8 commits into from
Sep 2, 2025
Merged

Make token and stat callback optional in multimodal runner #13664

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
bc8d3c4
Make token and stat callback optional in multimodal runner
jackzhxng Aug 25, 2025
9f8f723
Update on "Make token and stat callback optional in multimodal runner"
jackzhxng Aug 26, 2025
ac67303
Update on "Make token and stat callback optional in multimodal runner"
jackzhxng Aug 26, 2025
c3f1c2c
Update on "Make token and stat callback optional in multimodal runner"
jackzhxng Aug 26, 2025
ee3b3c3
Update on "Make token and stat callback optional in multimodal runner"
jackzhxng Aug 28, 2025
07106a0
Update on "Make token and stat callback optional in multimodal runner"
jackzhxng Aug 28, 2025
9fd7b2d
Update on "Make token and stat callback optional in multimodal runner"
jackzhxng Aug 28, 2025
e087e29
Update on "Make token and stat callback optional in multimodal runner"
jackzhxng Aug 28, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions extension/llm/runner/multimodal_runner.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,8 +65,8 @@ Error MultimodalRunner::load() {
Error MultimodalRunner::generate(
const std::vector<MultimodalInput>& inputs,
const GenerationConfig& config,
std::function<void(const std::string&)>& token_callback,
std::function<void(const Stats&)>& stats_callback) {
std::function<void(const std::string&)> token_callback,
std::function<void(const Stats&)> stats_callback) {
if (inputs.empty()) {
ET_LOG(Error, "MultimodalInput vector cannot be empty");
return Error::InvalidArgument;
Expand Down
4 changes: 2 additions & 2 deletions extension/llm/runner/multimodal_runner.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -116,8 +116,8 @@ class ET_EXPERIMENTAL MultimodalRunner {
virtual ::executorch::runtime::Error generate(
const std::vector<MultimodalInput>& inputs,
const GenerationConfig& config,
std::function<void(const std::string&)>& token_callback,
std::function<void(const Stats&)>& stats_callback);
std::function<void(const std::string&)> token_callback = {},
std::function<void(const Stats&)> stats_callback = {});

inline void stop() {
text_token_generator_->stop();
Expand Down
6 changes: 3 additions & 3 deletions extension/llm/runner/text_token_generator.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,9 +36,9 @@ class ET_EXPERIMENTAL TextTokenGenerator {

/**
* Token generation loop.
* @param tokens prompt tokens as well as the first token generated by
* prefill.
* @param start_pos the start position of the new tokens, based on how many
* @param tokens The first token generated by prefill, if using kv cache. Else
* the prompt tokens + the first token generated by prefill.
* @param start_pos The start position of the new tokens, based on how many
* prompt tokens is prefilled.
* @param max_new_tokens Maximum number of new tokens to generate.
* @param temperature controls the randomness of predictions by scaling the
Expand Down
Loading