Update cache position population and arg order for multimodal runner #14225
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/14225
Note: Links to docs will display an error until the docs builds have been completed. ❌ 2 New Failures, 2 Cancelled JobsAs of commit 5af64f8 with merge base 10e93fb ( NEW FAILURES - The following jobs have failed:
CANCELLED JOBS - The following jobs were cancelled. Please retry:
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
kirklandsign
requested review from
jackzhxng,
larryliu0820,
mergennachin and
swolchok
as code owners
meta-cla
bot
added
the
CLA Signed
| // e.g. if start_pos = 2 and encoder_output.size(1) = 5, | ||
| // cache_position_tensor should be [2, 3, 4, 5, 6]. | ||
| auto method_meta = ET_UNWRAP(module_->method_meta(kTextModelMethod)); | ||
| auto first_input_info = ET_UNWRAP(method_meta.input_tensor_meta(0)); |
There was a problem hiding this comment.
Change to second_input_info
| cache_positions.data(), | ||
| {static_cast<int>(seq_len)}, | ||
| executorch::aten::ScalarType::Long); | ||
| auto cache_position_tensor = (numel > 1) |
There was a problem hiding this comment.
Can you do something like
if (numel > 1) {
// `cache_position` goes from start_pos to start_pos + encoder_output.size(1).
// e.g. if start_pos = 2 and encoder_output.size(1) = 5,
// cache_position_tensor should be [2, 3, 4, 5, 6].
for (int64_t i = 0; i < seq_len; ++i) {
cache_positions[i] = start_pos + i;
}
auto cache_position_tensor = ::executorch::extension::from_blob(
cache_positions.data(),
{static_cast<int>(seq_len)},
executorch::aten::ScalarType::Long)
} else {
// Cache position is size 1.
auto cache_position_tensor = ::executorch::extension::from_blob(
&start_pos, {1}, executorch::aten::ScalarType::Long);
}
| // `cache_position` goes from start_pos to start_pos + encoder_output.size(1). | ||
| // e.g. if start_pos = 2 and encoder_output.size(1) = 5, | ||
| // cache_position_tensor should be [2, 3, 4, 5, 6]. | ||
| auto method_meta = ET_UNWRAP(module_->method_meta(kTextModelMethod)); |
There was a problem hiding this comment.
Add comment like
// Get expected shape of cache position tensor, which should be the second argument
| auto second_input_info = ET_UNWRAP(method_meta.input_tensor_meta(1)); | ||
| auto second_input_sizes = second_input_info.sizes(); | ||
| auto numel = second_input_sizes[0]; | ||
|
|
There was a problem hiding this comment.
Please reuse the logic here https://github.com/pytorch/executorch/blob/main/extension/llm/runner/text_decoder_runner.cpp#L44-L69
| {static_cast<int>(seq_len)}, | ||
| executorch::aten::ScalarType::Long); | ||
| auto prefill_result = module_->execute( | ||
| kTextModelMethod, {cache_position_tensor, encoder_output}); |
extension/llm/runner/util.h
Outdated
| } | ||
|
|
||
| inline runtime::Result<TensorPtr> | ||
| populate_start_pos_tensor(Module* module, int64_t& start_pos, int seq_len) { |
There was a problem hiding this comment.
Add some docstring please, like how we assume the second argument is cache position/ start pos and based on the shape to populate the tensor
There was a problem hiding this comment.
Also the name should be populate_start_pos_or_cache_position
kirklandsign
changed the title
| "The second input tensor is not 1D tensor. Got dimension (%zu)", | ||
| sizes.size()); | ||
| auto numel = sizes[0]; | ||
| std::vector<::executorch::aten::SizesType> sizes_vec = {numel}; |
There was a problem hiding this comment.
Seems like you can remove lines 35 ~ 54 since it's not being used anymore?
extension/llm/runner/util.h
Outdated
| const char* method_name, | ||
| Module* module, | ||
| int64_t& start_pos, | ||
| std::vector<int64_t>& cache_positions_underlying_vector, |
There was a problem hiding this comment.
Can be concise: cache_positions_vec
extension/llm/runner/util.h
Outdated
| // size 1 because model will populate the cache position tensor underneath), or | ||
| // a populated tensor for cache position, for the given start_pos and seq_len. | ||
| inline runtime::Result<TensorPtr> populate_start_pos_or_cache_position( | ||
| const char* method_name, |
There was a problem hiding this comment.
Give it a default value forward
|
@pytorchbot cheery-pick onto release/1.0 -c "fixnewfeature" |
|
❌ 🤖 pytorchbot command failed: Try |
|
@pytorchbot cherry-pick onto release/1.0 -c fixnewfeature |
|
❌ 🤖 pytorchbot command failed: Try |
|
@pytorchbot cherry-pick --onto release/1.0 -c fixnewfeature |
Cherry picking #14225The cherry pick PR is at #14343 and it is recommended to link a fixnewfeature cherry pick PR with an issue. The following tracker issues are updated: Details for Dev Infra teamRaised by workflow job |
…ytorch#14225) For voxtral and phi-3, we construct the cache_position_tensor like before; for llava, it will construct underneath so we pass in size 1.
Summary
For voxtral and phi-3, we construct the cache_position_tensor like before; for llava, it will construct underneath so we pass in size 1.
Test plan
CI