Skip to content

Commit 13125f4

Browse files
committed
Update on "[llm] Add a generic text only LLM runner"
Introducing `text_llm_runner`. This can be used to run all text only decoder only LLM models supported by ExecuTorch. * Metadata is being read out from the .pte file and being used to construct the runner object. * examples/models/llama/runner.h[.cpp] only contains a simple wrapper around `text_llm_runner.h[.cpp]`. In next PRs I will move examples/models/phi-3-mini/runner to use the generic runner. Will look into QNN and MediaTek runners as well. Differential Revision: [D75910889](https://our.internmc.facebook.com/intern/diff/D75910889/) [ghstack-poisoned]
2 parents 2b6243e + 61182e6 commit 13125f4

File tree

2 files changed

+2
-1
lines changed

2 files changed

+2
-1
lines changed

extension/llm/runner/test/CMakeLists.txt

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,4 +25,5 @@ et_cxx_test(
2525
${_test_srcs}
2626
EXTRA_LIBS
2727
executorch
28+
extension_llm_runner
2829
)

test/run_oss_cpp_tests.sh

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,7 @@ build_executorch() {
3232
if [ -x "$(command -v glslc)" ]; then
3333
BUILD_VULKAN="ON"
3434
fi
35+
# -DEXECUTORCH_BUILD_EXTENSION_LLM_RUNNER=ON \ TODO(larryliu0820): Fix the name collision between Abseil and XNNPACK and turn this on.
3536
cmake . \
3637
-DCMAKE_INSTALL_PREFIX=cmake-out \
3738
-DEXECUTORCH_USE_CPP_CODE_COVERAGE=ON \
@@ -40,7 +41,6 @@ build_executorch() {
4041
-DEXECUTORCH_BUILD_KERNELS_QUANTIZED=ON \
4142
-DEXECUTORCH_BUILD_EXTENSION_DATA_LOADER=ON \
4243
-DEXECUTORCH_BUILD_EXTENSION_FLAT_TENSOR=ON \
43-
# -DEXECUTORCH_BUILD_EXTENSION_LLM_RUNNER=ON \ TODO(larryliu0820): Fix the name collision between Abseil and XNNPACK and turn this on.
4444
-DEXECUTORCH_BUILD_EXTENSION_MODULE=ON \
4545
-DEXECUTORCH_BUILD_EXTENSION_RUNNER_UTIL=ON \
4646
-DEXECUTORCH_BUILD_EXTENSION_TENSOR=ON \

0 commit comments

Comments
 (0)