|
| 1 | +# Whisper example with WasmEdge WASI-NN MLX plugin |
| 2 | + |
| 3 | +This example demonstrates using WasmEdge WASI-NN MLX plugin to perform an inference task with whisper model. |
| 4 | + |
| 5 | +## Install WasmEdge with WASI-NN MLX plugin |
| 6 | + |
| 7 | +The MLX backend relies on [MLX](https://github.com/ml-explore/mlx), but we will auto-download MLX when you build WasmEdge. You do not need to install it yourself. If you want to custom MLX, install it yourself or set the `CMAKE_PREFIX_PATH` variable when configuring cmake. |
| 8 | + |
| 9 | +Build and install WasmEdge from source: |
| 10 | + |
| 11 | +``` bash |
| 12 | +cd <path/to/your/wasmedge/source/folder> |
| 13 | + |
| 14 | +cmake -GNinja -Bbuild -DCMAKE_BUILD_TYPE=Release -DWASMEDGE_PLUGIN_WASI_NN_BACKEND="mlx" |
| 15 | +cmake --build build |
| 16 | + |
| 17 | +# For the WASI-NN plugin, you should install this project. |
| 18 | +cmake --install build |
| 19 | +``` |
| 20 | + |
| 21 | +Then you will have an executable `wasmedge` runtime under `/usr/local/bin` and the WASI-NN with MLX backend plug-in under `/usr/local/lib/wasmedge/libwasmedgePluginWasiNN.so` after installation. |
| 22 | + |
| 23 | +## Download the model and tokenizer |
| 24 | + |
| 25 | +In this example, we will use `whisper-tiny`. |
| 26 | + |
| 27 | +``` bash |
| 28 | +git clone https://huggingface.co/grorge123/whisper-tiny |
| 29 | +cp -r whisper-tiny/assets . |
| 30 | +wget https://raw.githubusercontent.com/ml-explore/mlx-examples/refs/heads/main/whisper/mlx_whisper/assets/multilingual.tiktoken -P assets |
| 31 | +``` |
| 32 | + |
| 33 | +## Build wasm |
| 34 | + |
| 35 | +Run the following command to build wasm, the output WASM file will be at `target/wasm32-wasip1/release/`. |
| 36 | +Then we use AOT-compiled WASM to improve the performance. |
| 37 | + |
| 38 | +```bash |
| 39 | +cargo build --target wasm32-wasip1 --release |
| 40 | +``` |
| 41 | +## Execute |
| 42 | + |
| 43 | +Download the audio and save as `audio.mp3` |
| 44 | + |
| 45 | +Execute the WASM with the `wasmedge` using nn-preload to load model. |
| 46 | + |
| 47 | +``` bash |
| 48 | +wasmedge --dir .:. \ |
| 49 | +--nn-preload default:mlx:AUTO:whisper-tiny/weights.safetensors \ |
| 50 | +target/wasm32-wasip1/release/wasmedge-whisper.wasm default whisper-tiny audio.mp3 |
| 51 | +``` |
0 commit comments