LLM cannot run on the new neural accelerator computing unit added to the A19 Pro GPU

[GaoKaipeng2023]

While running some third-party LLMs through this project, I noticed that the GPU counters showed that they were not utilizing the new neural accelerator computing units added to the A19 Pro GPU; instead, they were using the general-purpose ALUs. Is there a need for adaptation here? Does anyone have any good ideas? I believe many large models in the future will want to leverage this computing power.

[GaoKaipeng2023]

LLMEval: Qwen3 1.7B

[GaoKaipeng2023]

LLMEval: Qwen3 1.7B

MLXChatExample: llama3.2 1B

[davidkoski]

This will require changes in MLX (https://github.com/ml-explore/mlx) and then an update to mlx-swift to pick it up -- watch that repo for updates.

[rudrankriyam]

Looking forward to it! Just got an M5; happy to benchmark 😄

[hfutjcd]

GaoKaipeng2023 你好， 请问这个GPU counters 图是用什么工具抓的啊

[davidkoski]

@hfutjcd that looks like Instruments.app with one of the GPU/Metal instruments doing the collecting/display.