Add IHV Qwen2.5-1.5B-Instruct NPU test models to azureml registry #4379
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,4 @@ | ||
| extra_config: model.yaml | ||
| spec: spec.yaml | ||
| type: model | ||
| categories: ["Local"] |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,11 @@ | ||
| This model is an optimized version of Qwen2.5-1.5B-Instruct to enable local inference on Intel NPUs. This model uses post-training quantization. | ||
|
|
||
| # Model Description | ||
| - **Developed by:** Microsoft | ||
| - **Model type:** ONNX | ||
| - **License:** apache-2.0 | ||
| - **Model Description:** This is a conversion of the Qwen2.5-1.5B-Instruct for local inference on Intel NPUs. | ||
| - **Disclaimer:** Model is only an optimization of the base model, any risk associated with the model is the responsibility of the user of the model. Please verify and test for your scenarios. There may be a slight difference in output from the base model with the optimizations applied. Note that optimizations applied are distinct from fine tuning and thus do not alter the intended uses or capabilities of the model. | ||
|
|
||
| # Base Model Information | ||
| See Hugging Face model [Qwen2.5-1.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-1.5B-Instruct) for details. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
| path: | ||
| container_name: models | ||
| container_path: foundrylocal/fl-perf-improvements/qwen2.5-1.5b-instruct/onnx/npu/qwen2.5-1.5b-instruct-openvino-npu | ||
| storage_name: automlcesdkdataresources | ||
| type: azureblob | ||
| publish: | ||
| description: description.md | ||
| type: custom_model | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
| @@ -0,0 +1,32 @@ | ||||||
| $schema: https://azuremlschemas.azureedge.net/latest/model.schema.json | ||||||
| name: qwen2.5-1.5b-instruct-openvino-npu | ||||||
|
There was a problem hiding this comment.
Suggested change
There was a problem hiding this comment. Might be best to add test to model name. If we publish without test in the name, this will publish as v1 of the model, which would be good to avoid as we're working out IHV model integration details. |
||||||
| version: 1 | ||||||
| path: ./ | ||||||
| tags: | ||||||
| foundryLocal: "test" | ||||||
| license: "apache-2.0" | ||||||
| licenseDescription: "This model is provided under the License Terms available at <https://huggingface.co/Qwen/Qwen2.5-1.5B-Instruct/blob/main/LICENSE>." | ||||||
| author: Microsoft | ||||||
| inputModalities: "text" | ||||||
| outputModalities: "text" | ||||||
| task: chat-completion | ||||||
| maxOutputTokens: 2048 | ||||||
| alias: qwen2.5-1.5b | ||||||
|
There was a problem hiding this comment.
Suggested change
There was a problem hiding this comment. This is worth double checking logic in Foundry Local / Neutron - I can't remember whether including an |
||||||
| directoryPath: qwen2.5-1.5b-instruct-openvino-npu | ||||||
| promptTemplate: "{\"system\": \"<|im_start|>system\\n{Content}<|im_end|>\", \"user\": \"<|im_start|>user\\n{Content}<|im_end|>\", \"assistant\": \"<|im_start|>assistant\\n{Content}<|im_end|>\", \"prompt\": \"<|im_start|>user\\n{Content}<|im_end|>\\n<|im_start|>assistant\"}" | ||||||
|
There was a problem hiding this comment.
Suggested change
There was a problem hiding this comment. Didn't have this in my original instructions b/c promptTemplate was required at the point that I wrote them, but promptTemplate should no longer be needed here. Leaving this tag out will automatically apply chat template from genai_config.json file. |
||||||
| supportsToolCalling: "" | ||||||
| toolCallStart: "<tool_call>" | ||||||
| toolCallEnd: "</tool_call>" | ||||||
| toolRegisterStart: "<tools>" | ||||||
| toolRegisterEnd: "</tools>" | ||||||
| type: custom_model | ||||||
| variantInfo: | ||||||
| parents: | ||||||
| - assetId: azureml://registries/azureml/models/qwen2.5-1.5b-instruct/versions/1 | ||||||
| variantMetadata: | ||||||
| modelType: 'ONNX' | ||||||
| quantization: ['PTQ'] | ||||||
|
There was a problem hiding this comment.
Suggested change
There was a problem hiding this comment. As discussed in our thread, |
||||||
| device: 'npu' | ||||||
| executionProvider: 'OpenVINOExecutionProvider' | ||||||
| fileSizeBytes: 864 | ||||||
|
There was a problem hiding this comment.
Suggested change
There was a problem hiding this comment. I think I gave you inaccurate info about calculating this field based on model sizes! Took a closer look, and the OpenVINO model also includes .xml files, which (I think) contribute to the So sum of all .bin and .xml files is 2.09 MiB + 9.36 KiB + 5.33 MiB + 24.55 KiB + 867.71 MiB + 2.21 MiB = 919,992,391 bytes. |
||||||
| vRamFootprintBytes: 917643791 | ||||||
|
There was a problem hiding this comment.
Suggested change
There was a problem hiding this comment. Also gave you the wrong info about calculating this field - this would be sum of previous field ( |
||||||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,4 @@ | ||
| extra_config: model.yaml | ||
| spec: spec.yaml | ||
| type: model | ||
| categories: ["Local"] |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,11 @@ | ||
| This model is an optimized version of Qwen2.5-1.5B-Instruct to enable local inference on Qualcomm NPUs. This model uses post-training quantization. | ||
|
|
||
| # Model Description | ||
| - **Developed by:** Microsoft | ||
| - **Model type:** ONNX | ||
| - **License:** apache-2.0 | ||
| - **Model Description:** This is a conversion of the Qwen2.5-1.5B-Instruct for local inference on Qualcomm NPUs. | ||
| - **Disclaimer:** Model is only an optimization of the base model, any risk associated with the model is the responsibility of the user of the model. Please verify and test for your scenarios. There may be a slight difference in output from the base model with the optimizations applied. Note that optimizations applied are distinct from fine tuning and thus do not alter the intended uses or capabilities of the model. | ||
|
|
||
| # Base Model Information | ||
| See Hugging Face model [Qwen2.5-1.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-1.5B-Instruct) for details. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
| @@ -0,0 +1,8 @@ | ||||||
| path: | ||||||
| container_name: models | ||||||
| container_path: foundrylocal/fl-perf-improvements/qwen2.5-1.5b-instruct/onnx/npu/qwen2.5-1.5b-instruct-qnn-npu | ||||||
|
There was a problem hiding this comment.
Suggested change
There was a problem hiding this comment. Similar to my comment above for OpenVINO model - I think you just need the files w/in the |
||||||
| storage_name: automlcesdkdataresources | ||||||
| type: azureblob | ||||||
| publish: | ||||||
| description: description.md | ||||||
| type: custom_model | ||||||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
| @@ -0,0 +1,32 @@ | ||||||
| $schema: https://azuremlschemas.azureedge.net/latest/model.schema.json | ||||||
| name: qwen2.5-1.5b-instruct-qnn-npu | ||||||
|
There was a problem hiding this comment.
Suggested change
|
||||||
| version: 1 | ||||||
| path: ./ | ||||||
| tags: | ||||||
| foundryLocal: "test" | ||||||
| license: "apache-2.0" | ||||||
| licenseDescription: "This model is provided under the License Terms available at <https://huggingface.co/Qwen/Qwen2.5-1.5B-Instruct/blob/main/LICENSE>." | ||||||
| author: Microsoft | ||||||
| inputModalities: "text" | ||||||
| outputModalities: "text" | ||||||
| task: chat-completion | ||||||
| maxOutputTokens: 2048 | ||||||
| alias: qwen2.5-1.5b | ||||||
|
There was a problem hiding this comment.
Suggested change
|
||||||
| directoryPath: qwen2.5-1.5b-instruct-qnn-npu | ||||||
| promptTemplate: "{\"system\": \"<|im_start|>system\\n{Content}<|im_end|>\", \"user\": \"<|im_start|>user\\n{Content}<|im_end|>\", \"assistant\": \"<|im_start|>assistant\\n{Content}<|im_end|>\", \"prompt\": \"<|im_start|>user\\n{Content}<|im_end|>\\n<|im_start|>assistant\"}" | ||||||
|
There was a problem hiding this comment.
Suggested change
|
||||||
| supportsToolCalling: "" | ||||||
| toolCallStart: "<tool_call>" | ||||||
| toolCallEnd: "</tool_call>" | ||||||
| toolRegisterStart: "<tools>" | ||||||
| toolRegisterEnd: "</tools>" | ||||||
| type: custom_model | ||||||
| variantInfo: | ||||||
| parents: | ||||||
| - assetId: azureml://registries/azureml/models/qwen2.5-1.5b-instruct/versions/1 | ||||||
| variantMetadata: | ||||||
| modelType: 'ONNX' | ||||||
| quantization: ['PTQ'] | ||||||
|
There was a problem hiding this comment.
Suggested change
|
||||||
| device: 'npu' | ||||||
| executionProvider: 'QNNExecutionProvider' | ||||||
| fileSizeBytes: 325137751 | ||||||
|
There was a problem hiding this comment.
Suggested change
There was a problem hiding this comment. I think I inverted the guidance I gave you for calculating this. The |
||||||
| vRamFootprintBytes: 998877199 | ||||||
|
There was a problem hiding this comment.
Suggested change
There was a problem hiding this comment. I got a slightly different value - I did sum of |
||||||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,4 @@ | ||
| extra_config: model.yaml | ||
| spec: spec.yaml | ||
| type: model | ||
| categories: ["Local"] |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,11 @@ | ||
| This model is an optimized version of Qwen2.5-1.5B-Instruct to enable local inference on AMD NPUs. This model uses post-training quantization. | ||
|
|
||
| # Model Description | ||
| - **Developed by:** Microsoft | ||
| - **Model type:** ONNX | ||
| - **License:** apache-2.0 | ||
| - **Model Description:** This is a conversion of the Qwen2.5-1.5B-Instruct for local inference on AMD NPUs. | ||
| - **Disclaimer:** Model is only an optimization of the base model, any risk associated with the model is the responsibility of the user of the model. Please verify and test for your scenarios. There may be a slight difference in output from the base model with the optimizations applied. Note that optimizations applied are distinct from fine tuning and thus do not alter the intended uses or capabilities of the model. | ||
|
|
||
| # Base Model Information | ||
| See Hugging Face model [Qwen2.5-1.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-1.5B-Instruct) for details. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
| @@ -0,0 +1,9 @@ | ||||||
| path: | ||||||
| container_name: models | ||||||
| container_path: foundrylocal/fl-perf-improvements/qwen2.5-1.5b-instruct/onnx/npu/qwen2.5-1.5b-instruct-vitis-npu | ||||||
|
There was a problem hiding this comment.
Suggested change
|
||||||
|
|
||||||
| storage_name: automlcesdkdataresources | ||||||
| type: azureblob | ||||||
| publish: | ||||||
| description: description.md | ||||||
| type: custom_model | ||||||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
| @@ -0,0 +1,32 @@ | ||||||
| $schema: https://azuremlschemas.azureedge.net/latest/model.schema.json | ||||||
| name: qwen2.5-1.5b-instruct-vitis-npu | ||||||
|
There was a problem hiding this comment.
Suggested change
|
||||||
| version: 1 | ||||||
| path: ./ | ||||||
| tags: | ||||||
| foundryLocal: "test" | ||||||
| license: "apache-2.0" | ||||||
| licenseDescription: "This model is provided under the License Terms available at <https://huggingface.co/Qwen/Qwen2.5-1.5B-Instruct/blob/main/LICENSE>." | ||||||
| author: Microsoft | ||||||
| inputModalities: "text" | ||||||
| outputModalities: "text" | ||||||
| task: chat-completion | ||||||
| maxOutputTokens: 2048 | ||||||
| alias: qwen2.5-1.5b | ||||||
|
There was a problem hiding this comment.
Suggested change
|
||||||
| directoryPath: qwen2.5-1.5b-instruct-vitis-npu | ||||||
| promptTemplate: "{\"system\": \"<|im_start|>system\\n{Content}<|im_end|>\", \"user\": \"<|im_start|>user\\n{Content}<|im_end|>\", \"assistant\": \"<|im_start|>assistant\\n{Content}<|im_end|>\", \"prompt\": \"<|im_start|>user\\n{Content}<|im_end|>\\n<|im_start|>assistant\"}" | ||||||
|
There was a problem hiding this comment.
Suggested change
|
||||||
| supportsToolCalling: "" | ||||||
| toolCallStart: "<tool_call>" | ||||||
| toolCallEnd: "</tool_call>" | ||||||
| toolRegisterStart: "<tools>" | ||||||
| toolRegisterEnd: "</tools>" | ||||||
| type: custom_model | ||||||
| variantInfo: | ||||||
| parents: | ||||||
| - assetId: azureml://registries/azureml/models/qwen2.5-1.5b-instruct/versions/1 | ||||||
| variantMetadata: | ||||||
| modelType: 'ONNX' | ||||||
| quantization: ['PTQ'] | ||||||
|
There was a problem hiding this comment.
Suggested change
|
||||||
| device: 'npu' | ||||||
| executionProvider: 'VitisAIExecutionProvider' | ||||||
| fileSizeBytes: 293141524 | ||||||
|
There was a problem hiding this comment.
Suggested change
There was a problem hiding this comment. In this case, this is the size of the .onnx.data file, which is 674,853,027 bytes. |
||||||
| vRamFootprintBytes: 967991316 | ||||||
|
There was a problem hiding this comment.
Suggested change
There was a problem hiding this comment. Got something slightly different when I added up the sum of .onnx.data file and all the .onnx files: 674,853,027.84 bytes + 693.54 KiB + 139.1 MiB + 693.54 KiB + 139.1 MiB = 967,987,240.96 bytes. |
||||||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Whether or not this is a valid suggestion will actually depend on which files you uploaded to cache for this model when testing manually through Foundry Local.
I'm fairly certain that you'll only need the files within the
modeldirectory to run this model, but did you include additional files when manually adding the model to your Foundry Local cache and running from there? i.e., Did you include the footprints.json, model_config.json, etc. files pictured here under themodeldirectory when running from your cache?