You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository was archived by the owner on Jun 3, 2025. It is now read-only.
[](https://youtu.be/zJy_8uPZd0o)
35
34
36
35
**How does Neural Magic make it work?**
37
36
@@ -53,7 +52,7 @@ We are open to opportunities to expand our support footprint for different CPU-b
53
52
54
53
**Do you run on ARM architecture?**
55
54
56
-
We currently do not support ARM and it’s on the Neural Magic roadmap; however, we’d still like to hear your use cases[Contact us to continue the conversation](https://neuralmagic.com/contact/).
55
+
We currently do not support ARM and it’s on the Neural Magic roadmap; however, we’d still like to hear your use cases. [Contact us to continue the conversation](https://neuralmagic.com/contact/).
57
56
58
57
**To what use cases is the Deep Sparse Platform best suited?**
59
58
@@ -63,6 +62,14 @@ We focus on the models and use cases related to computer vision due to cost sens
63
62
64
63
Today, we offer support for CNN-based computer vision models, specifically classification and object detection model types. We are continuously adding models to [our supported model list and SparseZoo](https://docs.neuralmagic.com/sparsezoo). Additionally, we are investigating model architectures beyond computer vision such as NLP models like BERT.
65
64
65
+
**Is dynamic shape supported?**
66
+
67
+
Dynamic shape is currently not supported; be sure to use models with fixed inputs and compile the model for a particular batch size. Dynamic shape and dynamic batch sizes are on the Neural Magic roadmap; [subscribe for updates](https://neuralmagic.com/subscribe/).
68
+
69
+
**Can multiple model inferences be executed?**
70
+
71
+
Model inferences are executed as a single stream; concurrent execution is unsupported at this time.
72
+
66
73
___
67
74
68
75
## Benchmarking FAQs
@@ -73,7 +80,7 @@ This will vary but, in some cases, we are seeing 3x-10x savings. Typically, we o
73
80
74
81
**Do you have benchmarks to compare and contrast?**
75
82
76
-
Yes. Check out our [benchmark demo video](https://neuralmagic.com/blog/neural-magic-demo/) or [contact us to](https://neuralmagic.com/contact/) discuss your particular performance requirements. If you’d rather observe performance for yourself, [head over to the Neural Magic GitHub repo](https://github.com/neuralmagic) to check out our tools and generate your own benchmarks in your environment.
83
+
Yes. Check out our [benchmark demo video](https://neuralmagic.com/blog/neural-magic-demo/) or [contact us](https://neuralmagic.com/contact/) to discuss your particular performance requirements. If you’d rather observe performance for yourself, [head over to the Neural Magic GitHub repo](https://github.com/neuralmagic) to check out our tools and generate your own benchmarks in your environment.
77
84
78
85
**Do you publish ML Perf inference benchmarks?**
79
86
@@ -83,11 +90,15 @@ ___
83
90
84
91
## Infrastructure FAQs
85
92
93
+
**Which instruction sets are supported and do we have to enable certain settings?**
94
+
95
+
AVX-512, AVX2, and VNNI. The DeepSparse Engine will automatically utilize the most effective available instruction set for the task. Generally, if AVX-512 is available then we have no reason to use AVX2 instruction set. AVX-512 VNNI only comes into use for quantized models i.e., INT8 or UINT8.
96
+
86
97
**Are you suitable for edge deployments (i.e., in-store devices, cameras)?**
87
98
88
99
Yes, absolutely. We can run anywhere you have a CPU with x86 instructions, including on bare metal, in the cloud, on-prem, or at the edge. Additionally, our model optimization tools are able to reduce the footprint of models across all architectures. We only guarantee performance in the DeepSparse Engine.
89
100
90
-
We’d love to hear from users highly interested in ML performance. If you want to chat about your use cases or how others are leveraging the Deep Sparse Platform, [please reach out](mailto: [email protected]). Or simply head over to the [Neural Magic GitHub repo](https://github.com/neuralmagic) and check out our tools.
101
+
We’d love to hear from users highly interested in ML performance. If you want to chat about your use cases or how others are leveraging the Deep Sparse Platform, [please reach out]([email protected]). Or simply head over to the [Neural Magic GitHub repo](https://github.com/neuralmagic) and check out our tools.
91
102
92
103
**Do you have available solutions or applications on the Microsoft/Azure platform?**
Copy file name to clipboardExpand all lines: docs/source/getstarted.md
+8-6Lines changed: 8 additions & 6 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -22,10 +22,10 @@ Neural Magic’s vision is no-hardware AI, shattering the hardware barriers that
22
22
23
23
To deliver on this vision, there are several components to the Deep Sparse Platform:
24
24
25
-
1.[Sparsify](https://docs.neuralmagic.com/sparsify): Open-source, easy-to-use interface to automatically sparsify and quantize deep learning models for CPUs & GPUs.
26
-
2.[SparseML](https://docs.neuralmagic.com/sparseml): Open-source libraries and optimization algorithms for CPUs & GPUs, enabling integration with a few lines of code.
27
-
3.[SparseZoo](https://docs.neuralmagic.com/sparsezoo): Open-source neural network model repository for highly sparse and sparse-quantized models with matching pruning recipes for CPUs and GPUs.
28
-
4.[DeepSparse Engine](https://docs.neuralmagic.com/deepsparse): Free CPU runtime that runs sparse models at GPU speeds.
25
+
1.[Sparsify](https://docs.neuralmagic.com/sparsify): Easy-to-use UI for automatically sparsifying neural networks and creating sparsification recipes for better inference performance and a smaller footprint
26
+
2.[SparseML](https://docs.neuralmagic.com/sparseml): Libraries for applying sparsification recipes to neural networks with a few lines of code, enabling faster and smaller models
27
+
3.[SparseZoo](https://docs.neuralmagic.com/sparsezoo): Neural network model repository for highly sparse and sparse-quantized models with matching sparsification recipes
28
+
4.[DeepSparse Engine](https://docs.neuralmagic.com/deepsparse): Neural network inference engine that delivers GPU-class performance for sparsified models on CPUs
29
29
30
30
Our inference engine and model optimization technologies enable companies to use ubiquitous and unconstrained CPU resources to achieve performance breakthroughs, at scale, with all the flexibility of software.
31
31
@@ -62,11 +62,13 @@ Today, we offer support for convolutional neural network-based computer vision m
62
62
63
63
We are continuously exploring models to add to our supported [model list](https://docs.neuralmagic.com/sparsezoo/models.html) and SparseZoo including model architectures beyond computer vision. Popular NLP models such as BERT are on the Neural Magic roadmap; [subscribe for updates](http://neuralmagic.com/subscribe).
64
64
65
-
### PyTorch and ONNX
65
+
### Notes
66
+
67
+
#### PyTorch and ONNX
66
68
67
69
Sparsify and the DeepSparse Engine inputs are standardized on the ONNX format. PyTorch has native ONNX export and requires fewer steps than other supported frameworks, such as [Keras or TensorFlow](https://docs.neuralmagic.com/sparseml/quicktour.html#exporting-to-onnx). If you have flexibility in frameworks, consider PyTorch to start.
68
70
69
-
### Model Considerations
71
+
####Model Considerations
70
72
71
73
Dynamic shape is currently not supported; be sure to use models with fixed inputs and compile the model for a particular batch size. Dynamic shape and dynamic batch sizes are on the Neural Magic roadmap; [subscribe for updates](http://neuralmagic.com/subscribe).
0 commit comments