You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.md
+92-2Lines changed: 92 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -18,5 +18,95 @@ limitations under the License.
18
18
19
19
# Triton Model Analyzer
20
20
21
-
> [!Warning]
22
-
> ##### THIS BRANCH IS UNDER ACTIVE DEVELOPMENT AND IS NOT READY FOR USE.
21
+
Triton Model Analyzer is a CLI tool which can help you find a more optimal configuration, on a given piece of hardware, for single, multiple, ensemble, or BLS models running on a [Triton Inference Server](https://github.com/triton-inference-server/server/). Model Analyzer will also generate reports to help you better understand the trade-offs of the different configurations along with their compute and memory requirements.
22
+
<br><br>
23
+
24
+
# Features
25
+
26
+
### Search Modes
27
+
28
+
-[Quick Search](docs/config_search.md#quick-search-mode) will **sparsely** search the [Max Batch Size](https://github.com/triton-inference-server/server/blob/r24.01/docs/user_guide/model_configuration.md#maximum-batch-size),
29
+
[Dynamic Batching](https://github.com/triton-inference-server/server/blob/r24.01/docs/user_guide/model_configuration.md#dynamic-batcher), and
30
+
[Instance Group](https://github.com/triton-inference-server/server/blob/r24.01/docs/user_guide/model_configuration.md#instance-groups) spaces by utilizing a heuristic hill-climbing algorithm to help you quickly find a more optimal configuration
31
+
32
+
-[Automatic Brute Search](docs/config_search.md#automatic-brute-search) will **exhaustively** search the
-[Manual Brute Search](docs/config_search.md#manual-brute-search) allows you to create manual sweeps for every parameter that can be specified in the model configuration
39
+
40
+
### Model Types
41
+
42
+
-[Ensemble Model Search](docs/config_search.md#ensemble-model-search): Model Analyzer can help you find the optimal
43
+
settings when profiling an ensemble model, utilizing the [Quick Search](docs/config_search.md#quick-search-mode) algorithm
44
+
45
+
-[BLS Model Search](docs/config_search.md#bls-model-search): Model Analyzer can help you find the optimal
46
+
settings when profiling a BLS model, utilizing the [Quick Search](docs/config_search.md#quick-search-mode) algorithm
47
+
48
+
-[Multi-Model Search](docs/config_search.md#multi-model-search-mode): **EARLY ACCESS** - Model Analyzer can help you
49
+
find the optimal settings when profiling multiple concurrent models, utilizing the [Quick Search](docs/config_search.md#quick-search-mode) algorithm
50
+
51
+
### Other Features
52
+
53
+
-[Detailed and summary reports](docs/report.md): Model Analyzer is able to generate
54
+
summarized and detailed reports that can help you better understand the trade-offs
55
+
between different model configurations that can be used for your model.
56
+
57
+
-[QoS Constraints](docs/config.md#constraint): Constraints can help you
58
+
filter out the Model Analyzer results based on your QoS requirements. For
59
+
example, you can specify a latency budget to filter out model configurations
60
+
that do not satisfy the specified latency threshold.
61
+
<br><br>
62
+
63
+
# Examples and Tutorials
64
+
65
+
### **Single Model**
66
+
67
+
See the [Single Model Quick Start](docs/quick_start.md) for a guide on how to use Model Analyzer to profile, analyze and report on a simple PyTorch model.
68
+
69
+
### **Multi Model**
70
+
71
+
See the [Multi-model Quick Start](docs/mm_quick_start.md) for a guide on how to use Model Analyzer to profile, analyze and report on two models running concurrently on the same GPU.
72
+
73
+
### **Ensemble Model**
74
+
75
+
See the [Ensemble Model Quick Start](docs/ensemble_quick_start.md) for a guide on how to use Model Analyzer to profile, analyze and report on a simple Ensemble model.
76
+
77
+
### **BLS Model**
78
+
79
+
See the [BLS Model Quick Start](docs/bls_quick_start.md) for a guide on how to use Model Analyzer to profile, analyze and report on a simple BLS model.
80
+
<br><br>
81
+
82
+
# Documentation
83
+
84
+
-[Installation](docs/install.md)
85
+
-[Model Analyzer CLI](docs/cli.md)
86
+
-[Launch Modes](docs/launch_modes.md)
87
+
-[Configuring Model Analyzer](docs/config.md)
88
+
-[Model Analyzer Metrics](docs/metrics.md)
89
+
-[Model Config Search](docs/config_search.md)
90
+
-[Checkpointing](docs/checkpoints.md)
91
+
-[Model Analyzer Reports](docs/report.md)
92
+
-[Deployment with Kubernetes](docs/kubernetes_deploy.md)
93
+
<br><br>
94
+
95
+
# Reporting problems, asking questions
96
+
97
+
We appreciate any feedback, questions or bug reporting regarding this
98
+
project. When help with code is needed, follow the process outlined in
99
+
the Stack Overflow (https://stackoverflow.com/help/mcve)
100
+
document. Ensure posted examples are:
101
+
102
+
- minimal – use as little code as possible that still produces the
103
+
same problem
104
+
105
+
- complete – provide all parts needed to reproduce the problem. Check
106
+
if you can strip external dependency and still show the problem. The
107
+
less time we spend on reproducing problems the more time we have to
108
+
fix it
109
+
110
+
- verifiable – test the code you're about to provide to make sure it
111
+
reproduces the problem. Remove all other problems that are not
0 commit comments