Skip to content

Commit ce28ec9

Browse files
authored
Update README and versions for 1.40.0 / 24.05 (#883)
* Update README and versions for 1.40.0 / 24.05
1 parent ad32523 commit ce28ec9

File tree

1 file changed

+96
-3
lines changed

1 file changed

+96
-3
lines changed

README.md

Lines changed: 96 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
<!--
2-
Copyright (c) 2020-2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2+
Copyright (c) 2020-2024, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
33
44
Licensed under the Apache License, Version 2.0 (the "License");
55
you may not use this file except in compliance with the License.
@@ -18,5 +18,98 @@ limitations under the License.
1818

1919
# Triton Model Analyzer
2020

21-
> [!Warning]
22-
> You are currently on the `24.05` branch which tracks under-development and unreleased features.
21+
Triton Model Analyzer is a CLI tool which can help you find a more optimal configuration, on a given piece of hardware, for single, multiple, ensemble, or BLS models running on a [Triton Inference Server](https://github.com/triton-inference-server/server/). Model Analyzer will also generate reports to help you better understand the trade-offs of the different configurations along with their compute and memory requirements.
22+
<br><br>
23+
24+
# Features
25+
26+
### Search Modes
27+
28+
- [Quick Search](docs/config_search.md#quick-search-mode) will **sparsely** search the [Max Batch Size](https://github.com/triton-inference-server/server/blob/r24.05/docs/user_guide/model_configuration.md#maximum-batch-size),
29+
[Dynamic Batching](https://github.com/triton-inference-server/server/blob/r24.05/docs/user_guide/model_configuration.md#dynamic-batcher), and
30+
[Instance Group](https://github.com/triton-inference-server/server/blob/r24.05/docs/user_guide/model_configuration.md#instance-groups) spaces by utilizing a heuristic hill-climbing algorithm to help you quickly find a more optimal configuration
31+
32+
- [Automatic Brute Search](docs/config_search.md#automatic-brute-search) will **exhaustively** search the
33+
[Max Batch Size](https://github.com/triton-inference-server/server/blob/r24.05/docs/user_guide/model_configuration.md#maximum-batch-size),
34+
[Dynamic Batching](https://github.com/triton-inference-server/server/blob/r24.05/docs/user_guide/model_configuration.md#dynamic-batcher), and
35+
[Instance Group](https://github.com/triton-inference-server/server/blob/r24.05/docs/user_guide/model_configuration.md#instance-groups)
36+
parameters of your model configuration
37+
38+
- [Manual Brute Search](docs/config_search.md#manual-brute-search) allows you to create manual sweeps for every parameter that can be specified in the model configuration
39+
40+
### Model Types
41+
42+
- [Ensemble Model Search](docs/config_search.md#ensemble-model-search): Model Analyzer can help you find the optimal
43+
settings when profiling an ensemble model, utilizing the [Quick Search](docs/config_search.md#quick-search-mode) algorithm
44+
45+
- [BLS Model Search](docs/config_search.md#bls-model-search): Model Analyzer can help you find the optimal
46+
settings when profiling a BLS model, utilizing the [Quick Search](docs/config_search.md#quick-search-mode) algorithm
47+
48+
- [Multi-Model Search](docs/config_search.md#multi-model-search-mode): Model Analyzer can help you
49+
find the optimal settings when profiling multiple concurrent models, utilizing the [Quick Search](docs/config_search.md#quick-search-mode) algorithm
50+
51+
- [LLM Search](docs/config_search.md#llm-search-mode): Model Analyzer can help you
52+
find the optimal settings when profiling large language models, utilizing the [Quick Search](docs/config_search.md#quick-search-mode) algorithm
53+
54+
### Other Features
55+
56+
- [Detailed and summary reports](docs/report.md): Model Analyzer is able to generate
57+
summarized and detailed reports that can help you better understand the trade-offs
58+
between different model configurations that can be used for your model.
59+
60+
- [QoS Constraints](docs/config.md#constraint): Constraints can help you
61+
filter out the Model Analyzer results based on your QoS requirements. For
62+
example, you can specify a latency budget to filter out model configurations
63+
that do not satisfy the specified latency threshold.
64+
<br><br>
65+
66+
# Examples and Tutorials
67+
68+
### **Single Model**
69+
70+
See the [Single Model Quick Start](docs/quick_start.md) for a guide on how to use Model Analyzer to profile, analyze and report on a simple PyTorch model.
71+
72+
### **Multi Model**
73+
74+
See the [Multi-model Quick Start](docs/mm_quick_start.md) for a guide on how to use Model Analyzer to profile, analyze and report on two models running concurrently on the same GPU.
75+
76+
### **Ensemble Model**
77+
78+
See the [Ensemble Model Quick Start](docs/ensemble_quick_start.md) for a guide on how to use Model Analyzer to profile, analyze and report on a simple Ensemble model.
79+
80+
### **BLS Model**
81+
82+
See the [BLS Model Quick Start](docs/bls_quick_start.md) for a guide on how to use Model Analyzer to profile, analyze and report on a simple BLS model.
83+
<br><br>
84+
85+
# Documentation
86+
87+
- [Installation](docs/install.md)
88+
- [Model Analyzer CLI](docs/cli.md)
89+
- [Launch Modes](docs/launch_modes.md)
90+
- [Configuring Model Analyzer](docs/config.md)
91+
- [Model Analyzer Metrics](docs/metrics.md)
92+
- [Model Config Search](docs/config_search.md)
93+
- [Checkpointing](docs/checkpoints.md)
94+
- [Model Analyzer Reports](docs/report.md)
95+
- [Deployment with Kubernetes](docs/kubernetes_deploy.md)
96+
<br><br>
97+
98+
# Reporting problems, asking questions
99+
100+
We appreciate any feedback, questions or bug reporting regarding this
101+
project. When help with code is needed, follow the process outlined in
102+
the Stack Overflow (https://stackoverflow.com/help/mcve)
103+
document. Ensure posted examples are:
104+
105+
- minimal – use as little code as possible that still produces the
106+
same problem
107+
108+
- complete – provide all parts needed to reproduce the problem. Check
109+
if you can strip external dependency and still show the problem. The
110+
less time we spend on reproducing problems the more time we have to
111+
fix it
112+
113+
- verifiable – test the code you're about to provide to make sure it
114+
reproduces the problem. Remove all other problems that are not
115+
related to your request/question.

0 commit comments

Comments
 (0)