Skip to content

Commit 43000fe

Browse files
committed
fixup: Resolve Code Review
1 parent 987cf37 commit 43000fe

File tree

1 file changed

+35
-38
lines changed

1 file changed

+35
-38
lines changed

README.md

Lines changed: 35 additions & 38 deletions
Original file line numberDiff line numberDiff line change
@@ -46,9 +46,7 @@ Ask questions or report problems using
4646

4747
Be sure to read all the information below as well as the
4848
[general Triton documentation](https://github.com/triton-inference-server/server#triton-inference-server)
49-
available in the
50-
[Triton Server](https://github.com/triton-inference-server/server)
51-
repository.
49+
available in the [Triton Server](https://github.com/triton-inference-server/server) repository.
5250

5351
## Build the PyTorch Backend
5452

@@ -103,6 +101,40 @@ make install
103101

104102
## Using the PyTorch Backend
105103

104+
### PyTorch 2.0 Models
105+
106+
The model repository should look like:
107+
108+
```bash
109+
model_repository/
110+
`-- model_directory
111+
|-- 1
112+
| |-- model.py
113+
| `-- [model.pt]
114+
`-- config.pbtxt
115+
```
116+
117+
The `model.py` contains the class definition of the PyTorch model.
118+
The class should extend the
119+
[`torch.nn.Module`](https://pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module).
120+
The `model.pt` may be optionally provided which contains the saved
121+
[`state_dict`](https://pytorch.org/tutorials/beginner/saving_loading_models.html#saving-loading-model-for-inference)
122+
of the model.
123+
124+
### TorchScript Models
125+
126+
The model repository should look like:
127+
128+
```bash
129+
model_repository/
130+
`-- model_directory
131+
|-- 1
132+
| `-- model.pt
133+
`-- config.pbtxt
134+
```
135+
136+
The `model.pt` is the TorchScript model file.
137+
106138
### Parameters
107139

108140
Triton exposes some flags to control the execution mode of the TorchScript models through the `Parameters` section of the model's `config.pbtxt` file.
@@ -248,7 +280,6 @@ Triton exposes some flags to control the execution mode of the TorchScript model
248280
* **Additional Optimizations**:
249281

250282
Three additional boolean parameters are available to disable certain Torch optimizations that can sometimes cause latency regressions in models with complex execution modes and dynamic shapes.
251-
252283
If not specified, all are enabled by default.
253284

254285
`ENABLE_JIT_EXECUTOR`
@@ -283,40 +314,6 @@ where the input tensors are placed as follows:
283314
> [!IMPORTANT]
284315
> If a device is not specified in the model, the backend uses the first available GPU device.
285316
286-
### PyTorch 2.0 Models
287-
288-
The model repository should look like:
289-
290-
```bash
291-
model_repository/
292-
`-- model_directory
293-
|-- 1
294-
| |-- model.py
295-
| `-- [model.pt]
296-
`-- config.pbtxt
297-
```
298-
299-
The `model.py` contains the class definition of the PyTorch model.
300-
The class should extend the
301-
[`torch.nn.Module`](https://pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module).
302-
The `model.pt` may be optionally provided which contains the saved
303-
[`state_dict`](https://pytorch.org/tutorials/beginner/saving_loading_models.html#saving-loading-model-for-inference)
304-
of the model.
305-
306-
### TorchScript Models
307-
308-
The model repository should look like:
309-
310-
```bash
311-
model_repository/
312-
`-- model_directory
313-
|-- 1
314-
| `-- model.pt
315-
`-- config.pbtxt
316-
```
317-
318-
The `model.pt` is the TorchScript model file.
319-
320317
### Customization
321318

322319
The following PyTorch settings may be customized by setting parameters on the

0 commit comments

Comments
 (0)