Skip to content

Commit 1fd9392

Browse files
committed
fixup: Resolve Code Review
1 parent 1182316 commit 1fd9392

File tree

1 file changed

+35
-38
lines changed

1 file changed

+35
-38
lines changed

README.md

Lines changed: 35 additions & 38 deletions
Original file line numberDiff line numberDiff line change
@@ -46,9 +46,7 @@ Ask questions or report problems using
4646

4747
Be sure to read all the information below as well as the
4848
[general Triton documentation](https://github.com/triton-inference-server/server#triton-inference-server)
49-
available in the
50-
[Triton Server](https://github.com/triton-inference-server/server)
51-
repository.
49+
available in the [Triton Server](https://github.com/triton-inference-server/server) repository.
5250

5351
## Build the PyTorch Backend
5452

@@ -103,6 +101,40 @@ make install
103101

104102
## Using the PyTorch Backend
105103

104+
### PyTorch 2.0 Models
105+
106+
The model repository should look like:
107+
108+
```bash
109+
model_repository/
110+
`-- model_directory
111+
|-- 1
112+
| |-- model.py
113+
| `-- [model.pt]
114+
`-- config.pbtxt
115+
```
116+
117+
The `model.py` contains the class definition of the PyTorch model.
118+
The class should extend the
119+
[`torch.nn.Module`](https://pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module).
120+
The `model.pt` may be optionally provided which contains the saved
121+
[`state_dict`](https://pytorch.org/tutorials/beginner/saving_loading_models.html#saving-loading-model-for-inference)
122+
of the model.
123+
124+
### TorchScript Models
125+
126+
The model repository should look like:
127+
128+
```bash
129+
model_repository/
130+
`-- model_directory
131+
|-- 1
132+
| `-- model.pt
133+
`-- config.pbtxt
134+
```
135+
136+
The `model.pt` is the TorchScript model file.
137+
106138
### Parameters
107139

108140
Triton exposes some flags to control the execution mode of the TorchScript models through the `Parameters` section of the model's `config.pbtxt` file.
@@ -253,7 +285,6 @@ Triton exposes some flags to control the execution mode of the TorchScript model
253285
* **Additional Optimizations**:
254286

255287
Three additional boolean parameters are available to disable certain Torch optimizations that can sometimes cause latency regressions in models with complex execution modes and dynamic shapes.
256-
257288
If not specified, all are enabled by default.
258289

259290
`ENABLE_JIT_EXECUTOR`
@@ -288,40 +319,6 @@ where the input tensors are placed as follows:
288319
> [!IMPORTANT]
289320
> If a device is not specified in the model, the backend uses the first available GPU device.
290321
291-
### PyTorch 2.0 Models
292-
293-
The model repository should look like:
294-
295-
```bash
296-
model_repository/
297-
`-- model_directory
298-
|-- 1
299-
| |-- model.py
300-
| `-- [model.pt]
301-
`-- config.pbtxt
302-
```
303-
304-
The `model.py` contains the class definition of the PyTorch model.
305-
The class should extend the
306-
[`torch.nn.Module`](https://pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module).
307-
The `model.pt` may be optionally provided which contains the saved
308-
[`state_dict`](https://pytorch.org/tutorials/beginner/saving_loading_models.html#saving-loading-model-for-inference)
309-
of the model.
310-
311-
### TorchScript Models
312-
313-
The model repository should look like:
314-
315-
```bash
316-
model_repository/
317-
`-- model_directory
318-
|-- 1
319-
| `-- model.pt
320-
`-- config.pbtxt
321-
```
322-
323-
The `model.pt` is the TorchScript model file.
324-
325322
### Customization
326323

327324
The following PyTorch settings may be customized by setting parameters on the

0 commit comments

Comments
 (0)