@@ -7,24 +7,28 @@ can be served with vLLM.
77
88TerraTorch models can be served with vLLM in _ tensor-to-tensor_ or
99_ image-to-image_ mode. The tensor-to-tensor mode is the default mode and is
10- natively enabled by vLLM. For the image-to-image mode instead , TerraTorch uses a
11- feature in vLLM called
10+ natively enabled by vLLM. For the image-to-image mode, TerraTorch uses a feature
11+ in vLLM called
1212[ IOProcessor plugins] ( https://docs.vllm.ai/en/v0.13.0/design/io_processor_plugins/#writing-an-io-processor-plugin ) ,
1313enabling processing and generation of data in any modality (e.g., geoTiff). In
14- TerraTorch we provide pre-defined IOProcessor plugins, check the list
15- [ here ] ( ./vllm_io_plugins.md#available-terratorch-ioprocessor-plugins ) .
14+ TerraTorch, we provide pre-defined IOProcessor plugins. Check the list of
15+ [ available plugins ] ( ./vllm_io_plugins.md#available-terratorch-ioprocessor-plugins ) .
1616
17- To enable your model to be served via vLLM, follow the below steps:
17+ To enable your model to be served via vLLM, follow the steps below :
1818
19- 1 . Verify the model you want to serve is either already a core model, or learn
20- how to [ add your model to TerraTorch] ( ../models.md#adding-a-new-model ) .
21- 2 . [ Prepare your model for serving with vLLM] ( ./prepare_your_model.md ) .
22- 3 . If serving in image-to-image mode
23- [ Learn about IOProcessor plugins] ( ./vllm_io_plugins.md ) , identify an existing
24- one suiting your model or
19+ 1 . ** Ensure TerraTorch Integration** : Verify the model you want to serve is
20+ either already a core model, or learn how to
21+ [ add your model to TerraTorch] ( ../models.md#adding-a-new-model ) .
22+ 2 . ** Create a Model _ config.json_ ** : Create a
23+ [ vLLM compatible _ config.json_ ] ( ./prepare_your_model.md ) .
24+ 3 . ** Determine IOProcessor Plugin Needs** : If serving in image-to-image mode,
25+ identify an [ IOProcessor plugin] ( ./vllm_io_plugins.md ) that suits your model
26+ or
2527 [ build one yourself] ( https://docs.vllm.ai/en/latest/design/io_processor_plugins/ ) .
26- 4 . Ensure your model weights and config.json are either hosted on Hugging Face,
27- or stored in a local directory and accessible by your vLLM instance.
28- 5 . Start a vLLM serving instance that loads your model and perform an inference
29- in [ tensor-to-tensor mode] ( ./serving_a_model_tensor.md ) or in
30- [ image-to-image mode] ( ./serving_a_model_image.md ) .
28+ 4 . ** Make your Model Accessible to vLLM** : Host your model weights and
29+ config.json on Hugging Face, or store them in a local directory accessible by
30+ the vLLM instance.
31+
32+ To validate the steps above, start a vLLM serving instance that loads your model
33+ and perform an inference in [ tensor-to-tensor mode] ( ./serving_a_model_tensor.md )
34+ or in [ image-to-image mode] ( ./serving_a_model_image.md ) .
0 commit comments