Skip to content

Commit 9c9d31b

Browse files
aniketmauryaaniketmaurya
authored
Document server hosting (#458)
* add deployment info on readme * update * update * update * update * add link * update layout * update * fix
1 parent c5ca70d commit 9c9d31b

File tree

1 file changed

+57
-35
lines changed

1 file changed

+57
-35
lines changed

‎README.md‎

Lines changed: 57 additions & 35 deletions
Original file line numberDiff line numberDiff line change
@@ -99,12 +99,30 @@ if __name__ == "__main__":
9999
server.run(port=8000)
100100
```
101101

102-
Now run the server via the command-line
102+
Now run the server anywhere (local or cloud) via the command-line.
103+
104+
### Run locally
103105

104106
```bash
107+
lightning serve api server.py
108+
```
109+
110+
You can also run the server directly in Python:
111+
112+
```bash
105113
python server.py
106114
```
107-
115+
116+
### Run on the cloud
117+
118+
Deploy the server to Lightning AI for fully managed hosting (autoscaling, security, etc...):
119+
120+
```bash
121+
lightning serve api server.py --cloud
122+
```
123+
124+
Learn more about deployment options and cloud hosting [here](https://lightning.ai/docs/litserve/features/deploy-on-cloud).
125+
108126
### Test the server
109127
Run the auto-generated test client:
110128
```bash
@@ -128,7 +146,7 @@ litgpt serve microsoft/phi-2
128146
- LitAPI lets you easily build complex AI systems with one or more models ([docs](https://lightning.ai/docs/litserve/api-reference/litapi)).
129147
- Use the setup method for one-time tasks like connecting models, DBs, and loading data ([docs](https://lightning.ai/docs/litserve/api-reference/litapi#setup)).
130148
- LitServer handles optimizations like batching, GPU autoscaling, streaming, etc... ([docs](https://lightning.ai/docs/litserve/api-reference/litserver)).
131-
- Self host on your own machines or use Lightning Studios for a fully managed deployment ([learn more](#hosting-options)).
149+
- Self host on your machines or create a fully managed deployment with Lightning ([learn more](https://lightning.ai/docs/litserve/features/deploy-on-cloud)).
132150

133151
[Learn how to make this server 200x faster](https://lightning.ai/docs/litserve/home/speed-up-serving-by-200x).
134152

@@ -163,6 +181,41 @@ Use LitServe to deploy any model or AI service: (Compound AI, Gen AI, classic ML
163181

164182
 
165183

184+
185+
# Hosting options
186+
LitServe can be hosted independently on your own machines or fully managed via Lightning Studios.
187+
188+
Self-hosting is ideal for hackers, students, and DIY developers, while fully managed hosting is ideal for enterprise developers needing easy autoscaling, security, release management, and 99.995% uptime and observability.
189+
190+
 
191+
192+
<div align="center">
193+
<a target="_blank" href="https://lightning.ai/docs/litserve/features/deploy-on-cloud">
194+
<img src="https://pl-bolts-doc-images.s3.us-east-2.amazonaws.com/app-2/host-on-lightning.svg" alt="Host on Lightning"/>
195+
</a>
196+
</div>
197+
198+
&nbsp;
199+
200+
<div align='center'>
201+
202+
| Feature | Self Managed | Fully Managed on Lightning |
203+
|----------------------------------|-----------------------------------|----------------------------|
204+
| Deployment | ✅ Do it yourself deployment | ✅ One-button cloud deploy |
205+
| Load balancing | ❌ | ✅ |
206+
| Autoscaling | ❌ | ✅ |
207+
| Scale to zero | ❌ | ✅ |
208+
| Multi-machine inference | ❌ | ✅ |
209+
| Authentication | ❌ | ✅ |
210+
| Own VPC | ❌ | ✅ |
211+
| AWS, GCP | ❌ | ✅ |
212+
| Use your own cloud commits | ❌ | ✅ |
213+
214+
</div>
215+
216+
&nbsp;
217+
218+
166219
# Features
167220
State-of-the-art features:
168221

@@ -173,6 +226,7 @@ State-of-the-art features:
173226
✅ [Batching](https://lightning.ai/docs/litserve/features/batching)
174227
✅ [Streaming](https://lightning.ai/docs/litserve/features/streaming)
175228
✅ [Worker autoscaling](https://lightning.ai/docs/litserve/features/autoscaling)
229+
✅ [Deploy with Lightning AI](https://lightning.ai/docs/litserve/features/deploy-on-cloud)
176230
✅ [Self-host on your machines](https://lightning.ai/docs/litserve/features/hosting-methods#host-on-your-own)
177231
✅ [Host fully managed on Lightning AI](https://lightning.ai/docs/litserve/features/hosting-methods#host-on-lightning-studios)
178232
✅ [Serve all models: (LLMs, vision, etc.)](https://lightning.ai/docs/litserve/examples)
@@ -206,40 +260,8 @@ These results are for image and text classification ML tasks. The performance re
206260

207261
***💡 Note on LLM serving:*** For high-performance LLM serving (like Ollama/vLLM), integrate [vLLM with LitServe](https://lightning.ai/lightning-ai/studios/deploy-a-private-llama-3-2-rag-api), use [LitGPT](https://github.com/Lightning-AI/litgpt?tab=readme-ov-file#deploy-an-llm), or build your custom vLLM-like server with LitServe. Optimizations like kv-caching, which can be done with LitServe, are needed to maximize LLM performance.
208262

209-
&nbsp;
210-
211-
# Hosting options
212-
LitServe can be hosted independently on your own machines or fully managed via Lightning Studios.
213-
214-
Self-hosting is ideal for hackers, students, and DIY developers, while fully managed hosting is ideal for enterprise developers needing easy autoscaling, security, release management, and 99.995% uptime and observability.
215-
216263
&nbsp;
217264

218-
<div align="center">
219-
<a target="_blank" href="https://lightning.ai/lightning-ai/studios/litserve-hello-world">
220-
<img src="https://pl-bolts-doc-images.s3.us-east-2.amazonaws.com/app-2/host-on-lightning.svg" alt="Host on Lightning"/>
221-
</a>
222-
</div>
223-
224-
&nbsp;
225-
226-
<div align='center'>
227-
228-
| Feature | Self Managed | Fully Managed on Studios |
229-
|----------------------------------|-----------------------------------|-------------------------------------|
230-
| Deployment | ✅ Do it yourself deployment | ✅ One-button cloud deploy |
231-
| Load balancing | ❌ | ✅ |
232-
| Autoscaling | ❌ | ✅ |
233-
| Scale to zero | ❌ | ✅ |
234-
| Multi-machine inference | ❌ | ✅ |
235-
| Authentication | ❌ | ✅ |
236-
| Own VPC | ❌ | ✅ |
237-
| AWS, GCP | ❌ | ✅ |
238-
| Use your own cloud commits | ❌ | ✅ |
239-
240-
</div>
241-
242-
&nbsp;
243265

244266
# Community
245267
LitServe is a [community project accepting contributions](https://lightning.ai/docs/litserve/community) - Let's make the world's most advanced AI inference engine.

0 commit comments

Comments
 (0)