Skip to content

Commit 12982d8

Browse files
committed
init blog for model booster controller
Signed-off-by: HunterChen <[email protected]>
1 parent 9ebbac7 commit 12982d8

File tree

1 file changed

+86
-0
lines changed
  • docs/kthena/blog/model-controller-manager

1 file changed

+86
-0
lines changed
Lines changed: 86 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,86 @@
1+
---
2+
slug: model-booster-controller-blog-post
3+
title: All you should know about model-booster controller
4+
authors: [ HunterChen ]
5+
tags: [ ]
6+
---
7+
8+
# All you should know about model-booster controller
9+
10+
## 1. Brief Introduction
11+
12+
Model-booster controller is a k8s [operator](https://kubernetes.io/docs/concepts/extend-kubernetes/operator/) that helps
13+
you manage your model deployments. It automates the deployment of models in a Kubernetes
14+
cluster. If you don't use model-booster to manager models in Kthena, you need to create `ModelServing`,
15+
`AutoscalingPolicy`,`AutoscalingPolicyBinding`, `ModelRoute`, `ModelServer` one by one.
16+
Model-booster controller will help you create these resources automatically based on the `ModelBooster` custom resource.
17+
18+
There are three kinds of conditions for `ModelBooster` custom resource:
19+
20+
- `Initialized`: The `ModelBooster` CR passed validation and is processing.
21+
- `Active`: Everything is ok, now you can use the model.
22+
- `Failed`: There is something wrong, check the message for details.
23+
24+
We use `true`, `false` and `unknown` to represent the status of each condition. `true` means the condition is met,
25+
`false` means the
26+
condition is not met, and `unknown` means we don't know for yet.
27+
28+
In kthena version v0.1.0, we only support `vLLM` as the model backend. In the future, we will support `SGLang`, `MindIE`.
29+
30+
## 2. Core Features
31+
32+
### 2.1 Automated Deployment
33+
34+
With model-booster controller, you can create/update/delete models with a single
35+
`ModelBooster` custom resource, the controller will create/update/delete the corresponding `ModelServing`,
36+
`AutoscalingPolicy`,`AutoscalingPolicyBinding`, `ModelRoute`, `ModelServer` resources automatically.
37+
38+
`ModelBooster` supports models from different sources, including:
39+
- `HuggingFace`(recommend): Models hosted on HuggingFace.
40+
- `S3`: Models stored in S3-compatible storage.
41+
- `PVC`: Models stored in Persistent Volume Claims.
42+
43+
When you first deploy a model using `ModelBooster`, the controller will download the model from the sources into the `cache` you specified, next time when you deploy the same model, the controller will use the cached model to speed up the deployment.
44+
45+
### 2.2 Validation
46+
47+
The webhook and the controller will validate the `ModelBooster` custom resource before
48+
creating/updating/deleting the corresponding resources to ensure that the resource is valid.
49+
50+
You can find the validations of CR in
51+
the [CRD Reference](https://kthena.volcano.sh/docs/reference/crd/workload.serving.volcano.sh#modelbooster)
52+
53+
### 2.3 Reconciliation
54+
55+
The controller will continuously monitor the state of the `ModelBooster` custom resource and the
56+
corresponding resources, and will take action to ensure that the desired state is always maintained. And when all the
57+
resources are ready, the controller will update the condition of the `ModelBooster` custom resource to `Active`.
58+
59+
### 2.4 Best Practice templates
60+
61+
We provide some best practice templates for `ModelBooster` custom resource to help you get started quickly. Including:
62+
- DeepSeek-R1
63+
- DeepSeek-R1-Distill-Qwen-7B
64+
- DeepSeek-R1-Distill-Qwen-32B
65+
- gemma-2-2b-it
66+
- gemma-2-27b-it
67+
- gemma-3-4b-it
68+
- gemma-3-27b-it
69+
- Llama-3.2-1B-Instruct
70+
- Llama-3.3-70B-Instruct
71+
- Llama-4-Maverick-17B-128E-Instruct-FP8
72+
- Llama-4-Scout-17B-16E-Instruct
73+
- Meta-Llama-3-8B
74+
- Mistral-Small-24B-Instruct-2501
75+
- Mixtral-8x7B-Instruct-v0.1
76+
- Mixtral-8x22B-Instruct-v0.1
77+
- gpt-oss-20b
78+
- gpt-oss-120b
79+
- Qwen3-32B
80+
81+
For more details, please refer to the [CLI documentation](/cli/kthena/README.md).
82+
83+
## 3. Conclusion
84+
85+
Model-booster controller simplifies the management of model deployments in Kthena. It automates the creation and
86+
management of multiple resources, ensuring that your models are always deployed and running as expected.

0 commit comments

Comments
 (0)