|
| 1 | +--- |
| 2 | +slug: model-booster-controller-blog-post |
| 3 | +title: All you should know about model-booster controller |
| 4 | +authors: [ HunterChen ] |
| 5 | +tags: [ ] |
| 6 | +--- |
| 7 | + |
| 8 | +# All you should know about model-booster controller |
| 9 | + |
| 10 | +## 1. Brief Introduction |
| 11 | + |
| 12 | +Model-booster controller is a k8s [operator](https://kubernetes.io/docs/concepts/extend-kubernetes/operator/) that helps |
| 13 | +you manage your model deployments. It automates the deployment of models in a Kubernetes |
| 14 | +cluster. If you don't use model-booster to manager models in Kthena, you need to create `ModelServing`, |
| 15 | +`AutoscalingPolicy`,`AutoscalingPolicyBinding`, `ModelRoute`, `ModelServer` one by one. |
| 16 | +Model-booster controller will help you create these resources automatically based on the `ModelBooster` custom resource. |
| 17 | + |
| 18 | +There are three kinds of conditions for `ModelBooster` custom resource: |
| 19 | + |
| 20 | +- `Initialized`: The `ModelBooster` CR passed validation and is processing. |
| 21 | +- `Active`: Everything is ok, now you can use the model. |
| 22 | +- `Failed`: There is something wrong, check the message for details. |
| 23 | + |
| 24 | +We use `true`, `false` and `unknown` to represent the status of each condition. `true` means the condition is met, |
| 25 | +`false` means the |
| 26 | +condition is not met, and `unknown` means we don't know for yet. |
| 27 | + |
| 28 | +In kthena version v0.1.0, we only support `vLLM` as the model backend. In the future, we will support `SGLang`, `MindIE`. |
| 29 | + |
| 30 | +## 2. Core Features |
| 31 | + |
| 32 | +### 2.1 Automated Deployment |
| 33 | + |
| 34 | +With model-booster controller, you can create/update/delete models with a single |
| 35 | +`ModelBooster` custom resource, the controller will create/update/delete the corresponding `ModelServing`, |
| 36 | +`AutoscalingPolicy`,`AutoscalingPolicyBinding`, `ModelRoute`, `ModelServer` resources automatically. |
| 37 | + |
| 38 | +`ModelBooster` supports models from different sources, including: |
| 39 | +- `HuggingFace`(recommend): Models hosted on HuggingFace. |
| 40 | +- `S3`: Models stored in S3-compatible storage. |
| 41 | +- `PVC`: Models stored in Persistent Volume Claims. |
| 42 | + |
| 43 | +When you first deploy a model using `ModelBooster`, the controller will download the model from the sources into the `cache` you specified, next time when you deploy the same model, the controller will use the cached model to speed up the deployment. |
| 44 | + |
| 45 | +### 2.2 Validation |
| 46 | + |
| 47 | +The webhook and the controller will validate the `ModelBooster` custom resource before |
| 48 | +creating/updating/deleting the corresponding resources to ensure that the resource is valid. |
| 49 | + |
| 50 | +You can find the validations of CR in |
| 51 | +the [CRD Reference](https://kthena.volcano.sh/docs/reference/crd/workload.serving.volcano.sh#modelbooster) |
| 52 | + |
| 53 | +### 2.3 Reconciliation |
| 54 | + |
| 55 | +The controller will continuously monitor the state of the `ModelBooster` custom resource and the |
| 56 | +corresponding resources, and will take action to ensure that the desired state is always maintained. And when all the |
| 57 | +resources are ready, the controller will update the condition of the `ModelBooster` custom resource to `Active`. |
| 58 | + |
| 59 | +### 2.4 Best Practice templates |
| 60 | + |
| 61 | +We provide some best practice templates for `ModelBooster` custom resource to help you get started quickly. Including: |
| 62 | +- DeepSeek-R1 |
| 63 | +- DeepSeek-R1-Distill-Qwen-7B |
| 64 | +- DeepSeek-R1-Distill-Qwen-32B |
| 65 | +- gemma-2-2b-it |
| 66 | +- gemma-2-27b-it |
| 67 | +- gemma-3-4b-it |
| 68 | +- gemma-3-27b-it |
| 69 | +- Llama-3.2-1B-Instruct |
| 70 | +- Llama-3.3-70B-Instruct |
| 71 | +- Llama-4-Maverick-17B-128E-Instruct-FP8 |
| 72 | +- Llama-4-Scout-17B-16E-Instruct |
| 73 | +- Meta-Llama-3-8B |
| 74 | +- Mistral-Small-24B-Instruct-2501 |
| 75 | +- Mixtral-8x7B-Instruct-v0.1 |
| 76 | +- Mixtral-8x22B-Instruct-v0.1 |
| 77 | +- gpt-oss-20b |
| 78 | +- gpt-oss-120b |
| 79 | +- Qwen3-32B |
| 80 | + |
| 81 | +For more details, please refer to the [CLI documentation](/cli/kthena/README.md). |
| 82 | + |
| 83 | +## 3. Conclusion |
| 84 | + |
| 85 | +Model-booster controller simplifies the management of model deployments in Kthena. It automates the creation and |
| 86 | +management of multiple resources, ensuring that your models are always deployed and running as expected. |
0 commit comments