File tree Expand file tree Collapse file tree 1 file changed +11
-1
lines changed
Expand file tree Collapse file tree 1 file changed +11
-1
lines changed Original file line number Diff line number Diff line change 11command : docker model configure
22short : Configure runtime options for a model
33long : Configure runtime options for a model
4- usage : docker model configure [--context-size=<n>] [--speculative-draft-model=<model>] [--hf_overrides=<json>] [--mode=<mode>] [--think] MODEL
4+ usage : docker model configure [--context-size=<n>] [--speculative-draft-model=<model>] [--hf_overrides=<json>] [--gpu-memory-utilization=<float>] [-- mode=<mode>] [--think] MODEL
55pname : docker model
66plink : docker_model.yaml
77options :
@@ -14,6 +14,16 @@ options:
1414 experimentalcli : false
1515 kubernetes : false
1616 swarm : false
17+ - option : gpu-memory-utilization
18+ value_type : float64
19+ description : |
20+ fraction of GPU memory to use for the model executor (0.0-1.0) - vLLM only
21+ deprecated : false
22+ hidden : false
23+ experimental : false
24+ experimentalcli : false
25+ kubernetes : false
26+ swarm : false
1727 - option : hf_overrides
1828 value_type : string
1929 description : HuggingFace model config overrides (JSON) - vLLM only
You can’t perform that action at this time.
0 commit comments