You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: content/manuals/ai/model-runner.md
+4-185Lines changed: 4 additions & 185 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -42,191 +42,6 @@ Models are pulled from Docker Hub the first time they're used and stored locally
42
42
43
43
You can now use the `docker model` command in the CLI and view and interact with your local models in the **Models** tab in the Docker Desktop Dashboard.
44
44
45
-
## Available commands
46
-
47
-
### Model runner status
48
-
49
-
Check whether the Docker Model Runner is active and displays the current inference engine:
50
-
51
-
```console
52
-
$ docker model status
53
-
```
54
-
55
-
### View all commands
56
-
57
-
Displays help information and a list of available subcommands.
58
-
59
-
```console
60
-
$ docker model help
61
-
```
62
-
63
-
Output:
64
-
65
-
```text
66
-
Usage: docker model COMMAND
67
-
68
-
Commands:
69
-
list List models available locally
70
-
pull Download a model from Docker Hub
71
-
rm Remove a downloaded model
72
-
run Run a model interactively or with a prompt
73
-
status Check if the model runner is running
74
-
version Show the current version
75
-
```
76
-
77
-
### Pull a model
78
-
79
-
Pulls a model from Docker Hub to your local environment.
80
-
81
-
```console
82
-
$ docker model pull <model>
83
-
```
84
-
85
-
Example:
86
-
87
-
```console
88
-
$ docker model pull ai/smollm2
89
-
```
90
-
91
-
Output:
92
-
93
-
```text
94
-
Downloaded: 257.71 MB
95
-
Model ai/smollm2 pulled successfully
96
-
```
97
-
98
-
The models also display in the Docker Desktop Dashboard.
99
-
100
-
#### Pull from Hugging Face
101
-
102
-
You can also pull GGUF models directly from [Hugging Face](https://huggingface.co/models?library=gguf).
103
-
104
-
```console
105
-
$ docker model pull hf.co/<model-you-want-to-pull>
106
-
```
107
-
108
-
For example:
109
-
110
-
```console
111
-
$ docker model pull hf.co/bartowski/Llama-3.2-1B-Instruct-GGUF
112
-
```
113
-
114
-
Pulls the [bartowski/Llama-3.2-1B-Instruct-GGUF](https://huggingface.co/bartowski/Llama-3.2-1B-Instruct-GGUF).
115
-
116
-
### List available models
117
-
118
-
Lists all models currently pulled to your local environment.
119
-
120
-
```console
121
-
$ docker model list
122
-
```
123
-
124
-
You will see something similar to:
125
-
126
-
```text
127
-
+MODEL PARAMETERS QUANTIZATION ARCHITECTURE MODEL ID CREATED SIZE
128
-
+ai/smollm2 361.82 M IQ2_XXS/Q4_K_M llama 354bf30d0aa3 3 days ago 256.35 MiB
129
-
```
130
-
131
-
### Run a model
132
-
133
-
Run a model and interact with it using a submitted prompt or in chat mode. When you run a model, Docker
134
-
calls an Inference Server API endpoint hosted by the Model Runner through Docker Desktop. The model
135
-
stays in memory until another model is requested, or until a pre-defined inactivity timeout is reached (currently 5 minutes).
136
-
137
-
You do not have to use `Docker model run` before interacting with a specific model from a
138
-
host process or from within a container. Model Runner transparently loads the requested model on-demand, assuming it has been
139
-
pulled beforehand and is locally available.
140
-
141
-
#### One-time prompt
142
-
143
-
```console
144
-
$ docker model run ai/smollm2 "Hi"
145
-
```
146
-
147
-
Output:
148
-
149
-
```text
150
-
Hello! How can I assist you today?
151
-
```
152
-
153
-
#### Interactive chat
154
-
155
-
```console
156
-
$ docker model run ai/smollm2
157
-
```
158
-
159
-
Output:
160
-
161
-
```text
162
-
Interactive chat mode started. Type '/bye' to exit.
163
-
> Hi
164
-
Hi there! It's SmolLM, AI assistant. How can I help you today?
165
-
> /bye
166
-
Chat session ended.
167
-
```
168
-
169
-
> [!TIP]
170
-
>
171
-
> You can also use chat mode in the Docker Desktop Dashboard when you select the model in the **Models** tab.
172
-
173
-
### Push a model to Docker Hub
174
-
175
-
To push your model to Docker Hub:
176
-
177
-
```console
178
-
$ docker model push <namespace>/<model>
179
-
```
180
-
181
-
### Tag a model
182
-
183
-
To specify a particular version or variant of the model:
184
-
185
-
```console
186
-
$ docker model tag
187
-
```
188
-
189
-
If no tag is provided, Docker defaults to `latest`.
190
-
191
-
### View the logs
192
-
193
-
Fetch logs from Docker Model Runner to monitor activity or debug issues.
194
-
195
-
```console
196
-
$ docker model logs
197
-
```
198
-
199
-
The following flags are accepted:
200
-
201
-
-`-f`/`--follow`: View logs with real-time streaming
202
-
-`--no-engines`: Exclude inference engine logs from the output
203
-
204
-
### Remove a model
205
-
206
-
Removes a downloaded model from your system.
207
-
208
-
```console
209
-
$ docker model rm <model>
210
-
```
211
-
212
-
Output:
213
-
214
-
```text
215
-
Model <model> removed successfully
216
-
```
217
-
218
-
### Package a model
219
-
220
-
Packages a GGUF file into a Docker model OCI artifact, with optional licenses, and pushes it to the specified registry.
221
-
222
-
```console
223
-
$ docker model package \
224
-
--gguf ./model.gguf \
225
-
--licenses license1.txt \
226
-
--licenses license2.txt \
227
-
--push registry.example.com/ai/custom-model
228
-
```
229
-
230
45
## Integrate the Docker Model Runner into your software development lifecycle
231
46
232
47
You can now start building your Generative AI application powered by the Docker Model Runner.
@@ -255,6 +70,10 @@ You can now interact with your own GenAI app, powered by a local model. Try a fe
255
70
256
71
All the available models are hosted in the [public Docker Hub namespace of `ai`](https://hub.docker.com/u/ai).
257
72
73
+
### What CLI commands are available?
74
+
75
+
See [the reference docs](/reference/cli/docker/model/).
76
+
258
77
### What API endpoints are available?
259
78
260
79
Once the feature is enabled, new API endpoints are available under the following base URLs:
0 commit comments