You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: content/manuals/ai/model-runner.md
+59-51Lines changed: 59 additions & 51 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -7,9 +7,9 @@ params:
7
7
text: Beta
8
8
group: AI
9
9
weight: 20
10
-
description: Learn how to use Docker Model Runner to manage and run AI models.
10
+
description: Learn how to use Docker Model Runner to manage and run AI models.
11
11
keywords: Docker, ai, model runner, docker deskotp, llm
12
-
aliases:
12
+
aliases:
13
13
- /desktop/features/model-runner/
14
14
- /ai/model-runner/
15
15
---
@@ -34,8 +34,8 @@ Models are pulled from Docker Hub the first time they're used and stored locally
34
34
35
35
1. Navigate to the **Features in development** tab in settings.
36
36
2. Under the **Experimental features** tab, select **Access experimental features**.
37
-
3. Select **Apply and restart**.
38
-
4. Quit and reopen Docker Desktop to ensure the changes take effect.
37
+
3. Select **Apply and restart**.
38
+
4. Quit and reopen Docker Desktop to ensure the changes take effect.
39
39
5. Open the **Settings** view in Docker Desktop.
40
40
6. Navigate to **Features in development**.
41
41
7. From the **Beta** tab, check the **Enable Docker Model Runner** setting.
@@ -46,7 +46,7 @@ You can now use the `docker model` command in the CLI and view and interact with
46
46
47
47
### Model runner status
48
48
49
-
Check whether the Docker Model Runner is active:
49
+
Check whether the Docker Model Runner is active and displays the current inference engine:
50
50
51
51
```console
52
52
$ docker model status
@@ -55,7 +55,7 @@ $ docker model status
55
55
### View all commands
56
56
57
57
Displays help information and a list of available subcommands.
58
-
58
+
59
59
```console
60
60
$ docker model help
61
61
```
@@ -74,15 +74,15 @@ Commands:
74
74
version Show the current version
75
75
```
76
76
77
-
### Pull a model
77
+
### Pull a model
78
78
79
79
Pulls a model from Docker Hub to your local environment.
80
80
81
81
```console
82
82
$ docker model pull <model>
83
83
```
84
84
85
-
Example:
85
+
Example:
86
86
87
87
```console
88
88
$ docker model pull ai/smollm2
@@ -114,7 +114,13 @@ You will see something similar to:
114
114
115
115
### Run a model
116
116
117
-
Run a model and interact with it using a submitted prompt or in chat mode.
117
+
Run a model and interact with it using a submitted prompt or in chat mode. When you run a model, Docker
118
+
calls an Inference Server API endpoint hosted by the Model Runner through Docker Desktop. The model
119
+
stays in memory until another model is requested, or until a pre-defined inactivity timeout is reached (currently 5 minutes).
120
+
121
+
You do not have to use `Docker model run` before interacting with a specific model from a
122
+
host process or from within a container. Model Runner transparently loads the requested model on-demand, assuming it has been
123
+
pulled beforehand and is locally available.
118
124
119
125
#### One-time prompt
120
126
@@ -150,18 +156,18 @@ Chat session ended.
150
156
151
157
### Push a model to Docker Hub
152
158
153
-
Use the following command to push your model to Docker Hub:
159
+
To push your model to Docker Hub:
154
160
155
161
```console
156
162
$ docker model push <namespace>/<model>
157
163
```
158
164
159
165
### Tag a model
160
166
161
-
You can specify a particular version or variant of the model:
167
+
To specify a particular version or variant of the model:
162
168
163
169
```console
164
-
$ docker model tag
170
+
$ docker model tag
165
171
```
166
172
167
173
If no tag is provided, Docker defaults to `latest`.
@@ -171,7 +177,7 @@ If no tag is provided, Docker defaults to `latest`.
171
177
Fetch logs from Docker Model Runner to monitor activity or debug issues.
172
178
173
179
```console
174
-
$ docker model logs
180
+
$ docker model logs
175
181
```
176
182
177
183
The following flags are accepted:
@@ -211,53 +217,54 @@ If you want to try an existing GenAI application, follow these instructions.
211
217
212
218
4. Open you app in the browser at the addresses specified in the repository [README](https://github.com/docker/hello-genai).
213
219
214
-
You'll see the GenAI app's interface where you can start typing your prompts.
220
+
You'll see the GenAI app's interface where you can start typing your prompts.
215
221
216
222
You can now interact with your own GenAI app, powered by a local model. Try a few prompts and notice how fast the responses are — all running on your machine with Docker.
217
223
218
224
## FAQs
219
225
220
226
### What models are available?
221
227
222
-
All the available models are hosted in the [public Docker Hub namespace of `ai`](https://hub.docker.com/u/ai).
228
+
All the available models are hosted in the [public Docker Hub namespace of `ai`](https://hub.docker.com/u/ai).
223
229
224
230
### What API endpoints are available?
225
231
226
-
Once the feature is enabled, the following new APIs are available:
232
+
Once the feature is enabled, new API endpoints are available under the following base URLs:
227
233
228
-
```text
229
-
#### Inside containers ####
234
+
- From containers: `http://model-runner.docker.internal/`
235
+
- From host processes: `http://localhost:12434/`, assuming you have enabled TCP host access on default port 12434.
230
236
231
-
http://model-runner.docker.internal/
237
+
Docker Model management endpoints:
232
238
233
-
# Docker Model management
234
-
POST /models/create
235
-
GET /models
236
-
GET /models/{namespace}/{name}
237
-
DELETE /models/{namespace}/{name}
239
+
```text
240
+
POST /models/create
241
+
GET /models
242
+
GET /models/{namespace}/{name}
243
+
DELETE /models/{namespace}/{name}
244
+
```
238
245
239
-
# OpenAI endpoints
240
-
GET /engines/llama.cpp/v1/models
241
-
GET /engines/llama.cpp/v1/models/{namespace}/{name}
242
-
POST /engines/llama.cpp/v1/chat/completions
243
-
POST /engines/llama.cpp/v1/completions
244
-
POST /engines/llama.cpp/v1/embeddings
245
-
Note: You can also omit llama.cpp.
246
-
E.g., POST /engines/v1/chat/completions.
246
+
OpenAI endpoints:
247
247
248
-
#### Inside or outside containers (host) ####
248
+
```text
249
+
GET /engines/llama.cpp/v1/models
250
+
GET /engines/llama.cpp/v1/models/{namespace}/{name}
251
+
POST /engines/llama.cpp/v1/chat/completions
252
+
POST /engines/llama.cpp/v1/completions
253
+
POST /engines/llama.cpp/v1/embeddings
254
+
```
249
255
250
-
Same endpoints on /var/run/docker.sock
256
+
To call these endpoints via a Unix socket (`/var/run/docker.sock`), prefix their path with
257
+
with `/exp/vDD4.40`.
258
+
259
+
> [!NOTE]
260
+
> You can omit `llama.cpp` from the path. For example: `POST /engines/v1/chat/completions`.
251
261
252
-
# While still in Beta
253
-
Prefixed with /exp/vDD4.40
254
-
```
255
262
256
263
### How do I interact through the OpenAI API?
257
264
258
265
#### From within a container
259
266
260
-
Examples of calling an OpenAI endpoint (`chat/completions`) from within another container using `curl`:
267
+
To call the `chat/completions` OpenAI endpoint from within another container using `curl`:
In case you want to interact with the API from the host, but use TCP instead of a Docker socket, you can enable the host-side TCP support from the Docker Desktop GUI, or via the [Docker Desktop CLI](/manuals/desktop/features/desktop-cli.md). For example, using `docker desktop enable model-runner --tcp <port>`.
318
+
#### From the host using a Unix socket
312
319
313
-
Afterwards, interact with it as previously documented using `localhost` and the chosen, or the default port.
320
+
To call the `chat/completions` OpenAI endpoint through the Docker socket from the host using `curl`:
@@ -354,15 +362,15 @@ Once linked, re-run the command.
354
362
355
363
### No safeguard for running oversized models
356
364
357
-
Currently, Docker Model Runner doesn't include safeguards to prevent you from launching models that exceed their system’s available resources. Attempting to run a model that is too large for the host machine may result in severe slowdowns or render the system temporarily unusable. This issue is particularly common when running LLMs models without sufficient GPU memory or system RAM.
365
+
Currently, Docker Model Runner doesn't include safeguards to prevent you from launching models that exceed their system's available resources. Attempting to run a model that is too large for the host machine may result in severe slowdowns or render the system temporarily unusable. This issue is particularly common when running LLMs models without sufficient GPU memory or system RAM.
358
366
359
367
### No consistent digest support in Model CLI
360
368
361
369
The Docker Model CLI currently lacks consistent support for specifying models by image digest. As a temporary workaround, you should refer to models by name instead of digest.
362
370
363
371
## Share feedback
364
372
365
-
Thanks for trying out Docker Model Runner. Give feedback or report any bugs you may find through the **Give feedback** link next to the **Enable Docker Model Runner** setting.
373
+
Thanks for trying out Docker Model Runner. Give feedback or report any bugs you may find through the **Give feedback** link next to the **Enable Docker Model Runner** setting.
366
374
367
375
## Disable the feature
368
376
@@ -371,4 +379,4 @@ To disable Docker Model Runner:
371
379
1. Open the **Settings** view in Docker Desktop.
372
380
2. Navigate to the **Beta** tab in **Features in development**.
373
381
3. Clear the **Enable Docker Model Runner** checkbox.
0 commit comments