Skip to content

Commit 5d19a63

Browse files
aevesdockeraevesdocker
committed
additions
1 parent ca50934 commit 5d19a63

File tree

1 file changed

+297
-5
lines changed

1 file changed

+297
-5
lines changed

‎content/manuals/desktop/features/model-runner.md‎

Lines changed: 297 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,6 @@
11
---
22
title: Docker Model Runner
33
params:
4-
sidebar:
5-
group: Products
64
badge:
75
color: blue
86
text: Beta
@@ -20,21 +18,315 @@ The Docker Model Runner plugin lets you:
2018
- Manage local models (list, remove)
2119
- Interact with models in prompt or chat mode
2220

21+
Models are pulled from Docker Hub and then loaded dynamically into memory based on request usage.
22+
The first pull may take a while; after that, model files are cached locally. You can interact with the model using [OpenAI-compatible APIs](#what-api-endpoints-are-available).
23+
2324
## Enable the feature
2425

25-
To enable Docker Model runner:
26+
To enable Docker Model Runner:
2627

2728
1. Open the **Settings** view in Docker Desktop.
2829
2. Navigate to the **Beta** tab in **Features in development**.
2930
3. Check the **Enable Docker Model Runner** checkbox.
3031
4. Select **Apply & restart**.
3132

32-
## Use the Docker Model Runner through the CLI
33+
## Available commands
34+
35+
### Model runner status
36+
37+
Check whether the Docker Model Runner is active:
38+
39+
```console
40+
$ docker model status
41+
```
42+
43+
### View all commands
44+
45+
Displays help information and a list of available subcommands.
46+
47+
```console
48+
$ docker model help
49+
```
50+
51+
Output:
52+
53+
```text
54+
Usage: docker model COMMAND
55+
56+
Commands:
57+
list List models available locally
58+
pull Download a model from Docker Hub
59+
rm Remove a downloaded model
60+
run Run a model interactively or with a prompt
61+
status Check if the model runner is running
62+
version Show the current version
63+
```
64+
65+
### Pull a model
66+
67+
Pulls a model from Docker Hub to your local environment.
68+
69+
```console
70+
$ docker model pull <model>
71+
```
72+
73+
Example:
74+
75+
```console
76+
$ docker model pull ignaciolopezluna020/llama3.2:1b
77+
```
78+
79+
Output:
80+
81+
```text
82+
Downloaded: 626.05 MB
83+
Model ignaciolopezluna020/llama3.2:1b pulled successfully
84+
```
85+
86+
### List available models
87+
88+
Lists all models currently pulled to your local environment.
89+
90+
```console
91+
$ docker model list
92+
```
93+
94+
If no models have been pulled yet, you will see:
95+
96+
```json
97+
{"object":"list","data":[]}
98+
```
99+
100+
For better readability, format the output using `jq`:
101+
102+
```console
103+
$ docker model list | jq .
104+
```
105+
106+
Expected formatted output:
107+
108+
```json
109+
{
110+
"object": "list",
111+
"data": []
112+
}
113+
```
114+
115+
### Run a model
116+
117+
Runs a model with an optional prompt or in interactive mode.
118+
119+
#### One-time prompt
120+
121+
```console
122+
$ docker model run ignaciolopezluna020/llama3.2:1b "Hi"
123+
```
124+
125+
Output:
126+
127+
```text
128+
Hi! How can I assist you today
129+
```
130+
131+
#### Interactive chat
132+
133+
```console
134+
docker model run ignaciolopezluna020/llama3.2:1b
135+
```
136+
137+
Output:
138+
139+
```text
140+
Interactive chat mode started. Type '/bye' to exit.
141+
> Hi
142+
Hi! How are you doing today?
143+
> /bye
144+
```
145+
146+
### Remove a model
147+
148+
Removes a downloaded model from your system.
149+
150+
```console
151+
$ docker model rm <model>
152+
```
153+
154+
Output:
155+
156+
```text
157+
Model <model> removed successfully
158+
```
159+
160+
## Integrate the Docker Model Runner into your software development lifecycle
161+
162+
You can now start building your Generative AI application powered by the Docker Model Runner.
163+
164+
If you want to try an existing GenAI application, follow these instructions.
33165

34-
## Intergrate the Docker Model Runner into your software development lifecycle
166+
1. Pull the required model from Docker Hub so it's ready for use in your app.
167+
168+
```console
169+
$ docker model pull ignaciolopezluna020/llama3.2:1b
170+
```
171+
172+
2. Set up the sample app. Download and unzip the following folder:
173+
174+
[myapp.zip](attachment:abc104c4-e0c9-4163-b90b-e1f06caab687:myapp.zip)
175+
176+
3. In your terminal, navigate to the `myapp` folder.
177+
4. Start the app with Docker Compose:
178+
179+
```console
180+
$ docker compose up
181+
```
182+
183+
5. Open you app in the browser at `http://localhost:3000`.
184+
185+
You'll see the GenAI app's interface where you can start typing your prompts.
186+
187+
You can now interact with your own GenAI app, powered by a local model. Try a few prompts and notice how fast the responses are — all running on your machine with Docker.
35188

36189
## FAQs
37190

191+
### What models are available?
192+
193+
Currently, all models are hosted in the public Docker Hub namespace of <CHANGE>. You can pull and use any of the following:
194+
195+
### What API endpoints are available?
196+
197+
Once the feature is enabled, the following new APIs are available:
198+
199+
```text
200+
#### Inside containers ####
201+
202+
http://model-runner.docker.internal/
203+
204+
# Docker Model management
205+
POST /models/create
206+
GET /models
207+
GET /models/{namespace}/{name}
208+
DELETE /models/{namespace}/{name}
209+
210+
# OpenAI endpoints (per-backend)
211+
GET /engines/{backend}/v1/models
212+
GET /engines/{backend}/v1/models/{namespace}/{name}
213+
POST /engines/{backend}/v1/chat/completions
214+
POST /engines/{backend}/v1/completions
215+
POST /engines/{backend}/v1/embeddings
216+
Note: You can also omit {backend} and it will default to llama.cpp
217+
E.g., POST /engines/v1/chat/completions.
218+
219+
#### Inside or outside containers (host) ####
220+
221+
Same endpoints on /var/run/docker.sock
222+
223+
# Until stable...
224+
Prefixed with /exp/vDD4.40
225+
```
226+
227+
### How do I interact through the OpenAI API?
228+
229+
#### From within a container
230+
231+
Examples of calling an OpenAI endpoint (`chat/completions`) from within another container using `curl`:
232+
233+
```bash
234+
#!/bin/sh
235+
236+
curl http://model-runner.docker.internal/engines/llama.cpp/v1/chat/completions \
237+
-H "Content-Type: application/json" \
238+
-d '{
239+
"model": "ignaciolopezluna020/llama3.2:1b",
240+
"messages": [
241+
{
242+
"role": "system",
243+
"content": "You are a helpful assistant."
244+
},
245+
{
246+
"role": "user",
247+
"content": "Please write 500 words about the fall of Rome."
248+
}
249+
]
250+
}'
251+
252+
```
253+
254+
#### From the host using a Unix socket
255+
256+
Examples of calling an OpenAI endpoint (`chat/completions`) through the Docker socket from the host using `curl`:
257+
258+
```bash
259+
#!/bin/sh
260+
261+
curl --unix-socket $HOME/.docker/run/docker.sock \
262+
localhost/exp/vDD4.40/engines/llama.cpp/v1/chat/completions \
263+
-H "Content-Type: application/json" \
264+
-d '{
265+
"model": "ignaciolopezluna020/llama3.2:1b",
266+
"messages": [
267+
{
268+
"role": "system",
269+
"content": "You are a helpful assistant."
270+
},
271+
{
272+
"role": "user",
273+
"content": "Please write 500 words about the fall of Rome."
274+
}
275+
]
276+
}'
277+
278+
```
279+
280+
#### From the host using TCP
281+
282+
In case you want to interact with the API from the host, but use TCP instead of a Docker socket, it is recommended you use a helper container as a reverse-proxy. For example, in order to forward the API to `8080`:
283+
284+
```bash
285+
docker run -d --name model-runner-proxy -p 8080:80 alpine/socat tcp-listen:80,fork,reuseaddr tcp:model-runner.docker.internal:80
286+
```
287+
288+
Afterwards, interact with it as previously documented using `localhost` and the forward port, in this case `8080`:
289+
290+
```bash
291+
#!/bin/sh
292+
293+
curl http://localhost:8080/engines/llama.cpp/v1/chat/completions \
294+
-H "Content-Type: application/json" \
295+
-d '{
296+
"model": "ignaciolopezluna020/llama3.2:1b",
297+
"messages": [
298+
{
299+
"role": "system",
300+
"content": "You are a helpful assistant."
301+
},
302+
{
303+
"role": "user",
304+
"content": "Please write 500 words about the fall of Rome."
305+
}
306+
]
307+
}'
308+
```
309+
38310
## Known issues
39311

312+
### `docker model` is not recognised
313+
314+
If you run a Docker Model Runner command and see:
315+
316+
```text
317+
docker: 'model' is not a docker command
318+
```
319+
320+
It means Docker can't find the plugin because it's not in the expected CLI plugins directory.
321+
322+
To fix this, create a symlink so Docker can detect it:
323+
324+
```console
325+
$ ln -s /Applications/Docker.app/Contents/Resources/cli-plugins/docker-model ~/.docker/cli-plugins/docker-model
326+
```
327+
328+
Once linked, re-run the command.
329+
40330
## Get support and share feedback
331+
332+
Thanks for trying out Docker Model Runner. Give feedback or report any bugs you may find through the **Give feedback** link next to the **Enable Docker Model Runner** setting.

0 commit comments

Comments
 (0)