You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
@@ -18,11 +21,16 @@ Compose lets you define AI models as core components of your application, so you
18
21
## Prerequisites
19
22
20
23
- Docker Compose v2.38 or later
21
-
- A platform that supports Compose models such as Docker Model Runner or compatible cloud providers
24
+
- A platform that supports Compose models such as Docker Model Runner or compatible cloud providers.
25
+
If you are using DMR:
26
+
27
+
- Docker Desktop 4.43 or later
28
+
- Docker Desktop for Mac with Apple Silicon or Docker Desktop for Windows with NVIDIA GPU
29
+
-[Docker Model Runner enabled in Docker Desktop](/manuals/ai/model-runner.md#enable-docker-model-runner)
22
30
23
31
## What are Compose models?
24
32
25
-
Compose `models` are a standardized way to define AI model dependencies in your application. By using the []`models` top-level element](/reference/compose-file/models.md) in your Compose file, you can:
33
+
Compose `models` are a standardized way to define AI model dependencies in your application. By using the [`models` top-level element](/reference/compose-file/models.md) in your Compose file, you can:
26
34
27
35
- Declare which AI models your application needs
28
36
- Specify model configurations and requirements
@@ -66,7 +74,14 @@ models:
66
74
Common configuration options include:
67
75
- `model` (required): The OCI artifact identifier for the model. This is what Compose pulls and runs via the model runner.
68
76
- `context_size`: Defines the maximum token context size for the model.
77
+
78
+
> [!NOTE]
79
+
> Each model has its own maximum context size. When increasing the context length,
80
+
> consider your hardware constraints. In general, try to use the smallest context size
81
+
> possible for your use case.
82
+
69
83
- `runtime_flags`: A list of raw command-line flags passed to the inference engine when the model is started.
84
+
For example, if If you use llama.cpp, you can pass any of [the available parameters](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md).
70
85
- Platform-specific options may also be available via extensions attributes `x-*`
71
86
72
87
## Service model binding
@@ -131,25 +146,58 @@ One of the key benefits of using Compose models is portability across different
131
146
132
147
### Docker Model Runner
133
148
134
-
When Docker Model Runner is enabled:
149
+
When [Docker Model Runner is enabled](/manuals/ai/model-runner/_index.md):
135
150
136
151
```yaml
137
152
services:
138
153
chat-app:
139
154
image: my-chat-app
140
155
models:
141
-
- llm
156
+
llm:
157
+
endpoint_var: AI_MODEL_URL
158
+
model_var: AI_MODEL_NAME
142
159
143
160
models:
144
161
llm:
145
162
model: ai/smollm2
163
+
context_size: 4096
164
+
runtime_flags:
165
+
- "--no-prefill-assistant"
146
166
```
147
167
148
168
Docker Model Runner will:
149
169
- Pull and run the specified model locally
150
170
- Provide endpoint URLs for accessing the model
151
171
- Inject environment variables into the service
152
172
173
+
#### Alternative configuration with Provider services
174
+
175
+
> [!TIP]
176
+
>
177
+
> This approach is deprecated. Use the [`models` top-level element](#use-models-definition) instead.
178
+
179
+
You can also use the `provider` service type, which allows you to declare platform capabilities required by your application.
180
+
For AI models, you can use the `model` type to declare model dependencies.
181
+
182
+
To define a model provider:
183
+
184
+
```yaml
185
+
services:
186
+
chat:
187
+
image: my-chat-app
188
+
depends_on:
189
+
- ai_runner
190
+
191
+
ai_runner:
192
+
provider:
193
+
type: model
194
+
options:
195
+
model: ai/smollm2
196
+
context-size: 1024
197
+
runtime-flags: "--no-prefill-assistant"
198
+
```
199
+
200
+
153
201
### Cloud providers
154
202
155
203
The same Compose file can run on cloud providers that support Compose models:
Copy file name to clipboardExpand all lines: content/manuals/desktop/release-notes.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -219,7 +219,7 @@ For more frequently asked questions, see the [FAQs](/manuals/desktop/troubleshoo
219
219
- Docker Model Runner is now available on x86 Windows machines with NVIDIA GPUs.
220
220
- You can now [push models](/manuals/ai/model-runner.md#push-a-model-to-docker-hub) to Docker Hub with Docker Model Runner.
221
221
- Added support for Docker Model Runner's model management and chat interface in Docker Desktop for Mac and Windows (on hardware supporting Docker Model Runner). Users can now view, interact with, and manage local AI models through a new dedicated interface.
222
-
-[Docker Compose](/manuals/ai/compose/model-runner.md) and Testcontainers [Java](https://java.testcontainers.org/modules/docker_model_runner/) and [Go](https://golang.testcontainers.org/modules/dockermodelrunner/) now support Docker Model Runner.
222
+
-[Docker Compose](/manuals/ai/compose/models-and-compose.md) and Testcontainers [Java](https://java.testcontainers.org/modules/docker_model_runner/) and [Go](https://golang.testcontainers.org/modules/dockermodelrunner/) now support Docker Model Runner.
223
223
- Introducing Docker Desktop in the [Microsoft App Store](https://apps.microsoft.com/detail/xp8cbj40xlbwkx?hl=en-GB&gl=GB).
0 commit comments