Skip to content

add context-size prop #22924

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 2 commits into from
Closed

add context-size prop #22924

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
096f4ac
add `context-size` prop
ArthurFlag Jun 26, 2025
a876ed8
add runtime-flag param
ArthurFlag Jun 26, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions content/manuals/ai/model-runner/_index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -385,3 +385,7 @@ The Docker Model CLI currently lacks consistent support for specifying models by
## Share feedback

Thanks for trying out Docker Model Runner. Give feedback or report any bugs you may find through the **Give feedback** link next to the **Enable Docker Model Runner** setting.

## Related pages

- [Use Model Runner with Compose](/manuals/compose/how-tos/model-runner.md)
24 changes: 18 additions & 6 deletions content/manuals/compose/how-tos/model-runner.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,15 +40,27 @@
type: model
options:
model: ai/smollm2
context-size: 1024
```

Notice the dedicated `provider` attribute in the `ai_runner` service.
This attribute specifies that the service is a model provider and lets you define options such as the name of the model to be used.
Notice the following:

There is also a `depends_on` attribute in the `chat` service.
This attribute specifies that the `chat` service depends on the `ai_runner` service.
This means that the `ai_runner` service will be started before the `chat` service to allow injection of model information to the `chat` service.
- In the `ai_runner` service:

- `provider.type`: Specifies that the service is a `model` provider.
- `provider.options`: Specifies the options of the model. In our case, we want to use

Check warning on line 51 in content/manuals/compose/how-tos/model-runner.md

View workflow job for this annotation

GitHub Actions / vale 

[vale] reported by reviewdog 🐶
[Docker.We] Avoid using first-person plural like 'our'.

Raw Output:
{"message": "[Docker.We] Avoid using first-person plural like 'our'.", "location": {"path": "content/manuals/compose/how-tos/model-runner.md", "range": {"start": {"line": 51, "column": 64}}}, "severity": "WARNING"}
`ai/smollm2`, and we set the context size to 1024 tokens.

Check warning on line 52 in content/manuals/compose/how-tos/model-runner.md

View workflow job for this annotation

GitHub Actions / vale 

[vale] reported by reviewdog 🐶
[Docker.We] Avoid using first-person plural like 'we'.

Raw Output:
{"message": "[Docker.We] Avoid using first-person plural like 'we'.", "location": {"path": "content/manuals/compose/how-tos/model-runner.md", "range": {"start": {"line": 52, "column": 23}}}, "severity": "WARNING"}

> [!NOTE]
> Each model has its own maximum context size. When increasing the context length,
> consider your hardware constraints. In general, try to use the smallest context size
> possible for your use case.

- In the `chat` service:

- `depends_on` specifies that the `chat` service depends on the `ai_runner` service. The
`ai_runner` service will be started before the `chat` service, to allow injection of model information to the `chat` service.

Check warning on line 62 in content/manuals/compose/how-tos/model-runner.md

View workflow job for this annotation

GitHub Actions / vale 

[vale] reported by reviewdog 🐶
[Docker.RecommendedWords] Consider using 'let' instead of 'allow'

Raw Output:
{"message": "[Docker.RecommendedWords] Consider using 'let' instead of 'allow'", "location": {"path": "content/manuals/compose/how-tos/model-runner.md", "range": {"start": {"line": 62, "column": 72}}}, "severity": "INFO"}

## How it works

During the `docker compose up` process, Docker Model Runner automatically pulls and runs the specified model.
Expand All @@ -61,6 +73,6 @@

This lets the `chat` service to interact with the model and use it for its own purposes.

## Reference
## Related pages

- [Docker Model Runner documentation](/manuals/ai/model-runner.md)