Skip to content

Commit 2509ef0

Browse files
committed
feat: auto-bootstrap missing DMR and MLX host components on harbor up
- dmr_install_components: auto-installs docker-model-plugin via apt/dnf on Linux, or enables Model Runner via Docker Desktop on macOS - dmr_install_runner: installs the DMR runtime when available - harbor up dmr now bootstraps gracefully instead of erroring out - harbor up mlx auto-installs mlx-serve via uv on macOS when missing - Updated all three backend docs to describe the automatic setup
1 parent 737bcb3 commit 2509ef0

5 files changed

Lines changed: 140 additions & 21 deletions

File tree

.facts

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -83,6 +83,9 @@
8383
- label: cli > routines > routines/utils.ts has explicit TypeScript annotations for shared utility exports so deno check routines/models.ts is not blocked by implicit-any diagnostics
8484
command: deno check routines/models.ts
8585
tags: [spec, routines-utils, implemented]
86+
- label: services > dmr/mlx > harbor up dmr and harbor up mlx automatically bootstrap missing host-managed backend components before starting their Harbor proxy services
87+
command: rg -q 'dmr_install_components' harbor.sh && rg -q 'mlx_install_components' harbor.sh && rg -q 'automatic.*install|automatically install|auto-installs' docs/2.2.22-Backend-Docker-Model-Runner.md docs/2.2.23-Backend-MLX.md
88+
tags: [spec, auto-host-bootstrap, implemented]
8689
# project
8790
- Harbor is a containerized LLM toolkit distributed as a Docker Compose project with a CLI and Tauri desktop app
8891
- label: the project is licensed under Apache 2.0

docs/2.2.22-Backend-Docker-Model-Runner.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@ DMR is especially useful on Apple Silicon because Metal inference runs on the ma
1313
harbor up dmr
1414
```
1515

16-
When `HARBOR_DMR_MANAGE_HOST=true`, Harbor checks for the Docker Model Runner CLI, enables the TCP endpoint when configured, and pulls `HARBOR_DMR_MODEL` before starting the proxy.
16+
When `HARBOR_DMR_MANAGE_HOST=true`, Harbor automatically installs or enables missing Docker Model Runner host components before starting the proxy. On Linux this uses the official `docker-model-plugin` package through `apt` or `dnf` when the `docker model` CLI is missing; on Docker Desktop systems Harbor uses Docker Desktop's Model Runner enablement command when it is available. Harbor then installs the runner runtime when supported, enables the TCP endpoint when configured, and pulls `HARBOR_DMR_MODEL`.
1717

1818
Start a frontend against DMR:
1919

@@ -96,7 +96,7 @@ harbor logs dmr
9696

9797
##### Docker Model Runner CLI is missing
9898

99-
Install or update Docker Desktop and verify:
99+
`harbor up dmr` attempts automatic setup when host management is enabled. If setup cannot proceed on your platform, install or update Docker Desktop, or install Docker's `docker-model-plugin` package on Linux, then verify:
100100

101101
```bash
102102
docker model --help

docs/2.2.23-Backend-MLX.md

Lines changed: 3 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -9,19 +9,13 @@ This service is intended for Apple Silicon systems. On other platforms the proxy
99

1010
#### Starting
1111

12-
Install `mlx-serve` on the host first:
13-
14-
```bash
15-
uv tool install mlx-serve
16-
```
17-
18-
Then start the Harbor service:
12+
Start the Harbor service:
1913

2014
```bash
2115
harbor up mlx
2216
```
2317

24-
When `HARBOR_MLX_MANAGE_HOST=true`, Harbor renders `services/mlx/models.yaml` from the template, starts `mlx-serve` from `services/mlx/`, and starts the Harbor proxy.
18+
When `HARBOR_MLX_MANAGE_HOST=true`, Harbor automatically installs missing MLX host components before starting the proxy. If `mlx-serve` is missing on macOS, Harbor installs it with `uv tool install mlx-serve`; if `uv` is missing and Homebrew is available, Harbor installs `uv` first. Harbor then renders `services/mlx/models.yaml` from the template, starts `mlx-serve` from `services/mlx/`, and starts the Harbor proxy.
2519

2620
Start a frontend against MLX:
2721

@@ -119,7 +113,7 @@ harbor logs mlx
119113

120114
##### mlx-serve is not installed
121115

122-
Install it on the macOS host:
116+
`harbor up mlx` automatically installs `mlx-serve` on macOS when host management is enabled. If automatic setup cannot proceed, install `uv` and `mlx-serve` manually:
123117

124118
```bash
125119
uv tool install mlx-serve

docs/8.5-OpenAI-Compatible-Local-LLM-Backends.md

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -75,6 +75,8 @@ harbor up dmr
7575

7676
On Apple Silicon, DMR is the preferred Docker-managed path for host-native Metal inference. Harbor starts a proxy container at `http://dmr:8080/v1`; Docker Model Runner itself stays on the host.
7777

78+
When host management is enabled, `harbor up dmr` also attempts to bootstrap missing Docker Model Runner components before the proxy starts.
79+
7880
### MLX
7981

8082
[MLX](./2.2.23-Backend-MLX.md) is useful when you specifically want Apple's MLX runtime on an Apple Silicon Mac. Harbor manages `mlx-serve` on the host and exposes it to the Compose network through a proxy container.
@@ -86,6 +88,8 @@ harbor launch --backend mlx --model mlx-qwen2.5-3b codex
8688

8789
MLX acceleration does not run inside Harbor's Linux containers. The Harbor service owns lifecycle, config, docs, and integration; Metal inference runs on macOS.
8890

91+
When host management is enabled, `harbor up mlx` automatically installs `mlx-serve` on macOS if it is missing.
92+
8993
## Choose or Swap Backends
9094

9195
The default Harbor profile starts Ollama and Open WebUI. You can still start another backend directly:
@@ -188,7 +192,7 @@ Use Docker Model Runner when:
188192
Use MLX when:
189193

190194
- You are on an Apple Silicon Mac and want direct MLX-backed inference.
191-
- You are comfortable installing `mlx-serve` on the host.
195+
- You want Harbor to manage `mlx-serve` startup and install it when it is missing.
192196
- You want Harbor-managed config and proxying for a host-native MLX process.
193197

194198
For many users, the practical workflow is simple: start with Ollama, add llama.cpp when you want direct GGUF serving, use vLLM when serving throughput or model format support matters more, and use DMR or MLX on Apple Silicon when Metal acceleration is the goal.

harbor.sh

Lines changed: 127 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -5033,8 +5033,85 @@ sed_replacement_escape() {
50335033
printf '%s' "$1" | sed 's/[&|]/\\&/g'
50345034
}
50355035
5036+
docker_cli_subcommand_available() {
5037+
local subcommand="$1"
5038+
command -v docker >/dev/null 2>&1 && docker "$subcommand" --help 2>&1 | grep -q "Usage: docker $subcommand"
5039+
}
5040+
5041+
docker_model_subcommand_available() {
5042+
local subcommand="$1"
5043+
docker_cli_subcommand_available model && docker model "$subcommand" --help 2>&1 | grep -q "docker model $subcommand"
5044+
}
5045+
50365046
docker_model_available() {
5037-
command -v docker >/dev/null 2>&1 && docker model --help >/dev/null 2>&1
5047+
docker_cli_subcommand_available model
5048+
}
5049+
5050+
run_privileged_install_command() {
5051+
if [ "$(id -u)" -eq 0 ]; then
5052+
"$@"
5053+
else
5054+
if ! command -v sudo >/dev/null 2>&1; then
5055+
log_error "sudo is required to install host packages automatically."
5056+
return 1
5057+
fi
5058+
sudo "$@"
5059+
fi
5060+
}
5061+
5062+
dmr_install_linux_plugin() {
5063+
if command -v apt-get >/dev/null 2>&1; then
5064+
log_info "Installing Docker Model Runner plugin with apt."
5065+
run_privileged_install_command apt-get update
5066+
run_privileged_install_command apt-get install -y docker-model-plugin
5067+
return
5068+
fi
5069+
5070+
if command -v dnf >/dev/null 2>&1; then
5071+
log_info "Installing Docker Model Runner plugin with dnf."
5072+
run_privileged_install_command dnf install -y docker-model-plugin
5073+
return
5074+
fi
5075+
5076+
log_error "Docker Model Runner CLI is missing and automatic installation is only supported through apt or dnf on Linux."
5077+
return 1
5078+
}
5079+
5080+
dmr_desktop_enable_available() {
5081+
docker_cli_subcommand_available desktop && docker desktop enable model-runner --help 2>&1 | grep -q 'model-runner'
5082+
}
5083+
5084+
dmr_install_components() {
5085+
if docker_model_available; then
5086+
return 0
5087+
fi
5088+
5089+
log_info "Docker Model Runner CLI is missing; attempting to install or enable it."
5090+
5091+
if [[ "$(uname -s)" == "Linux" ]]; then
5092+
dmr_install_linux_plugin || return 1
5093+
elif dmr_desktop_enable_available; then
5094+
docker desktop enable model-runner || return 1
5095+
else
5096+
log_error "Docker Model Runner CLI is not available. Install/update Docker Desktop or install docker-model-plugin, then retry."
5097+
return 1
5098+
fi
5099+
5100+
if ! docker_model_available; then
5101+
log_error "Docker Model Runner CLI is still unavailable after automatic setup."
5102+
return 1
5103+
fi
5104+
}
5105+
5106+
dmr_install_runner() {
5107+
if docker model status >/dev/null 2>&1; then
5108+
return 0
5109+
fi
5110+
5111+
if docker_model_subcommand_available install-runner; then
5112+
log_info "Installing Docker Model Runner runtime."
5113+
docker model install-runner || return 1
5114+
fi
50385115
}
50395116
50405117
dmr_host_start() {
@@ -5046,10 +5123,8 @@ dmr_host_start() {
50465123
return 0
50475124
fi
50485125
5049-
if ! docker_model_available; then
5050-
log_error "Docker Model Runner CLI is not available. Install/update Docker Desktop and ensure 'docker model' works."
5051-
return 1
5052-
fi
5126+
dmr_install_components || return 1
5127+
dmr_install_runner || return 1
50535128
50545129
enable_tcp=$(env_manager get dmr.enable.tcp)
50555130
runner_port=$(env_manager get dmr.runner.port)
@@ -5106,6 +5181,52 @@ mlx_render_config() {
51065181
"$template" > "$output"
51075182
}
51085183
5184+
mlx_add_uv_tool_bin_to_path() {
5185+
local tool_bin
5186+
tool_bin=$(uv tool dir --bin 2>/dev/null || true)
5187+
5188+
if [ -n "$tool_bin" ]; then
5189+
PATH="$tool_bin:$PATH"
5190+
export PATH
5191+
fi
5192+
}
5193+
5194+
mlx_install_uv() {
5195+
if command -v uv >/dev/null 2>&1; then
5196+
return 0
5197+
fi
5198+
5199+
if [[ "$(uname -s)" == "Darwin" ]] && command -v brew >/dev/null 2>&1; then
5200+
log_info "Installing uv with Homebrew."
5201+
brew install uv || return 1
5202+
return
5203+
fi
5204+
5205+
log_error "uv is required to install mlx-serve automatically. Install uv, then retry."
5206+
return 1
5207+
}
5208+
5209+
mlx_install_components() {
5210+
if command -v mlx-serve >/dev/null 2>&1; then
5211+
return 0
5212+
fi
5213+
5214+
if [[ "$(uname -s)" != "Darwin" ]]; then
5215+
log_error "mlx-serve is missing and automatic MLX host setup is only supported on macOS."
5216+
return 1
5217+
fi
5218+
5219+
mlx_install_uv || return 1
5220+
log_info "Installing mlx-serve with uv."
5221+
uv tool install mlx-serve || return 1
5222+
mlx_add_uv_tool_bin_to_path
5223+
5224+
if ! command -v mlx-serve >/dev/null 2>&1; then
5225+
log_error "mlx-serve is still unavailable after automatic setup. Ensure the uv tool bin directory is on PATH."
5226+
return 1
5227+
fi
5228+
}
5229+
51095230
mlx_host_start() {
51105231
local manage_host auto_pull model workspace api_key
51115232
@@ -5115,10 +5236,7 @@ mlx_host_start() {
51155236
return 0
51165237
fi
51175238
5118-
if ! command -v mlx-serve >/dev/null 2>&1; then
5119-
log_error "mlx-serve is not installed on the host. Install it first, then retry: uv tool install mlx-serve"
5120-
return 1
5121-
fi
5239+
mlx_install_components || return 1
51225240
51235241
mlx_render_config
51245242
workspace=$(mlx_workspace_path)

0 commit comments

Comments
 (0)