Skip to content

Commit 7ac3558

Browse files
committed
feat: harbor pull smart routing, volumes CLI, and down resilience
- harbor pull delegates to run_models_pull for HF metadata-based routing (GGUF -> llamacpp, other -> hf download) instead of hardcoding llamacpp for all HuggingFace repos - harbor down stops host services (MLX, DMR) before compose down so they are cleaned up even when Docker is unavailable - Add harbor volumes CLI and help text - Rewrite MLX CLI functions for mlx-lm, remove mlx-serve references - Extract config search/list to configSearch routine - Update README and install docs for DMR/MLX backends
1 parent 68c1d83 commit 7ac3558

3 files changed

Lines changed: 192 additions & 192 deletions

File tree

README.md

Lines changed: 19 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -71,6 +71,10 @@ harbor up ollama
7171
harbor up llamacpp
7272
harbor up vllm
7373

74+
# Host-native Metal/GPU inference on macOS
75+
harbor up dmr
76+
harbor up mlx
77+
7478
# Set and remember args for llama.cpp
7579
harbor llamacpp args -ngl 32
7680
```
@@ -97,11 +101,14 @@ Supported host tools include `claude`, `codex`, `copilot`, `droid`, `hermes`, `m
97101

98102
#### Cutting Edge Inference
99103

100-
Harbor supports most of the major inference engines as well as a few of the lesser-known ones.
104+
Harbor supports most of the major inference engines as well as a few of the lesser-known ones. On macOS, [Docker Model Runner](./docs/2.2.22-Backend-Docker-Model-Runner.md) and [MLX](./docs/2.2.23-Backend-MLX.md) run inference on the host with Metal acceleration — no containers needed for the compute.
101105

102106
```bash
103107
# We sincerely hope you'll never try to run all of them at once
104108
harbor up vllm llamacpp tgi litellm tabbyapi aphrodite sglang ktransformers mistralrs airllm
109+
110+
# Host-native inference (macOS Metal / Linux GPU)
111+
harbor up dmr mlx
105112
```
106113

107114
#### Tool Use
@@ -247,8 +254,8 @@ harbor eject searxng llamacpp > docker-compose.harbor.yml
247254
[KoboldCpp](https://github.com/av/harbor/wiki/2.2.16-Backend:-KoboldCpp) ⦁︎ [KTransformers](https://github.com/av/harbor/wiki/2.2.13-Backend:-KTransformers) ⦁︎ [Lemonade](https://github.com/av/harbor/wiki/2.2.19-Backend-Lemonade) ⦁︎ [llama.cpp](https://github.com/av/harbor/wiki/2.2.2-Backend:-llama.cpp)
248255
[lmdeploy](https://github.com/av/harbor/wiki/2.2.10-Backend:-lmdeploy) ⦁︎ [mistral.rs](https://github.com/av/harbor/wiki/2.2.6-Backend:-mistral.rs) ⦁︎ [Modular MAX](https://github.com/av/harbor/wiki/2.2.17-Backend-Modular-MAX) ⦁︎ [Needle](https://github.com/av/harbor/wiki/2.2.20-Backend-Needle)
249256
[Nexa SDK](https://github.com/av/harbor/wiki/2.2.15-Backend:-Nexa-SDK) ⦁︎ [Ollama](https://github.com/av/harbor/wiki/2.2.1-Backend:-Ollama) ⦁︎ [openedai-speech](https://github.com/av/harbor/wiki/2.2.7-Backend:-openedai-speech) ⦁︎ [Parler](https://github.com/av/harbor/wiki/2.2.8-Backend:-Parler)
250-
[SGLang](https://github.com/av/harbor/wiki/2.2.12-Backend:-SGLang) ⦁︎ [Speaches](https://github.com/av/harbor/wiki/2.2.14-Backend:-Speaches) ⦁︎ [TabbyAPI](https://github.com/av/harbor/wiki/2.2.4-Backend:-TabbyAPI) ⦁︎ [Text Generation Inference](https://github.com/av/harbor/wiki/2.2.9-Backend:-text-generation-inference)
251-
[vLLM](https://github.com/av/harbor/wiki/2.2.3-Backend:-vLLM)
257+
[Docker Model Runner](https://github.com/av/harbor/wiki/2.2.22-Backend-Docker-Model-Runner) ⦁︎ [MLX](https://github.com/av/harbor/wiki/2.2.23-Backend-MLX) ⦁︎ [SGLang](https://github.com/av/harbor/wiki/2.2.12-Backend:-SGLang) ⦁︎ [Speaches](https://github.com/av/harbor/wiki/2.2.14-Backend:-Speaches)
258+
[TabbyAPI](https://github.com/av/harbor/wiki/2.2.4-Backend:-TabbyAPI) ⦁︎ [Text Generation Inference](https://github.com/av/harbor/wiki/2.2.9-Backend:-text-generation-inference) ⦁︎ [vLLM](https://github.com/av/harbor/wiki/2.2.3-Backend:-vLLM)
252259

253260
##### Satellites
254261
[Activepieces](https://github.com/av/harbor/wiki/2.3.64-Satellite-Activepieces) ⦁︎ [Agent Zero](https://github.com/av/harbor/wiki/2.3.47-Satellite-Agent-Zero) ⦁︎ [aichat](https://github.com/av/harbor/wiki/2.3.14-Satellite:-aichat) ⦁︎ [Aider](https://github.com/av/harbor/wiki/2.3.13-Satellite:-aider)
@@ -296,6 +303,10 @@ harbor up speaches
296303
# Open Webui is automatically connected to them.
297304
harbor up llamacpp tgi litellm vllm tabbyapi aphrodite sglang ktransformers
298305

306+
# Host-native backends for macOS (Metal acceleration, no containers)
307+
harbor up dmr
308+
harbor up mlx
309+
299310
# Run different Frontends
300311
harbor up librechat chatui bionicgpt hollama
301312

@@ -309,6 +320,11 @@ harbor up comfyui
309320
# Use custom models for supported backends
310321
harbor llamacpp model https://huggingface.co/user/repo/model.gguf
311322

323+
# Manage models for host-native backends
324+
harbor dmr pull ai/mistral
325+
harbor dmr ls
326+
harbor mlx pull mlx-community/Qwen3.5-4B-4bit
327+
312328
# Access service CLIs without installing them
313329
# Caches are shared between services where possible
314330
harbor hf scan-cache

docs/1.0.-Installing-Harbor.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -186,7 +186,7 @@ Follow the [Harbor CLI installation methods](#harbor-cli-installation-methods) b
186186
- Administrator access
187187

188188
> [!NOTE]
189-
> **ARM64 Limitation:** Some services don't ship ARM64 images (Apple Silicon). GPU pass-through is not supported on macOS.
189+
> **ARM64 Limitation:** Some services don't ship ARM64 images (Apple Silicon). GPU pass-through is not supported on macOS. For Metal-accelerated inference, use host-native backends like [Docker Model Runner](./2.2.22-Backend-Docker-Model-Runner.md) (`harbor up dmr`) or [MLX](./2.2.23-Backend-MLX.md) (`harbor up mlx`) — they run on the host and are exposed to Harbor containers via a proxy.
190190
191191
#### Step 1: Install Docker Desktop (10 minutes)
192192

0 commit comments

Comments
 (0)