LibreChat and MLX working config for Apple Silicon

[vpnwiz]

I just got LibreChat working with the custom config for MLX on Mac. There is a general LibreChat MLX install guide here but it didn't work for me. Since I'm kinda new to this it took me a while to figure it out, so I wanted to share how I got it working since I couldn't find any tutorials or hits in this repo for mlx_lm. These instructions use the ~4GB model Mistral-7B-Instruct-v0.3-4bit from the mlx-community.

Pre-req's:

• LibreChat running on Docker Desktop.
• Homebrew is installed.
• If you have LibreChat up in a browser window right now, close it.

MLX and mlx_lm Installation

1. Open Terminal.
2. Get mlx (current version 0.21.1) with brew.
   • brew install mlx
3. Reference: https://huggingface.co/mlx-community - mlx_lm is required for MLX to work with LibreChat, but wasn't included with Brew's mlx package on my system. I needed pipx to install it (since pip and pip3 aren't allowed with Brew).
   • brew install pipx
   • pipx ensurepath
4. Install mlx_lm with pipx:
   • pipx install mlx_lm
5. Start the MLX server with the Mistral model from huggingface. It will automatically download:
   • mlx_lm.server --model mlx-community/Mistral-7B-Instruct-v0.3-4bit
     • You can watch the download process in Terminal.
   • Leave this Terminal window open for the server to continue running
   • When you are ready to stop it use,Ctrl+c
6. MLX is now running and available on port 8080.
   • Open a new Terminal window and confirm it is accessible with the following command...

curl localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
     "messages": [{"role": "user", "content": "Say this is a test!"}],
     "temperature": 0.7
   }'

...and MLX should return something like:

{"id": "chatcmpl-2f15245b-1d70-4b0d-9e22-f13d54580eba", "system_fingerprint": "0.20.6-0.21.1-macOS-15.2-arm64-arm-64bit-Mach-O-applegpu_g15s", "object": "chat.completion", "model": "default_model", "created": 1736363449, "choices": [{"index": 0, "logprobs": {"token_logprobs": [-0.234375, 0.0, -1.328125, -0.21875, -0.140625, -0.734375, 0.0, 0.0, -0.640625, -0.296875, -0.328125, 0.0, -0.015625, 0.0, -0.75, -0.125, -0.265625, -0.015625, -0.078125, -0.265625], "top_logprobs": [], "tokens": [7373, 8999, 29493, 1224, 1117, 8934, 1032, 2137, 29491, 2370, 1309, 1083, 6799, 1136, 4397, 1163, 1224, 2137, 29572, 2]}, "finish_reason": "stop", "message": {"role": "assistant", "content": "Understood, this is indeed a test. How can I assist you further with this test?"}}], "usage": {"prompt_tokens": 11, "completion_tokens": 20, "total_tokens": 31}}% 

Configure LibreChat to use MLX

1. Know where LibreChat is installed on your Mac. I installed it to ~/GitHub/LibreChat
2. Reference https://www.librechat.ai/docs/configuration/docker_override. Create a docker-compose.override.yml file in the LibreChat directory with the contents:

---
services:
  api:
    volumes:
      - ./librechat.yaml:/app/librechat.yaml

2. Reference https://www.librechat.ai/docs/configuration/librechat_yaml. Create the librechat.yaml file in the LibreChat directory.
3. This is my working config for MLX. Paste and save, or adapt if you already have this file.

---
# Configuration version (required)
version: 1.2.1

cache: true

endpoints:
  custom:
    - name: "MLX"
      apiKey: "mlx"
      baseURL: "http://host.docker.internal:8080/v1/"
      models:
        default: [
          "mlx-community/Mistral-7B-Instruct-v0.3-4bit"
        ]
        fetch: false  # fetching list of models is not supported
      titleConvo: true
      titleModel: "current_model"
      summarize: false
      summaryModel: "current_model"
      forcePrompt: false
      modelDisplayLabel: "Apple MLX"
      addParams:
        max_tokens: 2000
        "stop": [
          "<|eot_id|>"
        ]

4. Open a Terminal window and go to the LibreChat directory. In my case I go to cd ~/GitHub/LibreChat
5. Stop and restart LibreChat to use the new config:
   • docker compose down && docker compose up -d

Using MLX in LibreChat

1. Launch LibreChat in your browser.
2. Select the MLX provider. See the screenshot here
3. Ensure the MLX model in use (dropdown to the right of MLX) says "mlx-community/Mistral-7B-Instruct-v0.3-4bit"
4. MLX is ready to chat!