|
| 1 | +--- |
| 2 | +title: DeepSeek R1 with function calling |
| 3 | +date: 2025-02-20T22:00:00Z |
| 4 | +lastUpdated: false |
| 5 | +author: |
| 6 | + name: Gilad S. |
| 7 | + github: giladgd |
| 8 | +category: Release |
| 9 | +description: node-llama-cpp v3.6 is here, with full support for DeepSeek R1, including function calling! |
| 10 | +image: |
| 11 | + url: https://github.com/user-attachments/assets/9ed954f8-102d-4cdd-96d8-9b6710b8a1f5 |
| 12 | + alt: "node-llama-cpp + DeepSeek R1" |
| 13 | + width: 3072 |
| 14 | + height: 1536 |
| 15 | +--- |
| 16 | +[`node-llama-cpp`](https://node-llama-cpp.withcat.ai) v3.6 is here, with full support for [DeepSeek R1](https://github.com/deepseek-ai/DeepSeek-R1), including function calling! |
| 17 | + |
| 18 | +--- |
| 19 | + |
| 20 | +## Function Calling |
| 21 | +`node-llama-cpp` includes [many tricks](../guide/function-calling) used to make function calling work with most models. |
| 22 | +This release includes special adaptations for DeepSeek R1 to improve function calling performance and stability. |
| 23 | + |
| 24 | +Here's a basic example of function calling with DeepSeek R1: |
| 25 | +```typescript |
| 26 | +import {fileURLToPath} from "url"; |
| 27 | +import path from "path"; |
| 28 | +import { |
| 29 | + getLlama, LlamaChatSession, defineChatSessionFunction, resolveModelFile |
| 30 | +} from "node-llama-cpp"; |
| 31 | + |
| 32 | +const __dirname = path.dirname(fileURLToPath(import.meta.url)); |
| 33 | +const modelsDir = path.join(__dirname, "..", "models"); |
| 34 | + |
| 35 | +const modelUri = "hf:mradermacher/DeepSeek-R1-Distill-Qwen-7B-GGUF:Q4_K_M"; |
| 36 | + |
| 37 | + |
| 38 | +const llama = await getLlama(); |
| 39 | +const model = await llama.loadModel({ |
| 40 | + modelPath: await resolveModelFile(modelUri, modelsDir) |
| 41 | +}); |
| 42 | +const context = await model.createContext(); |
| 43 | +const session = new LlamaChatSession({ |
| 44 | + contextSequence: context.getSequence() |
| 45 | +}); |
| 46 | + |
| 47 | +const fruitPrices: Record<string, string> = { |
| 48 | + "apple": "$6", |
| 49 | + "banana": "$4" |
| 50 | +}; |
| 51 | +const functions = { |
| 52 | + getFruitPrice: defineChatSessionFunction({ |
| 53 | + description: "Get the price of a fruit", |
| 54 | + params: { |
| 55 | + type: "object", |
| 56 | + properties: { |
| 57 | + name: { |
| 58 | + type: "string" |
| 59 | + } |
| 60 | + } |
| 61 | + }, |
| 62 | + async handler(params) { |
| 63 | + const name = params.name.toLowerCase(); |
| 64 | + if (Object.keys(fruitPrices).includes(name)) |
| 65 | + return { |
| 66 | + name: name, |
| 67 | + price: fruitPrices[name] |
| 68 | + }; |
| 69 | + |
| 70 | + return `Unrecognized fruit "${params.name}"`; |
| 71 | + } |
| 72 | + }) |
| 73 | +}; |
| 74 | + |
| 75 | + |
| 76 | +const q1 = "Is an apple more expensive than a banana?"; |
| 77 | +console.log("User: " + q1); |
| 78 | + |
| 79 | +const a1 = await session.prompt(q1, {functions}); |
| 80 | +console.log("AI: " + a1.trim()); |
| 81 | +``` |
| 82 | + |
| 83 | + |
| 84 | +## Recommended Models |
| 85 | +Here are some recommended model URIs you can use to try out DeepSeek R1 with function calling. |
| 86 | + |
| 87 | +| Model | Size | URI | |
| 88 | +|---------------------------------------------------------------------------------------------------------|--------|-------------------------------------------------------------| |
| 89 | +| [DeepSeek R1 Distill Qwen 7B](https://huggingface.co/mradermacher/DeepSeek-R1-Distill-Qwen-7B-GGUF) | 4.68GB | `hf:mradermacher/DeepSeek-R1-Distill-Qwen-7B-GGUF:Q4_K_M` | |
| 90 | +| [DeepSeek R1 Distill Qwen 14B](https://huggingface.co/mradermacher/DeepSeek-R1-Distill-Qwen-14B-GGUF) | 8.99GB | `hf:mradermacher/DeepSeek-R1-Distill-Qwen-14B-GGUF:Q4_K_M` | |
| 91 | +| [DeepSeek R1 Distill Qwen 32B](https://huggingface.co/mradermacher/DeepSeek-R1-Distill-Qwen-32B-GGUF) | 23.3GB | `hf:mradermacher/DeepSeek-R1-Distill-Qwen-32B-GGUF:Q4_K_M` | |
| 92 | + |
| 93 | +::: info TIP |
| 94 | +Estimate the compatibility of a model with your machine before downloading it using the [`inspect estimate`](../cli/inspect/estimate.md) command: |
| 95 | +```shell |
| 96 | +npx -y node-llama-cpp inspect estimate <model URI> |
| 97 | +``` |
| 98 | +::: |
| 99 | + |
| 100 | +### Try It Using the CLI |
| 101 | +To try out function calling with a given model using the CLI, you can use the [`chat` command](../cli/chat.md) with the `--ef` flag |
| 102 | +to provide the model with date and time functions: |
| 103 | + |
| 104 | +```shell |
| 105 | +npx -y node-llama-cpp chat --ef --prompt "What is the time?" <model URI> |
| 106 | +``` |
| 107 | + |
| 108 | + |
| 109 | +## Chain of Thought Segmentation |
| 110 | +The thoughts generated by a reasoning model are now [separated into `thought` segments](../guide/chat-session.md#stream-response-segments) in the response, |
| 111 | +so you can choose whether to use them or not. |
| 112 | + |
| 113 | +By default, the [`.prompt(...)`](../api/classes/LlamaChatSession#prompt) method returns only the main response, without any `thought` segments. |
| 114 | +Use the [`.promptWithMeta(...)`](../api/classes/LlamaChatSession#promptwithmeta) method to get the full response. |
| 115 | + |
| 116 | +You can use the new [`onResponseChunk`](../api/type-aliases/LLamaChatPromptOptions.md#onresponsechunk) option to [stream `thought` segments as they are being generated](../guide/chat-session.md#stream-response-segments). |
| 117 | + |
| 118 | + |
| 119 | +## Electron App Template |
| 120 | +The [Electron app template](../guide/electron.md) has been updated to properly segment the thoughts in the response. |
| 121 | + |
| 122 | +Try it out by downloading the latest build [from GitHub](https://github.com/withcatai/node-llama-cpp/releases/latest), |
| 123 | +or by [scaffolding a new project](../guide/index.md#scaffold-new-project) based on the Electron template: |
| 124 | + |
| 125 | +```shell |
| 126 | +npm create node-llama-cpp@latest |
| 127 | +``` |
| 128 | + |
| 129 | +<YouTubePlayer id="IqfMs0lfIvQ" /> |
0 commit comments