Skip to content

Commit b1dff17

Browse files
committed
docs: responsePrefix example
1 parent 0e83243 commit b1dff17

File tree

2 files changed

+32
-1
lines changed

2 files changed

+32
-1
lines changed

docs/guide/chat-session.md

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -671,3 +671,34 @@ await new Promise(resolve => setTimeout(resolve, 1500));
671671
const cachedCompletion = completionEngine.complete("Hi there! How");
672672
console.log("Cached completion:", cachedCompletion);
673673
```
674+
675+
## Response Prefix {#response-prefix}
676+
You can force the model response to start with a specific prefix,
677+
to make the model follow a certain direction in its response.
678+
679+
```typescript
680+
import {fileURLToPath} from "url";
681+
import path from "path";
682+
import {getLlama, LlamaChatSession, GeneralChatWrapper} from "node-llama-cpp";
683+
684+
const __dirname = path.dirname(fileURLToPath(import.meta.url));
685+
686+
const llama = await getLlama();
687+
const model = await llama.loadModel({
688+
modelPath: path.join(__dirname, "models", "Meta-Llama-3.1-8B-Instruct.Q4_K_M.gguf")
689+
});
690+
const context = await model.createContext();
691+
const session = new LlamaChatSession({
692+
contextSequence: context.getSequence(),
693+
chatWrapper: new GeneralChatWrapper()
694+
});
695+
696+
697+
const q1 = "Hi there, how are you?";
698+
console.log("User: " + q1);
699+
700+
const a1 = await session.prompt(q1, {
701+
responsePrefix: "The weather today is"
702+
});
703+
console.log("AI: " + a1);
704+
```

src/evaluator/LlamaChatSession/LlamaChatSession.ts

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -149,7 +149,7 @@ export type LLamaChatPromptOptions<Functions extends ChatSessionModelFunctions |
149149
trimWhitespaceSuffix?: boolean,
150150

151151
/**
152-
* Force a given text prefix to be the start of the model response, to make the model follow a specific path.
152+
* Force a given text prefix to be the start of the model response, to make the model follow a certain direction.
153153
*
154154
* May cause some models to not use the given functions in some scenarios where they would have been used otherwise,
155155
* so avoid using it together with function calling if you notice unexpected behavior.

0 commit comments

Comments
 (0)