How to get Multi Turn responses?

[SpicyMelonYT]

Hi, I want to use node-llama-cpp for providing a multi turn array like in llama on python. It would look something like this:

[
  {
    "role": "system",
    "content": "You are a helpful assistant."
  },
  {
    "role": "user",
    "content": "Hello, how are you?"
  },
  {
    "role": "assistant",
    "content": "I'm good, thank you!"
  },
  {
    "role": "user",
    "content": "What is the weather in Tokyo?"
  }
]

Currently I see the LlamaContext and LlamaChatSession but I have not been successful at figuring out how to set the message DIRECTLY.

ChatGPT has suggested I format the message array into a format like this:

System: You are a helpful assistant.
User: Hello, how are you?
Assistant: I'm good, thank you!
User: What is the weather in Tokyo?
Assistant:

However, this does not work, it makes the model respond weird.

Please help thanks!

[giladgd]

You can set the current chat history of aLlamaChatSession and then call the .prompt(...) function with the new prompt:

import path from "path";
import {getLlama, LlamaChatSession, ChatHistoryItem} from "node-llama-cpp";

const llama = await getLlama();
const model = await llama.loadModel({
    modelPath: path.join(__dirname, "models", "Meta-Llama-3.1-8B-Instruct.Q4_K_M.gguf")
});
const context = await model.createContext();
const session = new LlamaChatSession({
    contextSequence: context.getSequence()
});

const chatHistory: ChatHistoryItem[] = [
    {
        type: "system",
        text: "You are a helpful assistant."
    },
    {
        type: "user",
        text: "Hello, how are you?"
    },
    {
        type: "model",
        response: ["I'm good, thank you!"]
    }
];
session.setChatHistory(chatHistory);

const res = await session.prompt("What is the weather in Tokyo?");
console.log("AI: " + res);

You can also use LlamaChat for more control over the generation, but doing that is more advanced and generally not needed in most cases.