See the raw prompt that is send to the LLM

[pguso]

How can I see the raw prompt that’s actually sent to the LLM by node-llama-cpp. Including all system messages, role formatting, special tokens, and any injected context or formatting (like <|system|> / <|user|> / <|assistant|> sections)?

I’d like to inspect the exact string that gets passed to the model for debugging or understanding how my inputs are structured before inference.

Is there a built-in way (e.g. a debug flag, callback, or verbose option) to log or print that full formatted prompt, or do I need to modify the source / intercept it manually?

[giladgd]

You can inspect the tokens evaluated onto a context sequence (the tokens representation of the current evaluation state) by either logging the textual representation of the tokens (including special tokens), or by converting the context tokens into a LlamaText and logging it to easily differentiate between regular tokens and special tokens:

import {LlamaText} from "node-llama-cpp";

console.log("contextState", model.detokenize(session.sequence.contextTokens, /* specialTokens */ true));
console.log("LlamaText", LlamaText.fromTokens(model.tokenizer, session.sequence.contextTokens));

I recommend the LlamaText route as it makes it easier to differentiate between regular tokens and special ones.

You can generate a LlamaText for a given chat history to see what will be evaluated onto a context sequence by calling the .generateContextState on the relevant chat wrapper with the chat history.
You can also access the chat wrapper of the current LlamaChatSession by accessing .chatWrapper.

It's worth noting that node-llama-cpp is safe against special tokens injection so you generally don't have to worry about sanitizing user input from special tokens.

[pguso]

Thank you so much for that fast and detailed response. That's exactly what I wanted.

I kind of did by myself what you mentioned:

const chatWrapper = session.chatWrapper
const state = chatWrapper.generateContextState({
    chatHistory: [{type: 'user', text: prompt}, {type: 'system', text: systemInstruction}],
    availableFunctions: functions
})
const formattedPrompt = state.contextText.toString()
console.log("Exact prompt:\n", formattedPrompt)

I do it because I want to look deeper into what the different models are doing under the hood and build up a deeper understanding.

I really love your package and I can't stop playing around with it!