getting called Result::unwrap() on an Err value: DecodeFailed(1) while building chat example. #76
Description
Hi there,
i am trying to build simple chat example with the llma_cpp-rs crate, following is my code
use std::io;
use std::io::Write;
use llama_cpp::standard_sampler::StandardSampler;
use llama_cpp::LlamaModel;
use llama_cpp::LlamaParams;
use llama_cpp::LlamaSession;
use llama_cpp::SessionParams;
fn main() {
let model = LlamaModel::load_from_file("data/Meta-Llama-3-8B-Instruct.Q5_K_S.gguf", LlamaParams::default())
.expect("unable to load model");
let mut ctx = model.create_session(SessionParams::default())
.expect("failed to create session");
loop {
println!("YOU:");
let mut promt = String::new();
io::stdin().read_line(&mut promt).unwrap();
println!("BOT:");
chat(&mut ctx, promt);
println!("");
}
}
fn chat(session:&mut LlamaSession,promt: String) {
session.advance_context(promt).unwrap();
let max_tokens = 2048;
let mut decoded_tokens = 0;
// `ctx.start_completing_with` creates a worker thread that generates tokens. When the completion
// handle is dropped, tokens stop generating!
let completions = session.start_completing_with(StandardSampler::default(), 1024).into_strings();
for completion in completions {
print!("{completion}");
let _ = io::stdout().flush();
decoded_tokens += 1;
if decoded_tokens > max_tokens {
return;
}
}
}
but after some time while giving next prompt i am facing below issue.
thread 'main' panicked at src/main.rs:44:36:
called `Result::unwrap()` on an `Err` value: DecodeFailed(1)
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
it will helpful if someone guide me, why i am facing this error.
as for the model i am using the Meta-Llama-3-8B-Instruct-GGUF (Meta-Llama-3-8B-Instruct.Q5_K_S.gguf) from following link
https://huggingface.co/QuantFactory/Meta-Llama-3-8B-Instruct-GGUF/tree/main