Some of these builds seem hardcoded to c++11, when we use a feature from c++14.

Any reason we aren't using say c++17

Any reasonable platform should be up-to date with c++17 I think

Converted to C++11 only

@slaren @ggerganov PTAL, I'm hoping to add other features to this example such as, read prompt from '-p' arg and read prompt from stdin

It would be good to have a more elaborated chat example, but the goal of this example is to show in the simplest way possible how to use the llama.cpp API. I don't think that these changes achieve that, I think that users will have a harder time understanding the llama.cpp API with all the extra boilerplate that is being added here.

If you want to use this as the base of a new example that adds more features that would be great, but I think we should keep this example as simple as possible.

Cool sounds good @slaren, mind if I call the new example ramalama-core ?

Maybe something like llama-chat. I mentioned before that I think it would be good to have a example focused on chat only, that does that very well, that in time could replace the current llama-cli as the main program of llama.cpp, which at this is point is basically unmaintainable and should be retired.

SGTM

It's also tempting to call this something like run and use a kinda RamaLama, LocalAI, Ollama type CLI interface to interact with models. Kinda like daemonless Ollama:

llama-run file://somedir/somefile.gguf

We could even possibly add https://, http://, ollama://, hf:// as valid syntaxes to pull models since they all are just a http pull in the end

That might be implemented as llama-pull that llama-run can fork/exec (or they share a common library)

I will be afk for 3 weeks, so expect inactivity in this PR. I did the rename in case we want to merge as is and not leave this go stale.

Although the syntax will completely change to:

llama-run [file://]somedir/somefile.gguf [prompt] [flags]

file:// will be optional, but will set up the possibility of adding the pullers discussed above.

This drives the compiler crazy FWIW:

diff --git a/include/llama.h b/include/llama.h
index 5e742642..c3285da3 100644
--- a/include/llama.h
+++ b/include/llama.h
@@ -537,6 +537,13 @@ extern "C" {
                          int32_t   il_start,
                          int32_t   il_end);

+#ifdef __cplusplus
+    // Smart pointers
+    typedef std::unique_ptr<llama_model, decltype(&llama_free_model)> llama_model_ptr;
+    typedef std::unique_ptr<llama_context, decltype(&llama_free)> llama_context_ptr;
+    typedef std::unique_ptr<llama_sampler, decltype(&llama_sampler_free)> llama_sampler;
+#endif
+
     //
     // KV cache
     //

seems like it only wants to build C code in that file or something.

You are probably doing something wrong. Put it at the end of the file, include <memory>, and rename the llama_sampler ptr to llama_sampler_ptr. You should also structs (function objects) as the deleter, see ggml-cpp.h for an example.

Added the smart pointers typedef stuff @slaren

Added as public header

Happy to merge this @slaren @ggerganov

Although there will be breaking changes in a follow on PR, want to change the syntax to this (ramalama/ollama style):

Description:
  Runs a llm

Usage:
  llama-run [options] MODEL [PROMPT]

Options:
  -c, --context-size <value>
      Context size (default: 2048)
  -n, --ngl <value>
      Number of GPU layers (default: 0)
  -h, --help
      Show help message

Examples:
  llama-run ~/.local/share/ramalama/models/ollama/smollm\:135m
  llama-run --ngl 99 ~/.local/share/ramalama/models/ollama/smollm\:135m
  llama-run --ngl 99 ~/.local/share/ramalama/models/ollama/smollm\:135m Hello World

Eventually want to add optional prefixes for the model also (like file://, hf://, maybe we implement http://, https://, ollama:// eventually also, its not that hard), but for no prefix we will just assume the string is a file path I guess.

Sounds good. Btw, to support dynamic loading of backends (#10469), you should add a call to ggml_backend_load_all at the beginning. The goal is to allow projects to distribute a single binary rather than needing different versions for each backend, which I imagine would be relevant to a project like ramalama.