Add a llama-cpp Model?

[joy-void-joy]

Description

I know it is already possible to use Ollama servers with pydantic using its OpenAI interface, however, there are a few reasons for why I would be interested in a direct llama-cpp binding:

• Tool-use and huggingface compatibility: When using Ollama on huggface models that support tool use (e.g. GLM-4-32B), Ollama says the model "does not support tool use", while llama-cpp works correctly
• Standalone file: llama-cpp does not need to boot the server separately, which matters for ease of use and portability as everything can be included in a single python file
• Better configurability/nativity? Unsure about this one, but it seems like llama-cpp-python might be faster and have overall more optimization builtin as compared to the ollama server

This is why I would like a llama-cpp Model to pydantic-ai.

Would there be interest for a PR regarding this? I already have started working on it for a personal project, so it is only a matter of packaging/adding tests etc...

References

No response

[Pranav-Official]

yeah.. i think this would be nice feature

[Kludex]

We are adding the HuggingFace model that uses a HuggingFace client. Would that be enough here?

[vitali87]

@Kludex when would that be?

[HDembinski]

We are adding the HuggingFace model that uses a HuggingFace client. Would that be enough here?

I don't think so. If you properly support llama.cpp, you can make use of its grammar-constrained sampling feature, which fits your library perfectly. In a nutshell, it ensures that the output follows a grammar (in your case typically a specific JSON schema) by disallowing tokens to be generated during sampling that would violate the grammar. This gets rid of retries when the output does not follow the spec, it saves time and energy.

https://github.com/ggml-org/llama.cpp/blob/master/examples/pydantic_models_to_grammar_examples.py

[karlla1220]

We are adding the HuggingFace model that uses a HuggingFace client. Would that be enough here?

I don't think so. If you properly support llama.cpp, you can make use of its grammar-constrained sampling feature, which fits your library perfectly. In a nutshell, it ensures that the output follows a grammar (in your case typically a specific JSON schema) by disallowing tokens to be generated during sampling that would violate the grammar. This gets rid of retries when the output does not follow the spec, it saves time and energy.

https://github.com/ggml-org/llama.cpp/blob/master/examples/pydantic_models_to_grammar_examples.py

Agree. At least the llama.cpp specific ModelProfile is necessary.

[HDembinski]

I will give this a try, because I think the grammar constraint is very powerful.

If you don't need that, it is very simple to connect the llama.cpp Server by adapting the OpenAIProvider

agent = Agent(
    OpenAIChatModel(
        "",
        provider=OpenAIProvider(base_url="http://localhost:8080/v1"),
        settings=ModelSettings(temperature=0.5, max_tokens=1000),
    ),
    ...
)

I wrote two blog articles where I use (and praise) PydanticAI here:

• https://hdembinski.github.io/posts/parsing_webpages_with_llm_revisited.html
• https://hdembinski.github.io/posts/llm_tag_posts.html

[HDembinski]

I think a LlamaCppChatModel should start the llama-server if it is not running, or not running with the right model under that url. Instead of the model name, you would pass the location of the gguf file, and in case of a vision model, the mmproj file.

The grammar mode should be activated if you use the NativeOutput tag for the output_type.

[DouweM]

@HDembinski Have a look at https://ai.pydantic.dev/models/outlines/, which supports llama.cpp and structured outputs enabled by https://dottxt-ai.github.io/outlines.

[HDembinski]

@DouweM I saw the Outlines PR but didn't realize they already do what this grammar feature of llama.cpp does. Great, so I think this can be closed.